Search

Frequency-domain multiplexing (fMux) is an established technique for the readout of large arrays of transition edge sensor (TES) bolometers. Each TES in a multiplexing module has a unique AC voltage bias that is selected by a resonant filter. This scheme enables the operation and readout of multiple bolometers on a single pair of wires, reducing thermal loading onto sub-Kelvin stages. The current receiver on the South Pole Telescope, SPT-3G, uses a 68x fMux system to operate its large-format camera of $\sim$16,000 TES bolometers. We present here the successful implementation and performance of the SPT-3G readout as measured on-sky. Characterization of the noise reveals a median pair-differenced 1/f knee frequency of 33 mHz, indicating that low-frequency noise in the readout will not limit SPT-3G's measurements of sky power on large angular scales. Measurements also show that the median readout white noise level in each of the SPT-3G observing bands is below the expectation for photon noise, demonstrating that SPT-3G is operating in the photon-noise-dominated regime., Comment: 9 pages, 5 figures submitted to the Journal of Low Temperature Physics: LTD18 Special Edition

We present measurements of the $E$-mode ($EE$) polarization power spectrum and temperature-$E$-mode ($TE$) cross-power spectrum of the cosmic microwave background using data collected by SPT-3G, the latest instrument installed on the South Pole Telescope. This analysis uses observations of a 1500 deg$^2$ region at 95, 150, and 220 GHz taken over a four month period in 2018. We report binned values of the $EE$ and $TE$ power spectra over the angular multipole range $300 \le \ell < 3000$, using the multifrequency data to construct six semi-independent estimates of each power spectrum and their minimum-variance combination. These measurements improve upon the previous results of SPTpol across the multipole ranges $300 \le \ell \le 1400$ for $EE$ and $300 \le \ell \le 1700$ for $TE$, resulting in constraints on cosmological parameters comparable to those from other current leading ground-based experiments. We find that the SPT-3G dataset is well-fit by a $\Lambda$CDM cosmological model with parameter constraints consistent with those from Planck and SPTpol data. From SPT-3G data alone, we find $H_0 = 68.8 \pm 1.5 \mathrm{km\,s^{-1}\,Mpc^{-1}}$ and $\sigma_8 = 0.789 \pm 0.016$, with a gravitational lensing amplitude consistent with the $\Lambda$CDM prediction ($A_L = 0.98 \pm 0.12$). We combine the SPT-3G and the Planck datasets and obtain joint constraints on the $\Lambda$CDM model. The volume of the 68% confidence region in six-dimensional $\Lambda$CDM parameter space is reduced by a factor of 1.5 compared to Planck-only constraints, with only slight shifts in central values. We note that the results presented here are obtained from data collected during just half of a typical observing season with only part of the focal plane operable, and that the active detector count has since nearly doubled for observations made with SPT-3G after 2018.

SPT-3G is a polarization-sensitive receiver, installed on the South Pole Telescope, that measures the anisotropy of the cosmic microwave background (CMB) from degree to arcminute scales. The receiver consists of ten 150~mm-diameter detector wafers, containing a total of 16,000 transition-edge sensor (TES) bolometers observing at 95, 150, and 220 GHz. During the 2018-2019 austral summer, one of these detector wafers was replaced by a new wafer fabricated with Al-Mn TESs instead of the Ti/Au design originally deployed for SPT-3G. We present the results of in-lab characterization and on-sky performance of this Al-Mn wafer, including electrical and thermal properties, optical efficiency measurements, and noise-equivalent temperature. In addition, we discuss and account for several calibration-related systematic errors that affect measurements made using frequency-domain multiplexing readout electronics., Comment: 9 pages, 5 figures, submitted to the Journal of Low Temperature Physics: LTD18 Special Edition

The File Transfer Service (FTS3) is a data movement service developed at CERN which is used to distribute the majority of the Large Hadron Collider’s data across the Worldwide LHC Computing Grid (WLCG) infrastructure. At Fermilab, we have deployed FTS3 instances for Intensity Frontier experiments (e.g. DUNE) to transfer data in America and Europe, using a container-based strategy. In this article we summarize our experience building docker images based on work from the SLATE project (slateci.io) and deployed in OKD, the community distribution of Red Hat OpenShift. Additionally, we discuss our method of certificate management and maintenance utilizing Kubernetes CronJobs. Finally, we also report on the configuration currently running at Fermilab.

The SLATE (Services Layer at the Edge) accelerates collaborative scientific computing through a secure container orchestration framework focused on the Science DMZ, enabling creation of advanced multi-institution platforms and novel science gateways. The goal of the SLATE project is to provide a secure federation platform to simplify deployment and operation of complex and often specialized services required by multi-institution scientific collaborations, utilizing where applicable open source, cloud native tooling such as Kubernetes. This paper outlines the design and operation of a monitoring infrastructure suitable for application developers and resource providers which gives visibility to resource utilization and service deployments across a network of independently managed Kubernetes clusters.

In this paper we showcase the support in Open Science Grid (OSG) of Midscale collaborations, the region of computing and storage scale where multi-institutional researchers collaborate to execute their science workflows on the grid without having dedicated technical support teams of their own. Collaboration Services enables such collaborations to take advantage of the distributed resources of the Open Science Grid by facilitating access to submission hosts, the deployment of their applications and supporting their data management requirements. Distributed computing software adopted from large scale collaborations, such as CVMFS, Rucio, xCache lower the barrier of intermediate scale research to integrate with existing infrastructure.

One of the most costly factors in providing a global computing infrastructure such as the WLCG is the human effort in deployment, integration, and operation of the distributed services supporting collaborative computing, data sharing and delivery, and analysis of extreme scale datasets. Furthermore, the time required to roll out global software updates, introduce new service components, or prototype novel systems requiring coordinated deployments across multiple facilities is often increased by communication latencies, staff availability, and in many cases expertise required for operations of bespoke services. While the WLCG (and distributed systems implemented throughout HEP) is a global service platform, it lacks the capability and flexibility of a modern platform-as-a-service including continuous integration/continuous delivery (CI/CD) methods, development-operations capabilities (DevOps, where developers assume a more direct role in the actual production infrastructure), and automation. Most importantly, tooling which reduces required training, bespoke service expertise, and the operational effort throughout the infrastructure, most notably at the resource endpoints (sites), is entirely absent in the current model. In this paper, we explore ideas and questions around potential NoOps models in this context: what is realistic given organizational policies and constraints? How should operational responsibility be organized across teams and facilities? What are the technical gaps? What are the social and cybersecurity challenges? Conversely what advantages does a NoOps model deliver for innovation and for accelerating the pace of delivery of new services needed for the HL-LHC era? We will describe initial work along these lines in the context of providing a data delivery network supporting IRIS-HEP DOMA R&D.

The Open Science Grid (OSG) provides a common service for resource providers and scientific institutions, and supports sciences such as High Energy Physics, Structural Biology, and other community sciences. As scientific frontiers expand, so does the need for resources to analyze new data. For example, high energy physics (LHC) sciences foresee an exponential growth in the amount of data collected, which comes with corresponding growth in the need for computing resources. Allowing resource providers an easy way to share their resources is paramount to ensure the grow of resources available to scientists. In this context, the OSG Hosted CE initiative provides site administrator a way to reduce the effort needed to install and maintain a Compute Element (CE), and represents a solution for sites who do not have the effort and expertise to run their own Grid middleware. An HTCondor Compute Element is installed on a remote VM at UChicago for each site that joins the Hosted CE initiative. The hardware/software stack is maintained by OSG Operations staff in a homogeneus and automated way, providing a reduction in the overall operational effort needed to maintain the CEs: one single organization does it in an uniform way, instead of each single resource provider doing it in their own way. Currently, more than 20 institutions joined the Hosted CE initiative. This contribution discusses the technical details behind a Hosted CE installation, highlighting key strenghts and common pitfalls, and outlining future plans to further reduce operational experience.

The SLATE (Services Layer at the Edge) platform supports collaborative, multi-institution scientific computing through federation of containerized edge services. This paper considers issues of trust between resource providers and developers of orchestrated services which span administrative domains. The context for discussion is the SLATE federation architecture and application deployment pattern. The major features are a custom central API server which implements the federation, partitioning of user and group applications on edge clusters, and a curated catalog of applications which can be installed within the federation.

Modern software development workflow patterns often involve the use of a developer's local machine as the first platform for testing code. SLATE mimics this paradigm with an implementation of a light-weight version, called MiniSLATE[? ], that runs completely contained on the developer's local machine or scales to larger machines (laptop, virtual machine, or another physical server). MiniSLATE resolves many development environment issues by providing an isolated and local configuration for the developer. Application developers are able to download MiniSLATE which provides a fully orchestrated set of containers on top of a production SLATE platform, complete with central information service, API server, and a local Kubernetes cluster. This approach mitigates the overhead of a hypervisor but still provides the requisite isolated environment. They are able to create the environment, iterate, destroy it, and repeat at will. A local MiniSLATE environment also allows the developer to explore the packaging of the edge service within a constrained security context in order to validate its full functionality within limited permissions. As a result, developers are able to test the functionality of their application with the complete complement of SLATE components local to their development environment without the overhead of building a cluster or virtual machine, registering a cluster, interacting with the production SLATE platform, etc.

We present two prescriptions for broadband (~77 - 252 GHz), millimeter-wave antireflection coatings for cryogenic, sintered polycrystalline aluminum oxide optics: one for large-format (700 mm diameter) planar and plano-convex elements, the other for densely packed arrays of quasi-optical elements, in our case 5 mm diameter half-spheres (called "lenslets"). The coatings comprise three layers of commercially-available, polytetrafluoroethylene-based, dielectric sheet material. The lenslet coating is molded to fit the 150 mm diameter arrays directly while the large-diameter lenses are coated using a tiled approach. We review the fabrication processes for both prescriptions then discuss laboratory measurements of their transmittance and reflectance. In addition, we present the inferred refractive indices and loss tangents for the coating materials and the aluminum oxide substrate. We find that at 150 GHz and 300 K the large-format coating sample achieves (97 +/- 2)% transmittance and the lenslet coating sample achieves (94 +/- 3)% transmittance., Comment: 19 pages, 11 figures; submitted 05 Dec 2019, accepted 26 Feb 2020

SPT-3G, the third generation camera on the South Pole Telescope (SPT), was deployed in the 2016-2017 Austral summer season. The SPT is a 10-meter telescope located at the geographic South Pole and designed for observations in the millimeter-wave and submillimeter-wave regions of the electromagnetic spectrum. The SPT is primarily used to study the cosmic microwave background (CMB). The upgraded camera produces an order of magnitude more data than the previous generations of SPT cameras. The telescope is expected to collect a petabyte (PB) of data over course of five years, which is a significantly larger data volume than any other CMB telescope in operation. The increase in data rate required radical changes to the SPT computing model both at the South Pole and University of Chicago. This paper will describe the overall integration of distributed storage and compute resources into a common interface, deployment of on-site data reduction and storage infrastructure, and the usage of the Open Science Grid (OSG) by the SPT collaboration.

The cosmic microwave background (CMB) encodes information about the content and evolution of the universe. The presence of light, weakly interacting particles impacts the expansion history of the early universe, which alters the temperature and polarization anisotropies of the CMB. In this way, current measurements of the CMB place interesting constraints on the neutrino energy density and mass, as well as on the abundance of other possible light relativistic particle species. We present the status of an on-going 1500 sq. deg. survey with the SPT-3G receiver, a new mm-wavelength camera on the 10-m diameter South Pole Telescope (SPT). The SPT-3G camera consists of 16,000 superconducting transition edge sensors, a 10x increase over the previous generation camera, which allows it to map the CMB with an unprecedented combination of sensitivity and angular resolution. We highlight projected constraints on the abundance of sterile neutrinos and the sum of the neutrino masses for the SPT-3G survey, which could help determine the neutrino mass hierarchy., 6 pages, 2 figures, TAUP 2019

The Open Science Grid (OSG) provides a common service for resource providers and scientific institutions, and supports sciences such as High Energy Physics, Structural Biology, and other community sciences. As scientific frontiers expand, so does the need for resources to analyze new data. For example, High Energy Physics experiments such as the LHC experiments foresee an exponential growth in the amount of data collected, which comes with corresponding growth in the need for computing resources. Allowing resource providers an easy way to share their resources is paramount to ensure the grow of resources available to scientists. In this context, the OSG Hosted CE initiative provides site administrator a way to reduce the effort needed to install and maintain a Compute Element (CE), and represents a solution for sites who do not have the effort and expertise to run their own Grid middleware. An HTCondor Compute Element is installed on a remote VM at UChicago for each site that joins the Hosted CE initiative. The hardware/software stack is maintained by OSG Operations staff in a homogeneus and automated way, providing a reduction in the overall operational effort needed to maintain the CEs: one single organization does it in an uniform way, instead of each single resource provider doing it in their own way. Currently, more than 20 institutions joined the Hosted CE initiative. This contribution discusses the technical details behind a Hosted CE installation, highlighting key strengths and common pitfalls, and outlining future plans to further reduce operational experience.

The Scalable Systems Laboratory (SSL), part of the IRIS-HEP Software Institute, provides Institute participants and HEP software developers generally with a means to transition their R&D from conceptual toys to testbeds to production-scale prototypes. The SSL enables tooling, infrastructure, and services supporting innovation of novel analysis and data architectures, development of software elements and tool-chains, reproducible functional and scalability testing of service components, and foundational systems R&D for accelerated services developed by the Institute. The SSL is constructed with a core team having expertise in scale testing and deployment of services across a wide range of cyberinfrastructure. The core team embeds and partners with other areas in the Institute, and with LHC and other HEP development and operations teams as appropriate, to define investigations and required service deployment patterns, to design concrete tests, and to execute and evaluate the results. We describe experiences with early deployments supporting the development of analysis platforms and intelligent data delivery systems.

We describe progress on building the SLATE (Services Layer at the Edge) platform. The high level goal of SLATE is to facilitate creation of multi-institutional science computing systems by augmenting the canonical Science DMZ pattern with a generic, "programmable", secure and trusted underlayment platform. This platform permits hosting of advanced container-centric services needed for higher-level capabilities such as data transfer nodes, software and data caches, workflow services and science gateway components. SLATE uses best-of-breed data center virtualization and containerization components, and where available, software defined networking, to enable distributed automation of deployment and service lifecycle management tasks by domain experts. As such it will simplify creation of scalable platforms that connect research teams, institutions and resources to accelerate science while reducing operational costs and development cycle times.

A traditional HPC computing facility provides a large amount of computing power but has a fixed environment designed to satisfy local needs. This makes it very challenging for users to deploy complex applications that span multiple sites and require specific application software, scheduling middleware, or sharing policies. The DOE-funded VC3 project aims to address these challenges by making it possible for researchers to easily aggregate and share resources, install custom software environments, and deploy clustering frameworks across multiple HPC facilities through the concept of "virtual clusters". This paper presents the design, implementation, and initial experience with our prototype self-service VC3 platform which automates deployment of cluster frameworks across diverse computing facilities. To create a virtual cluster, the VC3 platform materializes a custom head node in a secure private cloud, specifies a choice of scheduling middleware, then allocates resources from the remote facilities where the desired software and clustering framework is installed in user space. As resources become available from scheduled nodes from individual clusters, the research team simply sees a private cluster they can access directly or share with collaborators, such as a science gateway community. We discuss how this service can be used by research collaborations requiring shared resources, specific middleware frameworks, and complex applications and workflows in the areas of astrophysics, bioinformatics and high energy physics.

We present the distributed data and job management scheme on the Open Science Grid (OSG) and European Grid Infrastructure (EGI) that was developed for the XENON1T experiment. The experiment aims to detect dark matter using the first ton-scale detector of this kind, a ~2000-kg-target-mass dual-phase (liquid-gas) xenon time projection chamber in operation at the Laboratori Nazionali del Gran Sasso in Italy. The experiment's data is automatically spread across several sites on EGI and OSG. The data location and movement is managed using the rucio software package developed by the ATLAS collaboration. The job submission to OSG, EGI, SDSC's Comet Supercomputer, and campus HPC resources is done through the CI Connect infrastructure that is directly connected to the general OSG infrastructure. This paper will describe the overall integration of distributed storage and compute resources into a common interface, respectively, and integration into the existing XENON1T software

Big Data technologies have proven to be very useful for storage, processing and visualization of derived metrics associated with ATLAS distributed computing (ADC) services. Logfiles, database records, and metadata from a diversity of systems have been aggregated and indexed to create an analytics platform for ATLAS ADC operations analysis. Dashboards, wide area data access cost metrics, user analysis patterns, and resource utilization efficiency charts are produced flexibly through queries against a powerful analytics cluster. Here we explore whether these techniques and associated analytics ecosystem can be applied to add new modes of open, quick, and pervasive access to ATLAS event data. Such modes would simplify access and broaden the reach of ATLAS public data to new communities of users. An ability to efficiently store, filter, search and deliver ATLAS data at the event and/or sub-event level in a widely supported format would enable or significantly simplify usage of machine learning environments and tools like Spark, Jupyter, R, SciPy, Caffe, TensorFlow, etc. Machine learning challenges such as the Higgs Boson Machine Learning Challenge, the Tracking challenge, Event viewers (VP1, ATLANTIS, ATLASrift), and still to be developed educational and outreach tools would be able to access the data through a simple REST API. In this preliminary investigation we focus on derived xAOD data sets. These are much smaller than the primary xAODs having containers, variables, and events of interest to a particular analysis. Being encouraged with the performance of Elasticsearch for the ADC analytics platform, we developed an algorithm for indexing derived xAOD event data. We have made an appropriate document mapping and have imported a full set of standard model W/Z datasets. We compare the disk space efficiency of this approach to that of standard ROOT files, the performance in simple cut flow type of data analysis, and will present preliminary results on its scaling characteristics with different numbers of clients, query complexity, and size of the data retrieved.

High-angular-resolution cosmic microwave background experiments provide a unique opportunity to conduct a survey of time-variable sources at millimeter wavelengths, a population which has primarily been understood through follow-up measurements of detections in other bands. Here we report the first results of an astronomical transient survey with the South Pole Telescope (SPT) using the SPT-3G camera to observe 1500 square degrees of the southern sky. The observations took place from March to November 2020 in three bands centered at 95, 150, and 220 GHz. This survey yielded the detection of fifteen transient events from sources not previously detected by the SPT. The majority are associated with variable stars of different types, expanding the number of such detected flares by more than a factor of two. The stellar flares are unpolarized and bright, in some cases exceeding 1 Jy, and have durations from a few minutes to several hours. Another population of detected events last for 2--3 weeks and appear to be extragalactic in origin. Though data availability at other wavelengths is limited, we find evidence for concurrent optical activity for two of the stellar flares. Future data from SPT-3G and forthcoming instruments will provide real-time detection of millimeter-wave transients on timescales of minutes to months., 14 pages, 9 figures; accepted to ApJ 5/27

The third generation South Pole Telescope camera (SPT-3G) improves upon its predecessor (SPTpol) by an order of magnitude increase in detectors on the focal plane. The technology used to read out and control these detectors, digital frequency-domain multiplexing (DfMUX), is conceptually the same as used for SPTpol, but extended to accommodate more detectors. A nearly 5x expansion in the readout operating bandwidth has enabled the use of this large focal plane, and SPT-3G performance meets the forecasting targets relevant to its science objectives. However, the electrical dynamics of the higher-bandwidth readout differ from predictions based on models of the SPTpol system due to the higher frequencies used, and parasitic impedances associated with new cryogenic electronic architecture. To address this, we present an updated derivation for electrical crosstalk in higher-bandwidth DfMUX systems, and identify two previously uncharacterized contributions to readout noise, which become dominant at high bias frequency. The updated crosstalk and noise models successfully describe the measured crosstalk and readout noise performance of SPT-3G. These results also suggest specific changes to warm electronics component values, wire-harness properties, and SQUID parameters, to improve the readout system for future experiments using DfMUX, such as the LiteBIRD space telescope., Comment: Accepted to the Journal of Astronomical Telescopes, Instruments, and Systems