Search

Starting from a description of the Rubin Observatory Data Management System Architecture, and drawing on our experience with and involvement in a range of other projects including Gaia, SDSS, UKIRT, and JCMT, we derive a series of generic design patterns and lessons learned., Comment: 10 pages ADASS XXXII submission

The Vera C. Rubin Observatory will advance many areas of astronomy over the next decade with its unique wide-fast-deep multi-color imaging survey, the Legacy Survey of Space and Time (LSST). The LSST will produce approximately 20TB of raw data per night, which will be automatically processed by the LSST Science Pipelines to generate science-ready data products -- processed images, catalogs and alerts. To ensure that these data products enable transformative science with LSST, stringent requirements have been placed on their quality and scientific fidelity, for example on image quality and depth, astrometric and photometric performance, and object recovery completeness. In this paper we introduce faro, a framework for automatically and efficiently computing scientific performance metrics on the LSST data products for units of data of varying granularity, ranging from single-detector to full-survey summary statistics. By measuring and monitoring metrics, we are able to evaluate trends in algorithmic performance and conduct regression testing during development, compare the performance of one algorithm against another, and verify that the LSST data products will meet performance requirements by comparing to specifications. We present initial results using faro to characterize the performance of the data products produced on simulated and precursor data sets, and discuss plans to use faro to verify the performance of the LSST commissioning data products., Comment: 12 pages, 9 figues, Proc. SPIE 12189, Software and Cyberinfrastructure for Astronomy VII, Montreal, CA, July 2022

We describe Rubin Observatory's experience with offering a data access facility (and associated services including our Science Platform) deployed on Google Cloud infrastructure as part of our pre-Operations Data Preview program., Comment: 4 pages 1 figure

In preparation for cosmological analyses of the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST), the LSST Dark Energy Science Collaboration (LSST DESC) has created a 300 deg$^2$ simulated survey as part of an effort called Data Challenge 2 (DC2). The DC2 simulated sky survey, in six optical bands with observations following a reference LSST observing cadence, was processed with the LSST Science Pipelines (19.0.0). In this Note, we describe the public data release of the resulting object catalogs for the coadded images of five years of simulated observations along with associated truth catalogs. We include a brief description of the major features of the available data sets. To enable convenient access to the data products, we have developed a web portal connected to Globus data services. We describe how to access the data and provide example Jupyter Notebooks in Python to aid first interactions with the data. We welcome feedback and questions about the data release via a GitHub repository., Comment: 25 pages, 3 figures; 9 tables. A detailed changelog can be found in Appendix A. To obtain data, visit the DESC Data Portal at https://data.lsstdesc.org/

We describe the simulated sky survey underlying the second data challenge (DC2) carried out in preparation for analysis of the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) by the LSST Dark Energy Science Collaboration (LSST DESC). Significant connections across multiple science domains will be a hallmark of LSST; the DC2 program represents a unique modeling effort that stresses this interconnectivity in a way that has not been attempted before. This effort encompasses a full end-to-end approach: starting from a large N-body simulation, through setting up LSST-like observations including realistic cadences, through image simulations, and finally processing with Rubin's LSST Science Pipelines. This last step ensures that we generate data products resembling those to be delivered by the Rubin Observatory as closely as is currently possible. The simulated DC2 sky survey covers six optical bands in a wide-fast-deep (WFD) area of approximately 300 deg^2 as well as a deep drilling field (DDF) of approximately 1 deg^2. We simulate 5 years of the planned 10-year survey. The DC2 sky survey has multiple purposes. First, the LSST DESC working groups can use the dataset to develop a range of DESC analysis pipelines to prepare for the advent of actual data. Second, it serves as a realistic testbed for the image processing software under development for LSST by the Rubin Observatory. In particular, simulated data provide a controlled way to investigate certain image-level systematic effects. Finally, the DC2 sky survey enables the exploration of new scientific ideas in both static and time-domain cosmology., Comment: 39 pages, 19 figures, version accepted for publication in ApJS

LSST has chosen Kubernetes as the platform for deploying and operating the LSST Science Platform. We first present the background reasoning behind this decision, including both instrument-agnostic as well as LSST-specific requirements. We then discuss the basic principles of Kubernetes and Helm, and how they are used as the deployment base for the LSST Science Platform. Furthermore, we provide an example of how an external group may use these publicly available software resources to deploy their own instance of the LSST Science Platform, and customize it to their needs. Finally, we discuss how more astronomy software can follow these patterns to gain similar benefits., Comment: 10 pages, submitted to proceedings of ADASS XXIX, Groningen, The Netherlands, to be published in ASP Conf. Series

The Zwicky Transient Facility (ZTF) survey generates real-time alerts for optical transients, variables, and moving objects discovered in its wide-field survey. We describe the ZTF alert stream distribution and processing (filtering) system. The system uses existing open-source technologies developed in industry: Kafka, a real-time streaming platform, and Avro, a binary serialization format. The technologies used in this system provide a number of advantages for the ZTF use case, including (1) built-in replication, scalability, and stream rewind for the distribution mechanism; (2) structured messages with strictly enforced schemas and dynamic typing for fast parsing; and (3) a Python-based stream processing interface that is similar to batch for a familiar and user-friendly plug-in filter system, all in a modular, primarily containerized system. The production deployment has successfully supported streaming up to 1.2 million alerts or roughly 70 GB of data per night, with each alert available to a consumer within about 10 s of alert candidate production. Data transfer rates of about 80,000 alerts/minute have been observed. In this paper, we discuss this alert distribution and processing system, the design motivations for the technology choices for the framework, performance in production, and how this system may be generally suitable for other alert stream use cases, including the upcoming Large Synoptic Survey Telescope., Comment: Published in PASP Focus Issue on the Zwicky Transient Facility (doi: 10.1088/1538-3873/aae904). 9 Pages, 2 Figures

The Large Synoptic Survey Telescope (LSST) is an ambitious astronomical survey with a similarly ambitious Data Management component. Data Management for LSST includes processing on both nightly and yearly cadences to generate transient alerts, deep catalogs of the static sky, and forced photometry light-curves for billions of objects at hundreds of epochs, spanning at least a decade. The algorithms running in these pipelines are individually sophisticated and interact in subtle ways. This paper provides an overview of those pipelines, focusing more on those interactions than the details of any individual algorithm., Comment: Submitted to proceedings for ADASS XXVIII

The use of high-quality simulated sky catalogs is essential for the success of cosmological surveys. The catalogs have diverse applications, such as investigating signatures of fundamental physics in cosmological observables, understanding the effect of systematic uncertainties on measured signals and testing mitigation strategies for reducing these uncertainties, aiding analysis pipeline development and testing, and survey strategy optimization. The list of applications is growing with improvements in the quality of the catalogs and the details that they can provide. Given the importance of simulated catalogs, it is critical to provide rigorous validation protocols that enable both catalog providers and users to assess the quality of the catalogs in a straightforward and comprehensive way. For this purpose, we have developed the DESCQA framework for the Large Synoptic Survey Telescope Dark Energy Science Collaboration as well as for the broader community. The goal of DESCQA is to enable the inspection, validation, and comparison of an inhomogeneous set of synthetic catalogs via the provision of a common interface within an automated framework. In this paper, we present the design concept and first implementation of DESCQA. In order to establish and demonstrate its full functionality we use a set of interim catalogs and validation tests. We highlight several important aspects, both technical and scientific, that require thoughtful consideration when designing a validation framework, including validation metrics and how these metrics impose requirements on the synthetic sky catalogs., Comment: 16+4 pages, 9 figures, 2+1 tables. Published in the ApJ Supplement. Code repository at https://github.com/LSSTDESC/descqa

In this paper, we describe the optical imaging data processing pipeline developed for the Subaru Telescope's Hyper Suprime-Cam (HSC) instrument. The HSC Pipeline builds on the prototype pipeline being developed by the Large Synoptic Survey Telescope's Data Management system, adding customizations for HSC, large-scale processing capabilities, and novel algorithms that have since been reincorporated into the LSST codebase. While designed primarily to reduce HSC Subaru Strategic Program (SSP) data, it is also the recommended pipeline for reducing general-observer HSC data. The HSC pipeline includes high level processing steps that generate coadded images and science-ready catalogs as well as low-level detrending and image characterizations., Comment: 39 pages, 21 figures, 2 tables. Submitted to Publications of the Astronomical Society of Japan

The Large Synoptic Survey Telescope (LSST) is a large-aperture, wide-field, ground-based survey system that will image the sky in six optical bands from 320 to 1050 nm, uniformly covering approximately $18,000$deg$^2$ of the sky over 800 times. The LSST is currently under construction on Cerro Pach\'on in Chile, and expected to enter operations in 2022. Once operational, the LSST will explore a wide range of astrophysical questions, from discovering "killer" asteroids to examining the nature of Dark Energy. The LSST will generate on average 15 TB of data per night, and will require a comprehensive Data Management system to reduce the raw data to scientifically useful catalogs and images with minimum human intervention. These reductions will result in a real-time alert stream, and eleven data releases over the 10-year duration of LSST operations. To enable this processing, the LSST project is developing a new, general-purpose, high-performance, scalable, well documented, open source data processing software stack for O/IR surveys. Prototypes of this stack are already capable of processing data from existing cameras (e.g., SDSS, DECam, MegaCam), and form the basis of the Hyper-Suprime Cam (HSC) Survey data reduction pipeline., Comment: 10 pages, 3 figures, to appear in the Proceedings of ADASS XXV

The Large Synoptic Survey Telescope (LSST) will explore the entire southern sky over 10 years starting in 2022 with unprecedented depth and time sampling in six filters, $ugrizy$. Artificial power on the scale of the 3.5 deg LSST field-of-view will contaminate measurements of baryonic acoustic oscillations (BAO), which fall at the same angular scale at redshift $z \sim 1$. Using the HEALPix framework, we demonstrate the impact of an "un-dithered" survey, in which $17\%$ of each LSST field-of-view is overlapped by neighboring observations, generating a honeycomb pattern of strongly varying survey depth and significant artificial power on BAO angular scales. We find that adopting large dithers (i.e., telescope pointing offsets) of amplitude close to the LSST field-of-view radius reduces artificial structure in the galaxy distribution by a factor of $\sim$10. We propose an observing strategy utilizing large dithers within the main survey and minimal dithers for the LSST Deep Drilling Fields. We show that applying various magnitude cutoffs can further increase survey uniformity. We find that a magnitude cut of $r < 27.3$ removes significant spurious power from the angular power spectrum with a minimal reduction in the total number of observed galaxies over the ten-year LSST run. We also determine the effectiveness of the observing strategy for Type Ia SNe and predict that the main survey will contribute $\sim$100,000 Type Ia SNe. We propose a concentrated survey where LSST observes one-third of its main survey area each year, increasing the number of main survey Type Ia SNe by a factor of $\sim$1.5, while still enabling the successful pursuit of other science drivers., Comment: 9 pages, 6 figures, published in SPIE proceedings; corrected typo in equation 2

Supernova rates are directly coupled to high mass stellar birth and evolution. As such, they are one of the few direct measures of the history of cosmic stellar evolution. In this paper we describe an probabilistic technique for identifying supernovae within spectroscopic samples of galaxies. We present a study of 52 type Ia supernovae ranging in age from -14 days to +40 days extracted from a parent sample of \simeq 50,000 spectra from the SDSS DR5. We find a Supernova Rate (SNR) of 0.472^{+0.048}_{-0.039}(Systematic)^{+0.081}_{-0.071}(Statistical)SNu at a redshift of = 0.1. This value is higher than other values at low redshift at the 1{\sigma}, but is consistent at the 3{\sigma} level. The 52 supernova candidates used in this study comprise the third largest sample of supernovae used in a type Ia rate determination to date. In this paper we demonstrate the potential for the described approach for detecting supernovae in future spectroscopic surveys., Comment: 16 pages, 16 figures Accepted ApJ

A survey that can cover the sky in optical bands over wide fields to faint magnitudes with a fast cadence will enable many of the exciting science opportunities of the next decade. The Large Synoptic Survey Telescope (LSST) will have an effective aperture of 6.7 meters and an imaging camera with field of view of 9.6 deg^2, and will be devoted to a ten-year imaging survey over 20,000 deg^2 south of +15 deg. Each pointing will be imaged 2000 times with fifteen second exposures in six broad bands from 0.35 to 1.1 microns, to a total point-source depth of r~27.5. The LSST Science Book describes the basic parameters of the LSST hardware, software, and observing plans. The book discusses educational and outreach opportunities, then goes on to describe a broad range of science that LSST will revolutionize: mapping the inner and outer Solar System, stellar populations in the Milky Way and nearby galaxies, the structure of the Milky Way disk and halo and other objects in the Local Volume, transient and variable objects both at low and high redshift, and the properties of normal and active galaxies at low and high redshift. It then turns to far-field cosmological topics, exploring properties of supernovae to z~1, strong and weak lensing, the large-scale distribution of galaxies and baryon oscillations, and how these different probes may be combined to constrain cosmological models and the physics of dark energy., Comment: 596 pages. Also available at full resolution at http://www.lsst.org/lsst/scibook

(Abridged) We describe here the most ambitious survey currently planned in the optical, the Large Synoptic Survey Telescope (LSST). A vast array of science will be enabled by a single wide-deep-fast sky survey, and LSST will have unique survey capability in the faint time domain. The LSST design is driven by four main science themes: probing dark energy and dark matter, taking an inventory of the Solar System, exploring the transient optical sky, and mapping the Milky Way. LSST will be a wide-field ground-based system sited at Cerro Pach\'{o}n in northern Chile. The telescope will have an 8.4 m (6.5 m effective) primary mirror, a 9.6 deg$^2$ field of view, and a 3.2 Gigapixel camera. The standard observing sequence will consist of pairs of 15-second exposures in a given field, with two such visits in each pointing in a given night. With these repeats, the LSST system is capable of imaging about 10,000 square degrees of sky in a single filter in three nights. The typical 5$\sigma$ point-source depth in a single visit in $r$ will be $\sim 24.5$ (AB). The project is in the construction phase and will begin regular survey operations by 2022. The survey area will be contained within 30,000 deg$^2$ with $\delta<+34.5^\circ$, and will be imaged multiple times in six bands, $ugrizy$, covering the wavelength range 320--1050 nm. About 90\% of the observing time will be devoted to a deep-wide-fast survey mode which will uniformly observe a 18,000 deg$^2$ region about 800 times (summed over all six bands) during the anticipated 10 years of operations, and yield a coadded map to $r\sim27.5$. The remaining 10\% of the observing time will be allocated to projects such as a Very Deep and Fast time domain survey. The goal is to make LSST data products, including a relational database of about 32 trillion observations of 40 billion objects, available to the public and scientists around the world., Comment: 57 pages, 32 color figures, version with high-resolution figures available from https://www.lsst.org/overview

The large--scale structure (LSS) in the Universe comprises a complicated filamentary network of matter. We study this network using a high--resolution simulation of structure formation of a $\Lambda$ Cold Dark Matter cosmology. We investigate the distribution of matter between neighbouring large haloes whose masses are comparable to massive clusters of galaxies. We identify a total of 228 filaments between neighbouring clusters. Roughly half of the filaments are either warped or lie off the cluster--cluster axis. We find that straight filaments on the average are shorter than warped ones. More massive clusters are connected to more filaments than less massive ones on average. This finding indicates that the most massive clusters form at the intersections of the filamentary backbone of LSS. For straight filaments, we compute mass profiles. Radial profiles show a fairly well--defined radius, $r_s$, beyond which the profiles follow an $r^{-2}$ power law fairly closely. For the majority of filaments, $r_s$ lies between 1.5 $h^{-1}$ Mpc and 2.0 $h^{-1}$ Mpc. The enclosed overdensity inside $r_s$ varies between a few times up to 25 times mean density, independent of the length of the filaments. Along the filaments' axes, material is not distributed uniformly. Towards the clusters, the density rises, indicating the presence of the cluster infall regions. In addition, we also find some sheet--like connections between clusters. In roughly a fifth of all cluster--cluster connections where we could not identify a filament or sheet, projection effects lead to filamentary structures in the projected mass distribution. (abridged), Comment: 10 pages, 18 figures; submitted to MNRAS; updated: final version, accepted for publication