Search

The first generation of exascale systems will include a variety of machine architectures, featuring GPUs from multiple vendors. As a result, many developers are interested in adopting portable programming models to avoid maintaining multiple versions of their code. It is necessary to document experiences with such programming models to assist developers in understanding the advantages and disadvantages of different approaches. To this end, this paper evaluates the performance portability of a SYCL implementation of a large-scale cosmology application (CRK-HACC) running on GPUs from three different vendors: AMD, Intel, and NVIDIA. We detail the process of migrating the original code from CUDA to SYCL and show that specializing kernels for specific targets can greatly improve performance portability without significantly impacting programmer productivity. The SYCL version of CRK-HACC achieves a performance portability of 0.96 with a code divergence of almost 0, demonstrating that SYCL is a viable programming model for performance-portable applications., Comment: 12 pages, 13 figures, 2023 International Workshop on Performance, Portability & Productivity in HPC

The galaxy distribution in dark matter-dominated halos is expected to approximately trace the details of the underlying dark matter substructure. In this paper we introduce halo `core-tracking' as a way to efficiently follow the small-scale substructure in cosmological simulations and apply the technique to model the galaxy distribution in observed clusters. The method relies on explicitly tracking the set of particles identified as belonging to a halo's central density core, once a halo has attained a certain threshold mass. The halo cores are then followed throughout the entire evolution of the simulation. The aim of core-tracking is to simplify substructure analysis tasks by avoiding the use of subhalos and, at the same time, to more easily account for the so-called ``orphan'' galaxies, which have lost substantial dark mass due to tidal stripping. We show that simple models based on halo cores can reproduce the number and spatial distribution of galaxies found in optically-selected clusters in the Sloan Digital Sky Survey. We also discuss future applications of the core-tracking methodology in studying the galaxy-halo connection., Comment: 17 pages, 20 figures, 1 Appendix; version accepted by OJA

We compare two state-of-the-art numerical codes to study the overall accuracy in modeling the intergalactic medium and reproducing Lyman-$\alpha$ forest observables for DESI and high-resolution data sets. The codes employ different approaches to solving both gravity and modeling the gas hydrodynamics. The first code, Nyx, solves the Poisson equation using the Particle-Mesh (PM) method and the Euler equations using a finite volume method. The second code, \CRKHACC, uses a Tree-PM method to solve for gravity, and an improved Lagrangian smoothed particle hydrodynamics (SPH) technique, where fluid elements are modeled with particles, to treat the intergalactic gas. We compare the convergence behavior of the codes in flux statistics as well as the degree to which the codes agree in the converged limit. We find good agreement overall with differences being less than observational uncertainties, and a particularly notable $\lesssim$1\% agreement in the 1D flux power spectrum. This agreement was achieved by applying a tessellation methodology for reconstructing the density in \CRKHACC instead of using an SPH kernel as is standard practice. We show that use of the SPH kernel can lead to significant and unnecessary biases in flux statistics; this is especially prominent at high redshifts, $z \sim 5$, as the Lyman-$\alpha$ forest mostly comes from lower-density regions which are intrinsically poorly sampled by SPH particles., Comment: submitted to MNRAS

In this paper we introduce the Farpoint simulation, the latest member of the Hardware/Hybrid Accelerated Cosmology Code (HACC) gravity-only simulation family. The domain covers a volume of (1000$h^{-1}$Mpc)$^3$ and evolves close to two trillion particles, corresponding to a mass resolution of $m_p\sim 4.6\cdot 10^7 h^{-1}$M$_\odot$. These specifications enable comprehensive investigations of the galaxy-halo connection, capturing halos down to small masses. Further, the large volume resolves scales typical of modern surveys with good statistical coverage of high mass halos. The simulation was carried out on the GPU-accelerated system Summit, one of the fastest supercomputers currently available. We provide specifics about the Farpoint run and present an initial set of results. The high mass resolution facilitates precise measurements of important global statistics, such as the halo concentration-mass relation and the correlation function down to small scales. Selected subsets of the simulation data products are publicly available via the HACC Simulation Data Portal.

This paper introduces SMACC -- Subhalo Mass-loss Analysis using Core Catalogs. SMACC adds a mass model to substructure merger trees based on halo "core tracking." Our approach avoids the need for running expensive subhalo finding algorithms and instead uses subhalo mass-loss modeling to assign masses to halo cores. We present details of the SMACC methodology and demonstrate its excellent performance in describing halo substructure and its evolution. Validation of the approach is carried out using cosmological simulations at significantly different resolutions. We apply SMACC to the 1.24 trillion-particle Last Journey simulation and construct core catalogs with the additional mass information. These catalogs can be readily used as input to semi-analytic models or subhalo abundance matching approaches to determine approximate galaxy distributions, as well as for in-depth studies of small-scale structure evolution., Comment: 15 pages, 10 figures. Submitted to ApJ

Cosmological N-body simulations rank among the most computationally intensive efforts today. A key challenge is the analysis of structure, substructure, and the merger history for many billions of compact particle clusters, called halos. Effectively representing the merging history of halos is essential for many galaxy formation models used to generate synthetic sky catalogs, an important application of modern cosmological simulations. Generating realistic mock catalogs requires computing the halo formation history from simulations with large volumes and billions of halos over many time steps, taking hundreds of terabytes of analysis data. We present fast parallel algorithms for producing halo merger trees and tracking halo substructure from a single-level, density-based clustering algorithm. Merger trees are created from analyzing the halo-particle membership function in adjacent snapshots, and substructure is identified by tracking the "cores" of merging halos -- sets of particles near the halo center. Core tracking is performed after creating merger trees and uses the relationships found during tree construction to associate substructures with hosts. The algorithms are implemented with MPI and evaluated on a Cray XK7 supercomputer using up to 16,384 processes on data from HACC, a modern cosmological simulation framework. We present results for creating merger trees from 101 analysis snapshots taken from the Q Continuum, a large volume, high mass resolution, cosmological simulation evolving half a trillion particles., Comment: 2017 IEEE 24th International Conference on High Performance Computing

The Last Journey is a large-volume, gravity-only, cosmological N-body simulation evolving more than 1.24 trillion particles in a periodic box with a side-length of 5.025Gpc. It was implemented using the HACC simulation and analysis framework on the BG/Q system, Mira. The cosmological parameters are chosen to be consistent with the results from the Planck satellite. A range of analysis tools have been run in situ to enable a diverse set of science projects, and at the same time, to keep the resulting data amount manageable. Analysis outputs have been generated starting at redshift z~10 to allow for construction of synthetic galaxy catalogs using a semi-analytic modeling approach in post-processing. As part of our in situ analysis pipeline we employ a new method for tracking halo sub-structures, introducing the concept of subhalo cores. The production of multi-wavelength synthetic sky maps is facilitated by generating particle lightcones in situ, also beginning at z~10. We provide an overview of the simulation set-up and the generated data products; a first set of analysis results is presented. A subset of the data is publicly available., Comment: 14 pages, 9 figures. Accepted for publication in ApJS. New visualization and new results for the matter correlation function added, minor edits. The Last Journey data products can be accessed here: https://cosmology.alcf.anl.gov/

This paper introduces cosmoDC2, a large synthetic galaxy catalog designed to support precision dark energy science with the Large Synoptic Survey Telescope (LSST). CosmoDC2 is the starting point for the second data challenge (DC2) carried out by the LSST Dark Energy Science Collaboration (LSST DESC). The catalog is based on a trillion-particle, 4.225 Gpc^3 box cosmological N-body simulation, the `Outer Rim' run. It covers 440 deg^2 of sky area to a redshift of z=3 and is complete to a magnitude depth of 28 in the r-band. Each galaxy is characterized by a multitude of properties including stellar mass, morphology, spectral energy distributions, broadband filter magnitudes, host halo information and weak lensing shear. The size and complexity of cosmoDC2 requires an efficient catalog generation methodology; our approach is based on a new hybrid technique that combines data-driven empirical approaches with semi-analytic galaxy modeling. A wide range of observation-based validation tests has been implemented to ensure that cosmoDC2 enables the science goals of the planned LSST DESC DC2 analyses. This paper also represents the official release of the cosmoDC2 data set, including an efficient reader that facilitates interaction with the data., Comment: 27 pages, 17 figures, submitted to APJS

We describe the first major public data release from cosmological simulations carried out with Argonne's HACC code. This initial release covers a range of datasets from large gravity-only simulations. The data products include halo information for multiple redshifts, down-sampled particles, and lightcone outputs. We provide data from two very large LCDM simulations as well as beyond-LCDM simulations spanning eleven w0-wa cosmologies. Our release platform uses Petrel, a research data service, located at the Argonne Leadership Computing Facility. Petrel offers fast data transfer mechanisms and authentication via Globus, enabling simple and efficient access to stored datasets. Easy browsing of the available data products is provided via a web portal that allows the user to navigate simulation products efficiently. The data hub will be extended by adding more types of data products and by enabling computational capabilities to allow direct interactions with simulation results., Comment: 8 pages, 5 figures. Final version published in ApJS. The HACC Simulation Data Portal can be accessed here: https://cosmology.alcf.anl.gov/

We describe the Outer Rim cosmological simulation, one of the largest high-resolution N-body simulations performed to date, aimed at promoting science to be carried out with large-scale structure surveys. The simulation covers a volume of (4.225Gpc)^3 and evolves more than one trillion particles. It was executed on Mira, a BlueGene/Q system at the Argonne Leadership Computing Facility. We discuss some of the computational challenges posed by a system like Mira, a many-core supercomputer, and how the simulation code, HACC, has been designed to overcome these challenges. We have carried out a large range of analyses on the simulation data and we report on the results as well as the data products that have been generated. The full data set generated by the simulation totals more than 5PB of data, making data curation and data handling a large challenge in of itself. The simulation results have been used to generate synthetic catalogs for large-scale structure surveys, including DESI and eBOSS, as well as CMB experiments. A detailed catalog for the LSST DESC data challenges has been created as well. We publicly release some of the Outer Rim halo catalogs, downsampled particle information, and lightcone data., Comment: 10 pages, 10 figures. Submitted to ApJS. The Outer Rim data products can be accessed here: https://cosmology.alcf.anl.gov/

Cosmological cluster-scale strong gravitational lensing probes the mass distribution of the dense cores of massive dark matter halos and the structures along the line of sight from background sources to the observer. It is frequently assumed that the primary lens mass dominates the lensing, with the contribution of secondary masses along the line of sight being neglected. Secondary mass structures may, however, affect both the detectability of strong lensing in a given survey and modify the properties of the lensing that is detected. In this paper, we utilize a large cosmological N-body simulation and a multiple lens plane (and many source planes) ray-tracing technique to quantify the influence of line of sight halos on the detectability of cluster-scale strong lensing in a cluster sample with a mass limit that encompasses current cluster catalogs from the South Pole Telescope. We extract both primary and secondary halos from the "Outer Rim" simulation and consider two strong lensing realizations: one with only the primary halos included, and the other contains all secondary halos down to a mass limit. In both cases, we use the same source information extracted from the Hubble Ultra Deep Field, and create realistic lensed images consistent with moderately deep ground-based imaging. The results demonstrate that down to the mass limit considered the total number of lenses is boosted by about 13-21% when considering the complete multi-halo lightcone. The increment in strong lens counts peaks at lens redshifts of 0.6 approximately with no significant effect at z<0.3. The strongest trends are observed relative to the primary halo mass, with no significant impact in the most massive quintile of the halo sample, but increasingly boosting the observed lens counts toward small primary halo masses, with an enhancement greater than 50% in the least massive quintile of the halo masses considered., Comment: 11 pages, 4 figures, accepted for publication in The Astrophysical Journal

Gravitational lensing has become one of the most powerful tools available for investigating the 'dark side' of the universe. Cosmological strong gravitational lensing, in particular, probes the properties of the dense cores of dark matter halos over decades in mass and offers the opportunity to study the distant universe at flux levels and spatial resolutions otherwise unavailable. Studies of strongly-lensed variable sources offer yet further scientific opportunities. One of the challenges in realizing the potential of strong lensing is to understand the statistical context of both the individual systems that receive extensive follow-up study, as well as that of the larger samples of strong lenses that are now emerging from survey efforts. Motivated by these challenges, we have developed an image-simulation pipeline, PICS (Pipeline for Images of Cosmological Strong lensing) to generate realistic strong gravitational lensing signals from group and cluster scale lenses. PICS uses a low-noise and unbiased density estimator based on (resampled) Delaunay Tessellations to calculate the density field; lensed images are produced by ray-tracing images of actual galaxies from deep Hubble Space Telescope observations. Other galaxies, similarly sampled, are added to fill in the light cone. The pipeline further adds cluster-member galaxies and foreground stars into the lensed images. The entire image ensemble is then observed using a realistic point spread function which includes appropriate detector artifacts for bright stars. Noise is further added, including such non-Gaussian elements as noise window-paning from mosaiced observations, residual bad pixels, and cosmic rays. The aim is to produced simulated images that appear identical - to the eye (expert or otherwise) - to real observations in various imaging surveys., Comment: 20 pages, 13 figures, published

Precision measurements of the large scale structure of the Universe require large numbers of high fidelity mock catalogs to accurately assess, and account for, the presence of systematic effects. We introduce and test a scheme for generating mock catalogs rapidly using suitably derated N-body simulations. Our aim is to reproduce the large scale structure and the gross properties of dark matter halos with high accuracy, while sacrificing the details of the internal structure of the halos. By adjusting global and local time-steps in an N-body code, we demonstrate that we recover halo masses to better than 0.5% and the power spectrum to better than 1% both in real and redshift space for k = 1h/Mpc, while requiring a factor of 4 less CPU time. We also calibrate the redshift spacing of outputs required to generate simulated light cones. We find that outputs separated by every z = 0.05 allow us to interpolate particle positions and velocities to reproduce the real and redshift space power spectra to better than 1% (out to k = 1h/Mpc). We apply these ideas to generate a suite of simulations spanning a range of cosmologies, motivated by the Baryon Oscillation Spectroscopic Survey (BOSS) but broadly applicable to future large scale structure surveys including eBOSS and DESI. As an initial demonstration of the utility of such simulations, we calibrate the shift in the baryonic acoustic oscillation peak position as a function of galaxy bias with higher precision than has been possible so far. This paper also serves to document the simulations, which we make publicly available., Comment: 23 pages, 14 figures, submitted to JCAP, Mock catalogs are available at http://www.hep.anl.gov/cosmology/mock.html , fixed the link to the simulations. arXiv admin note: text overlap with arXiv:1410.2805 by other authors