Search

The unprecedented amount of scientific data has introduced heavy pressure on the current data storage and transmission systems. Progressive compression has been proposed to mitigate this problem, which offers data access with on-demand precision. However, existing approaches only consider precision control on primary data, leaving uncertainties on the quantities of interest (QoIs) derived from it. In this work, we present a progressive data retrieval framework with guaranteed error control on derivable QoIs. Our contributions are three-fold. (1) We carefully derive the theories to strictly control QoI errors during progressive retrieval. Our theory is generic and can be applied to any QoIs that can be composited by the basis of derivable QoIs proved in the paper. (2) We design and develop a generic progressive retrieval framework based on the proposed theories, and optimize it by exploring feasible progressive representations. (3) We evaluate our framework using five real-world datasets with a diverse set of QoIs. Experiments demonstrate that our framework can faithfully respect any user-specified QoI error bounds in the evaluated applications. This leads to over 2.02x performance gain in data transfer tasks compared to transferring the primary data while guaranteeing a QoI error that is less than 1E-5., Comment: SC'24

The next-generation radio astronomy instruments are providing a massive increase in sensitivity and coverage, through increased stations in the array and frequency span. Two primary problems encountered when processing the resultant avalanche of data are the need for abundant storage and I/O. An example of this is the data deluge expected from the SKA Telescopes of more than 60PB per day, all to be stored on the buffer filesystem. Compressing the data is an obvious solution. We used MGARD, an error-controlled compressor, and applied it to simulated and real visibility data, in noise-free and noise-dominated regimes. As the data has an implicit error level in the system temperature, using an error bound in compression provides a natural metric for compression. Measuring the degradation of images reconstructed using the lossy compressed data, we explore the trade-off between these error bounds and the corresponding compression ratios, as well as the impact on science quality derived from the lossy compressed data products through a series of experiments. We studied the global and local impacts on the output images. We found relative error bounds of as much as $10\%$, which provide compression ratios of about 20, have a limited impact on the continuum imaging as the increased noise is less than the image RMS. For extremely sensitive observations and for very precious data, we would recommend a $0.1\%$ error bound with compression ratios of about 4. These have noise impacts two orders of magnitude less than the image RMS levels. At these levels, the limits are due to instabilities in the deconvolution methods. We compared the results to the alternative compression tool DYSCO. MGARD provides better compression for similar results, and has a host of potentially powerful additional features., Comment: 10 figures

We present a general framework for compressing unstructured scientific data with known local connectivity. A common application is simulation data defined on arbitrary finite element meshes. The framework employs a greedy topology preserving reordering of original nodes which allows for seamless integration into existing data processing pipelines. This reordering process depends solely on mesh connectivity and can be performed offline for optimal efficiency. However, the algorithm's greedy nature also supports on-the-fly implementation. The proposed method is compatible with any compression algorithm that leverages spatial correlations within the data. The effectiveness of this approach is demonstrated on a large-scale real dataset using several compression methods, including MGARD, SZ, and ZFP., Comment: 6 pages, 9 figures

The next generation of radio astronomy telescopes are challenging existing data analysis paradigms, as they have an order of magnitude larger collecting area and bandwidth. The two primary problems encountered when processing this data are the need for storage and that processing is primarily I/O limited. An example of this is the data deluge expected from the SKA-Low Telescope of about 300 PB per year. To remedy these issues, we have demonstrated lossy and lossless compression of data on an existing precursor telescope, the Australian Square Kilometre Array Pathfinder (ASKAP), using MGARD and ADIOS2 libraries. We find data processing is faster by a factor of 7 and give compression ratios from a factor of 7 (lossless) up to 37 (lossy with an absolute error bound of 1e-3). We discuss the effectiveness of lossy MGARD compression and its adherence to the designated error bounds, the trade-off between these error bounds and the corresponding compression ratios, as well as the potential consequences of these I/O and storage improvements on the science quality of the data products., Comment: 9 pages, 10 figures. Included in the conference proceedings of Cray User Group Conference 2024

MGARD (MultiGrid Adaptive Reduction of Data) is an algorithm for compressing and refactoring scientific data, based on the theory of multigrid methods. The core algorithm is built around stable multilevel decompositions of conforming piecewise linear $C^0$ finite element spaces, enabling accurate error control in various norms and derived quantities of interest. In this work, we extend this construction to arbitrary order Lagrange finite elements $\mathbb{Q}_p$, $p \geq 0$, and propose a reformulation of the algorithm as a lifting scheme with polynomial predictors of arbitrary order. Additionally, a new formulation using a compactly supported wavelet basis is discussed, and an explicit construction of the proposed wavelet transform for uniform dyadic grids is described.

We present a new analysis of Kepler-21, the brightest (V = 8.5) Kepler system with a known transiting exoplanet, Kepler-21 b. Kepler-21 b is a radius valley planet ($R = 1.6\pm 0.2 R_{\oplus}$) with an Earth-like composition (8.38$\pm$1.62 g/cc), though its mass and radius fall in the regime of possible "water worlds." We utilize new Keck/HIRES and WIYN/NEID radial velocity (RV) data in conjunction with Kepler and TESS photometry to perform a detailed study of activity mitigation between photometry and RVs. We additionally refine the system parameters, and we utilize Gaia astrometry to place constraints on a long-term RV trend. Our activity analysis affirms the quality of Kepler photometry for removing correlated noise from RVs, despite its temporal distance, though we reveal some cases where TESS may be superior. Using refined orbital parameters and updated composition curves, we rule out a ``water world" scenario for Kepler-21 b, and we identify a long period super-Jupiter planetary candidate, Kepler-21 (c).

This paper presents a novel end-to-end framework for closed-form computation and visualization of critical point uncertainty in 2D uncertain scalar fields. Critical points are fundamental topological descriptors used in the visualization and analysis of scalar fields. The uncertainty inherent in data (e.g., observational and experimental data, approximations in simulations, and compression), however, creates uncertainty regarding critical point positions. Uncertainty in critical point positions, therefore, cannot be ignored, given their impact on downstream data analysis tasks. In this work, we study uncertainty in critical points as a function of uncertainty in data modeled with probability distributions. Although Monte Carlo (MC) sampling techniques have been used in prior studies to quantify critical point uncertainty, they are often expensive and are infrequently used in production-quality visualization software. We, therefore, propose a new end-to-end framework to address these challenges that comprises a threefold contribution. First, we derive the critical point uncertainty in closed form, which is more accurate and efficient than the conventional MC sampling methods. Specifically, we provide the closed-form and semianalytical (a mix of closed-form and MC methods) solutions for parametric (e.g., uniform, Epanechnikov) and nonparametric models (e.g., histograms) with finite support. Second, we accelerate critical point probability computations using a parallel implementation with the VTK-m library, which is platform portable. Finally, we demonstrate the integration of our implementation with the ParaView software system to demonstrate near-real-time results for real datasets., Comment: 9 pages paper + 2 page references, 8 figures, IEEE VIS 2024 paper to be published as a special issue of IEEE Transactions on Visualization and Computer Graphics (TVCG)

The Solar eruptioN Integral Field Spectrograph (SNIFS) is a solar-gazing spectrograph scheduled to fly in the summer of 2025 on a NASA sounding rocket. Its goal is to view the solar chromosphere and transition region at a high cadence (1s) both spatially (0.5") and spectrally (33 m{\AA}) viewing wavelengths around Lyman Alpha (1216 {\AA}), Si iii (1206 {\AA}) and O v (1218 {\AA}) to observe spicules, nanoflares, and possibly a solar flare. This time cadence will provide yet-unobserved detail about fast-changing features of the Sun. The instrument is comprised of a Gregorian-style reflecting telescope combined with a spectrograph via a specialized mirrorlet array that focuses the light from each spatial location in the image so that it may be spectrally dispersed without overlap from neighboring locations. This paper discusses the driving science, detailed instrument and subsystem design, and pre-integration testing of the SNIFS instrument., Comment: 22 pages (not including references), 7 figures, submitting to Solar Physics

Scientists conduct large-scale simulations to compute derived quantities-of-interest (QoI) from primary data. Often, QoI are linked to specific features, regions, or time intervals, such that data can be adaptively reduced without compromising the integrity of QoI. For many spatiotemporal applications, these QoI are binary in nature and represent presence or absence of a physical phenomenon. We present a pipelined compression approach that first uses neural-network-based techniques to derive regions where QoI are highly likely to be present. Then, we employ a Guaranteed Autoencoder (GAE) to compress data with differential error bounds. GAE uses QoI information to apply low-error compression to only these regions. This results in overall high compression ratios while still achieving downstream goals of simulation or data collections. Experimental results are presented for climate data generated from the E3SM Simulation model for downstream quantities such as tropical cyclone and atmospheric river detection and tracking. These results show that our approach is superior to comparable methods in the literature., Comment: 7 pages. arXiv admin note: text overlap with arXiv:2404.18063

We present an approach called guaranteed block autoencoder that leverages Tensor Correlations (GBATC) for reducing the spatiotemporal data generated by computational fluid dynamics (CFD) and other scientific applications. It uses a multidimensional block of tensors (spanning in space and time) for both input and output, capturing the spatiotemporal and interspecies relationship within a tensor. The tensor consists of species that represent different elements in a CFD simulation. To guarantee the error bound of the reconstructed data, principal component analysis (PCA) is applied to the residual between the original and reconstructed data. This yields a basis matrix, which is then used to project the residual of each instance. The resulting coefficients are retained to enable accurate reconstruction. Experimental results demonstrate that our approach can deliver two orders of magnitude in reduction while still keeping the errors of primary data under scientifically acceptable bounds. Compared to reduction-based approaches based on SZ, our method achieves a substantially higher compression ratio for a given error bound or a better error for a given compression ratio., Comment: 10 pages, 8 figures

We describe MGARD, a software providing MultiGrid Adaptive Reduction for floating-point scientific data on structured and unstructured grids. With exceptional data compression capability and precise error control, MGARD addresses a wide range of requirements, including storage reduction, high-performance I/O, and in-situ data analysis. It features a unified application programming interface (API) that seamlessly operates across diverse computing architectures. MGARD has been optimized with highly-tuned GPU kernels and efficient memory and device management mechanisms, ensuring scalable and rapid operations., Comment: 20 pages, 8 figures

Scientific discoveries are increasingly constrained by limited storage space and I/O capacities. For time-series simulations and experiments, their data often need to be decimated over timesteps to accommodate storage and I/O limitations. In this paper, we propose a technique that addresses storage costs while improving post-analysis accuracy through spatiotemporal adaptive, error-controlled lossy compression. We investigate the trade-off between data precision and temporal output rates, revealing that reducing data precision and increasing timestep frequency lead to more accurate analysis outcomes. Additionally, we integrate spatiotemporal feature detection with data compression and demonstrate that performing adaptive error-bounded compression in higher dimensional space enables greater compression ratios, leveraging the error propagation theory of a transformation-based compressor. To evaluate our approach, we conduct experiments using the well-known E3SM climate simulation code and apply our method to compress variables used for cyclone tracking. Our results show a significant reduction in storage size while enhancing the quality of cyclone tracking analysis, both quantitatively and qualitatively, in comparison to the prevalent timestep decimation approach. Compared to three state-of-the-art lossy compressors lacking feature preservation capabilities, our adaptive compression framework improves perfectly matched cases in TC tracking by 26.4-51.3% at medium compression ratios and by 77.3-571.1% at large compression ratios, with a merely 5-11% computational overhead., Comment: 10 pages, 13 figures, 2023 IEEE International Conference on e-Science and Grid Computing

Data compression is becoming critical for storing scientific data because many scientific applications need to store large amounts of data and post process this data for scientific discovery. Unlike image and video compression algorithms that limit errors to primary data, scientists require compression techniques that accurately preserve derived quantities of interest (QoIs). This paper presents a physics-informed compression technique implemented as an end-to-end, scalable, GPU-based pipeline for data compression that addresses this requirement. Our hybrid compression technique combines machine learning techniques and standard compression methods. Specifically, we combine an autoencoder, an error-bounded lossy compressor to provide guarantees on raw data error, and a constraint satisfaction post-processing step to preserve the QoIs within a minimal error (generally less than floating point error). The effectiveness of the data compression pipeline is demonstrated by compressing nuclear fusion simulation data generated by a large-scale fusion code, XGC, which produces hundreds of terabytes of data in a single day. Our approach works within the ADIOS framework and results in compression by a factor of more than 150 while requiring only a few percent of the computational resources necessary for generating the data, making the overall approach highly effective for practical scenarios.

Rapid growth in scientific data and a widening gap between computational speed and I/O bandwidth makes it increasingly infeasible to store and share all data produced by scientific simulations. Instead, we need methods for reducing data volumes: ideally, methods that can scale data volumes adaptively so as to enable negotiation of performance and fidelity tradeoffs in different situations. Multigrid-based hierarchical data representations hold promise as a solution to this problem, allowing for flexible conversion between different fidelities so that, for example, data can be created at high fidelity and then transferred or stored at lower fidelity via logically simple and mathematically sound operations. However, the effective use of such representations has been hindered until now by the relatively high costs of creating, accessing, reducing, and otherwise operating on such representations. We describe here highly optimized data refactoring kernels for GPU accelerators that enable efficient creation and manipulation of data in multigrid-based hierarchical forms. We demonstrate that our optimized design can achieve up to 264 TB/s aggregated data refactoring throughput -- 92% of theoretical peak -- on 1024 nodes of the Summit supercomputer. We showcase our optimized design by applying it to a large-scale scientific visualization workflow and the MGARD lossy compression software., Comment: arXiv admin note: text overlap with arXiv:2007.04457

This White Paper presents a mission concept called MIRECLE - the Mid-InfraRed Exoplanet CLimate Explorer. With a moderately sized aperture of 2 meters, broad wavelength coverage (4 - 25 um), and next generation instruments, MIRECLE will be capable of efficiently characterizing a statistically significant sample of terrestrial planets, many of which will be in their host stars's habitable zones. Spectroscopic characterization of terrestrial atmospheres will provide constraints for the distribution of planets with tenuous vs. substantial atmospheres, on the inner and outer edges of the habitable zone, and climate models to assess the potential for habitability. For the few brightest targets, the detection of specific combinations of molecules would provide evidence of biosignatures. For all other targets, this comprehensive survey would filter out the airless, desiccated, or lifeless worlds, thus providing a subset of potentially habitable worlds ready for in-depth atmospheric characterization using a larger aperture telescope., Comment: Astro2020: Decadal Survey on Astronomy and Astrophysics APC White Paper

Many state-of-the-art general object detection methods make use of shared full-image convolutional features (as in Faster R-CNN). This achieves a reasonable test-phase computation time while enjoys the discriminative power provided by large Convolutional Neural Network (CNN) models. Such designs excel on benchmarks which contain natural images but which have very unnatural distributions, i.e. they have an unnaturally high-frequency of the target classes and a bias towards a "friendly" or "dominant" object scale. In this paper we present further study of the use and adaptation of the Faster R-CNN object detection method for datasets presenting natural scale distribution and unbiased real-world object frequency. In particular, we show that better alignment of the detector scale sensitivity to the extant distribution improves vehicle detection performance. We do this by modifying both the selection of Region Proposals, and through using more scale-appropriate full-image convolution features within the CNN model. By selecting better scales in the region proposal input and by combining feature maps through careful design of the convolutional neural network, we improve performance on smaller objects. We significantly increase detection AP for the KITTI dataset car class from 76.3% on our baseline Faster R-CNN detector to 83.6% in our improved detector., Comment: Accepted by 2017 IEEE Intelligent Vehicles Symposium (IV). Link: http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7995812

A primary goal of direct imaging techniques is to spectrally characterize the atmospheres of planets around other stars at extremely high contrast levels. To achieve this goal, coronagraphic instruments have favored integral field spectrographs (IFS) as the science cameras to disperse the entire search area at once and obtain spectra at each location, since the planet position is not known a priori. These spectrographs are useful against confusion from speckles and background objects, and can also help in the speckle subtraction and wavefront control stages of the coronagraphic observation. We present a software package, the Coronagraph and Rapid Imaging Spectrograph in Python (crispy) to simulate the IFS of the WFIRST Coronagraph Instrument (CGI). The software propagates input science cubes using spatially and spectrally resolved coronagraphic focal plane cubes, transforms them into IFS detector maps and ultimately reconstructs the spatio-spectral input scene as a 3D datacube. Simulated IFS cubes can be used to test data extraction techniques, refine sensitivity analyses and carry out design trade studies of the flight CGI-IFS instrument. crispy is a publicly available Python package and can be adapted to other IFS designs., Comment: 15 pages

The Prototype Imaging Spectrograph for Coronagraphic Exoplanet Studies (PISCES) is a high contrast integral field spectrograph (IFS) whose design was driven by WFIRST coronagraph instrument requirements. We present commissioning and operational results using PISCES as a camera on the High Contrast Imaging Testbed at JPL. PISCES has demonstrated ability to achieve high contrast spectral retrieval with flight-like data reduction and analysis techniques., Comment: Author's copy - Proceedings of SPIE Volume 10400. Citation to SPIE proceedings volume will be added when available

We theoretically derive the lower and upper bounds of quantum Fisher information (QFI) of an SU(1,1) interferometer whatever the input state chosen. According to the QFI, the crucial resource for quantum enhancement is shown to be large intramode correlations indicated by the Mandel $Q$-parameter. For a photon-subtracted squeezed vacuum state with high super-Poissonian statistics in one input port and a coherent state in the other input port, the quantum Cram\'{e}r-Rao bound of the SU(1,1) interferometer can beat $1/\langle\hat{N}\rangle$ scaling in presence of large fluctuations in the number of photons, with a given fixed input mean number of photons. The definition of the Heisenberg limit (HL) should take into account the amount of fluctuations. The HL considering the number fluctuation effect may be the ultimate phase limit., Comment: 9 pages, 5 figures. arXiv admin note: text overlap with arXiv:1608.00389

We theoretically present the quantum Cram\'{e}r-Rao bounds (QCRB) of an SU(1,1) interferometer for Gaussian states input with and without the internal photonic losses. The phase shifts in the single arm and in the double arms are studied and the corresponding analytical expressions of quantum Fisher information with Gaussian input states are presented. Different from the traditional Mach-Zehnder interferometer, the QCRB of single arm case is slightly higher or lower than that of double arms case depending on the input states. With a fixed mean photon number and for pure Gaussian state input, the optimal sensitivity is achieved with a squeezed vacuum input in one mode and the vacuum input in the other. We compare the QCRB with the standard quantum limit and Heisenberg limit. In the case of small internal losses the QCRB can beat the standard quantum limit., Comment: 6 pages for the main text and 5 pages for the supplemental material

We describe a detailed radial velocity error budget for the NASA-NSF Extreme Precision Doppler Spectrometer instrument concept NEID (NN-explore Exoplanet Investigations with Doppler spectroscopy). Such an instrument performance budget is a necessity for both identifying the variety of noise sources currently limiting Doppler measurements, and estimating the achievable performance of next generation exoplanet hunting Doppler spectrometers. For these instruments, no single source of instrumental error is expected to set the overall measurement floor. Rather, the overall instrumental measurement precision is set by the contribution of many individual error sources. We use a combination of numerical simulations, educated estimates based on published materials, extrapolations of physical models, results from laboratory measurements of spectroscopic subsystems, and informed upper limits for a variety of error sources to identify likely sources of systematic error and construct our global instrument performance error budget. While natively focused on the performance of the NEID instrument, this modular performance budget is immediately adaptable to a number of current and future instruments. Such an approach is an important step in charting a path towards improving Doppler measurement precisions to the levels necessary for discovering Earth-like planets., Comment: 20 pages, 12 figures, published in Proc. of SPIE Astronomical Telescopes + Instrumentation 2016