Search

Uncertainty quantification (UQ) is essential for assessing the reliability of Earth observation (EO) products. However, the extensive use of machine learning models in EO introduces an additional layer of complexity, as those models themselves are inherently uncertain. While various UQ methods do exist for machine learning models, their performance on EO datasets remains largely unevaluated. A key challenge in the community is the absence of the ground truth for uncertainty, i.e. how certain the uncertainty estimates are, apart from the labels for the image/signal. This article fills this gap by introducing three benchmark datasets specifically designed for UQ in EO machine learning models. These datasets address three common problem types in EO: regression, image segmentation, and scene classification. They enable a transparent comparison of different UQ methods for EO machine learning models. We describe the creation and characteristics of each dataset, including data sources, preprocessing steps, and label generation, with a particular focus on calculating the reference uncertainty. We also showcase baseline performance of several machine learning models on each dataset, highlighting the utility of these benchmarks for model development and comparison. Overall, this article offers a valuable resource for researchers and practitioners working in artificial intelligence for EO, promoting a more accurate and reliable quality measure of the outputs of machine learning models. The dataset and code are accessible via https://gitlab.lrz.de/ai4eo/WG_Uncertainty., Comment: Submitted to IEEE Geoscience and Remote Sensing Magazine

A pulsar's scintillation bandwidth is inversely proportional to the scattering delay, making accurate measurements of scintillation bandwidth critical to characterize unmitigated delays in efforts to measure low-frequency gravitational waves with pulsar timing arrays. In this pilot work, we searched for a subset of known pulsars within $\sim$97% of the data taken with the PUPPI instrument for the AO327 survey with the Arecibo telescope, attempting to measure the scintillation bandwidths in the dataset by fitting to the 2D autocorrelation function of their dynamic spectra. We successfully measured 38 bandwidths from 23 pulsars (six without prior literature values), finding that: almost all of the measurements are larger than the predictions from NE2001 and YMW16 (two popular galactic models); NE2001 is more consistent with our measurements than YMW16; Gaussian fits to the bandwidth are more consistent with both electron density models than Lorentzian ones; and for the 17 pulsars with prior literature values, the measurements between various sources often vary by factors of a few. The success of Gaussian fits may be due to the use of Gaussian fits to train models in previous work. The variance of literature values over time could relate to the scaling factor used to compare measurements, but also seems consistent with time-varying interstellar medium parameters. This work can be extended to the rest of AO327 to further investigate these trends, highlighting the continuing importance of large archival datasets for projects beyond their initial conception., Comment: 21 pages, 8 figures, 2 tables

High-Performance Computing (HPC) systems are among the most energy-intensive scientific facilities, with electric power consumption reaching and often exceeding 20 megawatts per installation. Unlike other major scientific infrastructures such as particle accelerators or high-intensity light sources, which are few around the world, the number and size of supercomputers are continuously increasing. Even if every new system generation is more energy efficient than the previous one, the overall growth in size of the HPC infrastructure, driven by a rising demand for computational capacity across all scientific disciplines, and especially by artificial intelligence workloads (AI), rapidly drives up the energy demand. This challenge is particularly significant for HPC centers in Germany, where high electricity costs, stringent national energy policies, and a strong commitment to environmental sustainability are key factors. This paper describes various state-of-the-art strategies and innovations employed to enhance the energy efficiency of HPC systems within the national context. Case studies from leading German HPC facilities illustrate the implementation of novel heterogeneous hardware architectures, advanced monitoring infrastructures, high-temperature cooling solutions, energy-aware scheduling, and dynamic power management, among other optimizations. By reviewing best practices and ongoing research, this paper aims to share valuable insight with the global HPC community, motivating the pursuit of more sustainable and energy-efficient HPC operations., Comment: 30 pages, 3 figures, 4 tables

The Atomic Clock Ensemble in Space (ACES) mission is developing high performance clocks and links for space to test Einstein's theory of general relativity. From the International Space Station, the ACES payload will distribute a clock signal with fractional frequency stability and accuracy of 1E-16 establishing a worldwide network to compare clocks in space and on the ground. ACES will provide an absolute measurement of Einstein's gravitational redshift, it will search for time variations of fundamental constants, contribute to test topological dark matter models, and perform Standard Model Extension tests. Moreover, the ground clocks connected to the ACES network will be compared over different continents and used to measure geopotential differences at the clock locations. After solving some technical problems, the ACES flight model is now approaching its completion. System tests involving the laser-cooled Cs clock PHARAO, the active H-maser SHM and the on-board frequency comparator FCDP have measured the performance of the clock signal delivered by ACES. The ACES microwave link MWL is currently under test. The single-photon avalanche detector of the optical link ELT has been tested and will now be integrated in the ACES payload. The ACES mission concept, its scientific objectives, and the recent test results are discussed here together with the major milestones that will lead us to the ACES launch., Comment: 12 pages, 8 figures

Goal-directed interactive agents, which autonomously complete tasks through interactions with their environment, can assist humans in various domains of their daily lives. Recent advances in large language models (LLMs) led to a surge of new, more and more challenging tasks to evaluate such agents. To properly contextualize performance across these tasks, it is imperative to understand the different challenges they pose to agents. To this end, this survey compiles relevant tasks and environments for evaluating goal-directed interactive agents, structuring them along dimensions relevant for understanding current obstacles. An up-to-date compilation of relevant resources can be found on our project website: https://coli-saar.github.io/interactive-agents.

Phonocardiography has recently gained popularity in low-cost and remote monitoring, including passive fetal heart monitoring. Development for methods which analyse phonocardiographical data try to capitalize on this opportunity, and in recent years a multitude of such algorithms and models have been published. Although there is little to no standardization in these published algorithms and multiple parts of these models have to be reimplemented on a case-by-case basis. Datasets containing heart sound recordings also lack standardization in both data storage and labeling, especially in fetal phonocardiography. We are presenting a toolbox that can serve as a basis for a future standard framework for heart sound analysis. This toolbox contains some of the most widely used processing steps, and with these, complex analysis processes can be created. These functions can be individually tested. Due to the interdependence of the steps, we validated the current segmentation stage using a manually labeled fetal phonocardiogram dataset comprising 50 one-minute abdominal PCG recordings, which include 6,758 S1 and 6,729 S2 labels. Our results were compared to other common and available segmentation methods, peak detection with the Neurokit2 library, and the Hidden Semi-Markov Model by Springer et al. With a 30 ms tolerance our best model achieved a 97.1% F1 score and 10.8 +/- 7.9 ms mean absolute error for S1 detection. This detection accuracy outperformed all tested methods. With this a more accurate S2 detection method can be created as a multi-step process. After an accurate segmentation the extracted features should be representative of the selected segments, which allows for more accurate statistics or classification models. The toolbox contains functions for both feature extraction and statistics creation which are compatible with the previous steps., Comment: 25 pages, 11 figures, submitted to Open Source and Validated Computational Tools for Physiological Time Series Analysis, for associated program documentation, see https://pypcg-toolbox.readthedocs.io/en/latest/

Reinforcement learning often uses neural networks to solve complex control tasks. However, neural networks are sensitive to input perturbations, which makes their deployment in safety-critical environments challenging. This work lifts recent results from formally verifying neural networks against such disturbances to reinforcement learning in continuous state and action spaces using reachability analysis. While previous work mainly focuses on adversarial attacks for robust reinforcement learning, we train neural networks utilizing entire sets of perturbed inputs and maximize the worst-case reward. The obtained agents are verifiably more robust than agents obtained by related work, making them more applicable in safety-critical environments. This is demonstrated with an extensive empirical evaluation of four different benchmarks.

We introduce Modelizer - a novel framework that, given a black-box program, learns a _model from its input/output behavior_ using _neural machine translation_. The resulting model _mocks_ the original program: Given an input, the model predicts the output that would have been produced by the program. However, the model is also _reversible_ - that is, the model can predict the input that would have produced a given output. Finally, the model is _differentiable_ and can be efficiently restricted to predict only a certain aspect of the program behavior. Modelizer uses _grammars_ to synthesize inputs and to parse the resulting outputs, allowing it to learn sequence-to-sequence associations between token streams. Other than input and output grammars, Modelizer only requires the ability to execute the program. The resulting models are _small_, requiring fewer than 6.3 million parameters for languages such as Markdown or HTML; and they are _accurate_, achieving up to 95.4% accuracy and a BLEU score of 0.98 with standard error 0.04 in mocking real-world applications. We foresee several _applications_ of these models, especially as the output of the program can be any aspect of program behavior. Besides mocking and predicting program behavior, the model can also synthesize inputs that are likely to produce a particular behavior, such as failures or coverage., Comment: 42 pages, 6 figures, 8 tables

Models need appropriate inductive biases to effectively learn from small amounts of data and generalize systematically outside of the training distribution. While Transformers are highly versatile and powerful, they can still benefit from enhanced structural inductive biases for seq2seq tasks, especially those involving syntactic transformations, such as converting active to passive voice or semantic parsing. In this paper, we propose to strengthen the structural inductive bias of a Transformer by intermediate pre-training to perform synthetically generated syntactic transformations of dependency trees given a description of the transformation. Our experiments confirm that this helps with few-shot learning of syntactic tasks such as chunking, and also improves structural generalization for semantic parsing. Our analysis shows that the intermediate pre-training leads to attention heads that keep track of which syntactic transformation needs to be applied to which token, and that the model can leverage these attention heads on downstream tasks.

Discourse Representation Theory (DRT) distinguishes itself from other semantic representation frameworks by its ability to model complex semantic and discourse phenomena through structural nesting and variable binding. While seq2seq models hold the state of the art on DRT parsing, their accuracy degrades with the complexity of the sentence, and they sometimes struggle to produce well-formed DRT representations. We introduce the AMS parser, a compositional, neurosymbolic semantic parser for DRT. It rests on a novel mechanism for predicting quantifier scope. We show that the AMS parser reliably produces well-formed outputs and performs well on DRT parsing, especially on complex sentences.

Star formation rates (SFRs), gas-phase metallicities, and stellar masses are crucial for studying galaxy evolution. The different relations resulting from these properties give insights into the complex interplay of gas inside galaxies and their evolutionary trajectory and current characteristics. We aim to characterize these relations at $z\sim 0.3$, corresponding to a 3-4 Gyr lookback time. We utilized optical integral field spectroscopy of 65 emission-line galaxies from the MAGPI survey at a redshift of $0.28

The effect of non-planar substrate surface on homogeneity and quality of cold sprayed (CS) deposits was studied by scanning acoustic microscopy (SAM). Fe coatings were cold sprayed onto Al substrates containing artificially introduced grooves of square- and trapezoid-shaped geometries, with flat or cylindrical bottoms. The Al substrates were either wrought or cold sprayed, to comprehend their prospective influence on the Fe coatings build-up. SAM was then used to assess morphological properties of the materials from the cross-view and top-view directions. The microstructure below the surface of the studied samples was visualized by measuring the amplitudes of the reflection echoes and the velocity of the ultrasonic waves. The SAM analysis revealed that the regions of coating imperfections around the grooves are larger than what is suggested by standard scanning electron microscopy (SEM) observations. Furthermore, we found that the seemingly non-influenced coating regions that appear perfectly homogeneous and dense in SEM do, in fact, possess heterogeneous microstructure associated with the individual CS nozzle passes., Comment: Manuscript accepted to the Journal of Thermal Spray Technology (JTST)

There is an increasing trend towards evaluating NLP models with LLM-generated judgments instead of human judgments. In the absence of a comparison against human data, this raises concerns about the validity of these evaluations; in case they are conducted with proprietary models, this also raises concerns over reproducibility. We provide JUDGE-BENCH, a collection of 20 NLP datasets with human annotations, and comprehensively evaluate 11 current LLMs, covering both open-weight and proprietary models, for their ability to replicate the annotations. Our evaluations show that each LLM exhibits a large variance across datasets in its correlation to human judgments. We conclude that LLMs are not yet ready to systematically replace human judges in NLP.

We present a method for rewriting an input sentence to match specific values of nontrivial linguistic features, such as dependency depth. In contrast to earlier work, our method uses in-context learning rather than finetuning, making it applicable in use cases where data is sparse. We show that our model performs accurate rewrites and matches the state of the art on rewriting sentences to a specified school grade level.

Collaboration is an integral part of human dialogue. Typical task-oriented dialogue games assign asymmetric roles to the participants, which limits their ability to elicit naturalistic role-taking in collaboration and its negotiation. We present a novel and simple online setup that favors balanced collaboration: a two-player 2D object placement game in which the players must negotiate the goal state themselves. We show empirically that human players exhibit a variety of role distributions, and that balanced collaboration improves task performance. We also present an LLM-based baseline agent which demonstrates that automatic playing of our game is an interesting challenge for artificial systems.

We study the fixed points of outer-totalistic cellular automata on sparse random regular graphs. These can be seen as constraint satisfaction problems, where each variable must adhere to the same local constraint, which depends solely on its state and the total number of its neighbors in each possible state. Examples of this setting include classical problems such as independent sets or assortative/dissasortative partitions. We analyse the existence and number of fixed points in the large system limit using the cavity method, under both the replica symmetric (RS) and one-step replica symmetry breaking (1RSB) assumption. This method allows us to characterize the structure of the space of solutions, in particular, if the solutions are clustered and whether the clusters contain frozen variables. This last property is conjectured to be linked to the typical algorithmic hardness of the problem. We bring experimental evidence for this claim by studying the performance of the belief-propagation reinforcement algorithm, a message-passing-based solver for these constraint satisfaction problems., Comment: 26 pages, 7 figures

Recent research interest in the learning-based processing of garments, from virtual fitting to generation and reconstruction, stumbles on a scarcity of high-quality public data in the domain. We contribute to resolving this need by presenting the first large-scale synthetic dataset of 3D made-to-measure garments with sewing patterns, as well as its generation pipeline. GarmentCodeData contains 115,000 data points that cover a variety of designs in many common garment categories: tops, shirts, dresses, jumpsuits, skirts, pants, etc., fitted to a variety of body shapes sampled from a custom statistical body model based on CAESAR, as well as a standard reference body shape, applying three different textile materials. To enable the creation of datasets of such complexity, we introduce a set of algorithms for automatically taking tailor's measures on sampled body shapes, sampling strategies for sewing pattern design, and propose an automatic, open-source 3D garment draping pipeline based on a fast XPBD simulator, while contributing several solutions for collision resolution and drape correctness to enable scalability. Project Page: https://igl.ethz.ch/projects/GarmentCodeData/, Comment: Accepted to ECCV 2024. Sept 4th, 2024: release of GarmentCodeData(v2)

Statistical inference is a major scientific endeavor for many researchers. In terms of inferential methods implemented to mixed-effects models, significant progress has been made in the R software. However, these advances primarily concern classical estimators (ML, REML) and mainly focus on fixed effects. In the confintROB package, we have implemented various bootstrap methods for computing confidence intervals (CIs) not only for fixed effects but also for variance components. These methods can be implemented with the widely used lmer function from the lme4 package, as well as with the rlmer function from the robustlmm package and the varComprob function from the robustvarComp package. These functions implement robust estimation methods suitable for data with outliers. The confintROB package implements the Wald method for fixed effects, whereas for both fixed effects and variance components, two bootstrap methods are implemented: the parametric bootstrap and the wild bootstrap. Moreover, the confintROB package can obtain both the percentile and the bias-corrected accelerated versions of CIs., Comment: 19 pages, 2 figures

Quantum communication promises unprecedented communication capabilities enabled by the transmission of quantum states of light. However, current implementations face severe limitations in communication distance due to photon loss. Silicon carbide (SiC) defects have emerged as a promising quantum device platform, offering strong optical transitions, long spin coherence lifetimes and the opportunity for integration with semiconductor devices. Some defects with optical transitions in the telecom range have been identified, allowing to interface with fiber networks without the need for wavelength conversion. These unique properties make SiC an attractive platform for the implementation of quantum nodes for quantum communication networks. We provide an overview of the most prominent defects in SiC and their implementation in spin-photon interfaces. Furthermore, we model a memory-enhanced quantum communication protocol in order to extract the parameters required to surpass a direct point-to-point link performance. Based on these insights, we summarize the key steps required towards the deployment of SiC devices in large-scale quantum communication networks., Comment: 20 pages, 8 figures

Catalyst particles or complexes suspended in liquid films can trigger chemical reactions leading to inhomogeneous concentrations of reactants and products in the film. We demonstrate that the sensitivity of the liquid film's gas-liquid surface tension to these inhomogeneous concentrations strongly impacts the film stability. Using linear stability analysis, we identify novel scenarios in which the film can be either stabilized or destabilized by the reactions. Furthermore, we find so far unrevealed rupture mechanisms which are absent in the chemically inactive case. The linear stability predictions are confirmed by numerical simulations, which also demonstrate that the shape of chemically active droplets can depart from the spherical cap and that unsteady states such as traveling and standing waves might appear. Finally, we critically discuss the relevance of our predictions by showing that the range of our selected parameters is well accessible by typical experiments.

Our Sun lies within 300 pc of the 2.7-kpc-long sinusoidal chain of dense gas clouds known as the Radcliffe Wave. The structure's wave-like shape was discovered using 3D dust mapping, but initial kinematic searches for oscillatory motion were inconclusive. Here we present evidence that the Radcliffe Wave is oscillating through the Galactic plane while also drifting radially away from the Galactic Center. We use measurements of line-of-sight velocity for 12CO and 3D velocities of young stellar clusters to show that the most massive star-forming regions spatially associated with the Radcliffe Wave (including Orion, Cepheus, North America, and Cygnus X) move as if they are part of an oscillating wave driven by the gravitational acceleration of the Galactic potential. By treating the Radcliffe Wave as a coherently oscillating structure, we can derive its motion independently of the local Galactic mass distribution, and directly measure local properties of the Galactic potential as well as the Sun's vertical oscillation period. In addition, the measured drift of the Radcliffe Wave radially outward from the Galactic Center suggests that the cluster whose supernovae ultimately created today's expanding Local Bubble may have been born in the Radcliffe Wave., Comment: Published in Nature on 20 February 2024. For data, interactive visualizations, and more information see https://sites.google.com/cfa.harvard.edu/radcliffewave

The prominent radio quasar PKS 2215+020 (J2217+0220) was once labelled as a new laboratory for core--jet physics at redshift z=3.572 because of its exceptionally extended jet structure traceable with very long baseline interferometric (VLBI) observations up to a ~600 pc projected distance from the compact core and a hint of an arcsec-scale radio and an X-ray jet. While the presence of an X-ray jet could not be confirmed later, this active galactic nucleus is still unique at high redshift with its long VLBI jet. Here, we analyse archival multi-epoch VLBI imaging data at five frequency bands from 1.7 to 15.4 GHz covering a period of more than 25 years from 1995 to 2020. We constrain apparent proper motions of jet components in PKS 2215+020 for the first time. Brightness distribution modeling at 8 GHz reveals a nearly 0.02 mas/yr proper motion (moderately superluminal with apparently two times the speed of light), and provides delta=11.5 for the Doppler-boosting factor in the inner relativistic jet that is inclined within 2 deg to the line of sight and has a Gamma=6 bulk Lorentz factor. These values qualify PKS 2215+020 as a blazar, with rather typical jet properties in a small sample of only about 20 objects at z>3.5 that have similar measurements to date. According to the 2-GHz VLBI data, the diffuse and extended outer emission feature at ~60 mas from the core, probably a place where the jet interacts with and decelerated by the ambient galactic medium, is consistent with being stationary, albeit slow motion cannot be excluded based on the presently available data., Comment: 18 pages, 2 figures, 5 tables, accepted for publication in Universe

Neural networks are vulnerable to adversarial attacks, i.e., small input perturbations can significantly affect the outputs of a neural network. In safety-critical environments, the inputs often contain noisy sensor data; hence, in this case, neural networks that are robust against input perturbations are required. To ensure safety, the robustness of a neural network must be formally verified. However, training and formally verifying robust neural networks is challenging. We address both of these challenges by employing, for the first time, an end-to-end set-based training procedure that trains robust neural networks for formal verification. Our training procedure trains neural networks, which can be easily verified using simple polynomial-time verification algorithms. Moreover, our extensive evaluation demonstrates that our set-based training procedure effectively trains robust neural networks, which are easier to verify. Set-based trained neural networks consistently match or outperform those trained with state-of-the-art robust training approaches.

High-order behavior of the perturbative expansion for short-distance observables in QCD is intimately related to the contributions of small momenta in the corresponding Feynman diagrams and this correspondence provides one with a useful tool to investigate power-suppressed nonperturbative corrections. We use this technique to study the structure of power corrections to parton quasi- and pseudo-GPDs which are used in lattice calculations of generalized parton distributions. As the main result, we predict the functional dependence of the leading power corrections to quasi(pseudo)-GPDs on $x$ variable for nonzero skewedness parameter $\xi$. The kinematic point $x=\pm\xi$ turns out to be special. We find that the nonperturbative corrections to quasi-GPDs at this point are suppressed by the first power of the hard scale only. These contributions come from soft momenta and have nothing to do with the known UV renormalon in the Wilson line. We also show that power corrections can be strongly suppressed by the normalization procedure.