Search

Goal-directed interactive agents, which autonomously complete tasks through interactions with their environment, can assist humans in various domains of their daily lives. Recent advances in large language models (LLMs) led to a surge of new, more and more challenging tasks to evaluate such agents. To properly contextualize performance across these tasks, it is imperative to understand the different challenges they pose to agents. To this end, this survey compiles relevant tasks and environments for evaluating goal-directed interactive agents, structuring them along dimensions relevant for understanding current obstacles. An up-to-date compilation of relevant resources can be found on our project website: https://coli-saar.github.io/interactive-agents.

Phonocardiography has recently gained popularity in low-cost and remote monitoring, including passive fetal heart monitoring. Development for methods which analyse phonocardiographical data try to capitalize on this opportunity, and in recent years a multitude of such algorithms and models have been published. Although there is little to no standardization in these published algorithms and multiple parts of these models have to be reimplemented on a case-by-case basis. Datasets containing heart sound recordings also lack standardization in both data storage and labeling, especially in fetal phonocardiography. We are presenting a toolbox that can serve as a basis for a future standard framework for heart sound analysis. This toolbox contains some of the most widely used processing steps, and with these, complex analysis processes can be created. These functions can be individually tested. Due to the interdependence of the steps, we validated the current segmentation stage using a manually labeled fetal phonocardiogram dataset comprising 50 one-minute abdominal PCG recordings, which include 6,758 S1 and 6,729 S2 labels. Our results were compared to other common and available segmentation methods, peak detection with the Neurokit2 library, and the Hidden Semi-Markov Model by Springer et al. With a 30 ms tolerance our best model achieved a 97.1% F1 score and 10.8 +/- 7.9 ms mean absolute error for S1 detection. This detection accuracy outperformed all tested methods. With this a more accurate S2 detection method can be created as a multi-step process. After an accurate segmentation the extracted features should be representative of the selected segments, which allows for more accurate statistics or classification models. The toolbox contains functions for both feature extraction and statistics creation which are compatible with the previous steps., Comment: 25 pages, 11 figures, submitted to Open Source and Validated Computational Tools for Physiological Time Series Analysis, for associated program documentation, see https://pypcg-toolbox.readthedocs.io/en/latest/

Reinforcement learning often uses neural networks to solve complex control tasks. However, neural networks are sensitive to input perturbations, which makes their deployment in safety-critical environments challenging. This work lifts recent results from formally verifying neural networks against such disturbances to reinforcement learning in continuous state and action spaces using reachability analysis. While previous work mainly focuses on adversarial attacks for robust reinforcement learning, we train neural networks utilizing entire sets of perturbed inputs and maximize the worst-case reward. The obtained agents are verifiably more robust than agents obtained by related work, making them more applicable in safety-critical environments. This is demonstrated with an extensive empirical evaluation of four different benchmarks.

We introduce Modelizer - a novel framework that, given a black-box program, learns a _model from its input/output behavior_ using _neural machine translation_. The resulting model _mocks_ the original program: Given an input, the model predicts the output that would have been produced by the program. However, the model is also _reversible_ - that is, the model can predict the input that would have produced a given output. Finally, the model is _differentiable_ and can be efficiently restricted to predict only a certain aspect of the program behavior. Modelizer uses _grammars_ to synthesize inputs and to parse the resulting outputs, allowing it to learn sequence-to-sequence associations between token streams. Other than input and output grammars, Modelizer only requires the ability to execute the program. The resulting models are _small_, requiring fewer than 6.3 million parameters for languages such as Markdown or HTML; and they are _accurate_, achieving up to 95.4% accuracy and a BLEU score of 0.98 with standard error 0.04 in mocking real-world applications. We foresee several _applications_ of these models, especially as the output of the program can be any aspect of program behavior. Besides mocking and predicting program behavior, the model can also synthesize inputs that are likely to produce a particular behavior, such as failures or coverage., Comment: 42 pages, 6 figures, 8 tables

Models need appropriate inductive biases to effectively learn from small amounts of data and generalize systematically outside of the training distribution. While Transformers are highly versatile and powerful, they can still benefit from enhanced structural inductive biases for seq2seq tasks, especially those involving syntactic transformations, such as converting active to passive voice or semantic parsing. In this paper, we propose to strengthen the structural inductive bias of a Transformer by intermediate pre-training to perform synthetically generated syntactic transformations of dependency trees given a description of the transformation. Our experiments confirm that this helps with few-shot learning of syntactic tasks such as chunking, and also improves structural generalization for semantic parsing. Our analysis shows that the intermediate pre-training leads to attention heads that keep track of which syntactic transformation needs to be applied to which token, and that the model can leverage these attention heads on downstream tasks.

Discourse Representation Theory (DRT) distinguishes itself from other semantic representation frameworks by its ability to model complex semantic and discourse phenomena through structural nesting and variable binding. While seq2seq models hold the state of the art on DRT parsing, their accuracy degrades with the complexity of the sentence, and they sometimes struggle to produce well-formed DRT representations. We introduce the AMS parser, a compositional, neurosymbolic semantic parser for DRT. It rests on a novel mechanism for predicting quantifier scope. We show that the AMS parser reliably produces well-formed outputs and performs well on DRT parsing, especially on complex sentences.

Star formation rates (SFRs), gas-phase metallicities, and stellar masses are crucial for studying galaxy evolution. The different relations resulting from these properties give insights into the complex interplay of gas inside galaxies and their evolutionary trajectory and current characteristics. We aim to characterize these relations at $z\sim 0.3$, corresponding to a 3-4 Gyr lookback time. We utilized optical integral field spectroscopy of 65 emission-line galaxies from the MAGPI survey at a redshift of $0.28

The effect of non-planar substrate surface on homogeneity and quality of cold sprayed (CS) deposits was studied by scanning acoustic microscopy (SAM). Fe coatings were cold sprayed onto Al substrates containing artificially introduced grooves of square- and trapezoid-shaped geometries, with flat or cylindrical bottoms. The Al substrates were either wrought or cold sprayed, to comprehend their prospective influence on the Fe coatings build-up. SAM was then used to assess morphological properties of the materials from the cross-view and top-view directions. The microstructure below the surface of the studied samples was visualized by measuring the amplitudes of the reflection echoes and the velocity of the ultrasonic waves. The SAM analysis revealed that the regions of coating imperfections around the grooves are larger than what is suggested by standard scanning electron microscopy (SEM) observations. Furthermore, we found that the seemingly non-influenced coating regions that appear perfectly homogeneous and dense in SEM do, in fact, possess heterogeneous microstructure associated with the individual CS nozzle passes., Comment: Manuscript accepted to the Journal of Thermal Spray Technology (JTST)

There is an increasing trend towards evaluating NLP models with LLM-generated judgments instead of human judgments. In the absence of a comparison against human data, this raises concerns about the validity of these evaluations; in case they are conducted with proprietary models, this also raises concerns over reproducibility. We provide JUDGE-BENCH, a collection of 20 NLP datasets with human annotations, and comprehensively evaluate 11 current LLMs, covering both open-weight and proprietary models, for their ability to replicate the annotations. Our evaluations show that each LLM exhibits a large variance across datasets in its correlation to human judgments. We conclude that LLMs are not yet ready to systematically replace human judges in NLP.

We present a method for rewriting an input sentence to match specific values of nontrivial linguistic features, such as dependency depth. In contrast to earlier work, our method uses in-context learning rather than finetuning, making it applicable in use cases where data is sparse. We show that our model performs accurate rewrites and matches the state of the art on rewriting sentences to a specified school grade level.

Collaboration is an integral part of human dialogue. Typical task-oriented dialogue games assign asymmetric roles to the participants, which limits their ability to elicit naturalistic role-taking in collaboration and its negotiation. We present a novel and simple online setup that favors balanced collaboration: a two-player 2D object placement game in which the players must negotiate the goal state themselves. We show empirically that human players exhibit a variety of role distributions, and that balanced collaboration improves task performance. We also present an LLM-based baseline agent which demonstrates that automatic playing of our game is an interesting challenge for artificial systems.

We study the fixed points of outer-totalistic cellular automata on sparse random regular graphs. These can be seen as constraint satisfaction problems, where each variable must adhere to the same local constraint, which depends solely on its state and the total number of its neighbors in each possible state. Examples of this setting include classical problems such as independent sets or assortative/dissasortative partitions. We analyse the existence and number of fixed points in the large system limit using the cavity method, under both the replica symmetric (RS) and one-step replica symmetry breaking (1RSB) assumption. This method allows us to characterize the structure of the space of solutions, in particular, if the solutions are clustered and whether the clusters contain frozen variables. This last property is conjectured to be linked to the typical algorithmic hardness of the problem. We bring experimental evidence for this claim by studying the performance of the belief-propagation reinforcement algorithm, a message-passing-based solver for these constraint satisfaction problems., Comment: 26 pages, 7 figures

Recent research interest in the learning-based processing of garments, from virtual fitting to generation and reconstruction, stumbles on a scarcity of high-quality public data in the domain. We contribute to resolving this need by presenting the first large-scale synthetic dataset of 3D made-to-measure garments with sewing patterns, as well as its generation pipeline. GarmentCodeData contains 115,000 data points that cover a variety of designs in many common garment categories: tops, shirts, dresses, jumpsuits, skirts, pants, etc., fitted to a variety of body shapes sampled from a custom statistical body model based on CAESAR, as well as a standard reference body shape, applying three different textile materials. To enable the creation of datasets of such complexity, we introduce a set of algorithms for automatically taking tailor's measures on sampled body shapes, sampling strategies for sewing pattern design, and propose an automatic, open-source 3D garment draping pipeline based on a fast XPBD simulator, while contributing several solutions for collision resolution and drape correctness to enable scalability. Project Page: https://igl.ethz.ch/projects/GarmentCodeData/, Comment: Accepted to ECCV 2024. Sept 4th, 2024: release of GarmentCodeData(v2)

Statistical inference is a major scientific endeavor for many researchers. In terms of inferential methods implemented to mixed-effects models, significant progress has been made in the R software. However, these advances primarily concern classical estimators (ML, REML) and mainly focus on fixed effects. In the confintROB package, we have implemented various bootstrap methods for computing confidence intervals (CIs) not only for fixed effects but also for variance components. These methods can be implemented with the widely used lmer function from the lme4 package, as well as with the rlmer function from the robustlmm package and the varComprob function from the robustvarComp package. These functions implement robust estimation methods suitable for data with outliers. The confintROB package implements the Wald method for fixed effects, whereas for both fixed effects and variance components, two bootstrap methods are implemented: the parametric bootstrap and the wild bootstrap. Moreover, the confintROB package can obtain both the percentile and the bias-corrected accelerated versions of CIs., Comment: 19 pages, 2 figures

Quantum communication promises unprecedented communication capabilities enabled by the transmission of quantum states of light. However, current implementations face severe limitations in communication distance due to photon loss. Silicon carbide (SiC) defects have emerged as a promising quantum device platform, offering strong optical transitions, long spin coherence lifetimes and the opportunity for integration with semiconductor devices. Some defects with optical transitions in the telecom range have been identified, allowing to interface with fiber networks without the need for wavelength conversion. These unique properties make SiC an attractive platform for the implementation of quantum nodes for quantum communication networks. We provide an overview of the most prominent defects in SiC and their implementation in spin-photon interfaces. Furthermore, we model a memory-enhanced quantum communication protocol in order to extract the parameters required to surpass a direct point-to-point link performance. Based on these insights, we summarize the key steps required towards the deployment of SiC devices in large-scale quantum communication networks., Comment: 20 pages, 8 figures

Catalyst particles or complexes suspended in liquid films can trigger chemical reactions leading to inhomogeneous concentrations of reactants and products in the film. We demonstrate that the sensitivity of the liquid film's gas-liquid surface tension to these inhomogeneous concentrations strongly impacts the film stability. Using linear stability analysis, we identify novel scenarios in which the film can be either stabilized or destabilized by the reactions. Furthermore, we find so far unrevealed rupture mechanisms which are absent in the chemically inactive case. The linear stability predictions are confirmed by numerical simulations, which also demonstrate that the shape of chemically active droplets can depart from the spherical cap and that unsteady states such as traveling and standing waves might appear. Finally, we critically discuss the relevance of our predictions by showing that the range of our selected parameters is well accessible by typical experiments.

Our Sun lies within 300 pc of the 2.7-kpc-long sinusoidal chain of dense gas clouds known as the Radcliffe Wave. The structure's wave-like shape was discovered using 3D dust mapping, but initial kinematic searches for oscillatory motion were inconclusive. Here we present evidence that the Radcliffe Wave is oscillating through the Galactic plane while also drifting radially away from the Galactic Center. We use measurements of line-of-sight velocity for 12CO and 3D velocities of young stellar clusters to show that the most massive star-forming regions spatially associated with the Radcliffe Wave (including Orion, Cepheus, North America, and Cygnus X) move as if they are part of an oscillating wave driven by the gravitational acceleration of the Galactic potential. By treating the Radcliffe Wave as a coherently oscillating structure, we can derive its motion independently of the local Galactic mass distribution, and directly measure local properties of the Galactic potential as well as the Sun's vertical oscillation period. In addition, the measured drift of the Radcliffe Wave radially outward from the Galactic Center suggests that the cluster whose supernovae ultimately created today's expanding Local Bubble may have been born in the Radcliffe Wave., Comment: Published in Nature on 20 February 2024. For data, interactive visualizations, and more information see https://sites.google.com/cfa.harvard.edu/radcliffewave

The prominent radio quasar PKS 2215+020 (J2217+0220) was once labelled as a new laboratory for core--jet physics at redshift z=3.572 because of its exceptionally extended jet structure traceable with very long baseline interferometric (VLBI) observations up to a ~600 pc projected distance from the compact core and a hint of an arcsec-scale radio and an X-ray jet. While the presence of an X-ray jet could not be confirmed later, this active galactic nucleus is still unique at high redshift with its long VLBI jet. Here, we analyse archival multi-epoch VLBI imaging data at five frequency bands from 1.7 to 15.4 GHz covering a period of more than 25 years from 1995 to 2020. We constrain apparent proper motions of jet components in PKS 2215+020 for the first time. Brightness distribution modeling at 8 GHz reveals a nearly 0.02 mas/yr proper motion (moderately superluminal with apparently two times the speed of light), and provides delta=11.5 for the Doppler-boosting factor in the inner relativistic jet that is inclined within 2 deg to the line of sight and has a Gamma=6 bulk Lorentz factor. These values qualify PKS 2215+020 as a blazar, with rather typical jet properties in a small sample of only about 20 objects at z>3.5 that have similar measurements to date. According to the 2-GHz VLBI data, the diffuse and extended outer emission feature at ~60 mas from the core, probably a place where the jet interacts with and decelerated by the ambient galactic medium, is consistent with being stationary, albeit slow motion cannot be excluded based on the presently available data., Comment: 18 pages, 2 figures, 5 tables, accepted for publication in Universe

Neural networks are vulnerable to adversarial attacks, i.e., small input perturbations can significantly affect the outputs of a neural network. In safety-critical environments, the inputs often contain noisy sensor data; hence, in this case, neural networks that are robust against input perturbations are required. To ensure safety, the robustness of a neural network must be formally verified. However, training and formally verifying robust neural networks is challenging. We address both of these challenges by employing, for the first time, an end-to-end set-based training procedure that trains robust neural networks for formal verification. Our training procedure trains neural networks, which can be easily verified using simple polynomial-time verification algorithms. Moreover, our extensive evaluation demonstrates that our set-based training procedure effectively trains robust neural networks, which are easier to verify. Set-based trained neural networks consistently match or outperform those trained with state-of-the-art robust training approaches.

High-order behavior of the perturbative expansion for short-distance observables in QCD is intimately related to the contributions of small momenta in the corresponding Feynman diagrams and this correspondence provides one with a useful tool to investigate power-suppressed nonperturbative corrections. We use this technique to study the structure of power corrections to parton quasi- and pseudo-GPDs which are used in lattice calculations of generalized parton distributions. As the main result, we predict the functional dependence of the leading power corrections to quasi(pseudo)-GPDs on $x$ variable for nonzero skewedness parameter $\xi$. The kinematic point $x=\pm\xi$ turns out to be special. We find that the nonperturbative corrections to quasi-GPDs at this point are suppressed by the first power of the hard scale only. These contributions come from soft momenta and have nothing to do with the known UV renormalon in the Wilson line. We also show that power corrections can be strongly suppressed by the normalization procedure.

LLMs are being increasingly used for planning-style tasks, but their capabilities for planning and reasoning are poorly understood. We present AutoPlanBench, a novel method for automatically converting planning benchmarks written in PDDL into textual descriptions and offer a benchmark dataset created with our method. We show that while the best LLM planners do well on some planning tasks, others remain out of reach of current methods.

Uncertainty in machine learning models is a timely and vast field of research. In supervised learning, uncertainty can already occur in the first stage of the training process, the annotation phase. This scenario is particularly evident when some instances cannot be definitively classified. In other words, there is inevitable ambiguity in the annotation step and hence, not necessarily a "ground truth" associated with each instance. The main idea of this work is to drop the assumption of a ground truth label and instead embed the annotations into a multidimensional space. This embedding is derived from the empirical distribution of annotations in a Bayesian setup, modeled via a Dirichlet-Multinomial framework. We estimate the model parameters and posteriors using a stochastic Expectation Maximization algorithm with Markov Chain Monte Carlo steps. The methods developed in this paper readily extend to various situations where multiple annotators independently label instances. To showcase the generality of the proposed approach, we apply our approach to three benchmark datasets for image classification and Natural Language Inference. Besides the embeddings, we can investigate the resulting correlation matrices, which reflect the semantic similarities of the original classes very well for all three exemplary datasets.

The ability to predict an NLP model's accuracy on unseen, potentially out-of-distribution data is a prerequisite for trustworthiness. We present a novel model that establishes upper and lower bounds on the accuracy, without requiring gold labels for the unseen data. We achieve this by training a discriminator which predicts whether the output of a given sequence-to-sequence model is correct or not. We show across a variety of tagging, parsing, and semantic parsing tasks that the gold accuracy is reliably between the predicted upper and lower bounds, and that these bounds are remarkably close together.

Large Language Models (LLMs) are increasingly being used for interactive decision-making tasks requiring planning and adapting to the environment. Recent works employ LLMs-as-agents in broadly two ways: iteratively determining the next action (iterative executors) or generating plans and executing sub-tasks using LLMs (plan-and-execute). However, these methods struggle with task complexity, as the inability to execute any sub-task may lead to task failure. To address these shortcomings, we introduce As-Needed Decomposition and Planning for complex Tasks (ADaPT), an approach that explicitly plans and decomposes complex sub-tasks as-needed, i.e., when the LLM is unable to execute them. ADaPT recursively decomposes sub-tasks to adapt to both task complexity and LLM capability. Our results demonstrate that ADaPT substantially outperforms established strong baselines, achieving success rates up to 28.3% higher in ALFWorld, 27% in WebShop, and 33% in TextCraft -- a novel compositional dataset that we introduce. Through extensive analysis, we illustrate the importance of multilevel decomposition and establish that ADaPT dynamically adjusts to the capabilities of the executor LLM as well as to task complexity., Comment: NAACL 2024 (findings) camera-ready. Project Page: https://allenai.github.io/adaptllm

For classifying digital whole slide images in the absence of pixel level annotation, typically multiple instance learning methods are applied. Due to the generic applicability, such methods are currently of very high interest in the research community, however, the issue of data augmentation in this context is rarely explored. Here we investigate linear and multilinear interpolation between feature vectors, a data augmentation technique, which proved to be capable of improving the generalization performance classification networks and also for multiple instance learning. Experiments, however, have been performed on only two rather small data sets and one specific feature extraction approach so far and a strong dependence on the data set has been identified. Here we conduct a large study incorporating 10 different data set configurations, two different feature extraction approaches (supervised and self-supervised), stain normalization and two multiple instance learning architectures. The results showed an extraordinarily high variability in the effect of the method. We identified several interesting aspects to bring light into the darkness and identified novel promising fields of research., Comment: for code and data, see gitlab repo: https://gitlab.com/mgadermayr/mixupmil. arXiv admin note: substantial text overlap with arXiv:2211.05862

The goal of compositional generalization benchmarks is to evaluate how well models generalize to new complex linguistic expressions. Existing benchmarks often focus on lexical generalization, the interpretation of novel lexical items in syntactic structures familiar from training; structural generalization tasks, where a model needs to interpret syntactic structures that are themselves unfamiliar from training, are often underrepresented, resulting in overly optimistic perceptions of how well models can generalize. We introduce SLOG, a semantic parsing dataset that extends COGS (Kim and Linzen, 2020) with 17 structural generalization cases. In our experiments, the generalization accuracy of Transformer models, including pretrained ones, only reaches 40.6%, while a structure-aware parser only achieves 70.8%. These results are far from the near-perfect accuracy existing models achieve on COGS, demonstrating the role of SLOG in foregrounding the large discrepancy between models' lexical and structural generalization capacities., Comment: Accepted to EMNLP 2023

Despite their ubiquity in language generation, it remains unknown why truncation sampling heuristics like nucleus sampling are so effective. We provide a theoretical explanation for the effectiveness of the truncation sampling by proving that truncation methods that discard tokens below some probability threshold (the most common type of truncation) can guarantee that all sampled tokens have nonzero true probability. However, thresholds are a coarse heuristic, and necessarily discard some tokens with nonzero true probability as well. In pursuit of a more precise sampling strategy, we show that we can leverage a known source of model errors, the softmax bottleneck, to prove that certain tokens have nonzero true probability, without relying on a threshold. Based on our findings, we develop an experimental truncation strategy and the present pilot studies demonstrating the promise of this type of algorithm. Our evaluations show that our method outperforms its threshold-based counterparts under automatic and human evaluation metrics for low-entropy (i.e., close to greedy) open-ended text generation. Our theoretical findings and pilot experiments provide both insight into why truncation sampling works, and make progress toward more expressive sampling algorithms that better surface the generative capabilities of large language models.

Strong inductive biases enable learning from little data and help generalization outside of the training distribution. Popular neural architectures such as Transformers lack strong structural inductive biases for seq2seq NLP tasks on their own. Consequently, they struggle with systematic generalization beyond the training distribution, e.g. with extrapolating to longer inputs, even when pre-trained on large amounts of text. We show how a structural inductive bias can be efficiently injected into a seq2seq model by pre-training it to simulate structural transformations on synthetic data. Specifically, we inject an inductive bias towards Finite State Transducers (FSTs) into a Transformer by pre-training it to simulate FSTs given their descriptions. Our experiments show that our method imparts the desired inductive bias, resulting in improved systematic generalization and better few-shot learning for FST-like tasks. Our analysis shows that fine-tuned models accurately capture the state dynamics of the unseen underlying FSTs, suggesting that the simulation process is internalized by the fine-tuned model., Comment: ACL 2024 camera-ready

We have measured the geopotential difference between two locations separated by 457 km by comparison of two optical lattice clocks via an interferometric fiber link, utilizing the gravitational redshift of the clock transition frequency. The $^{87}$Sr clocks have been compared side-by-side before and after one of the clocks was moved to the remote location. The chronometrically measured geopotential difference of $3918.1(2.4)\,\mathrm{m^2 \, s^{-2}}$ agrees with an independent geodetic determination of $3915.88(0.30)\,\mathrm{m^2 \, s^{-2}}$. The uncertainty of the chronometric geopotential difference is equivalent to an uncertainty of $24~\mathrm{cm}$ in height.