Search

This paper introduces a method to enhance Interactive Imitation Learning (IIL) by extracting touch interaction points and tracking object movement from video demonstrations. The approach extends current IIL systems by providing robots with detailed knowledge of both where and how to interact with objects, particularly complex articulated ones like doors and drawers. By leveraging cutting-edge techniques such as 3D Gaussian Splatting and FoundationPose for tracking, this method allows robots to better understand and manipulate objects in dynamic environments. The research lays the foundation for more effective task learning and execution in autonomous robotic systems., Comment: CoRL 2024, Workshop on Lifelong Learning for Home Robots, Munich, Germany

Neural differential equations offer a powerful approach for learning dynamics from data. However, they do not impose known constraints that should be obeyed by the learned model. It is well-known that enforcing constraints in surrogate models can enhance their generalizability and numerical stability. In this paper, we introduce projected neural differential equations (PNDEs), a new method for constraining neural differential equations based on projection of the learned vector field to the tangent space of the constraint manifold. In tests on several challenging examples, including chaotic dynamical systems and state-of-the-art power grid models, PNDEs outperform existing methods while requiring fewer hyperparameters. The proposed approach demonstrates significant potential for enhancing the modeling of constrained dynamical systems, particularly in complex domains where accuracy and reliability are essential., Comment: 17 pages, 6 figures

There is a scarcity of multilingual vision-language models that properly account for the perceptual differences that are reflected in image captions across languages and cultures. In this work, through a multimodal, multilingual retrieval case study, we quantify the existing lack of model flexibility. We empirically show performance gaps between training on captions that come from native German perception and captions that have been either machine-translated or human-translated from English into German. To address these gaps, we further propose and evaluate caption augmentation strategies. While we achieve mean recall improvements (+1.3), gaps still remain, indicating an open area of future work for the community., Comment: EMNLP 2024 Main - Short

The increasing integration of renewable energy sources (RESs) into power systems requires the deployment of grid-forming inverters to ensure a stable operation. Accurate modeling of these devices is necessary. In this paper, a system identification approach to obtain low-dimensional models of grid-forming inverters is presented. The proposed approach is based on a Hammerstein-Wiener parametrization of the normal-form model. The normal-form is a gray-box model that utilizes complex frequency and phase to capture non-linear inverter dynamics. The model is validated on two well-known control strategies: droop-control and dispatchable virtual oscillators. Simulations and hardware-in-the-loop experiments demonstrate that the normal-form accurately models inverter dynamics across various operating conditions. The approach shows great potential for enhancing the modeling of RES-dominated power systems, especially when component models are unavailable or computationally expensive.

The evaluation of image generators remains a challenge due to the limitations of traditional metrics in providing nuanced insights into specific image regions. This is a critical problem as not all regions of an image may be learned with similar ease. In this work, we propose a novel approach to disentangle the cosine similarity of mean embeddings into the product of cosine similarities for individual pixel clusters via central kernel alignment. Consequently, we can quantify the contribution of the cluster-wise performance to the overall image generation performance. We demonstrate how this enhances the explainability and the likelihood of identifying pixel regions of model misbehavior across various real-world use cases., Comment: Preprint

Electrolytically generated gas bubbles can significantly hamper the overall electrolysis efficiency. Therefore it is crucial to understand their dynamics in order to optimise water electrolyzer systems. Here we demonstrate a distinct transport mechanism where coalescence with microbubbles drives electrolyte droplets, resulting from the fragmentation of the Worthington jet, into the gas phase during hydrogen evolution reaction, both in normal and microgravity environments. This indicates that the H$_2$ bubble is not only composed of hydrogen gas and vapor but also includes electrolyte fractions. Reminiscent of bursting bubbles on a liquid-gas interface, this behavior results in a flow inside the bubble, which is further affected by Marangoni convection at the gas-electrolyte interface, highlighting interface mobility. In the case of electrode-attached bubbles, the sprayed droplets form electrolyte puddles at the bubble-electrode contact area, affecting the dynamics near the three-phase contact line and favoring bubble detachment from the electrode. The results of this work unravel important insights into the physicochemical aspects of electrolytic gas bubbles, integral for optimizing gas-evolving electrochemical systems. Besides, our findings are essential for studying the limits of jet formation and rupture relevant to acid mist formation in electrowinning, generation of sea spray aerosols, impact of droplets on liquid surfaces, etc., Comment: manuscript: 25 pages, 6 figures; SI: 12 pages, 5 figures, 1 table

Monitoring and maintaining machine learning models are among the most critical challenges in translating recent advances in the field into real-world applications. However, current monitoring methods lack the capability of provide actionable insights answering the question of why the performance of a particular model really degraded. In this work, we propose a novel approach to explain the behavior of a black-box model under feature shifts by attributing an estimated performance change to interpretable input characteristics. We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation (XPE). We analyze the underlying assumptions and demonstrate the superiority of our approach over several baselines on different data sets across various data modalities such as images, audio, and tabular data. We also indicate how the generated results can lead to valuable insights, enabling explanatory model monitoring by revealing potential root causes for model deterioration and guiding toward actionable countermeasures., Comment: ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 24)

Many real-world applications require machine-learning models to be able to deal with non-stationary data distributions and thus learn autonomously over an extended period of time, often in an online setting. One of the main challenges in this scenario is the so-called catastrophic forgetting (CF) for which the learning model tends to focus on the most recent tasks while experiencing predictive degradation on older ones. In the online setting, the most effective solutions employ a fixed-size memory buffer to store old samples used for replay when training on new tasks. Many approaches have been presented to tackle this problem. However, it is not clear how predictive uncertainty information for memory management can be leveraged in the most effective manner and conflicting strategies are proposed to populate the memory. Are the easiest-to-forget or the easiest-to-remember samples more effective in combating CF? Starting from the intuition that predictive uncertainty provides an idea of the samples' location in the decision space, this work presents an in-depth analysis of different uncertainty estimates and strategies for populating the memory. The investigation provides a better understanding of the characteristics data points should have for alleviating CF. Then, we propose an alternative method for estimating predictive uncertainty via the generalised variance induced by the negative log-likelihood. Finally, we demonstrate that the use of predictive uncertainty measures helps in reducing CF in different settings., Comment: arXiv admin note: substantial text overlap with arXiv:2405.18925

In order to develop robots that can effectively serve as versatile and capable home assistants, it is crucial for them to reliably perceive and interact with a wide variety of objects across diverse environments. To this end, we proposed Open Vocabulary Mobile Manipulation as a key benchmark task for robotics: finding any object in a novel environment and placing it on any receptacle surface within that environment. We organized a NeurIPS 2023 competition featuring both simulation and real-world components to evaluate solutions to this task. Our baselines on the most challenging version of this task, using real perception in simulation, achieved only an 0.8% success rate; by the end of the competition, the best participants achieved an 10.8\% success rate, a 13x improvement. We observed that the most successful teams employed a variety of methods, yet two common threads emerged among the best solutions: enhancing error detection and recovery, and improving the integration of perception with decision-making processes. In this paper, we detail the results and methodologies used, both in simulation and real-world settings. We discuss the lessons learned and their implications for future research. Additionally, we compare performance in real and simulated environments, emphasizing the necessity for robust generalization to novel settings.

Due to the increasing share of renewables, the analysis of the dynamical behavior of power grids gains importance. Effective risk assessments necessitate the analysis of large number of fault scenarios. The computational costs inherent in dynamic simulations impose constraints on the number of configurations that can be analyzed. Machine Learning (ML) has proven to efficiently predict complex power grid properties. Hence, we analyze the potential of ML for predicting dynamic stability of future power grids with large shares of inverters. For this purpose, we generate a new dataset consisting of synthetic power grid models and perform dynamical simulations. As targets for the ML training, we calculate the fault-ride-through probability, which we define as the probability of staying within a ride-through curve after a fault at a bus has been cleared. Importantly, we demonstrate that ML models accurately predict the fault-ride-through probability of synthetic power grids. Finally, we also show that the ML models generalize to an IEEE-96 Test System, which emphasizes the potential of deploying ML methods to study probabilistic stability of power grids., Comment: 11 pages, 6 figures

Using feature attributions for post-hoc explanations is a common practice to understand and verify the predictions of opaque machine learning models. Despite the numerous techniques available, individual methods often produce inconsistent and unstable results, putting their overall reliability into question. In this work, we aim to systematically improve the quality of feature attributions by combining multiple explanations across distinct methods or their variations. For this purpose, we propose a novel approach to derive optimal convex combinations of feature attributions that yield provable improvements of desired quality criteria such as robustness or faithfulness to the model behavior. Through extensive experiments involving various model architectures and popular feature attribution techniques, we demonstrate that our combination strategy consistently outperforms individual methods and existing baselines., Comment: International Conference on Machine Learning (ICML) 2024

Given the ability to model more realistic and dynamic problems, Federated Continual Learning (FCL) has been increasingly investigated recently. A well-known problem encountered in this setting is the so-called catastrophic forgetting, for which the learning model is inclined to focus on more recent tasks while forgetting the previously learned knowledge. The majority of the current approaches in FCL propose generative-based solutions to solve said problem. However, this setting requires multiple training epochs over the data, implying an offline setting where datasets are stored locally and remain unchanged over time. Furthermore, the proposed solutions are tailored for vision tasks solely. To overcome these limitations, we propose a new approach to deal with different modalities in the online scenario where new data arrive in streams of mini-batches that can only be processed once. To solve catastrophic forgetting, we propose an uncertainty-aware memory-based approach. Specifically, we suggest using an estimator based on the Bregman Information (BI) to compute the model's variance at the sample level. Through measures of predictive uncertainty, we retrieve samples with specific characteristics, and - by retraining the model on such samples - we demonstrate the potential of this approach to reduce the forgetting effect in realistic settings while maintaining data confidentiality and competitive communication efficiency compared to state-of-the-art approaches.

In light of the GDPR, data controllers (DC) need to allow data subjects (DS) to exercise certain data subject rights. A key requirement here is that DCs can reliably authenticate a DS. Due to a lack of clear technical specifications, this has been realized in different ways, such as by requesting copies of ID documents or by email address verification. However, previous research has shown that this is associated with various security and privacy risks and that identifying DSs can be a non-trivial task. In this paper, we review different authentication schemes and propose an architecture that enables DCs to authenticate DSs with the help of independent Identity Providers in a secure and privacy-preserving manner by utilizing attribute-based credentials and eIDs. Our work contributes to a more standardized and privacy-preserving way of authenticating DSs, which will benefit both DCs and DSs., Comment: 17 pages, 6 figures, presented and published at IFIP Summer School on Privacy and Identity Management 2023

Convection of liquid metals drives large natural processes and is important in technical processes. Model experiments are conducted for research purposes where simulations are expensive and the clarification of open questions requires novel flow mapping methods with an increased spatial resolution. In this work, the method of Ultrasound Localization Microscopy (ULM) is investigated for this purpose. Known from microvasculature imaging, this method provides an increased spatial resolution beyond the diffraction limit. Its applicability in liquid metal flows is promising, however the realization and reliability is challenging, as artificial scattering particles or microbubbles cannot be utilized. To solve this issue an approach using nonlinear adaptive beamforming is proposed. This allowed the reliable tracking of particles of which super-resolved flow maps can be deduced. Furthermore, the application in fluid physics requires quantified results. Therefore, an uncertainty quantification model based on the spatial resolution, velocity gradient and measurement parameters is proposed, which allows to estimate the flow maps validity under experimental conditions. The proposed method is demonstrated in magnetohydrodynamic convection experiments. In some occasions, ULM was able to measure velocity vectors within the boundary layer of the flow, which will help for future in-depth flow studies. Furthermore, the proposed uncertainty model of ULM is of generic use in other applications.

Nowadays, most online services offer different authentication methods that users can set up for multi-factor authentication but also as a recovery method. This configuration must be done thoroughly to prevent an adversary's access while ensuring the legitimate user does not lose access to their account. This is particularly important for fundamental everyday services, where either failure would have severe consequences. Nevertheless, little research has been done on the authentication of actual users regarding security and the risk of being locked out of their accounts. To foster research in this direction, this paper presents a study on the account settings of Google and Apple users. Considering the multi-factor authentication configuration and recovery options, we analyzed the account security and lock-out risks. Our results provide insights into the usage of multi-factor authentication in practice, show significant security differences between Google and Apple accounts, and reveal that many users would miss access to their accounts when losing a single authentication device., Comment: 10 pages, published and presented at ICISSP 2024

Poor generalization performance caused by distribution shifts in unseen domains often hinders the trustworthy deployment of deep neural networks. Many domain generalization techniques address this problem by adding a domain invariant regularization loss terms during training. However, there is a lack of modular software that allows users to combine the advantages of different methods with minimal effort for reproducibility. DomainLab is a modular Python package for training user specified neural networks with composable regularization loss terms. Its decoupled design allows the separation of neural networks from regularization loss construction. Hierarchical combinations of neural networks, different domain generalization methods, and associated hyperparameters, can all be specified together with other experimental setup in a single configuration file. Hierarchical combinations of neural networks, different domain generalization methods, and associated hyperparameters, can all be specified together with other experimental setup in a single configuration file. In addition, DomainLab offers powerful benchmarking functionality to evaluate the generalization performance of neural networks in out-of-distribution data. The package supports running the specified benchmark on an HPC cluster or on a standalone machine. The package is well tested with over 95 percent coverage and well documented. From the user perspective, it is closed to modification but open to extension. The package is under the MIT license, and its source code, tutorial and documentation can be found at https://github.com/marrlab/DomainLab.

Risk-based authentication (RBA) is used in online services to protect user accounts from unauthorized takeover. RBA commonly uses contextual features that indicate a suspicious login attempt when the characteristic attributes of the login context deviate from known and thus expected values. Previous research on RBA and anomaly detection in authentication has mainly focused on the login process. However, recent attacks have revealed vulnerabilities in other parts of the authentication process, specifically in the account recovery function. Consequently, to ensure comprehensive authentication security, the use of anomaly detection in the context of account recovery must also be investigated. This paper presents the first study to investigate risk-based account recovery (RBAR) in the wild. We analyzed the adoption of RBAR by five prominent online services (that are known to use RBA). Our findings confirm the use of RBAR at Google, LinkedIn, and Amazon. Furthermore, we provide insights into the different RBAR mechanisms of these services and explore the impact of multi-factor authentication on them. Based on our findings, we create a first maturity model for RBAR challenges. The goal of our work is to help developers, administrators, and policy-makers gain an initial understanding of RBAR and to encourage further research in this direction.

Over the past several years, the misidentification of SpaceX Starlink satellites as Unidentified Aerial Phenomena (UAP) by pilots and laypersons has generated unnecessary aviation risk and confusion. The many deployment and orbital evolution strategies, coupled with changing sun specular reflection angles, contribute to this gap in space situational awareness. In this paper we present a case analysis of an incident that generated multiple, corroborating reports of a UAP from five pilots on two commercial airline flights over the Pacific Ocean on August 10th, 2022. This incident included two cell phone photos and a video of an unrecognizable and possibly anomalous phenomenon. We then use supplemental two-line elements (TLEs) for the Starlink train of satellites launched that same day and Automatic Dependent Surveillance Broadcast (ADS-B) data from the flight with the photographs to reconstruct a view of these satellites from the cockpit at the time and place of the sighting. The success of this work demonstrates an approach that could, in principle, warn aviators about satellites that could be visible in unusual or novel illumination configurations, thus increasing space situational awareness and supporting aviation safety. We conclude with recommendations for governments and satellite operators to provide better a-priori data that can be used to create advisories to aviators and the public. The automated simulation of known specular reflection off constellations of satellites could also support researchers investigating sightings of unfamiliar aerial or aerospace objects as likely being from normal versus novel space events., Comment: 35 pages, 10 figures, submitted to the 4th IAA Conference on Space Situational Awareness (ICSSA), May 8-10 2024, Daytona Beach, FL, USA; Abstract number AH-92328

Spin-orbit torques (SOTs) act as efficient drivers for nanoscale magnetic systems, such as in magnetic tunnel junctions, nano-oscillators and racetrack geometries. In particular, in combination with materials exhibiting high Dzyaloshinskii--Moriya interaction, SOTs are considered to result in well-controlled deterministic magnetisation dynamics and are, therefore, used as robust drives to move and create magnetic skyrmions. In contrast to these expectations, we here find unpredictable, transiently chaotic dynamics induced by SOT at an artificial anisotropy-engineered defect in a magnetic racetrack. Based on these controlled conditions, we directly observe the nanoscale dynamics with holography-based, time-resolved x-ray imaging. In concert with micromagnetic simulations, we disclose a regime of violent picosecond fluctuations, including topological instabilities that, remarkably, result in deterministic final configurations. In addition, our images expose previously unseen skyrmion shedding and highlight the potential of transiently chaotic pathways for topological switching. Our approach offers new perspectives for the investigation and application of highly non-linear SOT dynamics in spintronics materials.

Considering the growing interest in magnetic materials for unconventional computing, data storage, and sensor applications, there is active research not only on material synthesis but also characterisation of their properties. In addition to structural and integral magnetic characterisations, imaging of magnetization patterns, current distributions and magnetic fields at nano- and microscale is of major importance to understand the material responses and qualify them for specific applications. In this roadmap, we aim to cover a broad portfolio of techniques to perform nano- and microscale magnetic imaging using SQUIDs, spin center and Hall effect magnetometries, scanning probe microscopies, x-ray- and electron-based methods as well as magnetooptics and nanoMRI. The roadmap is aimed as a single access point of information for experts in the field as well as the young generation of students outlining prospects of the development of magnetic imaging technologies for the upcoming decade with a focus on physics, materials science, and chemistry of planar, 3D and geometrically curved objects of different material classes including 2D materials, complex oxides, semi-metals, multiferroics, skyrmions, antiferromagnets, frustrated magnets, magnetic molecules/nanoparticles, ionic conductors, superconductors, spintronic and spinorbitronic materials.

Existing object recognition models have been shown to lack robustness in diverse geographical scenarios due to domain shifts in design and context. Class representations need to be adapted to more accurately reflect an object concept under these shifts. In the absence of training data from target geographies, we hypothesize that geographically diverse descriptive knowledge of categories can enhance robustness. For this purpose, we explore the feasibility of probing a large language model for geography-based object knowledge, and we examine the effects of integrating knowledge into zero-shot and learnable soft prompting with CLIP. Within this exploration, we propose geography knowledge regularization to ensure that soft prompts trained on a source set of geographies generalize to an unseen target set. Accuracy gains over prompting baselines on DollarStreet while training only on Europe data are up to +2.8/1.2/1.6 on target data from Africa/Asia/Americas, and +4.6 overall on the hardest classes. Competitive performance is shown vs. few-shot target training, and analysis is provided to direct future study of geographical robustness., Comment: To appear in IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR), 2024

This report introduces our UniTeam agent - an improved baseline for the "HomeRobot: Open Vocabulary Mobile Manipulation" challenge. The challenge poses problems of navigation in unfamiliar environments, manipulation of novel objects, and recognition of open-vocabulary object classes. This challenge aims to facilitate cross-cutting research in embodied AI using recent advances in machine learning, computer vision, natural language, and robotics. In this work, we conducted an exhaustive evaluation of the provided baseline agent; identified deficiencies in perception, navigation, and manipulation skills; and improved the baseline agent's performance. Notably, enhancements were made in perception - minimizing misclassifications; navigation - preventing infinite loop commitments; picking - addressing failures due to changing object visibility; and placing - ensuring accurate positioning for successful object placement.

Proper scoring rules evaluate the quality of probabilistic predictions, playing an essential role in the pursuit of accurate and well-calibrated models. Every proper score decomposes into two fundamental components -- proper calibration error and refinement -- utilizing a Bregman divergence. While uncertainty calibration has gained significant attention, current literature lacks a general estimator for these quantities with known statistical properties. To address this gap, we propose a method that allows consistent, and asymptotically unbiased estimation of all proper calibration errors and refinement terms. In particular, we introduce Kullback--Leibler calibration error, induced by the commonly used cross-entropy loss. As part of our results, we prove the relation between refinement and f-divergences, which implies information monotonicity in neural networks, regardless of which proper scoring rule is optimized. Our experiments validate empirically the claimed properties of the proposed estimator and suggest that the selection of a post-hoc calibration method should be determined by the particular calibration error of interest., Comment: Preprint