Search

This paper proposes the RePAIR dataset that represents a challenging benchmark to test modern computational and data driven methods for puzzle-solving and reassembly tasks. Our dataset has unique properties that are uncommon to current benchmarks for 2D and 3D puzzle solving. The fragments and fractures are realistic, caused by a collapse of a fresco during a World War II bombing at the Pompeii archaeological park. The fragments are also eroded and have missing pieces with irregular shapes and different dimensions, challenging further the reassembly algorithms. The dataset is multi-modal providing high resolution images with characteristic pictorial elements, detailed 3D scans of the fragments and meta-data annotated by the archaeologists. Ground truth has been generated through several years of unceasing fieldwork, including the excavation and cleaning of each fragment, followed by manual puzzle solving by archaeologists of a subset of approx. 1000 pieces among the 16000 available. After digitizing all the fragments in 3D, a benchmark was prepared to challenge current reassembly and puzzle-solving methods that often solve more simplistic synthetic scenarios. The tested baselines show that there clearly exists a gap to fill in solving this computationally complex problem., Comment: NeurIPS 2024, Track Datasets and Benchmarks, 10 pages

Large Language Models (LLMs) have shown promise in highly-specialized domains, however challenges are still present in aspects of accuracy and costs. These limitations restrict the usage of existing models in domain-specific tasks. While fine-tuning pre-trained models have shown promising results, this process can be computationally expensive and require massive datasets of the specialized application in hand. In this work, we bridge that gap. We have developed Phi-2-Legal and Mistral-Legal-7B, which are language models specifically designed for legal applications. These models are based on Phi-2 and Mistral-7B-v0.1, and have gone through continued pre-training with over 500 million tokens of legal texts. Our innovative approach significantly improves capabilities in legal tasks by using Large Language Models (LLMs) to convert raw training data into reading comprehension text. Our legal LLMs have demonstrated superior performance in legal benchmarks, even outperforming models trained on much larger datasets with more resources. This work emphasizes the effectiveness of continued pre-training on domain-specific texts, while using affordable LLMs for data conversion, which gives these models domain expertise while retaining general language understanding capabilities. While this work uses the legal domain as a test case, our method can be scaled and applied to any pre-training dataset, resulting in significant improvements across different tasks. These findings underscore the potential of domain-adaptive pre-training and reading comprehension for the development of highly effective domain-specific language models.

GPT-4o is an autoregressive omni model that accepts as input any combination of text, audio, image, and video, and generates any combination of text, audio, and image outputs. It's trained end-to-end across text, vision, and audio, meaning all inputs and outputs are processed by the same neural network. GPT-4o can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time in conversation. It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages, while also being much faster and 50\% cheaper in the API. GPT-4o is especially better at vision and audio understanding compared to existing models. In line with our commitment to building AI safely and consistent with our voluntary commitments to the White House, we are sharing the GPT-4o System Card, which includes our Preparedness Framework evaluations. In this System Card, we provide a detailed look at GPT-4o's capabilities, limitations, and safety evaluations across multiple categories, focusing on speech-to-speech while also evaluating text and image capabilities, and measures we've implemented to ensure the model is safe and aligned. We also include third-party assessments on dangerous capabilities, as well as discussion of potential societal impacts of GPT-4o's text and vision capabilities.

Training and fine-tuning large language models (LLMs) come with challenges related to memory and computational requirements due to the increasing size of the model weights and the optimizer states. Various techniques have been developed to tackle these challenges, such as low-rank adaptation (LoRA), which involves introducing a parallel trainable low-rank matrix to the fixed pre-trained weights at each layer. However, these methods often fall short compared to the full-rank weight training approach, as they restrict the parameter search to a low-rank subspace. This limitation can disrupt training dynamics and require a full-rank warm start to mitigate the impact. In this paper, we introduce a new method inspired by a phenomenon we formally prove: as training progresses, the rank of the estimated layer gradients gradually decreases, and asymptotically approaches rank one. Leveraging this, our approach involves adaptively reducing the rank of the gradients during Adam optimization steps, using an efficient online-updating low-rank projections rule. We further present a randomized SVD scheme for efficiently finding the projection matrix. Our technique enables full-parameter fine-tuning with adaptive low-rank gradient updates, significantly reducing overall memory requirements during training compared to state-of-the-art methods while improving model performance in both pretraining and fine-tuning. Finally, we provide a convergence analysis of our method and demonstrate its merits for training and fine-tuning language and biological foundation models.

We use PyDynamicaLC, a model using the least number of, and the least correlated, degrees of freedom needed to derive a photodynamical model, to describe some of the smallest -- and lowest TTV (transit timing variations) amplitude -- of the Kepler planets. We successfully analyze 64 systems containing 218 planets, for 88 of which we were able to determine significant masses (to better than $3\sigma$). We demonstrate consistency with literature results over two orders of magnitude in mass, and for the planets that already had literature mass estimations, we were able to reduce the relative mass error by $\sim22\%$ (median value). Of the planets with determined masses 23 are new mass determinations with no previous significant literature value, including a planet smaller and lighter than Earth (KOI-1977.02 / Kepler-345 b). These results demonstrate the power of photodynamical modeling with the appropriately chosen degrees of freedom. This will become increasingly more important as smaller planets are detected, especially as the TESS mission gathers ever longer-baseline light curves and for the analysis of the future PLATO mission data, Comment: 10 pages, 3 figures, 3 tables. AJ accepted

We present an inference scheme of long timescale, non-exponential kinetics from Molecular Dynamics simulations accelerated by stochastic resetting. Standard simulations provide valuable insight into chemical processes but are limited to timescales shorter than $\sim 1 \mu s$. Slower processes require the use of enhanced sampling methods to expedite them, and inference schemes to obtain the unbiased kinetics. However, most kinetics inference schemes assume an underlying exponential first-passage time distribution and are inappropriate for other distributions, e.g., with a power-law decay. We propose an inference scheme that is designed for such cases, based on simulations enhanced by stochastic resetting. We show that resetting promotes enhanced sampling of the first-passage time distribution at short timescales, but often also provides sufficient information to estimate the long-time asymptotics, which allows the kinetics inference. We apply our method to a model system and a short peptide in an explicit solvent, successfully estimating the unbiased mean first-passage time while accelerating the sampling by more than an order of magnitude.

Autonomous systems for software engineering are now capable of fixing bugs and developing features. These systems are commonly evaluated on SWE-bench (Jimenez et al., 2024a), which assesses their ability to solve software issues from GitHub repositories. However, SWE-bench uses only Python repositories, with problem statements presented predominantly as text and lacking visual elements such as images. This limited coverage motivates our inquiry into how existing systems might perform on unrepresented software engineering domains (e.g., front-end, game development, DevOps), which use different programming languages and paradigms. Therefore, we propose SWE-bench Multimodal (SWE-bench M), to evaluate systems on their ability to fix bugs in visual, user-facing JavaScript software. SWE-bench M features 617 task instances collected from 17 JavaScript libraries used for web interface design, diagramming, data visualization, syntax highlighting, and interactive mapping. Each SWE-bench M task instance contains at least one image in its problem statement or unit tests. Our analysis finds that top-performing SWE-bench systems struggle with SWE-bench M, revealing limitations in visual problem-solving and cross-language generalization. Lastly, we show that SWE-agent's flexible language-agnostic features enable it to substantially outperform alternatives on SWE-bench M, resolving 12% of task instances compared to 6% for the next best system.

Bosonic Josephson junctions provide a versatile platform for exploring quantum tunneling and coherence phenomena in ultracold atomic systems. While extensive research has examined the Josephson-junction dynamics in various double-well configurations, most studies have been limited to inertial reference frames. In the present work, we posed the question how placing a Josephson junction in a non-inertial reference frame would impact the quantum tunnelling. Our findings demonstrate that accelerating a Josephson junction alters the tunneling dynamics. Conversely, tunneling behavior can be used to assess the acceleration of the system. By analyzing the changes in physical properties, we can assess the acceleration of the double-well. We begin with the most simple non-inertial frame: moving with constant acceleration. The tunneling time decreases exponentially as acceleration increases, making it effective for measuring larger accelerations. However, for smaller accelerations, accurate assessment requires accounting for many-body depletion, which decreases linearly as acceleration rises. Next, we explore a more complex scenario where the acceleration is time dependent. In this case, the acceleration is mapped onto the tunneling time period and depletion, which again serve as predictors of acceleration. We go further by conducting a detailed analysis of the change in tunnelling dynamics when the system deviates from constant or zero acceleration. The quantitative analysis show that the depletion changes exponentially near constant acceleration, while around zero acceleration, the change follows a polynomial pattern. All in all, we quantify how the tunneling process, as well as the mean-field and many-body properties, evolve in a non-inertial system of increasing complexity., Comment: 26 pages, 11 figures

Although language model (LM) agents are demonstrating growing potential in many domains, their success in cybersecurity has been limited due to simplistic design and the lack of fundamental features for this domain. We present EnIGMA, an LM agent for autonomously solving Capture The Flag (CTF) challenges. EnIGMA introduces new Agent-Computer Interfaces (ACIs) to improve the success rate on CTF challenges. We establish the novel Interactive Agent Tool concept, which enables LM agents to run interactive command-line utilities essential for these challenges. Empirical analysis of EnIGMA on over 350 CTF challenges from three different benchmarks indicates that providing a robust set of new tools with demonstration of their usage helps the LM solve complex problems and achieves state-of-the-art results on the NYU CTF and Intercode-CTF benchmarks. Finally, we discuss insights on ACI design and agent behavior on cybersecurity tasks that highlight the need to adapt real-world tools for LM agents.

Stochastic resetting, the procedure of stopping and re-initializing random processes, has recently emerged as a powerful tool for accelerating processes ranging from queuing systems to molecular simulations. However, its usefulness is severely limited by assuming that the resetting protocol is completely decoupled from the state and age of the process that is being reset. We present a general formulation for state- and time-dependent resetting of stochastic processes, which we call adaptive resetting. This allows us to predict, using a single set of trajectories without resetting and via a simple reweighing procedure, all key observables of processes with adaptive resetting. These include the first-passage time distribution, the propagator, and the steady-state. Our formulation enables efficient exploration of informed search strategies and facilitates the prediction and design of complex non-equilibrium steady states, eliminating the need for extensive brute-force sampling across different resetting protocols.

Crafting a single, versatile physics-based controller that can breathe life into interactive characters across a wide spectrum of scenarios represents an exciting frontier in character animation. An ideal controller should support diverse control modalities, such as sparse target keyframes, text instructions, and scene information. While previous works have proposed physically simulated, scene-aware control models, these systems have predominantly focused on developing controllers that each specializes in a narrow set of tasks and control modalities. This work presents MaskedMimic, a novel approach that formulates physics-based character control as a general motion inpainting problem. Our key insight is to train a single unified model to synthesize motions from partial (masked) motion descriptions, such as masked keyframes, objects, text descriptions, or any combination thereof. This is achieved by leveraging motion tracking data and designing a scalable training method that can effectively utilize diverse motion descriptions to produce coherent animations. Through this process, our approach learns a physics-based controller that provides an intuitive control interface without requiring tedious reward engineering for all behaviors of interest. The resulting controller supports a wide range of control modalities and enables seamless transitions between disparate tasks. By unifying character control through motion inpainting, MaskedMimic creates versatile virtual characters. These characters can dynamically adapt to complex scenes and compose diverse motions on demand, enabling more interactive and immersive experiences., Comment: ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2024) Project page: https://research.nvidia.com/labs/par/maskedmimic/

We present a procedure for enhanced sampling of molecular dynamics simulations through informed stochastic resetting. Many phenomena, such as protein folding and crystal nucleation, occur over time scales that are inaccessible using standard simulation methods. We recently showed that stochastic resetting can accelerate molecular simulations that exhibit broad transition time distributions. However, standard stochastic resetting does not exploit any information about the reaction progress. Here, we demonstrate that an informed resetting protocol leads to greater accelerations than standard stochastic resetting, both for molecular dynamics and Metadynamics simulations. This is achieved by resetting only when a certain condition is met, e.g., when the distance from the target along the reaction coordinate is larger than some threshold. We then employ recently obtained theoretical results to identify the condition that leads to the greatest acceleration and to infer the unbiased mean transition time from accelerated simulations. Our work significantly extends the applicability of stochastic resetting for enhanced sampling of molecular simulations.

Stochastic resetting, a method for accelerating target search in random processes, often incurs temporal and energetic costs. For a diffusing particle, a lower bound exists for the energetic cost of reaching the target, which is attained at low resetting rates and equals the direct linear transportation cost against fluid drag. Here, we study ``smart resetting," a strategy that aims to beat this lower bound. By strategically resetting the particle only when this benefits its progress toward the target, smart resetting leverages information to minimize energy consumption. We analytically calculate the energetic cost per mean first passage time and show that smart resetting consistently reduces the energetic cost compared to regular resetting. Surprisingly, smart resting achieves the minimum energy cost previously established for regular resetting, irrespective of the resetting rate. Yet, it fails to reduce this cost further. We extend our findings in two ways: first, by examining nonlinear energetic cost functions, and second, by considering smart resetting of drift-diffusion processes.

We introduce Segmentation by Factorization (F-SEG), an unsupervised segmentation method for pathology that generates segmentation masks from pre-trained deep learning models. F-SEG allows the use of pre-trained deep neural networks, including recently developed pathology foundation models, for semantic segmentation. It achieves this without requiring additional training or finetuning, by factorizing the spatial features extracted by the models into segmentation masks and their associated concept features. We create generic tissue phenotypes for H&E images by training clustering models for multiple numbers of clusters on features extracted from several deep learning models on The Cancer Genome Atlas Program (TCGA), and then show how the clusters can be used for factorizing corresponding segmentation masks using off-the-shelf deep learning models. Our results show that F-SEG provides robust unsupervised segmentation capabilities for H&E pathology images, and that the segmentation quality is greatly improved by utilizing pathology foundation models. We discuss and propose methods for evaluating the performance of unsupervised segmentation in pathology.

We present the updated design of HighSpec, a high-resolution $\mathcal{R} \sim 20,000$ spectrograph designed for the Multi Aperture Spectroscopic Telescope (MAST). HighSpec offers three observing modes centered at the Ca II H&K, Mg b triplet, and H$\alpha$ lines. Each mode is supported by a highly optimized ion-etched grating, contributing to an exceptional instrument peak efficiency of $\gtrsim85\%$ for the two latter bands ($\gtrsim55\%$ for the Ca II H&K band). Optimizing throughput over wavelength coverage ($\Delta \lambda=10-17$ nm), HighSpec enables the precise measurement of spectral lines from faint targets. This approach is especially relevant for stellar object studies, specifically of WDs, which are intrinsically faint and have few spectroscopic lines. Each observing mode was tailored to target spectral features essential for WD research. Its integration with MAST, an array of 20 custom-designed telescopes that can function as a single large telescope (equivalent to a $2.7$ m telescope in collecting area) or multiplexing over the entire sky, provides unique adaptability for extensive and effective spectroscopic campaigns. Currently in its final assembly and testing stages, HighSpec's on-sky commissioning is scheduled for 2025., Comment: SPIE Conference "Space Telescopes and Instrumentation 2024: Ground-based and Airborne Instrumentation for Astronomy X", Yokahama, Japan

In this paper, we investigate a variant of the classical stochastic Multi-armed Bandit (MAB) problem, where the payoff received by an agent (either cost or reward) is both delayed, and directly corresponds to the magnitude of the delay. This setting models faithfully many real world scenarios such as the time it takes for a data packet to traverse a network given a choice of route (where delay serves as the agent's cost); or a user's time spent on a web page given a choice of content (where delay serves as the agent's reward). Our main contributions are tight upper and lower bounds for both the cost and reward settings. For the case that delays serve as costs, which we are the first to consider, we prove optimal regret that scales as $\sum_{i:\Delta_i > 0}\frac{\log T}{\Delta_i} + d^*$, where $T$ is the maximal number of steps, $\Delta_i$ are the sub-optimality gaps and $d^*$ is the minimal expected delay amongst arms. For the case that delays serves as rewards, we show optimal regret of $\sum_{i:\Delta_i > 0}\frac{\log T}{\Delta_i} + \bar{d}$, where $\bar d$ is the second maximal expected delay. These improve over the regret in the general delay-dependent payoff setting, which scales as $\sum_{i:\Delta_i > 0}\frac{\log T}{\Delta_i} + D$, where $D$ is the maximum possible delay. Our regret bounds highlight the difference between the cost and reward scenarios, showing that the improvement in the cost scenario is more significant than for the reward. Finally, we accompany our theoretical results with an empirical evaluation.

We apply advanced machine learning techniques to two challenging jet classification problems at the LHC. The first is strange-quark tagging, in particular distinguishing strange-quark jets from down-quark jets. The second, which we term fragmentation tagging, involves identifying the fragmentation channel of a quark. We exemplify the latter by training neural networks to differentiate between bottom jets containing a bottom baryon and those containing a bottom meson. The common challenge in these two problems is that neither quark lifetimes and masses nor parton showering provide discriminating tools, making it necessary to rely on differences in the distributions of the hadron types contained in each type of jet and their kinematics. For these classification tasks, we employ variations of Graph Attention Networks and the Particle Transformer, which receive jet and all constituent properties as inputs. We compare their performance to a simple Multilayer Perceptron that uses simple variables. We find that the more sophisticated architectures do not improve $s$-quark versus $d$-quark jet differentiation by a significant amount, but they do lead to a significant gain in $b$-baryon versus $b$-meson jet differentiation., Comment: 33 pages, 18 figures

Multistationarity - the existence of multiple equilibrium points - is a common phenomenon in dynamical systems from a variety of fields, including neuroscience, opinion dynamics, systems biology, and power systems. A recently proposed generalization of contraction theory, called $k$-contraction, is a promising approach for analyzing the asymptotic behaviour of multistationary systems. In particular, all bounded trajectories of a time-invariant 2-contracting system converge to an equilibrium point, but the system may have multiple equilibrium points where more than one is locally stable. An important challenge is to study $k$-contraction in large-scale interconnected systems. Inspired by a recent small-gain theorem for 2-contraction by Angeli et al., we derive a new sufficient condition for 2-contraction of a feedback interconnection of two nonlinear dynamical systems. Our condition is based on (i) deriving new formulas for the 2-multiplicative [2-additive] compound of block matrices using block Kronecker products [sums], (ii) a hierarchical approach for proving standard contraction, and (iii) a network small-gain theorem for Metzler matrices. We demonstrate our results by deriving a simple sufficient condition for 2-contraction in a network of FitzHugh-Nagumo neurons.

We consider the problem of asynchronous stochastic optimization, where an optimization algorithm makes updates based on stale stochastic gradients of the objective that are subject to an arbitrary (possibly adversarial) sequence of delays. We present a procedure which, for any given $q \in (0,1]$, transforms any standard stochastic first-order method to an asynchronous method with convergence guarantee depending on the $q$-quantile delay of the sequence. This approach leads to convergence rates of the form $O(\tau_q/qT+\sigma/\sqrt{qT})$ for non-convex and $O(\tau_q^2/(q T)^2+\sigma/\sqrt{qT})$ for convex smooth problems, where $\tau_q$ is the $q$-quantile delay, generalizing and improving on existing results that depend on the average delay. We further show a method that automatically adapts to all quantiles simultaneously, without any prior knowledge of the delays, achieving convergence rates of the form $O(\inf_{q} \tau_q/qT+\sigma/\sqrt{qT})$ for non-convex and $O(\inf_{q} \tau_q^2/(q T)^2+\sigma/\sqrt{qT})$ for convex smooth problems. Our technique is based on asynchronous mini-batching with a careful batch-size selection and filtering of stale gradients., Comment: 18 pages

Fine-tuning Large Language Models (LLMs) is an effective method to enhance their performance on downstream tasks. However, choosing the appropriate setting of tuning hyperparameters (HPs) is a labor-intensive and computationally expensive process. Here, we provide recommended HP configurations for practical use-cases that represent a better starting point for practitioners, when considering two SOTA LLMs and two commonly used tuning methods. We describe Coverage-based Search (CBS), a process for ranking HP configurations based on an offline extensive grid search, such that the top ranked configurations collectively provide a practical robust recommendation for a wide range of datasets and domains. We focus our experiments on Llama-3-8B and Mistral-7B, as well as full fine-tuning and LoRa, conducting a total of > 10,000 tuning experiments. Our results suggest that, in general, Llama-3-8B and LoRA should be preferred, when possible. Moreover, we show that for both models and tuning methods, exploring only a few HP configurations, as recommended by our analysis, can provide excellent results in practice, making this work a valuable resource for practitioners.

SOXS (Son Of X-Shooter) is the new single object spectrograph for the ESO New Technology Telescope (NTT) at the La Silla Observatory, able to cover simultaneously both the UV-VIS and NIR bands (350-2000 nm). The instrument is currently in the integration and test phase, approaching the Preliminary Acceptance in Europe (PAE) before shipment to Chile for commissioning. After the assembly and preliminary test of the control electronics at INAF - Astronomical Observatory of Capodimonte (Napoli), the two main control cabinets of SOXS are now hosted in Padova, connected to the real hardware. This contribution describes the final electronic cabinets layout, the control strategy and the different integration phases, waiting for the Preliminary Acceptance in Europe and the installation of the instrument in Chile.

The Son Of X-Shooter (SOXS) will be the specialized facility to observe any transient event with a flexible scheduler at the ESO New Technology Telescope (NTT) at La Silla, Chile. SOXS is a single object spectrograph offering simultaneous spectral coverage in UV-VIS (350-850 nm) and NIR (800-2000 nm) wavelength regimes with an average of R~4500 for a 1arcsec slit. SOXS also has imaging capabilities in the visible wavelength regime. Currently, SOXS is being integrated at the INAF-Astronomical Observatory of Padova. Subsystem- and system-level tests and verification are ongoing to ensure and confirm that every requirement and performance are met. In this paper, we report on the integration and verification of SOXS as the team and the instrument prepare for the Preliminary Acceptance Europe (PAE).

The Son Of X-Shooter (SOXS) is a single object spectrograph, built by an international consortium for the 3.58-m ESO New Technology Telescope at the La Silla Observatory [1]. It offers a simultaneous spectral coverage over 350-2000 nm, with two separate spectrographs. In this paper we present the status of the Near InfraRed (NIR) cryogenic echelle cross-dispersed spectrograph [1], in the range 0.80-2.00 {\mu}m with 15 orders, equipped with an 2k x 2k Hawaii H2RG IR array from Teledyne, working at 40K, that is currently assembled and tested on the SOXS instrument, in the premises of INAF in Padova. We describe the different tests and results of the cryo, vacuum, opto-mechanics and detector subsystems that finally will be part of the PAE by ESO., Comment: 8 Pages, 7 Figures, Astronomical Telescopes and Instrumentation, SPIE Proceedings 2024

We present the advancements in the development of the scheduler for the Son Of X-shooter instrument at the ESO-NTT 3.58-m telescope in La Silla, Chile. SOXS is designed as a single-object spectroscopic facility and features a high-efficiency spectrograph with two arms covering the spectral range of 350-2000 nm and a mean resolving power of approximately R=4500. It will conduct UV-visible and near-infrared follow-up observations of astrophysical transients, drawing from a broad pool of targets accessible through the streaming services of wide-field telescopes, both current and future, as well as high-energy satellites. The instrument will cater to various scientific objectives within the astrophysical community, each entailing specific requirements for observation planning. SOXS will operate at the European Southern Observatory (ESO) in La Silla, without the presence of astronomers on the mountain. This poses a unique challenge for the scheduling process, demanding a fully automated algorithm that is autonomously interacting with the appropriate databases and the La Silla Weather API, and is capable of presenting the operator not only with an ordered list of optimal targets (in terms of observing constraints) but also with optimal backups in the event of changing weather conditions. This imposes the necessity for a scheduler with rapid-response capabilities without compromising the optimization process, ensuring the high quality of observations and best use of the time at the telescope. We thus developed a new highly available and scalable architecture, implementing API Restful applications like Docker Containers, API Gateway, and Python-based Flask frameworks. We provide an overview of the current state of the scheduler, which is now ready for the approaching on-site testing during Commissioning phase, along with insights into its web interface and preliminary performance tests.

The Instrument Control Software of SOXS (Son Of X-Shooter), the forthcoming spectrograph for the ESO New Technology Telescope at the La Silla Observatory, has reached a mature state of development and is approaching the crucial Preliminary Acceptance in Europe phase. Now that all the subsystems have been integrated in the laboratories of the Padova Astronomical Observatory, the team operates for testing purposes with the whole instrument at both engineering and scientific level. These activities will make use of a set of software peculiarities that will be discussed in this contribution. In particular, we focus on the synoptic panel, the co-rotator system special device, on the Active Flexure Compensation system which controls two separate piezo tip-tilt devices., Comment: 6 pages, 3 figures, SPIE conference

Language agents, built on top of language models (LMs), are systems that can interact with complex environments, such as the open web. In this work, we examine whether such agents can perform realistic and time-consuming tasks on the web, e.g., monitoring real-estate markets or locating relevant nearby businesses. We introduce AssistantBench, a challenging new benchmark consisting of 214 realistic tasks that can be automatically evaluated, covering different scenarios and domains. We find that AssistantBench exposes the limitations of current systems, including language models and retrieval-augmented language models, as no model reaches an accuracy of more than 26 points. While closed-book LMs perform well in terms of accuracy, they exhibit low precision and tend to hallucinate facts. State-of-the-art web agents reach a score of near zero. Additionally, we introduce SeePlanAct (SPA), a new web agent that significantly outperforms previous agents, and an ensemble of SPA and closed-book models reaches the best overall performance. Moreover, we analyze failures of current systems and highlight that open web navigation remains a major challenge.

Since language models (LMs) now outperform average humans on many challenging tasks, it has become increasingly difficult to develop challenging, high-quality, and realistic evaluations. We address this issue by examining LMs' capabilities to generate code for solving real scientific research problems. Incorporating input from scientists and AI researchers in 16 diverse natural science sub-fields, including mathematics, physics, chemistry, biology, and materials science, we created a scientist-curated coding benchmark, SciCode. The problems in SciCode naturally factorize into multiple subproblems, each involving knowledge recall, reasoning, and code synthesis. In total, SciCode contains 338 subproblems decomposed from 80 challenging main problems. It offers optional descriptions specifying useful scientific background information and scientist-annotated gold-standard solutions and test cases for evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can solve only 4.6% of the problems in the most realistic setting. We believe that SciCode demonstrates both contemporary LMs' progress towards becoming helpful scientific assistants and sheds light on the development and evaluation of scientific AI in the future., Comment: 25 pages, 9 figures, 7 tables

Recent advancements in Language Models (LMs) have catalyzed the creation of multiple benchmarks, designed to assess these models' general capabilities. A crucial task, however, is assessing the validity of the benchmarks themselves. This is most commonly done via Benchmark Agreement Testing (BAT), where new benchmarks are validated against established ones using some agreement metric (e.g., rank correlation). Despite the crucial role of BAT for benchmark builders and consumers, there are no standardized procedures for such agreement testing. This deficiency can lead to invalid conclusions, fostering mistrust in benchmarks and upending the ability to properly choose the appropriate benchmark to use. By analyzing over 40 prominent benchmarks, we demonstrate how some overlooked methodological choices can significantly influence BAT results, potentially undermining the validity of conclusions. To address these inconsistencies, we propose a set of best practices for BAT and demonstrate how utilizing these methodologies greatly improves BAT robustness and validity. To foster adoption and facilitate future research,, we introduce BenchBench, a python package for BAT, and release the BenchBench-leaderboard, a meta-benchmark designed to evaluate benchmarks using their peers. Our findings underscore the necessity for standardized BAT, ensuring the robustness and validity of benchmark evaluations in the evolving landscape of language model research. BenchBench Package: github.com/IBM/BenchBench Leaderboard: hf.co/spaces/IBM/BenchBench, Comment: Under Review

In recent years, notable advancements have been made in the domain of visual document understanding, with the prevailing architecture comprising a cascade of vision and language models. The text component can either be extracted explicitly with the use of external OCR models in OCR-based approaches, or alternatively, the vision model can be endowed with reading capabilities in OCR-free approaches. Typically, the queries to the model are input exclusively to the language component, necessitating the visual features to encompass the entire document. In this paper, we present VisFocus, an OCR-free method designed to better exploit the vision encoder's capacity by coupling it directly with the language prompt. To do so, we replace the down-sampling layers with layers that receive the input prompt and allow highlighting relevant parts of the document, while disregarding others. We pair the architecture enhancements with a novel pre-training task, using language masking on a snippet of the document text fed to the visual encoder in place of the prompt, to empower the model with focusing capabilities. Consequently, VisFocus learns to allocate its attention to text patches pertinent to the provided prompt. Our experiments demonstrate that this prompt-guided visual encoding approach significantly improves performance, achieving state-of-the-art results on various benchmarks., Comment: 32 pages, 18 figures

Choosing a meaningful subset of features from high-dimensional observations in unsupervised settings can greatly enhance the accuracy of downstream analysis, such as clustering or dimensionality reduction, and provide valuable insights into the sources of heterogeneity in a given dataset. In this paper, we propose a self-supervised graph-based approach for unsupervised feature selection. Our method's core involves computing robust pseudo-labels by applying simple processing steps to the graph Laplacian's eigenvectors. The subset of eigenvectors used for computing pseudo-labels is chosen based on a model stability criterion. We then measure the importance of each feature by training a surrogate model to predict the pseudo-labels from the observations. Our approach is shown to be robust to challenging scenarios, such as the presence of outliers and complex substructures. We demonstrate the effectiveness of our method through experiments on real-world datasets, showing its robustness across multiple domains, particularly its effectiveness on biological datasets., Comment: 20 pages, 5 figures

The data revolution holds significant promise for the health sector. Vast amounts of data collected from individuals will be transformed into knowledge, AI models, predictive systems, and best practices. One area of health that stands to benefit greatly is the genomic domain. Progress in AI, machine learning, and data science has opened new opportunities for genomic research, promising breakthroughs in personalized medicine. However, increasing awareness of privacy and cybersecurity necessitates robust solutions to protect sensitive data in collaborative research. This paper presents a practical deployment of a privacy-preserving framework for genomic research, developed in collaboration with Lynx$.$MD, a platform for secure health data collaboration. The framework addresses critical cybersecurity and privacy challenges, enabling the privacy-preserving sharing and analysis of genomic data while mitigating risks associated with data breaches. By integrating advanced privacy-preserving algorithms, the solution ensures the protection of individual privacy without compromising data utility. A unique feature of the system is its ability to balance trade-offs between data sharing and privacy, providing stakeholders tools to quantify privacy risks and make informed decisions. Implementing the framework within Lynx$.$MD involves encoding genomic data into binary formats and applying noise through controlled perturbation techniques. This approach preserves essential statistical properties of the data, facilitating effective research and analysis. Moreover, the system incorporates real-time data monitoring and advanced visualization tools, enhancing user experience and decision-making. The paper highlights the need for tailored privacy attacks and defenses specific to genomic data. Addressing these challenges fosters collaboration in genomic research, advancing personalized medicine and public health., Comment: The first two authors contributed equally to this work. Due to the limitation "The abstract field cannot be longer than 1,920 characters", the abstract here is shorter than that in the PDF file

Communication network engineering in enterprise environments is traditionally a complex, time-consuming, and error-prone manual process. Most research on network engineering automation has concentrated on configuration synthesis, often overlooking changes in the physical network topology. This paper introduces GeNet, a multimodal co-pilot for enterprise network engineers. GeNet is a novel framework that leverages a large language model (LLM) to streamline network design workflows. It uses visual and textual modalities to interpret and update network topologies and device configurations based on user intents. GeNet was evaluated on enterprise network scenarios adapted from Cisco certification exercises. Our results demonstrate GeNet's ability to interpret network topology images accurately, potentially reducing network engineers' efforts and accelerating network design processes in enterprise environments. Furthermore, we show the importance of precise topology understanding when handling intents that require modifications to the network's topology.

Thousands of new scientific papers are published each month. Such information overload complicates researcher efforts to stay current with the state-of-the-art as well as to verify and correctly attribute claims. We pose the following research question: Given a text excerpt referencing a paper, could an LM act as a research assistant to correctly identify the referenced paper? We advance efforts to answer this question by building a benchmark that evaluates the abilities of LMs in citation attribution. Our benchmark, CiteME, consists of text excerpts from recent machine learning papers, each referencing a single other paper. CiteME use reveals a large gap between frontier LMs and human performance, with LMs achieving only 4.2-18.5% accuracy and humans 69.7%. We close this gap by introducing CiteAgent, an autonomous system built on the GPT-4o LM that can also search and read papers, which achieves an accuracy of 35.3\% on CiteME. Overall, CiteME serves as a challenging testbed for open-ended claim attribution, driving the research community towards a future where any claim made by an LM can be automatically verified and discarded if found to be incorrect.

The phenomenon of double descent has recently gained attention in supervised learning. It challenges the conventional wisdom of the bias-variance trade-off by showcasing a surprising behavior. As the complexity of the model increases, the test error initially decreases until reaching a certain point where the model starts to overfit the train set, causing the test error to rise. However, deviating from classical theory, the error exhibits another decline when exceeding a certain degree of over-parameterization. We study the presence of double descent in unsupervised learning, an area that has received little attention and is not yet fully understood. We conduct extensive experiments using under-complete auto-encoders (AEs) for various applications, such as dealing with noisy data, domain shifts, and anomalies. We use synthetic and real data and identify model-wise, epoch-wise, and sample-wise double descent for all the aforementioned applications. Finally, we assessed the usability of the AEs for detecting anomalies and mitigating the domain shift between datasets. Our findings indicate that over-parameterized models can improve performance not only in terms of reconstruction, but also in enhancing capabilities for the downstream task.

Large Language Models (LLMs) can become outdated over time as they may lack updated world knowledge, leading to factual knowledge errors and gaps. Knowledge Editing (KE) aims to overcome this challenge using weight updates that do not require expensive retraining. We propose treating KE as an LLM alignment problem. Toward this goal, we introduce Knowledge Direct Preference Optimization (KDPO), a variation of the Direct Preference Optimization (DPO) that is more effective for knowledge modifications. Our method is based on an online approach that continually updates the knowledge stored in the model. We use the current knowledge as a negative sample and the new knowledge we want to introduce as a positive sample in a process called DPO. We also use teacher-forcing for negative sample generation and optimize using the positive sample, which helps maintain localized changes. We tested our KE method on various datasets and models, comparing it to several cutting-edge methods, with 100 and 500 sequential edits. Additionally, we conducted an ablation study comparing our method to the standard DPO approach. Our experimental results show that our modified DPO method allows for more refined KE, achieving similar or better performance compared to previous methods., Comment: 9 pages, 4 figures

With the increasing prevalence of voice-activated devices and applications, keyword spotting (KWS) models enable users to interact with technology hands-free, enhancing convenience and accessibility in various contexts. Deploying KWS models on edge devices, such as smartphones and embedded systems, offers significant benefits for real-time applications, privacy, and bandwidth efficiency. However, these devices often possess limited computational power and memory. This necessitates optimizing neural network models for efficiency without significantly compromising their accuracy. To address these challenges, we propose a novel keyword-spotting model based on sparse input representation followed by a linear classifier. The model is four times faster than the previous state-of-the-art edge device-compatible model with better accuracy. We show that our method is also more robust in noisy environments while being fast. Our code is available at: https://github.com/jsvir/sparknet.

PLATO (PLAnetary Transits and Oscillations of stars) is ESA's M3 mission designed to detect and characterise extrasolar planets and perform asteroseismic monitoring of a large number of stars. PLATO will detect small planets (down to <2 R_(Earth)) around bright stars (<11 mag), including terrestrial planets in the habitable zone of solar-like stars. With the complement of radial velocity observations from the ground, planets will be characterised for their radius, mass, and age with high accuracy (5 %, 10 %, 10 % for an Earth-Sun combination respectively). PLATO will provide us with a large-scale catalogue of well-characterised small planets up to intermediate orbital periods, relevant for a meaningful comparison to planet formation theories and to better understand planet evolution. It will make possible comparative exoplanetology to place our Solar System planets in a broader context. In parallel, PLATO will study (host) stars using asteroseismology, allowing us to determine the stellar properties with high accuracy, substantially enhancing our knowledge of stellar structure and evolution. The payload instrument consists of 26 cameras with 12cm aperture each. For at least four years, the mission will perform high-precision photometric measurements. Here we review the science objectives, present PLATO's target samples and fields, provide an overview of expected core science performance as well as a description of the instrument and the mission profile at the beginning of the serial production of the flight cameras. PLATO is scheduled for a launch date end 2026. This overview therefore provides a summary of the mission to the community in preparation of the upcoming operational phases.

Let $\alpha$ be a Steinhaus or a Rademacher random multiplicative function. For a wide class of multiplicative functions $f$ we show that the sum $\sum_{n \le x}\alpha(n) f(n)$, normalised to have mean square $1$, has a non-Gaussian limiting distribution. More precisely, we establish a generalised central limit theorem with random variance determined by the total mass of a random measure associated with $\alpha f$. Our result applies to $d_z$, the $z$-th divisor function, as long as $z$ is strictly between $0$ and $\tfrac{1}{\sqrt{2}}$. Other examples of admissible $f$-s include any multiplicative indicator function with the property that $f(p)=1$ holds for a set of primes of density strictly between $0$ and $\tfrac{1}{2}$., Comment: 42 pages, 2 figures. Typos fixed, abstract and discussion of previous works updated. Codes for simulation experiment available on authors' personal page; comments are still welcome

Let $\alpha \colon \mathbb{N} \to S^1$ be the Steinhaus multiplicative function: a completely multiplicative function such that $(\alpha(p))_{p\text{ prime}}$ are i.i.d.~random variables uniformly distributed on $S^1$. Helson conjectured that $\mathbb{E}|\sum_{n\le x}\alpha(n)|=o(\sqrt{x})$ as $x \to \infty$, and this was solved in strong form by Harper. We give a short proof of the conjecture using a result of Saksman and Webb on a random model for the zeta function., Comment: 7 pages. Typos fixed and discussion of previous works added; comments are welcome

Modern cameras typically offer two types of image states: a minimally processed linear raw RGB image representing the raw sensor data, and a highly-processed non-linear image state, such as the sRGB state. The CIE-XYZ color space is a device-independent linear space used as part of the camera pipeline and can be helpful for computer vision tasks, such as image deblurring, dehazing, and color recognition tasks in medical applications, where color accuracy is important. However, images are usually saved in non-linear states, and achieving CIE-XYZ color images using conventional methods is not always possible. To tackle this issue, classical methodologies have been developed that focus on reversing the acquisition pipeline. More recently, supervised learning has been employed, using paired CIE-XYZ and sRGB representations of identical images. However, obtaining a large-scale dataset of CIE-XYZ and sRGB pairs can be challenging. To overcome this limitation and mitigate the reliance on large amounts of paired data, self-supervised learning (SSL) can be utilized as a substitute for relying solely on paired data. This paper proposes a framework for using SSL methods alongside paired data to reconstruct CIE-XYZ images and re-render sRGB images, outperforming existing approaches. The proposed framework is applied to the sRGB2XYZ dataset., Comment: 10 pages, 4 figures, WSCG 2024 conference

We present MedConceptsQA, a dedicated open source benchmark for medical concepts question answering. The benchmark comprises of questions of various medical concepts across different vocabularies: diagnoses, procedures, and drugs. The questions are categorized into three levels of difficulty: easy, medium, and hard. We conducted evaluations of the benchmark using various Large Language Models. Our findings show that pre-trained clinical Large Language Models achieved accuracy levels close to random guessing on this benchmark, despite being pre-trained on medical data. However, GPT-4 achieves an absolute average improvement of nearly 27%-37% (27% for zero-shot learning and 37% for few-shot learning) when compared to clinical Large Language Models. Our benchmark serves as a valuable resource for evaluating the understanding and reasoning of medical concepts by Large Language Models. Our benchmark is available at https://huggingface.co/datasets/ofir408/MedConceptsQA

Language model (LM) agents are increasingly being used to automate complicated tasks in digital environments. Just as humans benefit from powerful software applications, such as integrated development environments, for complex tasks like software engineering, we posit that LM agents represent a new category of end users with their own needs and abilities, and would benefit from specially-built interfaces to the software they use. We investigate how interface design affects the performance of language model agents. As a result of this exploration, we introduce SWE-agent: a system that facilitates LM agents to autonomously use computers to solve software engineering tasks. SWE-agent's custom agent-computer interface (ACI) significantly enhances an agent's ability to create and edit code files, navigate entire repositories, and execute tests and other programs. We evaluate SWE-agent on SWE-bench and HumanEvalFix, achieving state-of-the-art performance on both with a pass@1 rate of 12.5% and 87.7%, respectively, far exceeding the previous state-of-the-art achieved with non-interactive LMs. Finally, we provide insight on how the design of the ACI can impact agents' behavior and performance., Comment: Code, data, and demo available at https://swe-agent.com

Large-scale text-to-image models that can generate high-quality and diverse images based on textual prompts have shown remarkable success. These models aim ultimately to create complex scenes, and addressing the challenge of multi-subject generation is a critical step towards this goal. However, the existing state-of-the-art diffusion models face difficulty when generating images that involve multiple subjects. When presented with a prompt containing more than one subject, these models may omit some subjects or merge them together. To address this challenge, we propose a novel approach based on a guiding principle. We allow the diffusion model to initially propose a layout, and then we rearrange the layout grid. This is achieved by enforcing cross-attention maps (XAMs) to adhere to proposed masks and by migrating pixels from latent maps to new locations determined by us. We introduce new loss terms aimed at reducing XAM entropy for clearer spatial definition of subjects, reduce the overlap between XAMs, and ensure that XAMs align with their respective masks. We contrast our approach with several alternative methods and show that it more faithfully captures the desired concepts across a variety of text prompts.

Deep neural networks that achieve remarkable performance in image classification have previously been shown to be easily fooled by tiny transformations such as a one pixel translation of the input image. In order to address this problem, two approaches have been proposed in recent years. The first approach suggests using huge datasets together with data augmentation in the hope that a highly varied training set will teach the network to learn to be invariant. The second approach suggests using architectural modifications based on sampling theory to deal explicitly with image translations. In this paper, we show that these approaches still fall short in robustly handling 'natural' image translations that simulate a subtle change in camera orientation. Our findings reveal that a mere one-pixel translation can result in a significant change in the predicted image representation for approximately 40% of the test images in state-of-the-art models (e.g. open-CLIP trained on LAION-2B or DINO-v2) , while models that are explicitly constructed to be robust to cyclic translations can still be fooled with 1 pixel realistic (non-cyclic) translations 11% of the time. We present Robust Inference by Crop Selection: a simple method that can be proven to achieve any desired level of consistency, although with a modest tradeoff with the model's accuracy. Importantly, we demonstrate how employing this method reduces the ability to fool state-of-the-art models with a 1 pixel translation to less than 5% while suffering from only a 1% drop in classification accuracy. Additionally, we show that our method can be easy adjusted to deal with circular shifts as well. In such case we achieve 100% robustness to integer shifts with state-of-the-art accuracy, and with no need for any further training., Comment: 14 pages, 6 appendices, 17 figures

Quantum metrology leverages quantum correlations for enhanced parameter estimation. Recently, structured light enabled increased resolution and sensitivity in quantum metrology systems. However, lossy and complex setups impacting photon flux, hinder true quantum advantage while using high dimensional structured light. We introduce a straightforward mechanical rotation quantum sensing mechanism, employing high-dimensional structured light and a compact high-flux (45,000 coincidence counts per second) N00N state source with N=2. The system utilizes two opposite spiral phase plates with topological charge of up to l=16 that convert mechanical rotation into wavefront phase shifts, and exhibit a 16-fold enhanced super-resolution and 25-fold enhanced sensitivity between different topological charges, while retaining the acquisition times and with negligible change in coincidence count. Furthermore, the high photon flux enables to detect mechanical angular acceleration in real-time. Our approach paves the way for highly sensitive quantum measurements, applicable to various interferometric schemes., Comment: {\dag} equal contribution

Decoherence in qubits, caused by their interaction with a noisy environment, poses a significant challenge to developing reliable quantum processors. Monitoring the qubit's environment enables not only to identify decoherence events but also to reverse these errors, thereby restoring the qubit coherence. This approach is particularly beneficial for superconducting cavity qubits, whose unavoidable interaction with auxiliary transmons impacts their coherence. In this work, we uncover the intricate dynamics of cavity decoherence by tracking the noisy trajectory of a transmon acting as the cavity's environment. Using real-time feedback, we successfully recover the lost coherence of the cavity qubit, achieving a fivefold increase in its dephasing time. Alternatively, by detecting transmon errors and converting them into erasures, we improve the cavity phase coherence by more than an order of magnitude. These advances are essential for implementing long-lived cavity qubits with high-fidelity gates and can enable more efficient bosonic quantum error correction codes., Comment: 17 pages, 10 figures, including supplementary information