Search

The sub-Jovian desert is a region in the mass-period and radius-period parameter space, typically encompassing short-period ranges between super-Earths and hot Jupiters, that exhibits an intrinsic dearth of planets. This scarcity is likely shaped by photoevaporation caused by the stellar irradiation received by giant planets that have migrated inward. We report the detection and characterization of TOI-3568 b, a transiting super-Neptune with a mass of $26.4\pm1.0$ M$_\oplus$, a radius of $5.30\pm0.27$ R$_\oplus$, a bulk density of $0.98\pm0.15$ g cm$^{-3}$, and an orbital period of 4.417965(5) d situated in the vicinity of the sub-Jovian desert. This planet orbiting a K dwarf star with solar metallicity, was identified photometrically by TESS. It was characterized as a planet by our high-precision radial velocity monitoring program using MAROON-X at Gemini North, supplemented by additional observations from the SPICE large program with SPIRou at CFHT. We performed a Bayesian MCMC joint analysis of the TESS and ground-based photometry, MAROON-X and SPIRou radial velocities, to measure the orbit, radius, and mass of the planet, as well as a detailed analysis of the high-resolution flux and polarimetric spectra to determine the physical parameters and elemental abundances of the host star. Our results reveal TOI-3568 b as a hot super-Neptune, rich in hydrogen and helium with a core of heavier elements with a mass between 10 and 25 M$_\oplus$. We analyzed the photoevaporation status of TOI-3568 b and found that it experiences one of the highest EUV luminosities among planets with a mass M$_{\rm p}$ $<2$ M$_{\rm Nep}$, yet it has an evaporation lifetime exceeding 5 Gyr. Positioned in the transition between two significant populations of exoplanets on the mass-period and energy diagrams, this planet presents an opportunity to test theories concerning the origin of the sub-Jovian desert., Comment: Accepted for publication in A&A on September 4, 2024

Learned language-conditioned robot policies often struggle to effectively adapt to new real-world tasks even when pre-trained across a diverse set of instructions. We propose a novel approach for few-shot adaptation to unseen tasks that exploits the semantic understanding of task decomposition provided by vision-language models (VLMs). Our method, Policy Adaptation via Language Optimization (PALO), combines a handful of demonstrations of a task with proposed language decompositions sampled from a VLM to quickly enable rapid nonparametric adaptation, avoiding the need for a larger fine-tuning dataset. We evaluate PALO on extensive real-world experiments consisting of challenging unseen, long-horizon robot manipulation tasks. We find that PALO is able of consistently complete long-horizon, multi-tier tasks in the real world, outperforming state of the art pre-trained generalist policies, and methods that have access to the same demonstrations., Comment: 27 pages, 14 figures

Offline-to-online reinforcement learning (RL), a framework that trains a policy with offline RL and then further fine-tunes it with online RL, has been considered a promising recipe for data-driven decision-making. While sensible, this framework has drawbacks: it requires domain-specific offline RL pre-training for each task, and is often brittle in practice. In this work, we propose unsupervised-to-online RL (U2O RL), which replaces domain-specific supervised offline RL with unsupervised offline RL, as a better alternative to offline-to-online RL. U2O RL not only enables reusing a single pre-trained model for multiple downstream tasks, but also learns better representations, which often result in even better performance and stability than supervised offline-to-online RL. To instantiate U2O RL in practice, we propose a general recipe for U2O RL to bridge task-agnostic unsupervised offline skill-based policy pre-training and supervised online fine-tuning. Throughout our experiments in nine state-based and pixel-based environments, we empirically demonstrate that U2O RL achieves strong performance that matches or even outperforms previous offline-to-online RL approaches, while being able to reuse a single pre-trained model for a number of different downstream tasks.

Modern machine learning systems rely on large datasets to attain broad generalization, and this often poses a challenge in robot learning, where each robotic platform and task might have only a small dataset. By training a single policy across many different kinds of robots, a robot learning method can leverage much broader and more diverse datasets, which in turn can lead to better generalization and robustness. However, training a single policy on multi-robot data is challenging because robots can have widely varying sensors, actuators, and control frequencies. We propose CrossFormer, a scalable and flexible transformer-based policy that can consume data from any embodiment. We train CrossFormer on the largest and most diverse dataset to date, 900K trajectories across 20 different robot embodiments. We demonstrate that the same network weights can control vastly different robots, including single and dual arm manipulation systems, wheeled robots, quadcopters, and quadrupeds. Unlike prior work, our model does not require manual alignment of the observation or action spaces. Extensive experiments in the real world show that our method matches the performance of specialist policies tailored for each embodiment, while also significantly outperforming the prior state of the art in cross-embodiment learning., Comment: Project website at https://crossformer-model.github.io/

As Artificial Intelligence (AI) models are increasingly integrated into critical systems, the need for a robust framework to establish the trustworthiness of AI is increasingly paramount. While collaborative efforts have established conceptual foundations for such a framework, there remains a significant gap in developing concrete, technically robust methods for assessing AI model quality and performance. A critical drawback in the traditional methods for assessing the validity and generalizability of models is their dependence on internal developer datasets, rendering it challenging to independently assess and verify their performance claims. This paper introduces a novel approach for assessing a newly trained model's performance based on another known model by calculating correlation between neural networks. The proposed method evaluates correlations by determining if, for each neuron in one network, there exists a neuron in the other network that produces similar output. This approach has implications for memory efficiency, allowing for the use of smaller networks when high correlation exists between networks of different sizes. Additionally, the method provides insights into robustness, suggesting that if two highly correlated networks are compared and one demonstrates robustness when operating in production environments, the other is likely to exhibit similar robustness. This contribution advances the technical toolkit for responsible AI, supporting more comprehensive and nuanced evaluations of AI models to ensure their safe and effective deployment. Code is available at https://github.com/aheldis/Cross-model-correlation.git.

Offline reinforcement learning algorithms hold the promise of enabling data-driven RL methods that do not require costly or dangerous real-world exploration and benefit from large pre-collected datasets. This in turn can facilitate real-world applications, as well as a more standardized approach to RL research. Furthermore, offline RL methods can provide effective initializations for online finetuning to overcome challenges with exploration. However, evaluating progress on offline RL algorithms requires effective and challenging benchmarks that capture properties of real-world tasks, provide a range of task difficulties, and cover a range of challenges both in terms of the parameters of the domain (e.g., length of the horizon, sparsity of rewards) and the parameters of the data (e.g., narrow demonstration data or broad exploratory data). While considerable progress in offline RL in recent years has been enabled by simpler benchmark tasks, the most widely used datasets are increasingly saturating in performance and may fail to reflect properties of realistic tasks. We propose a new benchmark for offline RL that focuses on realistic simulations of robotic manipulation and locomotion environments, based on models of real-world robotic systems, and comprising a variety of data sources, including scripted data, play-style data collected by human teleoperators, and other data sources. Our proposed benchmark covers state-based and image-based domains, and supports both offline RL and online fine-tuning evaluation, with some of the tasks specifically designed to require both pre-training and fine-tuning. We hope that our proposed benchmark will facilitate further progress on both offline RL and fine-tuning algorithms. Website with code, examples, tasks, and data is available at \url{https://sites.google.com/view/d5rl/}, Comment: RLC 2024

Diffusion models excel at capturing the natural design spaces of images, molecules, DNA, RNA, and protein sequences. However, rather than merely generating designs that are natural, we often aim to optimize downstream reward functions while preserving the naturalness of these design spaces. Existing methods for achieving this goal often require ``differentiable'' proxy models (\textit{e.g.}, classifier guidance or DPS) or involve computationally expensive fine-tuning of diffusion models (\textit{e.g.}, classifier-free guidance, RL-based fine-tuning). In our work, we propose a new method to address these challenges. Our algorithm is an iterative sampling method that integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the future, into the standard inference procedure of pre-trained diffusion models. Notably, our approach avoids fine-tuning generative models and eliminates the need to construct differentiable models. This enables us to (1) directly utilize non-differentiable features/reward feedback, commonly used in many scientific domains, and (2) apply our method to recent discrete diffusion models in a principled way. Finally, we demonstrate the effectiveness of our algorithm across several domains, including image generation, molecule generation, and DNA/RNA sequence generation. The code is available at \href{https://github.com/masa-ue/SVDD}{https://github.com/masa-ue/SVDD}., Comment: The code is available at https://github.com/masa-ue/SVDD

Topological abstractions offer a method to summarize the behavior of vector fields but computing them robustly can be challenging due to numerical precision issues. One alternative is to represent the vector field using a discrete approach, which constructs a collection of pairs of simplices in the input mesh that satisfies criteria introduced by Forman's discrete Morse theory. While numerous approaches exist to compute pairs in the restricted case of the gradient of a scalar field, state-of-the-art algorithms for the general case of vector fields require expensive optimization procedures. This paper introduces a fast, novel approach for pairing simplices of two-dimensional, triangulated vector fields that do not vary in time. The key insight of our approach is that we can employ a local evaluation, inspired by the approach used to construct a discrete gradient field, where every simplex in a mesh is considered by no more than one of its vertices. Specifically, we observe that for any edge in the input mesh, we can uniquely assign an outward direction of flow. We can further expand this consistent notion of outward flow at each vertex, which corresponds to the concept of a downhill flow in the case of scalar fields. Working with outward flow enables a linear-time algorithm that processes the (outward) neighborhoods of each vertex one-by-one, similar to the approach used for scalar fields. We couple our approach to constructing discrete vector fields with a method to extract, simplify, and visualize topological features. Empirical results on analytic and simulation data demonstrate drastic improvements in running time, produce features similar to the current state-of-the-art, and show the application of simplification to large, complex flows., Comment: 11 pages, Accepted at IEEE Vis Conference 2024

We build on a 1990 paper of Bukovsky and Coplakova-Hartova. First, we remove the hypothesis of $\textsf{CH}$ from one of their minimality results. Then, using a measurable cardinal, we show that there is a $|\aleph_2^V|=\aleph_1$-minimal extension that is not a $|\aleph_3^V|=\aleph_1$-extension, answering the first of their questions.

Wide-separation lensed quasars (WSLQs) are a rare class of strongly lensed quasars, magnified by foreground massive galaxy clusters, with typically large magnifications of the multiple quasar images. They are a relatively unexplored opportunity for detailed study of quasar host galaxies. The current small sample of known WSLQs has a median redshift of $z\approx 2.1$, larger than most other samples of quasar host galaxies studied to date. Here, we derive precise constraints on the properties of six WSLQs and their host galaxies, using parametric surface brightness fitting, measurements of quasar emission lines, and stellar population synthesis of host galaxies in six WSLQ systems. Our results, with significant uncertainty, indicate that these six hosts are a mixture of star-forming and quiescent galaxies. To probe for co-evolution between AGNs and host galaxies, we model the offset from the `local' ($z=0$) $M_{\rm{BH}}\unicode{x2013}M_\star$ relation as a simple power-law in redshift. Accounting for selection effects, a WSLQ-based model for evolution in the $M_{\rm{BH}}\unicode{x2013}M_\star$ relation has a power-law index of $\gamma_M=-0.42\pm0.31$, consistent with no evolution. Compared to several literature samples, which mostly probe unlensed quasars at $z<2$, the WSLQ sample shows less evolution from the local relation, at $\sim 4\sigma$. We find that selection affects and choices of $M_{\rm{BH}}$ calibration are the most important systematics in these comparisons. Given that we resolve host galaxy flux confidently even from the ground in some instances, our work demonstrates that WSLQs and highly magnified AGNs are exceptional systems for future AGN$\unicode{x2013}$host co-evolution studies., Comment: Submitted to ApJ. 25 pages + 7-page appendix, 12+4 figures. Key results are shown starting with Figure 6. Comments welcome

Locality of bulk operators in AdS imposes stringent constraints on their description in terms of the boundary CFT. These constraints are encoded as sum rules on the bulk-to-boundary expansion coefficients. In this paper, we construct families of sum rules that are (i) complete and (ii) `dual' to sparse CFT spectra. The sum rules trivialise the reconstruction of bulk operators in strongly interacting QFTs in AdS space and allow us to write down explicit, exact, interacting solutions to the locality problem. Technically, we characterise `completeness' of a set of sum rules by constructing Schauder bases for a certain space of real-analytic functions. In turn, this allows us to prove a Paley-Weiner type theorem characterising the space of sum rules. Remarkably, with control over this space, it is possible to write down closed-form `designer sum rules', dual to a chosen spectrum of CFT operators meeting certain criteria. We discuss the consequences of our results for both analytic and numerical bootstrap applications., Comment: 50 pages + appendices

The exponential scaling of complete active space (CAS) and full configuration interaction (CI) calculations limits the ability of quantum chemists to simulate the electronic structures of strongly correlated systems. Herein, we present corner hierarchically approximated CI (CHACI), an approach to wave function compression based on corner hierarchical matrices (CH-matrices) -- a new variant of hierarchical matrices based on a block-wise low-rank decomposition. By application to dodecacene, a strongly correlated molecule, we demonstrate that CH matrix compression provides superior compression compared to a truncated global singular value decomposition. The compression ratio is shown to improve with increasing active space size. By comparison of several alternative schemes, we demonstrate that superior compression is achieved by a) using a blocking approach that emphasizes the upper-left corner of the CI vector, b) sorting the CI vector prior to compression, and c) optimizing the rank of each block to maximize information density.

Intelligent instruction-following robots capable of improving from autonomously collected experience have the potential to transform robot learning: instead of collecting costly teleoperated demonstration data, large-scale deployment of fleets of robots can quickly collect larger quantities of autonomous data that can collectively improve their performance. However, autonomous improvement requires solving two key problems: (i) fully automating a scalable data collection procedure that can collect diverse and semantically meaningful robot data and (ii) learning from non-optimal, autonomous data with no human annotations. To this end, we propose a novel approach that addresses these challenges, allowing instruction-following policies to improve from autonomously collected data without human supervision. Our framework leverages vision-language models to collect and evaluate semantically meaningful experiences in new environments, and then utilizes a decomposition of instruction following tasks into (semantic) language-conditioned image generation and (non-semantic) goal reaching, which makes it significantly more practical to improve from this autonomously collected data without any human annotations. We carry out extensive experiments in the real world to demonstrate the effectiveness of our approach, and find that in a suite of unseen environments, the robot policy can be improved significantly with autonomously collected data. We open-source the code for our semantic autonomous improvement pipeline, as well as our autonomous dataset of 30.5K trajectories collected across five tabletop environments.

The Ehrenfest with collapse-to-a-block (TAB) molecular dynamics approach was recently introduced to allow accurate simulation of nonadiabatic dynamics on many electronic states. Previous benchmarking work has demonstrated it to be highly accurate for modeling dynamics in one-dimensional analytical models, but nonadiabatic dynamics often involves conical intersections, which are inherently two-dimensional. In this report, we assess the performance of TAB on two-dimensional models of cascades of conical intersections in dense manifolds of states. Several variants of TAB are considered, including TAB-w, which is based on the assumption of a Gaussian rather than exponential decay of the coherence, and TAB-DMS, which incorporates an efficient collapse procedure based on approximate eigenstates. Upon comparison to numerically exact quantum dynamics simulations, it is found that all TAB variants provide a suitable description of the dynamical passage through a cascade of conical intersections. The TAB-w approach is found to provide a somewhat more accurate description of population dynamics than the original TAB method, with final absolute population errors $\leq$0.013 in all cases. Even when only four approximate eigenstates are computed, the use of approximate eigenstates was found to introduce minimal additional error (absolute population error $\leq$0.018 in all models).

A defining property of Hawking radiation is that states with very low entanglement masquerade as highly mixed states; this property is captured by a quantum computational phenomenon known as spoofing entanglement. Motivated by the potential implications for black hole information and the emergence of spacetime connectivity, as well as possible applications of spoofing entanglement, we investigate the geometrization of two types of entanglement spoofers in AdS/CFT: so-called EFI pairs and pseudoentangled state ensembles. We show that (a strengthened version of) EFI pairs with a semiclassical bulk dual have a Python's Lunch; the maximally mixed state over the pseudoentangled state ensemble likewise features a Python's Lunch. Since a Python's Lunch must lie behind an event horizon, we find that black holes are the exclusive gravitational source of entanglement spoofing in the semiclassical limit. Finally, we use an extant construction of holographic pseudorandom states to yield a candidate example of a pseudoentangled state ensemble with a semiclassical bulk dual., Comment: 25+1 pages, 1 figure

This tutorial provides a comprehensive survey of methods for fine-tuning diffusion models to optimize downstream reward functions. While diffusion models are widely known to provide excellent generative modeling capability, practical applications in domains such as biology require generating samples that maximize some desired metric (e.g., translation efficiency in RNA, docking score in molecules, stability in protein). In these cases, the diffusion model can be optimized not only to generate realistic samples but also to explicitly maximize the measure of interest. Such methods are based on concepts from reinforcement learning (RL). We explain the application of various RL algorithms, including PPO, differentiable optimization, reward-weighted MLE, value-weighted sampling, and path consistency learning, tailored specifically for fine-tuning diffusion models. We aim to explore fundamental aspects such as the strengths and limitations of different RL-based fine-tuning algorithms across various scenarios, the benefits of RL-based fine-tuning compared to non-RL-based approaches, and the formal objectives of RL-based fine-tuning (target distributions). Additionally, we aim to examine their connections with related topics such as classifier guidance, Gflownets, flow-based diffusion models, path integral control theory, and sampling from unnormalized distributions such as MCMC. The code of this tutorial is available at https://github.com/masa-ue/RLfinetuning_Diffusion_Bioseq, Comment: We plan to add more content/codes. Please let us know if there are any comments

This paper presents a practical approach for the optimization of topological simplification, a central pre-processing step for the analysis and visualization of scalar data. Given an input scalar field f and a set of "signal" persistence pairs to maintain, our approach produces an output field g that is close to f and which optimizes (i) the cancellation of "non-signal" pairs, while (ii) preserving the "signal" pairs. In contrast to pre-existing simplification algorithms, our approach is not restricted to persistence pairs involving extrema and can thus address a larger class of topological features, in particular saddle pairs in three-dimensional scalar data. Our approach leverages recent generic persistence optimization frameworks and extends them with tailored accelerations specific to the problem of topological simplification. Extensive experiments report substantial accelerations over these frameworks, thereby making topological simplification optimization practical for real-life datasets. Our approach enables a direct visualization and analysis of the topologically simplified data, e.g., via isosurfaces of simplified topology (fewer components and handles). We apply our approach to the extraction of prominent filament structures in three-dimensional data. Specifically, we show that our pre-simplification of the data leads to practical improvements over standard topological techniques for removing filament loops. We also show how our approach can be used to repair genus defects in surface processing. Finally, we provide a C++ implementation for reproducibility purposes., Comment: 13 pages, 10 figures, IEEE VIS 2024

Ancient manuscripts are frequently damaged, containing gaps in the text known as lacunae. In this paper, we present a bidirectional RNN model for character prediction of Coptic characters in manuscript lacunae. Our best model performs with 72% accuracy on single character reconstruction, but falls to 37% when reconstructing lacunae of various lengths. While not suitable for definitive manuscript reconstruction, we argue that our RNN model can help scholars rank the likelihood of textual reconstructions. As evidence, we use our RNN model to rank reconstructions in two early Coptic manuscripts. Our investigation shows that neural models can augment traditional methods of textual restoration, providing scholars with an additional tool to assess lacunae in Coptic manuscripts., Comment: Machine Learning for Ancient Languages, ACL 2024 Workshop, 15 August 2024

A key limitation of learned robot control policies is their inability to generalize outside their training data. Recent works on vision-language-action models (VLAs) have shown that the use of large, internet pre-trained vision-language models as the backbone of learned robot policies can substantially improve their robustness and generalization ability. Yet, one of the most exciting capabilities of large vision-language models in other domains is their ability to reason iteratively through complex problems. Can that same capability be brought into robotics to allow policies to improve performance by reasoning about a given task before acting? Naive use of "chain-of-thought" (CoT) style prompting is significantly less effective with standard VLAs because of the relatively simple training examples that are available to them. Additionally, purely semantic reasoning about sub-tasks, as is common in regular CoT, is insufficient for robot policies that need to ground their reasoning in sensory observations and the robot state. To this end, we introduce Embodied Chain-of-Thought Reasoning (ECoT) for VLAs, in which we train VLAs to perform multiple steps of reasoning about plans, sub-tasks, motions, and visually grounded features like object bounding boxes and end effector positions, before predicting the robot action. We design a scalable pipeline for generating synthetic training data for ECoT on large robot datasets. We demonstrate, that ECoT increases the absolute success rate of OpenVLA, the current strongest open-source VLA policy, by 28% across challenging generalization tasks, without any additional robot training data. Additionally, ECoT makes it easier for humans to interpret a policy's failures and correct its behavior using natural language., Comment: Project Website: https://embodied-cot.github.io

An elusive goal in navigation research is to build an intelligent agent that can understand multimodal instructions including natural language and image, and perform useful navigation. To achieve this, we study a widely useful category of navigation tasks we call Multimodal Instruction Navigation with demonstration Tours (MINT), in which the environment prior is provided through a previously recorded demonstration video. Recent advances in Vision Language Models (VLMs) have shown a promising path in achieving this goal as it demonstrates capabilities in perceiving and reasoning about multimodal inputs. However, VLMs are typically trained to predict textual output and it is an open research question about how to best utilize them in navigation. To solve MINT, we present Mobility VLA, a hierarchical Vision-Language-Action (VLA) navigation policy that combines the environment understanding and common sense reasoning power of long-context VLMs and a robust low-level navigation policy based on topological graphs. The high-level policy consists of a long-context VLM that takes the demonstration tour video and the multimodal user instruction as input to find the goal frame in the tour video. Next, a low-level policy uses the goal frame and an offline constructed topological graph to generate robot actions at every timestep. We evaluated Mobility VLA in a 836m^2 real world environment and show that Mobility VLA has a high end-to-end success rates on previously unsolved multimodal instructions such as "Where should I return this?" while holding a plastic bin. A video demonstrating Mobility VLA can be found here: https://youtu.be/-Tof__Q8_5s

This work presents HiLMa-Res, a hierarchical framework leveraging reinforcement learning to tackle manipulation tasks while performing continuous locomotion using quadrupedal robots. Unlike most previous efforts that focus on solving a specific task, HiLMa-Res is designed to be general for various loco-manipulation tasks that require quadrupedal robots to maintain sustained mobility. The novel design of this framework tackles the challenges of integrating continuous locomotion control and manipulation using legs. It develops an operational space locomotion controller that can track arbitrary robot end-effector (toe) trajectories while walking at different velocities. This controller is designed to be general to different downstream tasks, and therefore, can be utilized in high-level manipulation planning policy to address specific tasks. To demonstrate the versatility of this framework, we utilize HiLMa-Res to tackle several challenging loco-manipulation tasks using a quadrupedal robot in the real world. These tasks span from leveraging state-based policy to vision-based policy, from training purely from the simulation data to learning from real-world data. In these tasks, HiLMa-Res shows better performance than other methods., Comment: IROS 2024

As large language models (LLMs) are deployed in more and more real-world situations, it is crucial to understand their decision-making when faced with moral dilemmas. Inspired by a large-scale cross-cultural study of human moral preferences, "The Moral Machine Experiment", we set up the same set of moral choices for LLMs. We translate 1K vignettes of moral dilemmas, parametrically varied across key axes, into 100+ languages, and reveal the preferences of LLMs in each of these languages. We then compare the responses of LLMs to that of human speakers of those languages, harnessing a dataset of 40 million human moral judgments. We discover that LLMs are more aligned with human preferences in languages such as English, Korean, Hungarian, and Chinese, but less aligned in languages such as Hindi and Somali (in Africa). Moreover, we characterize the explanations LLMs give for their moral choices and find that fairness is the most dominant supporting reason behind GPT-4's decisions and utilitarianism by GPT-3. We also discover "language inequality" (which we define as the model's different development levels in different languages) in a series of meta-properties of moral decision making.

Legged robots are physically capable of navigating a diverse variety of environments and overcoming a wide range of obstructions. For example, in a search and rescue mission, a legged robot could climb over debris, crawl through gaps, and navigate out of dead ends. However, the robot's controller needs to respond intelligently to such varied obstacles, and this requires handling unexpected and unusual scenarios successfully. This presents an open challenge to current learning methods, which often struggle with generalization to the long tail of unexpected situations without heavy human supervision. To address this issue, we investigate how to leverage the broad knowledge about the structure of the world and commonsense reasoning capabilities of vision-language models (VLMs) to aid legged robots in handling difficult, ambiguous situations. We propose a system, VLM-Predictive Control (VLM-PC), combining two key components that we find to be crucial for eliciting on-the-fly, adaptive behavior selection with VLMs: (1) in-context adaptation over previous robot interactions and (2) planning multiple skills into the future and replanning. We evaluate VLM-PC on several challenging real-world obstacle courses, involving dead ends and climbing and crawling, on a Go1 quadruped robot. Our experiments show that by reasoning over the history of interactions and future plans, VLMs enable the robot to autonomously perceive, navigate, and act in a wide range of complex scenarios that would otherwise require environment-specific engineering or human guidance., Comment: 27 pages

We introduce a new family of video prediction models designed to support downstream control tasks. We call these models Video Occupancy models (VOCs). VOCs operate in a compact latent space, thus avoiding the need to make predictions about individual pixels. Unlike prior latent-space world models, VOCs directly predict the discounted distribution of future states in a single step, thus avoiding the need for multistep roll-outs. We show that both properties are beneficial when building predictive models of video for use in downstream control. Code is available at \href{https://github.com/manantomar/video-occupancy-models}{\texttt{github.com/manantomar/video-occupancy-models}}.

Temporal distances lie at the heart of many algorithms for planning, control, and reinforcement learning that involve reaching goals, allowing one to estimate the transit time between two states. However, prior attempts to define such temporal distances in stochastic settings have been stymied by an important limitation: these prior approaches do not satisfy the triangle inequality. This is not merely a definitional concern, but translates to an inability to generalize and find shortest paths. In this paper, we build on prior work in contrastive learning and quasimetrics to show how successor features learned by contrastive learning (after a change of variables) form a temporal distance that does satisfy the triangle inequality, even in stochastic settings. Importantly, this temporal distance is computationally efficient to estimate, even in high-dimensional and stochastic settings. Experiments in controlled settings and benchmark suites demonstrate that an RL algorithm based on these new temporal distances exhibits combinatorial generalization (i.e., "stitching") and can sometimes learn more quickly than prior methods, including those based on quasimetrics., Comment: Proceedings of the 41st International Conference on Machine Learning (ICML 2024)

Diffusion models are powerful generative models that allow for precise control over the characteristics of the generated samples. While these diffusion models trained on large datasets have achieved success, there is often a need to introduce additional controls in downstream fine-tuning processes, treating these powerful models as pre-trained diffusion models. This work presents a novel method based on reinforcement learning (RL) to add additional controls, leveraging an offline dataset comprising inputs and corresponding labels. We formulate this task as an RL problem, with the classifier learned from the offline dataset and the KL divergence against pre-trained models serving as the reward functions. We introduce our method, $\textbf{CTRL}$ ($\textbf{C}$onditioning pre-$\textbf{T}$rained diffusion models with $\textbf{R}$einforcement $\textbf{L}$earning), which produces soft-optimal policies that maximize the abovementioned reward functions. We formally demonstrate that our method enables sampling from the conditional distribution conditioned on additional controls during inference. Our RL-based approach offers several advantages over existing methods. Compared to commonly used classifier-free guidance, our approach improves sample efficiency, and can greatly simplify offline dataset construction by exploiting conditional independence between the inputs and additional controls. Furthermore, unlike classifier guidance, we avoid the need to train classifiers from intermediate states to additional controls., Comment: Under review

Training corpuses for vision language models (VLMs) typically lack sufficient amounts of decision-centric data. This renders off-the-shelf VLMs sub-optimal for decision-making tasks such as in-the-wild device control through graphical user interfaces (GUIs). While training with static demonstrations has shown some promise, we show that such methods fall short for controlling real GUIs due to their failure to deal with real-world stochasticity and non-stationarity not captured in static observational data. This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents through fine-tuning a pre-trained VLM in two stages: offline RL to initialize the model, followed by offline-to-online RL. To do this, we build a scalable and parallelizable Android learning environment equipped with a VLM-based evaluator and develop a simple yet effective RL approach for learning in this domain. Our approach runs advantage-weighted RL with advantage estimators enhanced to account for stochasticity along with an automatic curriculum for deriving maximal learning signal. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our 1.3B VLM trained with RL achieves a 49.5% absolute improvement -- from 17.7 to 67.2% success rate -- over supervised fine-tuning with static human demonstration data. These results significantly surpass not only the prior best agents, including AppAgent with GPT-4V (8.3% success rate) and the 17B CogAgent trained with AitW data (38.5%), but also the prior best autonomous RL approach based on filtered behavior cloning (57.8%), thereby establishing a new state-of-the-art for digital agents for in-the-wild device control., Comment: 11 pages of main text, 28 pages in total

While imitation learning requires access to high-quality data, offline reinforcement learning (RL) should, in principle, perform similarly or better with substantially lower data quality by using a value function. However, current results indicate that offline RL often performs worse than imitation learning, and it is often unclear what holds back the performance of offline RL. Motivated by this observation, we aim to understand the bottlenecks in current offline RL algorithms. While poor performance of offline RL is typically attributed to an imperfect value function, we ask: is the main bottleneck of offline RL indeed in learning the value function, or something else? To answer this question, we perform a systematic empirical study of (1) value learning, (2) policy extraction, and (3) policy generalization in offline RL problems, analyzing how these components affect performance. We make two surprising observations. First, we find that the choice of a policy extraction algorithm significantly affects the performance and scalability of offline RL, often more so than the value learning objective. For instance, we show that common value-weighted behavioral cloning objectives (e.g., AWR) do not fully leverage the learned value function, and switching to behavior-constrained policy gradient objectives (e.g., DDPG+BC) often leads to substantial improvements in performance and scalability. Second, we find that a big barrier to improving offline RL performance is often imperfect policy generalization on test-time states out of the support of the training data, rather than policy learning on in-distribution states. We then show that the use of suboptimal but high-coverage data or test-time policy training techniques can address this generalization issue in practice. Specifically, we propose two simple test-time policy improvement methods and show that these methods lead to better performance.

In this paper we present an algorithm for computing a congruence on an inverse semigroup from a collection of generating pairs. This algorithm uses a myriad of techniques from computational group theory, automata, and the theory of inverse semigroups. An initial implementation of this algorithm outperforms existing implementations by several orders of magnitude., Comment: 16 pages, 3 figures

Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such vision-language-action (VLA) models to obtain robust, generalizable policies for visuomotor control. Yet, widespread adoption of VLAs for robotics has been challenging as 1) existing VLAs are largely closed and inaccessible to the public, and 2) prior work fails to explore methods for efficiently fine-tuning VLAs for new tasks, a key component for adoption. Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations. OpenVLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP. As a product of the added data diversity and new model components, OpenVLA demonstrates strong results for generalist manipulation, outperforming closed models such as RT-2-X (55B) by 16.5% in absolute task success rate across 29 tasks and multiple robot embodiments, with 7x fewer parameters. We further show that we can effectively fine-tune OpenVLA for new settings, with especially strong generalization results in multi-task environments involving multiple objects and strong language grounding abilities, and outperform expressive from-scratch imitation learning methods such as Diffusion Policy by 20.4%. We also explore compute efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned on consumer GPUs via modern low-rank adaptation methods and served efficiently via quantization without a hit to downstream success rate. Finally, we release model checkpoints, fine-tuning notebooks, and our PyTorch codebase with built-in support for training VLAs at scale on Open X-Embodiment datasets., Comment: Website: https://openvla.github.io/

Perovskite solar cells (PSCs) offer an efficient, inexpensive alternative to current photovoltaic technologies, with the potential for manufacture via high-throughput coating methods. However, challenges for commercial-scale solution-processing of metal-halide perovskites include the use of harmful solvents, the expense of maintaining controlled atmospheric conditions, and the inherent instabilities of PSCs under operation. Here, we address these challenges by introducing a high volatility, low toxicity, biorenewable solvent system to fabricate a range of 2D perovskites, which highly effective precursor phases for subsequent transformation to alpha-formamidinium lead triiodide (FAPbI3), fully processed under ambient conditions. PSCs utilising our FAPbI3 reproducibly show remarkable stability under illumination and elevated temperature (ISOS-L-2) and "damp heat" (ISOS-D-3) stressing, surpassing other state-of-the-art perovskite compositions. We determine that this enhancement is a consequence of the 2D precursor phase crystallisation route, which simultaneously avoids retention of residual low-volatility solvents (such as DMF and DMSO) and reduces the rate of degradation of FA+ in the material. Our findings highlight both the critical role of the initial crystallisation process in determining the operational stability of perovskite materials, and that neat FA+-based perovskites can be competitively stable despite the inherent metastability of the alpha-phase.

Transformers, and the attention mechanism in particular, have become ubiquitous in machine learning. Their success in modeling nonlocal, long-range correlations has led to their widespread adoption in natural language processing, computer vision, and time-series problems. Neural operators, which map spaces of functions into spaces of functions, are necessarily both nonlinear and nonlocal if they are universal; it is thus natural to ask whether the attention mechanism can be used in the design of neural operators. Motivated by this, we study transformers in the function space setting. We formulate attention as a map between infinite dimensional function spaces and prove that the attention mechanism as implemented in practice is a Monte Carlo or finite difference approximation of this operator. The function space formulation allows for the design of transformer neural operators, a class of architectures designed to learn mappings between function spaces, for which we prove a universal approximation result. The prohibitive cost of applying the attention operator to functions defined on multi-dimensional domains leads to the need for more efficient attention-based architectures. For this reason we also introduce a function space generalization of the patching strategy from computer vision, and introduce a class of associated neural operators. Numerical results, on an array of operator learning problems, demonstrate the promise of our approaches to function space formulations of attention and their use in neural operators.

Skill discovery methods enable agents to learn diverse emergent behaviors without explicit rewards. To make learned skills useful for unknown downstream tasks, obtaining a semantically diverse repertoire of skills is essential. While some approaches introduce a discriminator to distinguish skills and others aim to increase state coverage, no existing work directly addresses the "semantic diversity" of skills. We hypothesize that leveraging the semantic knowledge of large language models (LLMs) can lead us to improve semantic diversity of resulting behaviors. In this sense, we introduce Language Guided Skill Discovery (LGSD), a skill discovery framework that aims to directly maximize the semantic diversity between skills. LGSD takes user prompts as input and outputs a set of semantically distinctive skills. The prompts serve as a means to constrain the search space into a semantically desired subspace, and the generated LLM outputs guide the agent to visit semantically diverse states within the subspace. We demonstrate that LGSD enables legged robots to visit different user-intended areas on a plane by simply changing the prompt. Furthermore, we show that language guidance aids in discovering more diverse skills compared to five existing skill discovery methods in robot-arm manipulation environments. Lastly, LGSD provides a simple way of utilizing learned skills via natural language.

Offline reinforcement learning (RL) is a compelling paradigm to extend RL's practical utility by leveraging pre-collected, static datasets, thereby avoiding the limitations associated with collecting online interactions. The major difficulty in offline RL is mitigating the impact of approximation errors when encountering out-of-distribution (OOD) actions; doing so ineffectively will lead to policies that prefer OOD actions, which can lead to unexpected and potentially catastrophic results. Despite the variety of works proposed to address this issue, they tend to excessively suppress the value function in and around OOD regions, resulting in overly pessimistic value estimates. In this paper, we propose a novel framework called Strategically Conservative Q-Learning (SCQ) that distinguishes between OOD data that is easy and hard to estimate, ultimately resulting in less conservative value estimates. Our approach exploits the inherent strengths of neural networks to interpolate, while carefully navigating their limitations in extrapolation, to obtain pessimistic yet still property calibrated value estimates. Theoretical analysis also shows that the value function learned by SCQ is still conservative, but potentially much less so than that of Conservative Q-learning (CQL). Finally, extensive evaluation on the D4RL benchmark tasks shows our proposed method outperforms state-of-the-art methods. Our code is available through \url{https://github.com/purewater0901/SCQ}.

We report new limits on the kinetic mixing strength of the dark photon spanning the mass range 0.21 -- 1.24 $\mu$eV corresponding to a frequency span of 50 -- 300 MHz. The Dark E-Field Radio experiment is a wide-band search for dark photon dark matter. In this paper we detail changes in calibration and upgrades since our proof-of-concept pilot run. Our detector employs a wide bandwidth E-field antenna moved to multiple positions in a shielded room, a low noise amplifier, wideband ADC, followed by a $2^{24}$-point FFT. An optimal filter searches for signals with Q $\approx10^6$. In nine days of integration, this system is capable of detecting dark photon signals corresponding to $\epsilon$ several orders of magnitude lower than previous limits. We find a 95% exclusion limit on $\epsilon$ over this mass range between $6\times 10^{-15}$ and $6\times 10^{-13}$, tracking the complex resonant mode structure in the shielded room., Comment: 15 pages, 12 figures, submitted to PRD

In this paper we provide an overview of the class of inverse semigroups $S$ such that every congruence on $S$ relates at least one idempotent to a non-idempotent; such inverse semigroups are called $E$-disjunctive. This overview includes the study of the inverse semigroup theoretic structure of $E$-disjunctive semigroups; a large number of natural examples; some asymptotic results establishing the rarity of such inverse semigroups; and a general structure theorem for all inverse semigroups where the building blocks are $E$-disjunctive., Comment: 37 pages, 3 figures, 1 table

Aeronomy, the study of Earth's upper atmosphere and its interaction with the local space environment, has long traced changes in the thermospheres of Earth and other solar system planets to solar variability in the X-ray and extreme ultraviolet (collectively, "XUV") bands. Extending comparative aeronomy to the short-period extrasolar planets may illuminate whether stellar XUV irradiation powers atmospheric outflows that change planetary radii on astronomical timescales. In recent years, near-infrared transit spectroscopy of metastable HeI has been a prolific tracer of high-altitude planetary gas. We present a case study of exoplanet aeronomy using metastable HeI transit observations from Palomar/WIRC and follow-up high-energy data from the Neil Gehrels Swift Observatory that were taken within one month of the WASP-69 system, a K-type main sequence star with a well-studied hot Jupiter companion. Supplemented by archival data, we find that WASP-69's X-ray flux in 2023 was less than 50% of what was recorded in 2016 and that the metastable HeI absorption from WASP-69b was lower in 2023 versus past epochs from 2017-2019. Via atmospheric modeling, we show that this time-variable metastable HeI signal is in the expected direction given the observed change in stellar XUV, possibly stemming from WASP-69's magnetic activity cycle. Our results underscore the ability of multi-epoch, multi-wavelength observations to paint a cohesive picture of the interaction between an exoplanet's atmosphere and its host star., Comment: 25 pages, 15 figures, 6 tables, accepted to The Astronomical Journal

AI-driven design problems, such as DNA/protein sequence design, are commonly tackled from two angles: generative modeling, which efficiently captures the feasible design space (e.g., natural images or biological sequences), and model-based optimization, which utilizes reward models for extrapolation. To combine the strengths of both approaches, we adopt a hybrid method that fine-tunes cutting-edge diffusion models by optimizing reward models through RL. Although prior work has explored similar avenues, they primarily focus on scenarios where accurate reward models are accessible. In contrast, we concentrate on an offline setting where a reward model is unknown, and we must learn from static offline datasets, a common scenario in scientific domains. In offline scenarios, existing approaches tend to suffer from overoptimization, as they may be misled by the reward model in out-of-distribution regions. To address this, we introduce a conservative fine-tuning approach, BRAID, by optimizing a conservative reward model, which includes additional penalization outside of offline data distributions. Through empirical and theoretical analysis, we demonstrate the capability of our approach to outperform the best designs in offline data, leveraging the extrapolation capabilities of reward models while avoiding the generation of invalid designs through pre-trained diffusion models., Comment: Under review

The Scintillating Bubble Chamber (SBC) collaboration purchased 32 Hamamatsu VUV4 silicon photomultipliers (SiPMs) for use in SBC-LAr10, a bubble chamber containing 10~kg of liquid argon. A dark-count characterization technique, which avoids the use of a single-photon source, was used at two temperatures to measure the VUV4 SiPMs breakdown voltage ($V_{\text{BD}}$), the SiPM gain ($g_{\text{SiPM}}$), the rate of change of $g_{\text{SiPM}}$ with respect to voltage ($m$), the dark count rate (DCR), and the probability of a correlated avalanche (P$_{\text{CA}}$) as well as the temperature coefficients of these parameters. A Peltier-based chilled vacuum chamber was developed at Queen's University to cool down the Quads to $233.15\pm0.2$~K and $255.15\pm0.2$~K with average stability of $\pm20$~mK. An analysis framework was developed to estimate $V_{\text{BD}}$ to tens of mV precision and DCR close to Poissonian error. The temperature dependence of $V_{\text{BD}}$ was found to be $56\pm2$~mV~K$^{-1}$, and $m$ on average across all Quads was found to be $(459\pm3(\rm{stat.})\pm23(\rm{sys.}))\times 10^{3}~e^-$~PE$^{-1}$~V$^{-1}$. The average DCR temperature coefficient was estimated to be $0.099\pm0.008$~K$^{-1}$ corresponding to a reduction factor of 7 for every 20~K drop in temperature. The average temperature dependence of P$_{\text{CA}}$ was estimated to be $4000\pm1000$~ppm~K$^{-1}$. P$_{\text{CA}}$ estimated from the average across all SiPMs is a better estimator than the P$_{\text{CA}}$ calculated from individual SiPMs, for all of the other parameters, the opposite is true. All the estimated parameters were measured to the precision required for SBC-LAr10, and the Quads will be used in conditions to optimize the signal-to-noise ratio., Comment: 25 pages, 19 figures

A major challenge facing the world is the provision of equitable and universal access to quality education. Recent advances in generative AI (gen AI) have created excitement about the potential of new technologies to offer a personal tutor for every learner and a teaching assistant for every teacher. The full extent of this dream, however, has not yet materialised. We argue that this is primarily due to the difficulties with verbalising pedagogical intuitions into gen AI prompts and the lack of good evaluation practices, reinforced by the challenges in defining excellent pedagogy. Here we present our work collaborating with learners and educators to translate high level principles from learning science into a pragmatic set of seven diverse educational benchmarks, spanning quantitative, qualitative, automatic and human evaluations; and to develop a new set of fine-tuning datasets to improve the pedagogical capabilities of Gemini, introducing LearnLM-Tutor. Our evaluations show that LearnLM-Tutor is consistently preferred over a prompt tuned Gemini by educators and learners on a number of pedagogical dimensions. We hope that this work can serve as a first step towards developing a comprehensive educational evaluation framework, and that this can enable rapid progress within the AI and EdTech communities towards maximising the positive impact of gen AI in education.

Large policies pretrained on diverse robot datasets have the potential to transform robotic learning: instead of training new policies from scratch, such generalist robot policies may be finetuned with only a little in-domain data, yet generalize broadly. However, to be widely applicable across a range of robotic learning scenarios, environments, and tasks, such policies need to handle diverse sensors and action spaces, accommodate a variety of commonly used robotic platforms, and finetune readily and efficiently to new domains. In this work, we aim to lay the groundwork for developing open-source, widely applicable, generalist policies for robotic manipulation. As a first step, we introduce Octo, a large transformer-based policy trained on 800k trajectories from the Open X-Embodiment dataset, the largest robot manipulation dataset to date. It can be instructed via language commands or goal images and can be effectively finetuned to robot setups with new sensory inputs and action spaces within a few hours on standard consumer GPUs. In experiments across 9 robotic platforms, we demonstrate that Octo serves as a versatile policy initialization that can be effectively finetuned to new observation and action spaces. We also perform detailed ablations of design decisions for the Octo model, from architecture to training data, to guide future research on building generalist robot models., Comment: Project website: https://octo-models.github.io

Large vision-language models (VLMs) fine-tuned on specialized visual instruction-following data have exhibited impressive language reasoning capabilities across various scenarios. However, this fine-tuning paradigm may not be able to efficiently learn optimal decision-making agents in multi-step goal-directed tasks from interactive environments. To address this challenge, we propose an algorithmic framework that fine-tunes VLMs with reinforcement learning (RL). Specifically, our framework provides a task description and then prompts the VLM to generate chain-of-thought (CoT) reasoning, enabling the VLM to efficiently explore intermediate reasoning steps that lead to the final text-based action. Next, the open-ended text output is parsed into an executable action to interact with the environment to obtain goal-directed task rewards. Finally, our framework uses these task rewards to fine-tune the entire VLM with RL. Empirically, we demonstrate that our proposed framework enhances the decision-making capabilities of VLM agents across various tasks, enabling 7b models to outperform commercial models such as GPT4-V or Gemini. Furthermore, we find that CoT reasoning is a crucial component for performance improvement, as removing the CoT reasoning results in a significant decrease in the overall performance of our method.

The field of robotics has made significant advances towards generalist robot manipulation policies. However, real-world evaluation of such policies is not scalable and faces reproducibility challenges, which are likely to worsen as policies broaden the spectrum of tasks they can perform. We identify control and visual disparities between real and simulated environments as key challenges for reliable simulated evaluation and propose approaches for mitigating these gaps without needing to craft full-fidelity digital twins of real-world environments. We then employ these approaches to create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot setups. Through paired sim-and-real evaluations of manipulation policies, we demonstrate strong correlation between policy performance in SIMPLER environments and in the real world. Additionally, we find that SIMPLER evaluations accurately reflect real-world policy behavior modes such as sensitivity to various distribution shifts. We open-source all SIMPLER environments along with our workflow for creating new environments at https://simpler-env.github.io to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks.

Reinforcement learning provides an appealing framework for robotic control due to its ability to learn expressive policies purely through real-world interaction. However, this requires addressing real-world constraints and avoiding catastrophic failures during training, which might severely impede both learning progress and the performance of the final policy. In many robotics settings, this amounts to avoiding certain "unsafe" states. The high-speed off-road driving task represents a particularly challenging instantiation of this problem: a high-return policy should drive as aggressively and as quickly as possible, which often requires getting close to the edge of the set of "safe" states, and therefore places a particular burden on the method to avoid frequent failures. To both learn highly performant policies and avoid excessive failures, we propose a reinforcement learning framework that combines risk-sensitive control with an adaptive action space curriculum. Furthermore, we show that our risk-sensitive objective automatically avoids out-of-distribution states when equipped with an estimator for epistemic uncertainty. We implement our algorithm on a small-scale rally car and show that it is capable of learning high-speed policies for a real-world off-road driving task. We show that our method greatly reduces the number of safety violations during the training process, and actually leads to higher-performance policies in both driving and non-driving simulation environments with similar challenges., Comment: In review, RSS 2024

We present the largest optical photometry compilation of Gamma-Ray Bursts (GRBs) with redshifts ($z$). We include 64813 observations of 535 events (including upper limits) from 28 February 1997 up to 18 August 2023. We also present a user-friendly web tool \textit{grbLC} which allows users the visualization of photometry, coordinates, redshift, host galaxy extinction, and spectral indices for each event in our database. Furthermore, we have added a Gamma Ray Coordinate Network (GCN) scraper that can be used to collect data by gathering magnitudes from the GCNs. The web tool also includes a package for uniformly investigating colour evolution. We compute the optical spectral indices for 138 GRBs for which we have at least 4 filters at the same epoch in our sample and craft a procedure to distinguish between GRBs with and without colour evolution. By providing a uniform format and repository for the optical catalogue, this web-based archive is the first step towards unifying several community efforts to gather the photometric information for all GRBs with known redshifts. This catalogue will enable population studies by providing light curves (LCs) with better coverage since we have gathered data from different ground-based locations. Consequently, these LCs can be used to train future LC reconstructions for an extended inference of the redshift. The data gathering also allows us to fill some of the orbital gaps from Swift in crucial points of the LCs, e.g., at the end of the plateau emission or where a jet break is identified., Comment: 20 pages, 16 figures, 2 tables. Submitted to MNRAS, this version matches the third revision. The Online Materials and data will be available after the publication