Search

We introduce proximity morphisms between MT-algebras and show that the resulting category is equivalent to the category of frames. This is done by utilizing the Funayama envelope of a frame, which is viewed as the $T_D$-hull. Our results have some spatial ramifications, including a generalization of the $T_D$-duality of Banaschewski and Pultr.

Meshes are used to represent complex objects in high fidelity physics simulators across a variety of domains, such as radar sensing and aerodynamics. There is growing interest in using neural networks to accelerate physics simulations, and also a growing body of work on applying neural networks directly to irregular mesh data. Since multiple mesh topologies can represent the same object, mesh augmentation is typically required to handle topological variation when training neural networks. Due to the sensitivity of physics simulators to small changes in mesh shape, it is challenging to use these augmentations when training neural network-based physics simulators. In this work, we show that variations in mesh topology can significantly reduce the performance of neural network simulators. We evaluate whether pretraining can be used to address this issue, and find that employing an established autoencoder pretraining technique with graph embedding models reduces the sensitivity of neural network simulators to variations in mesh topology. Finally, we highlight future research directions that may further reduce neural simulator sensitivity to mesh topology., Comment: 5 pages, 3 figures

Group theory has been used in machine learning to provide a theoretically grounded approach for incorporating known symmetry transformations in tasks from robotics to protein modeling. In these applications, equivariant neural networks use known symmetry groups with predefined representations to learn over geometric input data. We propose MatrixNet, a neural network architecture that learns matrix representations of group element inputs instead of using predefined representations. MatrixNet achieves higher sample efficiency and generalization over several standard baselines in prediction tasks over the several finite groups and the Artin braid group. We also show that MatrixNet respects group relations allowing generalization to group elements of greater word length than in the training set., Comment: NeurIPS 2024

The Simons Observatory (SO) is a ground-based cosmic microwave background (CMB) survey experiment that currently consists of three 0.42m small-aperture telescopes (SATs) and one 6m large-aperture telescope (LAT), located at an elevation of 5200m in the Atacama Desert in Chile. At the LAT's focal plane, SO will install >62,000 transition-edge sensor detectors across 13 optics tubes (OTs) within the Large Aperture Telescope Receiver (LATR), the largest cryogenic camera ever built to observe the CMB. Here we report on the validation of the LATR in the laboratory and the subsequent dark testing and validation within the LAT. We show that the LATR meets cryogenic, optical, and detector specifications required for high-sensitivity measurements of the CMB. At the time of writing, the LATR is installed in the LAT with six OTs (corresponding to >31,000 detectors), and the LAT mirrors and remaining seven OTs are undergoing development.

Electrochemistry workflows utilize various instruments and computing systems to execute workflows consisting of electrocatalyst synthesis, testing and evaluation tasks. The heterogeneity of the software and hardware of these ecosystems makes it challenging to orchestrate a complete workflow from production to characterization by automating its tasks. We propose an autonomous electrochemistry computing platform for a multi-site ecosystem that provides the services for remote experiment steering, real-time measurement transfer, and AI/ML-driven analytics. We describe the integration of a mobile robot and synthesis workstation into the ecosystem by developing custom hub-networks and software modules to support remote operations over the ecosystem's wireless and wired networks. We describe a workflow task for generating I-V voltammetry measurements using a potentiostat, and a machine learning framework to ensure their normality by detecting abnormal conditions such as disconnected electrodes. We study a number of machine learning methods for the underlying detection problem, including smooth, non-smooth, structural and statistical methods, and their fusers. We present experimental results to illustrate the effectiveness of this platform, and also validate the proposed ML method by deriving its rigorous generalization equations., Comment: 10 pages, 14 figures, accepted in the IEEE 20th International Conference on e-Science (e-Science), 2024

AI Agent, powered by large language models (LLMs) as its cognitive core, is an intelligent agentic system capable of autonomously controlling and determining the execution paths under user's instructions. With the burst of capabilities of LLMs and various plugins, such as RAG, text-to-image/video/3D, etc., the potential of AI Agents has been vastly expanded, with their capabilities growing stronger by the day. However, at the intersection between AI and web3, there is currently no ideal agentic framework that can seamlessly integrate web3 applications into AI agent functionalities. In this paper, we propose Eliza, the first open-source web3-friendly Agentic framework that makes the deployment of web3 applications effortless. We emphasize that every aspect of Eliza is a regular Typescript program under the full control of its user, and it seamlessly integrates with web3 (i.e., reading and writing blockchain data, interacting with smart contracts, etc.). Furthermore, we show how stable performance is achieved through the pragmatic implementation of the key components of Eliza's runtime. Our code is publicly available at https://github.com/ai16z/eliza., Comment: 20 pages, 5 figures

Advancement of solid state electrolytes (SSEs) for all solid state batteries typically focuses on modification of a parent structural framework for improved conductivity, \textit{e.g.} cation substitution for an immobile ion or varying the concentration of the mobile ion. Therefore, novel frameworks can be disruptive by enabling fast ion conduction aided by different structure and diffusion mechanisms, and unlocking optimal conductors with different properties (\textit{e.g.} mechanical properties, sintering needs, electrochemical stability) than previously published. Herein, we perform a high throughput survey of an understudied structural framework for sodium ion conduction, Na$_{8-x}$A$^x$P$_2$O$_9$ (NAP), to understand the family's thermodynamic stability, synthesizability, and ionic conduction. We first show that the parent phase Na$_4$TiP$_2$O$_9$ (NTP) undergoes a structural distortion (with accompanying conductivity transition) due to unstable phonons from a pseduo-Jahn Teller mode in the 1D titanium chains. Then, screening of cation-substituted structural candidates with \textit{ab initio} and machine-learned potential calculations reveal a number of candidates that are thermodynamically stable, likely synthesizable, and have high predicted ionic conductivities. High throughput experimental trials and subsequent methodology optimization of one Na$_4$SnP$_2$O$_9$ (NSP) highlight collective challenges to the synthesis pathways for sodium phosphate materials via solid state synthesis. Our results demonstrate that NAP is a highly tunable conduction framework whose high temperature conductivity transition has heretofore eliminated it from significant research interest. By expanding the structural toolkit for SSE design, we increase the number of useful sodium ion electrolytes for integration into safe and accessible solid state batteries., Comment: 15 page manuscript, 42 total pages

Removing unwanted concepts from large-scale text-to-image (T2I) diffusion models while maintaining their overall generative quality remains an open challenge. This difficulty is especially pronounced in emerging paradigms, such as Stable Diffusion (SD) v3 and Flux, which incorporate flow matching and transformer-based architectures. These advancements limit the transferability of existing concept-erasure techniques that were originally designed for the previous T2I paradigm (e.g., SD v1.4). In this work, we introduce EraseAnything, the first method specifically developed to address concept erasure within the latest flow-based T2I framework. We formulate concept erasure as a bi-level optimization problem, employing LoRA-based parameter tuning and an attention map regularizer to selectively suppress undesirable activations. Furthermore, we propose a self-contrastive learning strategy to ensure that removing unwanted concepts does not inadvertently harm performance on unrelated ones. Experimental results demonstrate that EraseAnything successfully fills the research gap left by earlier methods in this new T2I paradigm, achieving state-of-the-art performance across a wide range of concept erasure tasks., Comment: 24 pages, 18 figures

In fishery science, harvest management of size-structured stochastic populations is a long-standing and difficult problem. Rectilinear precautionary policies based on biomass and harvesting reference points have now become a standard approach to this problem. While these standard feedback policies are adapted from analytical or dynamic programming solutions assuming relatively simple ecological dynamics, they are often applied to more complicated ecological settings in the real world. In this paper we explore the problem of designing harvest control rules for partially observed, age-structured, spasmodic fish populations using tools from reinforcement learning (RL) and Bayesian optimization. Our focus is on the case of Walleye fisheries in Alberta, Canada, whose highly variable recruitment dynamics have perplexed managers and ecologists. We optimized and evaluated policies using several complementary performance metrics. The main questions we addressed were: 1. How do standard policies based on reference points perform relative to numerically optimized policies? 2. Can an observation of mean fish weight, in addition to stock biomass, aid policy decisions?, Comment: 25 pages, 4 figures, 2 tables

Reinforcement learning (RL) algorithms for continuous control tasks require accurate sampling-based action selection. Many tasks, such as robotic manipulation, contain inherent problem symmetries. However, correctly incorporating symmetry into sampling-based approaches remains a challenge. This work addresses the challenge of preserving symmetry in sampling-based planning and control, a key component for enhancing decision-making efficiency in RL. We introduce an action sampling approach that enforces the desired symmetry. We apply our proposed method to a coordinate regression problem and show that the symmetry aware sampling method drastically outperforms the naive sampling approach. We furthermore develop a general framework for sampling-based model-based planning with Model Predictive Path Integral (MPPI). We compare our MPPI approach with standard sampling methods on several continuous control tasks. Empirical demonstrations across multiple continuous control environments validate the effectiveness of our approach, showcasing the importance of symmetry preservation in sampling-based action selection., Comment: Published at International Workshop on the Algorithmic Foundations of Robotics (WAFR) 2024. Website: http://lfzhao.com/EquivSampling

We present two developments which enhance the predictive power of empirical shell model Hamiltonians for cases in which calibration data is sparse. A recent improvement in the ab initio derivation of effective Hamiltonians leads to a much better starting point for the optimization procedure. In addition, we introduce a protocol to avoid over-fitting, enabling a more reliable extrapolation beyond available data. These developments will enable more robust predictions for exotic isotopes produced at rare isotope beam facilities and in astrophysical environments., Comment: 11 pages, 6 figures

The Hydrogen Intensity and Real-time Analysis eXperiment (HIRAX) is an upcoming radio interferometric telescope designed to constrain dark energy through the 21cm intensity mapping of Baryon Acoustic Oscillations (BAO). Instrumental systematics must be controlled and carefully characterized to measure the 21cm power spectrum with fidelity and achieve high-precision constraints on the cosmological parameters. The chromaticity of the primary beam is one such complicated systematic, which can leak the power of spectrally smooth foregrounds beyond the ideal horizon limits due to the complex spatial and spectral structures of the sidelobes and the mainlobe. This paper studies the chromaticity of the HIRAX Stokes I primary beam and its effects on accurate measurements of the 21cm power spectrum. To investigate the effect of chromaticity in the 21cm power spectrum, we present a physically motivated beam modeling technique, which uses a flexible basis derived from traditional optics that can account for higher-order radial and azimuthal structures in the primary beam. We investigate the impact of imperfect knowledge of the mainlobe and sidelobes chromaticity in the power spectrum space by subtracting a simple foreground model in simulated snapshot visibilities to recover the H$\textsc{i}$ power spectrum. Additionally, we find that modeling up to the octupolar azimuthal order feature (fourth-order angular variation) in the primary beam is sufficient to reduce the leakage outside the wedge with minimal bias., Comment: 20 pages, 12 figures. Prepared for submitting in the Astrophysical Journal (ApJ)

The diffusion learning method, DiffDock, for docking small-molecule ligands into protein binding sites was recently introduced. Results included comparisons to more conventional docking approaches, with DiffDock showing superior performance. Here, we employ a fully automatic workflow using the Surflex-Dock methods to generate a fair baseline for conventional docking approaches. Results were generated for the common and expected situation where a binding site location is known and also for the condition of an unknown binding site. For the known binding site condition, Surflex-Dock success rates at 2.0 Angstroms RMSD far exceeded those for DiffDock (Top-1/Top-5 success rates, respectively, were 68/81% compared with 45/51%). Glide performed with similar success rates (67/73%) to Surflex-Dock for the known binding site condition, and results for AutoDock Vina and Gnina followed this pattern. For the unknown binding site condition, using an automated method to identify multiple binding pockets, Surflex-Dock success rates again exceeded those of DiffDock, but by a somewhat lesser margin. DiffDock made use of roughly 17,000 co-crystal structures for learning (98% of PDBBind version 2020, pre-2019 structures) for a training set in order to predict on 363 test cases (2% of PDBBind 2020) from 2019 forward. DiffDock's performance was inextricably linked with the presence of near-neighbor cases of close to identical protein-ligand complexes in the training set for over half of the test set cases. DiffDock exhibited a 40 percentage point difference on near-neighbor cases (two-thirds of all test cases) compared with cases with no near-neighbor training case. DiffDock has apparently encoded a type of table-lookup during its learning process, rendering meaningful applications beyond its reach. Further, it does not perform even close to competitively with a competently run modern docking workflow., Comment: Post-Conclusion addendum added with additional reference and context, 19 pages including references and appendices, 7 figures

In many computer vision domains the collection of sufficient real-world data is challenging and can severely impact model performance, particularly when running inference on samples that are unseen or underrepresented in training. Synthetically generated images provide a promising solution, but it remains unclear how to design synthetic data to optimally improve model performance, for example whether to introduce more realism or more abstraction in such datasets. Here we propose a novel conceptual approach to improve the performance of computer vision models trained on synthetic images, by using robust Explainable AI (XAI) techniques to guide the modification of 3D models used to generate these images. Importantly, this framework allows both modifications that increase and decrease realism in synthetic data, which can both improve model performance. We illustrate this concept using a real-world example where data are sparse; the detection of vehicles in infrared imagery. We fine-tune an initial YOLOv8 model on the ATR DSIAC infrared dataset and synthetic images generated from 3D mesh models in the Unity gaming engine, and then use XAI saliency maps to guide modification of our Unity models. We show that synthetic data can improve detection of vehicles in orientations unseen in training by 4.6\% (to mAP50 scores of 94.6\%). We further improve performance by an additional 1.5\% (to 96.1\%) through our new XAI-guided approach, which reduces misclassifications through both increasing and decreasing the realism of different parts of the synthetic data. These proof-of-concept results pave the way for fine, XAI-controlled curation of synthetic datasets through detailed feature modifications, tailored to improve object detection performance.

In this work, we developed a rigorous procedure for mapping the exact non-Markovian propagator to the generalized Lindblad form. It allows us to extract the negative decay rate that is the indicator of the non-Markovian effect. As a consequence, we can investigate the influence of the non-Markovian bath on the system's properties such as coherence and equilibrium state distribution. The understanding of the non-Markovian contribution to the dynamical process points to the possibility of leveraging non-Markovianity for quantum control.

The simulation of non-Markovian quantum dynamics plays an important role in the understanding of charge and exciton dynamics in the condensed phase environment, and yet it remains computationally expensive on classical computers. We have developed a variational quantum algorithm that is capable of simulating non-Markovian quantum dynamics. The algorithm captures the non-Markovian effect by employing the Ehrenfect trajectories in the path integral formulation and the Monte Carlo sampling of the thermal distribution. We tested the algorithm with the spin-boson model on the quantum simulator and the results match well with the exact ones. The algorithm naturally fits into the parallel computing platform of the NISQ devices and is well suited for anharmonic system-bath interactions and multi-state systems.

The QCD axion, originally proposed to solve the strong CP problem in QCD, is a prominent candidate for dark matter (DM). In the presence of strong magnetic fields, such as those around neutron stars, axions can theoretically convert into photons, producing detectable electromagnetic signals. This axion-photon coupling provides a unique experimental pathway to probe axions within a specific mass range. We investigate a novel observational approach using the Green Bank Telescope (GBT) to search for radio transients that could arise from interactions between neutron stars and dense DM clumps known as axion miniclusters. By observing the core of Andromeda with the VErsatile GBT Astronomical Spectrometer (VEGAS) and the X-band receiver (8 to 10 GHz), we achieve sensitivity to axions with masses in the range of (33 - 42)$\,\mu$eV, with a mass resolution of $3.8 \times 10^{-4}\,\mu$eV. We detail our observational and analytical strategies developed to capture transient signals from axion-photon conversion, achieving an instrumental sensitivity of $2\,$mJy per spectral channel. Despite our sensitivity threshold, no candidate signals exceeding the 5$\sigma$ level were identified. Future implementations will extend this search across additional spectral bands and refine the modeling used for the processes involved, strengthening the constraints on axion DM models., Comment: 9 pages, 1 figure. Contribution to: 2nd General Meeting of the COST Action: Cosmic WISPers (CA21106), 3-6 September 2024, Istanbul (Turkey). Based on arXiv:2407.13060, arXiv:2011.05378, arXiv:2011.05377

In this work, we developed an efficient quantum algorithm for the simulation of non-Markovian quantum dynamics, based on the Feynman path integral formulation. The algorithm scales polynomially with the number of native gates and the number of qubits, and has no classical overhead. It demonstrates the quantum advantage by overcoming the exponential cost on classical computers. In addition, the algorithm is efficient regardless of whether the temporal entanglement due to non-Markovianity is low or high, making it a unified framework for simulating non-Markovian dynamics in open quantum system.

The integration of machine learning in magnetic resonance imaging (MRI), specifically in neuroimaging, is proving to be incredibly effective, leading to better diagnostic accuracy, accelerated image analysis, and data-driven insights, which can potentially transform patient care. Deep learning models utilize multiple layers of processing to capture intricate details of complex data, which can then be used on a variety of tasks, including brain tumor classification, segmentation, image synthesis, and registration. Previous research demonstrates high accuracy in tumor segmentation using various model architectures, including nn-UNet and Swin-UNet. U-Mamba, which uses state space modeling, also achieves high accuracy in medical image segmentation. To leverage these models, we propose a deep learning framework that ensembles these state-of-the-art architectures to achieve accurate segmentation and produce finely synthesized images.

Equivariant neural networks have shown great success in reinforcement learning, improving sample efficiency and generalization when there is symmetry in the task. However, in many problems, only approximate symmetry is present, which makes imposing exact symmetry inappropriate. Recently, approximately equivariant networks have been proposed for supervised classification and modeling physical systems. In this work, we develop approximately equivariant algorithms in reinforcement learning (RL). We define approximately equivariant MDPs and theoretically characterize the effect of approximate equivariance on the optimal Q function. We propose novel RL architectures using relaxed group convolutions and experiment on several continuous control domains and stock trading with real financial data. Our results demonstrate that approximate equivariance matches prior work when exact symmetries are present, and outperforms them when domains exhibit approximate symmetry. As an added byproduct of these techniques, we observe increased robustness to noise at test time., Comment: Preprint

GPT-4o is an autoregressive omni model that accepts as input any combination of text, audio, image, and video, and generates any combination of text, audio, and image outputs. It's trained end-to-end across text, vision, and audio, meaning all inputs and outputs are processed by the same neural network. GPT-4o can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time in conversation. It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages, while also being much faster and 50\% cheaper in the API. GPT-4o is especially better at vision and audio understanding compared to existing models. In line with our commitment to building AI safely and consistent with our voluntary commitments to the White House, we are sharing the GPT-4o System Card, which includes our Preparedness Framework evaluations. In this System Card, we provide a detailed look at GPT-4o's capabilities, limitations, and safety evaluations across multiple categories, focusing on speech-to-speech while also evaluating text and image capabilities, and measures we've implemented to ensure the model is safe and aligned. We also include third-party assessments on dangerous capabilities, as well as discussion of potential societal impacts of GPT-4o's text and vision capabilities.

With the aim of studying subspaces in pointfree bitopology, we characterize extremal epimorphism in biframes and show that a smallest dense one always exists, providing an analogue of Isbell's Density Theorem. Further we study the functoriality of assigning to each biframe its lattice of subbilocales and its smallest dense subbilocale.

Many manipulation tasks require the robot to rearrange objects relative to one another. Such tasks can be described as a sequence of relative poses between parts of a set of rigid bodies. In this work, we propose MATCH POLICY, a simple but novel pipeline for solving high-precision pick and place tasks. Instead of predicting actions directly, our method registers the pick and place targets to the stored demonstrations. This transfers action inference into a point cloud registration task and enables us to realize nontrivial manipulation policies without any training. MATCH POLICY is designed to solve high-precision tasks with a key-frame setting. By leveraging the geometric interaction and the symmetries of the task, it achieves extremely high sample efficiency and generalizability to unseen configurations. We demonstrate its state-of-the-art performance across various tasks on RLBench benchmark compared with several strong baselines and test it on a real robot with six tasks., Comment: project url: https://haojhuang.github.io/match_page/

We utilize the Bruns-Lakser completion to introduce Bruns-Lakser towers of a meet-semilattice. This machinery enables us to develop various hierarchies inside the class of bounded distributive lattices, which measure $\kappa$-degrees of distributivity of bounded distributive lattices and their Dedekind-MacNeille completions. We also use Priestley duality to obtain a dual characterization of the resulting hierarchies. Among other things, this yields a natural generalization of Esakia's representation of Heyting lattices to proHeyting lattices.

We investigate a random geometric graph model introduced by Bonato and Janssen. The vertices are the points of a countable dense set $S$ in a (necessarily separable) normed vector space $X$, and each pair of points are joined independently with some fixed probability $p$ (with $0

Instabilities driven by pressure anisotropy play a critical role in modulating the energy transfer in space and astrophysical plasmas. For the first time, we simulate the evolution and saturation of the parallel proton firehose instability using a multi-fluid model without adding artificial viscosity. These simulations are performed using a 10-moment, multi-fluid model with local and gradient relaxation heat-flux closures in high-$\beta$ proton-electron plasmas. When these higher-order moments are included and pressure anisotropy is permitted to develop in all species, we find that the electrons have a significant impact on the saturation of the parallel proton firehose instability, modulating the proton pressure anisotropy as the instability saturates. Even for lower $\beta$s more relevant to heliospheric plasmas, we observe a pronounced electron energization in simulations using the gradient relaxation closure. Our results indicate that resolving the electron pressure anisotropy is important to correctly describe the behavior of multi-species plasma systems., Comment: 10 pages, 7 figures, under review in ApJ

3D Euclidean symmetry equivariant neural networks have demonstrated notable success in modeling complex physical systems. We introduce a framework for relaxed $E(3)$ graph equivariant neural networks that can learn and represent symmetry breaking within continuous groups. Building on the existing e3nn framework, we propose the use of relaxed weights to allow for controlled symmetry breaking. We show empirically that these relaxed weights learn the correct amount of symmetry breaking., Comment: Extended abstract presented at the Geometry-grounded Representation Learning and Generative Modeling Workshop (GRaM) at the 41st International Conference on Machine Learning, July 2024, Vienna, Austria

Estimating many-body effects that deviate from an independent particle approach, has long been a key research interest in condensed matter physics. Layered cuprates are prototypical systems, where electron-electron interactions are found to strongly affect the dynamics of single-particle excitations. It is however, still unclear how the electron correlations influence charge excitations, such as plasmons, which have been variously treated with either weak or strong correlation models. In this work, we demonstrate the hybridised nature of collective valence charge fluctuations leading to dispersing acoustic-like plasmons in hole-doped La$_{1.84}$Sr$_{0.16}$CuO$_{4}$ and electron-doped La$_{1.84}$Ce$_{0.16}$CuO$_{4}$ using the two-particle probe, resonant inelastic x-ray scattering. We then describe the plasmon dispersions in both systems, within both the weak mean-field Random Phase Approximation (RPA) and strong coupling $t$-$J$-$V$ models. The $t$-$J$-$V$ model, which includes the correlation effects implicitly, accurately describes the plasmon dispersions as resonant excitations outside the single-particle intra-band continuum. In comparison, a quantitative description of the plasmon dispersion in the RPA approach is obtained only upon explicit consideration of re-normalized electronic band parameters. Our comparative analysis shows that electron correlations significantly impact the low-energy plasmon excitations across the cuprate doping phase diagram, even at long wavelengths. Thus, complementary information on the evolution of electron correlations, influenced by the rich electronic phases in condensed matter systems, can be extracted through the study of two-particle charge response., Comment: 6 Figures

The QCD axion and axion-like particles are compelling candidates for galactic dark matter. Theoretically, axions can convert into photons in the presence of a strong external magnetic field, which means it is possible to search for them experimentally. One approach is to use radio telescopes with high-resolution spectrometers to look for axion-photon conversion in the magnetospheres of neutron stars. In this paper, we describe the results obtained using a novel approach where we used the Green Bank Telescope (GBT) to search for radio transients produced by collisions between neutron stars and dark matter clumps known as axion miniclusters. We used the VErsatile GBT Astronomical Spectrometer (VEGAS) and the X-band receiver (8 to 10 GHz) to observe the core of Andromeda. Our measurements are sensitive to axions with masses between 33 and 42 $\mu$eV with $\Delta$$m_a$ = 3.8$\times10^{-4}$ $\mu$eV. This paper gives a description of the search method we developed, including observation and analysis strategies. Given our analysis algorithm choices and the instrument sensitivity ($\sim$2 mJy in each spectral channel), we did not find any candidate signals greater than 5$\sigma$. We are currently implementing this search method in other spectral bands., Comment: 19 pages, 12 figures, accepted to PRD

While grasp detection is an important part of any robotic manipulation pipeline, reliable and accurate grasp detection in $SE(3)$ remains a research challenge. Many robotics applications in unstructured environments such as the home or warehouse would benefit a lot from better grasp performance. This paper proposes a novel framework for detecting $SE(3)$ grasp poses based on point cloud input. Our main contribution is to propose an $SE(3)$-equivariant model that maps each point in the cloud to a continuous grasp quality function over the 2-sphere $S^2$ using spherical harmonic basis functions. Compared with reasoning about a finite set of samples, this formulation improves the accuracy and efficiency of our model when a large number of samples would otherwise be needed. In order to accomplish this, we propose a novel variation on EquiFormerV2 that leverages a UNet-style encoder-decoder architecture to enlarge the number of points the model can handle. Our resulting method, which we name $\textit{OrbitGrasp}$, significantly outperforms baselines in both simulation and physical experiments., Comment: Conference on Robot Learning 2024

Recent work has shown diffusion models are an effective approach to learning the multimodal distributions arising from demonstration data in behavior cloning. However, a drawback of this approach is the need to learn a denoising function, which is significantly more complex than learning an explicit policy. In this work, we propose Equivariant Diffusion Policy, a novel diffusion policy learning method that leverages domain symmetries to obtain better sample efficiency and generalization in the denoising function. We theoretically analyze the $\mathrm{SO}(2)$ symmetry of full 6-DoF control and characterize when a diffusion model is $\mathrm{SO}(2)$-equivariant. We furthermore evaluate the method empirically on a set of 12 simulation tasks in MimicGen, and show that it obtains a success rate that is, on average, 21.9% higher than the baseline Diffusion Policy. We also evaluate the method on a real-world system to show that effective policies can be learned with relatively few training samples, whereas the baseline Diffusion Policy cannot., Comment: Conference on Robot Learning 2024, Oral Presentation

Controlling robots through natural language instructions in open-vocabulary scenarios is pivotal for enhancing human-robot collaboration and complex robot behavior synthesis. However, achieving this capability poses significant challenges due to the need for a system that can generalize from limited data to a wide range of tasks and environments. Existing methods rely on large, costly datasets and struggle with generalization. This paper introduces Grounded Equivariant Manipulation (GEM), a novel approach that leverages the generative capabilities of pre-trained vision-language models and geometric symmetries to facilitate few-shot and zero-shot learning for open-vocabulary robot manipulation tasks. Our experiments demonstrate GEM's high sample efficiency and superior generalization across diverse pick-and-place tasks in both simulation and real-world experiments, showcasing its ability to adapt to novel instructions and unseen objects with minimal data requirements. GEM advances a significant step forward in the domain of language-conditioned robot control, bridging the gap between semantic understanding and action generation in robotic systems.

Humans can imagine goal states during planning and perform actions to match those goals. In this work, we propose Imagination Policy, a novel multi-task key-frame policy network for solving high-precision pick and place tasks. Instead of learning actions directly, Imagination Policy generates point clouds to imagine desired states which are then translated to actions using rigid action estimation. This transforms action inference into a local generative task. We leverage pick and place symmetries underlying the tasks in the generation process and achieve extremely high sample efficiency and generalizability to unseen configurations. Finally, we demonstrate state-of-the-art performance across various tasks on the RLbench benchmark compared with several strong baselines and validate our approach on a real robot.