Search

For AI agents to be helpful to humans, they should be able to follow natural language instructions to complete everyday cooperative tasks in human environments. However, real human instructions inherently possess ambiguity, because the human speakers assume sufficient prior knowledge about their hidden goals and intentions. Standard language grounding and planning methods fail to address such ambiguities because they do not model human internal goals as additional partially observable factors in the environment. We propose a new framework, Follow Instructions with Social and Embodied Reasoning (FISER), aiming for better natural language instruction following in collaborative embodied tasks. Our framework makes explicit inferences about human goals and intentions as intermediate reasoning steps. We implement a set of Transformer-based models and evaluate them over a challenging benchmark, HandMeThat. We empirically demonstrate that using social reasoning to explicitly infer human intentions before making action plans surpasses purely end-to-end approaches. We also compare our implementation with strong baselines, including Chain of Thought prompting on the largest available pre-trained language models, and find that FISER provides better performance on the embodied social reasoning tasks under investigation, reaching the state-of-the-art on HandMeThat.

We introduce a novel technique for enhancing the robustness of light-pulse atom interferometers against the pulse infidelities that typically limit their sensitivities. The technique uses quantum optimal control to favorably harness the multipath interference of the stray trajectories produced by imperfect atom-optics operations. We apply this method to a resonant atom interferometer and achieve thousand-fold phase amplification, representing a fifty-fold improvement over the performance observed without optimized control. Moreover, we find that spurious interference can arise from the interplay of spontaneous emission and many-pulse sequences and demonstrate optimization strategies to mitigate this effect. Given the ubiquity of spontaneous emission in quantum systems, these results may be valuable for improving the performance of a diverse array of quantum sensors. We anticipate our findings will significantly benefit the performance of matter-wave interferometers for a variety of applications, including dark matter, dark energy, and gravitational wave detection.

The sliding window model of computation captures scenarios in which data are continually arriving in the form of a stream, and only the most recent $w$ items are used for analysis. In this setting, an algorithm needs to accurately track some desired statistics over the sliding window using a small space. When data streams contain sensitive information about individuals, the algorithm is also urgently needed to provide a provable guarantee of privacy. In this paper, we focus on the two fundamental problems of privately (1) estimating the frequency of an arbitrary item and (2) identifying the most frequent items (i.e., \emph{heavy hitters}), in the sliding window model. We propose \textsc{DPSW-Sketch}, a sliding window framework based on the count-min sketch that not only satisfies differential privacy over the stream but also approximates the results for frequency and heavy-hitter queries within bounded errors in sublinear time and space w.r.t.~$w$. Extensive experiments on five real-world and synthetic datasets show that \textsc{DPSW-Sketch} provides significantly better utility-privacy trade-offs than state-of-the-art methods., Comment: Accepted for publication at KDD 2024

Data selection has emerged as a core issue for large-scale visual-language model pretaining (e.g., CLIP), particularly with noisy web-curated datasets. Three main data selection approaches are: (1) leveraging external non-CLIP models to aid data selection, (2) training new CLIP-style embedding models that are more effective at selecting high-quality data than the original OpenAI CLIP model, and (3) designing better metrics or strategies universally applicable to any CLIP embedding without requiring specific model properties (e.g., CLIPScore is one popular metric). While the first two approaches have been extensively studied, the third remains under-explored. In this paper, we advance the third approach by proposing two new methods. Firstly, instead of classical CLIP scores that only consider the alignment between two modalities from a single sample, we introduce negCLIPLoss, a CLIP loss-inspired method that adds the alignment between one sample and its contrastive pairs as an extra normalization term for better quality measurement. Secondly, when downstream tasks are known, we propose a new norm-based metric, NormSim, to measure the similarity between pretraining data and target data. We test our methods on the data selection benchmark, DataComp~\cite{gadre2023datacomp}. Compared to the best baseline using only OpenAI's CLIP-L/14, our methods achieve a 5.3\% improvement on ImageNet-1k and a 2.8\% improvement on 38 downstream evaluation tasks. Moreover, both negCLIPLoss and NormSim are compatible with existing techniques. By combining our methods with the current best methods DFN~\cite{fang2023data} and HYPE~\cite{kim2024hype}, we can boost average performance on downstream tasks by 0.9\%, achieving a new state-of-the-art., Comment: This paper supercedes our previous VAS paper (arXiv:2402.02055)

{\alpha}-RuCl$_3$, a narrow-band Mott insulator with large work function, offers intriguing potential as a quantum material or as a charge acceptor for electrical contacts in van der Waals devices. In this work, we perform a systematic study of the optical reflection contrast of {\alpha}-RuCl$_3$ nanoflakes on oxidized silicon wafers and estimate the accuracy of this imaging technique to assess the crystal thickness. Via spectroscopic micro-ellipsometry measurements, we characterize the wavelength-dependent complex refractive index of {\alpha}-RuCl$_3$ nanoflakes of varying thickness in the visible and near-infrared. Building on these results, we simulate the optical contrast of {\alpha}-RuCl$_3$ nanoflakes with thicknesses below 100 nm on SiO$_2$/Si substrates under different illumination conditions. We compare the simulated optical contrast with experimental values extracted from optical microscopy images and obtain good agreement. Finally, we show that optical contrast imaging allows us to retrieve the thickness of the RuCl$_3$ nanoflakes exfoliated on an oxidized silicon substrate with a mean deviation of -0.2 nm for thicknesses below 100 nm with a standard deviation of only 1 nm. Our results demonstrate that optical contrast can be used as a non-invasive, fast, and reliable technique to estimate the {\alpha}-RuCl$_3$ thickness.

The convergence of topology and correlations represents a highly coveted realm in the pursuit of novel quantum states of matter. Introducing electron correlations to a quantum spin Hall (QSH) insulator can lead to the emergence of a fractional topological insulator and other exotic time-reversal-symmetric topological order, not possible in quantum Hall and Chern insulator systems. However, the QSH insulator with quantized edge conductance remains rare, let alone that with significant correlations. In this work, we report a novel dual QSH insulator within the intrinsic monolayer crystal of TaIrTe4, arising from the interplay of its single-particle topology and density-tuned electron correlations. At charge neutrality, monolayer TaIrTe4 demonstrates the QSH insulator that aligns with single-particle band structure calculations, manifesting enhanced nonlocal transport and quantized helical edge conductance. Interestingly, upon introducing electrons from charge neutrality, TaIrTe4 only shows metallic behavior in a small range of charge densities but quickly goes into a new insulating state, entirely unexpected based on TaIrTe4's single-particle band structure. This insulating state could arise from a strong electronic instability near the van Hove singularities (VHS), likely leading to a charge density wave (CDW). Remarkably, within this correlated insulating gap, we observe a resurgence of the QSH state, marked by the revival of nonlocal transport and quantized helical edge conduction. Our observation of helical edge conduction in a CDW gap could bridge spin physics and charge orders. The discovery of a dual QSH insulator introduces a new method for creating topological flat minibands via CDW superlattices, which offer a promising platform for exploring time-reversal-symmetric fractional phases and electromagnetism., Comment: 23 pages, 15 figures, submitted version

In recent years, data selection has emerged as a core issue for large-scale visual-language model pretraining, especially on noisy web-curated datasets. One widely adopted strategy assigns quality scores such as CLIP similarity for each sample and retains the data pairs with the highest scores. However, these approaches are agnostic of data distribution and always fail to select the most informative samples. To solve this problem, we propose a simple yet theoretically principled metric named Variance Alignment Score (VAS), which has the form $\langle \Sigma_{\text{test}}, \Sigma_i\rangle$. Here, $\Sigma_{\text{test}}$ represents the target (cross-)covariance matrix we aim to align, potentially based on prior knowledge, while $\Sigma_i$ denotes the tensor product of single or multi-modal representations for the $i$-th sample. We further design a new data selection method that maximizes the total VAS. We provide theoretical analysis in a simplified setting to demonstrate the theoretical advantage of VAS over random or other existing data selection. Experimentally, applying VAS and CLIP scores together can outperform baselines by a margin of $1.3\%$ average on 38 evaluation sets for noisy dataset DataComp and $2.5\%$ on VTAB for high-quality dataset CC12M. Additionally, our ablation study also shows visual features are better than text for calculating VAS, and the related classical experimental design methods may fail under this context., Comment: 17 pages, 4 figures

Exchange-coupled interfaces are pivotal in exploiting two-dimensional (2D) ferromagnetism. Due to the extraordinary correlations among charge, spin, orbital and lattice degrees of freedom, layered magnetic transition metal chalcogenides (TMCs) bode well for exotic topological phenomena. Here we report the realization of wafer-scale Cr2Te3 down to monolayer (ML) on insulating SrTiO3(111) and/or Al2O3(001) substrates using molecular beam epitaxy. Robust ferromagnetism persists in the 2D limit. In particular, the Curie temperature TC of 2 ML Cr2Te3 increases from 100 K to ~ 120 K when proximitized to topological insulator (TI) (Bi,Sb)2Te3, with substantially boosted magnetization as observed via polarized neutron reflectometry. Our experiments and theory strongly indicate that the Bloembergen-Rowland interaction is likely universal underlying TC enhancement in TI-coupled magnetic heterostructures. The topological-surface-enhanced magnetism in 2D TMC enables further exchange coupling physics and quantu, Comment: Main: 7 pages, 3 figures, 1 table; SI: 6 pages, 8 figures

The Coriolis force is a significant source of systematic phase errors and dephasing in atom interferometry and is often compensated by counter-rotating the interferometry laser beam against Earth's rotation. We present a novel method for performing Coriolis force compensation for long-baseline atom interferometry which mitigates atom-beam misalignment due to beam rotation, an effect which is magnified by the long lever arm of the baseline length. The method involves adjustment of the angle of the interferometer beam prior to a magnifying telescope, enabling the beam to pivot around a tunable position along the interferometer baseline. By tuning the initial atom kinematics, and adjusting the angle with which the interferometer beam pivots about this point, we can ensure that the atoms align with the center of the beam during the atom optics laser pulses. This approach will be used in the MAGIS-100 atom interferometer and could also be applied to other long-baseline atom interferometers. An additional challenge associated with long baseline interferometry is that since long-baseline atom interferometers are often located outside of typical laboratory environments, facilities constraints may require lasers to be housed in a climate-controlled room a significant distance away from the main experiment. Nonlinear effects in optical fibers restrict the use of fiber-based transport of the high-power interferometry beam from the laser room to the experiment. We present the design of and prototype data from a laser transport system for MAGIS-100 that maintains robustness against alignment drifts despite the absence of a long fiber.

We propose Joint MLP/Attention (JoMA) dynamics, a novel mathematical framework to understand the training procedure of multilayer Transformer architectures. This is achieved by integrating out the self-attention layer in Transformers, producing a modified dynamics of MLP layers only. JoMA removes unrealistic assumptions in previous analysis (e.g., lack of residual connection) and predicts that the attention first becomes sparse (to learn salient tokens), then dense (to learn less salient tokens) in the presence of nonlinear activations, while in the linear case, it is consistent with existing works that show attention becomes sparse over time. We leverage JoMA to qualitatively explains how tokens are combined to form hierarchies in multilayer Transformers, when the input tokens are generated by a latent hierarchical generative model. Experiments on models trained from real-world dataset (Wikitext2/Wikitext103) and various pre-trained models (OPT, Pythia) verify our theoretical findings. Code can be found in https://github.com/facebookresearch/luckmatters/tree/yuandong3., Comment: ICLR'24 camera ready. Improve theorem 3 and theorem 4. Polish writing and add code link

To leverage the copious amount of data from source tasks and overcome the scarcity of the target task samples, representation learning based on multi-task pretraining has become a standard approach in many applications. However, up until now, most existing works design a source task selection strategy from a purely empirical perspective. Recently, \citet{chen2022active} gave the first active multi-task representation learning (A-MTRL) algorithm which adaptively samples from source tasks and can provably reduce the total sample complexity using the L2-regularized-target-source-relevance parameter $\nu^2$. But their work is theoretically suboptimal in terms of total source sample complexity and is less practical in some real-world scenarios where sparse training source task selection is desired. In this paper, we address both issues. Specifically, we show the strict dominance of the L1-regularized-relevance-based ($\nu^1$-based) strategy by giving a lower bound for the $\nu^2$-based strategy. When $\nu^1$ is unknown, we propose a practical algorithm that uses the LASSO program to estimate $\nu^1$. Our algorithm successfully recovers the optimal result in the known case. In addition to our sample complexity results, we also characterize the potential of our $\nu^1$-based strategy in sample-cost-sensitive settings. Finally, we provide experiments on real-world computer vision datasets to illustrate the effectiveness of our proposed method., Comment: Accepted by ICML 2023

Transformer architecture has shown impressive performance in multiple research domains and has become the backbone of many neural network models. However, there is limited understanding on how it works. In particular, with a simple predictive loss, how the representation emerges from the gradient \emph{training dynamics} remains a mystery. In this paper, for 1-layer transformer with one self-attention layer plus one decoder layer, we analyze its SGD training dynamics for the task of next token prediction in a mathematically rigorous manner. We open the black box of the dynamic process of how the self-attention layer combines input tokens, and reveal the nature of underlying inductive bias. More specifically, with the assumption (a) no positional encoding, (b) long input sequence, and (c) the decoder layer learns faster than the self-attention layer, we prove that self-attention acts as a \emph{discriminative scanning algorithm}: starting from uniform attention, it gradually attends more to distinct key tokens for a specific next token to be predicted, and pays less attention to common key tokens that occur across different next tokens. Among distinct tokens, it progressively drops attention weights, following the order of low to high co-occurrence between the key and the query token in the training set. Interestingly, this procedure does not lead to winner-takes-all, but decelerates due to a \emph{phase transition} that is controllable by the learning rates of the two layers, leaving (almost) fixed token combination. We verify this \textbf{\emph{scan and snap}} dynamics on synthetic and real-world data (WikiText)., Comment: Camera Ready for NeurIPS 2023

We present the development of a laser system for performing single-photon atom interferometry on the 698 nm clock transition in ultracold Strontium. We coherently combine the power of two Titanium:Sapphire lasers and demonstrate chirps of 200 MHz in 2.5 ms while phase-locked to an optical reference. Moreover, we demonstrate a novel scheme to deliver 4 W pulsed beams to the atoms via a mode-cleaning optical fiber using active noise cancellation., Comment: 5 pages, 3 figures

Improving the generalization of deep networks is an important open challenge, particularly in domains without plentiful data. The mixup algorithm improves generalization by linearly interpolating a pair of examples and their corresponding labels. These interpolated examples augment the original training set. Mixup has shown promising results in various classification tasks, but systematic analysis of mixup in regression remains underexplored. Using mixup directly on regression labels can result in arbitrarily incorrect labels. In this paper, we propose a simple yet powerful algorithm, C-Mixup, to improve generalization on regression tasks. In contrast with vanilla mixup, which picks training examples for mixing with uniform probability, C-Mixup adjusts the sampling probability based on the similarity of the labels. Our theoretical analysis confirms that C-Mixup with label similarity obtains a smaller mean square error in supervised regression and meta-regression than vanilla mixup and using feature similarity. Another benefit of C-Mixup is that it can improve out-of-distribution robustness, where the test distribution is different from the training distribution. By selectively interpolating examples with similar labels, it mitigates the effects of domain-associated information and yields domain-invariant representations. We evaluate C-Mixup on eleven datasets, ranging from tabular to video data. Compared to the best prior approach, C-Mixup achieves 6.56%, 4.76%, 5.82% improvements in in-distribution generalization, task generalization, and out-of-distribution robustness, respectively. Code is released at https://github.com/huaxiuyao/C-Mixup., Comment: Accepted by NeurIPS 2022

Strontium clock atom interferometry is a promising new technique, with multiple experiments under development to explore its potential for dark matter and gravitational wave detection. In these detectors, large momentum transfer (LMT) using sequences of many laser pulses is necessary, and thus high fidelity of each pulse is important since small infidelities become magnified. Quantum Optimal Control (QOC) is a framework for developing control pulse waveforms that achieve high fidelity and are robust against experimental imperfections. Resonant single-photon transitions using the narrow clock transition of strontium involve significantly different quantum dynamics than more established atom interferometry methods based on far-detuned two-photon Raman or Bragg transitions, which leads to new opportunities and challenges when applying QOC. Here, we study QOC pulses for strontium clock interferometry and demonstrate their advantage over basic square pulses (primitive pulses) and composite pulses in terms of robustness against multiple noise channels. This could improve the scale of large momentum transfer in Sr clock interferometers, paving the way to achieve these scientific goals., Comment: 7 pages, 5 figures

We investigate the magnetic excitations of the hyperhoneycomb Kitaev magnet $\beta$-$\text{Li}_2\text{IrO}_3$ by means of inelastic Raman scattering. The spectra exhibits a coexistence of a broad scattering continuum and two sharp low-energy peaks at 2.5 meV and 3 meV, with a distinctive polarization dependence. While the continuum is suggestive of fractional quasi-particles emerging from a proximate quantum spin liquid phase, the sharp peaks provide the first experimental signature of the `non-Loudon-Fleury' one-magnon scattering processes proposed recently [Phys. Rev. B 104, 144412 (2021)]. The corresponding microscopic mechanism is similar to the one leading to the symmetric off-diagonal exchange interaction $\Gamma$ (as it involves a combination of both direct and ligand-mediated exchange paths), but is otherwise completely unexpected within the traditional Loudon-Fleury theory of Raman scattering. The present experimental verification therefore calls for a drastic reevaluation of Raman scattering in similar systems with strong spin orbit coupling and multiple exchange paths., Comment: 5 pages, 4 figures, Supplementary material

The observation of the Higgs boson solidified the standard model of particle physics. However, explanations of anomalies (e.g. dark matter) rely on further symmetry breaking calling for an undiscovered axial Higgs mode. In condensed matter the Higgs was seen in magnetic, superconducting and charge density wave(CDW) systems. Uncovering a low energy mode's vector properties is challenging, requiring going beyond typical spectroscopic or scattering techniques. Here, we discover an axial Higgs mode in the CDW system RTe3 using the interference of quantum pathways. In RTe3 (R=La,Gd), the electronic ordering couples bands of equal or different angular momenta. As such, the Raman scattering tensor associated to the Higgs mode contains both symmetric and antisymmetric components, which can be excited via two distinct, but degenerate pathways. This leads to constructive or destructive interference of these pathways, depending on the choice of the incident and Raman scattered light polarization. The qualitative behavior of the Raman spectra is well-captured by an appropriate tight-binding model including an axial Higgs mode. The elucidation of the antisymmetric component provides direct evidence that the Higgs mode contains an axial vector representation (i.e. a pseudo-angular momentum) and hints the CDW in RTe3 is unconventional. Thus we provide a means for measuring collective modes quantum properties without resorting to extreme experimental conditions.