Search

This paper explores how non-experts can teach robots desired skills in their environments. We argue that natural language is an intuitive and accessible interface for robot learning. To this end, we investigate two key aspects: (1) how non-experts collect robotic data using natural language supervision and (2) how pre-trained vision-language models learn end-to-end policies directly from this supervision. We propose a data collection framework that collects robot demonstrations based on natural language supervision (e.g., "move forward") and further augments these demonstrations. Next, we introduce a model that learns language-conditioned policies from natural language supervision called CLIP-RT. Our model employs pre-trained CLIP models and learns to predict actions represented in language via contrastive imitation learning. We first train CLIP-RT on large-scale robotic data and then enable it to learn desired skills using data collected from our framework. CLIP-RT shows strong capabilities in acquiring novel manipulation skills, outperforming the state-of-the-art model, OpenVLA (7B parameters), by 17% in average success rates, while using 7x fewer parameters (1B)., Comment: 27 pages, 27 figures

Chirality, the breaking of improper rotational symmetry, is a fundamental concept spanning diverse scientific domains. In condensed matter physics, chiral phonons, originating from circular atomic motions that carry angular momentum, have sparked intense interest due to their coupling to magnetic degrees of freedom, enabling potential phonon-controlled spintronics. However, modes and their counter-rotating counterparts are typically degenerate at the Brillouin zone center. Selective excitation of a single-handed circulating phonon requires external stimuli that break the degeneracy. Whether energetically nondegenerate circularly polarized phonons can appear spontaneously without structural or external symmetry breaking remains an open question. Here, we demonstrate that nondegenerate elliptically polarized phonon pairs can be induced by coupling to magnons with same helicity in the van der Waals antiferromagnet $\mathrm{FePSe_3}$. We confirm the presence of magnon-phonon hybrids, also known as magnon polarons, which exhibit inherent elliptical polarization with opposite helicities and distinct energies. This nondegeneracy enables their coherent excitation with linearly polarized terahertz pulses, which also endows these rotating modes with chirality. By tuning the polarization of the terahertz drive and measuring phase-resolved polarimetry of the resulting coherent oscillations, we determine the ellipticity and map the trajectory of these hybrid quasiparticles. Our findings establish a general approach to search for intrinsically nondegenerate phonons with angular momentum at the center of the Brillouin zone and introduce a new methodology for characterizing their ellipticity, outlining a roadmap towards chiral-phonon-controlled spintronic functionalities., Comment: 14 pages, 4 figures

Coherent manipulation of magnetism through the lattice provides unprecedented opportunities for controlling spintronic functionalities on the ultrafast timescale. Such nonthermal control conventionally involves nonlinear excitation of Raman-active phonons which are coupled to the magnetic order. Linear excitation, in contrast, holds potential for more efficient and selective modulation of magnetic properties. However, the linear channel remains uncharted, since it is conventionally considered forbidden in inversion symmetric quantum materials. Here, we harness strong coupling between magnons and Raman-active phonons to achieve both linear and quadratic excitation regimes of magnon-polarons, magnon-phonon hybrid quasiparticles. We demonstrate this by driving magnon-polarons with an intense terahertz pulse in the van der Waals antiferromagnet $\mathrm{FePS_3}$. Such excitation behavior enables a unique way to coherently control the amplitude of magnon-polaron oscillations by tuning the terahertz field strength and its polarization. The polarimetry of the resulting coherent oscillation amplitude breaks the crystallographic $C_2$ symmetry due to strong interference between different excitation channels. Our findings unlock a wide range of possibilities to manipulate material properties, including modulation of exchange interactions by phonon-Floquet engineering., Comment: 15 pages, 4 figures

Van der Waals (vdW) magnets have opened a new avenue of novel opportunities covering various interesting phases. Co1/3TaS2-an intercalated metallic vdW antiferromagnet-is one of the latest important additions to the growing list of materials due to its unique triple-Q (3Q) ground state possessing topological characteristics. Careful bulk characterisations have shown the ground state of CoxTaS2 to be a rare 3Q tetrahedral structure for x less than 1/3. The uniqueness of this ground state arises from the dense real-space Berry curvature due to scalar spin chirality, giving rise to a noticeable anomalous Hall effect. In this work, we demonstrate that we can control this topological phase via gating. Using three kinds of CoxTaS2 devices with different Co compositions, we have established that we can cover the whole 3Q topological phase with ionic gating. This work reports a rare demonstration of electrical gating control of layered antiferromagnetic metal. More importantly, our work constitutes one of the first examples of the electrical control of the scalar spin chirality using antiferromagnetic metal., Comment: 19 pages, 4 figures

Van der Waals (vdW) magnets have rapidly emerged as a fertile playground for novel fundamental physics and exciting applications. Despite the impressive developments over the past few years, technical limitations pose a severe challenge to many other potential breakthroughs. High on the list is the lack of suitable experimental tools for studying spin dynamics on atomically thin samples. Here, Raman scattering techniques are employed to observe directly the low-lying magnon (~1 meV) even in bilayer NiPS3. The unique advantage is that it offers excellent energy resolutions far better on low-energy sides than most inelastic neutron spectrometers can offer. More importantly, with appropriate theoretical analysis, the polarization dependence of the Raman scattering by those low-lying magnons also provides otherwise hidden information on the dominant spin-exchange scattering paths for different magnons. By comparing with high-resolution inelastic neutron scattering data, these low-energy Raman modes are confirmed to be indeed of magnon origin. Because of the different scattering mechanisms involved in inelastic neutron and Raman scattering, this new information is fundamental in pinning down the final spin Hamiltonian. This work demonstrates the capability of Raman spectroscopy to probe the genuine two-dimensional spin dynamics in atomically-thin vdW magnets, which can provide novel insights that are obscured in bulk spin dynamics., Comment: 37 pages, 5 figures, supplementary information 11 pages

Embodied Instruction Following (EIF) is the task of executing natural language instructions by navigating and interacting with objects in 3D environments. One of the primary challenges in EIF is compositional task planning, which is often addressed with supervised or in-context learning with labeled data. To this end, we introduce the Socratic Planner, the first zero-shot planning method that infers without the need for any training data. Socratic Planner first decomposes the instructions into substructural information of the task through self-questioning and answering, translating it into a high-level plan, i.e., a sequence of subgoals. Subgoals are executed sequentially, with our visually grounded re-planning mechanism adjusting plans dynamically through a dense visual feedback. We also introduce an evaluation metric of high-level plans, RelaxedHLP, for a more comprehensive evaluation. Experiments demonstrate the effectiveness of the Socratic Planner, achieving competitive performance on both zero-shot and few-shot task planning in the ALFRED benchmark, particularly excelling in tasks requiring higher-dimensional inference. Additionally, a precise adjustments in the plan were achieved by incorporating environmental visual information., Comment: 14 pages, 6 figures

The control of antiferromagnets with ultrashort optical pulses has emerged as a prominent field of research. Tailored laser excitation can launch coherent spin waves at terahertz frequencies, yet a comprehensive description of their generation mechanisms is still lacking despite extensive efforts. Using terahertz emission spectroscopy, we investigate the generation of a coherent magnon mode in the van der Waals antiferromagnet NiPS$_3$ under a range of photoexcitation conditions. By tuning the pump photon energy from transparency to resonant with a $d$-$d$ transition, we reveal a striking change in the coherent magnon's dependence on the pump polarization, indicating two distinct excitation mechanisms. Our findings provide a strategy for the manipulation of magnetic modes via photoexcitation around sub-gap electronic states., Comment: 15 pages, 14 figures

The unique discovery of magnetic exciton in van der Waals antiferromagnet NiPS3 arises between two quantum many-body states of a Zhang-Rice singlet excited state and a Zhang-Rice triplet ground state. Simultaneously, the spectral width of photoluminescence originating from this exciton is exceedingly narrow as 0.4 meV. These extraordinary properties, including the extreme coherence of the magnetic exciton in NiPS3, beg many questions. We studied doping effects using Ni1-xCdxPS3 using two experimental techniques and theoretical studies. Our experimental results show that the magnetic exciton is drastically suppressed upon a few % Cd doping. All these happen while the width of the exciton only gradually increases, and the antiferromagnetic ground state is robust. These results highlight the lattice uniformity's hidden importance as a prerequisite for coherent magnetic exciton. Finally, an exciting scenario emerges: the broken charge transfer forbids the otherwise uniform formation of the coherent magnetic exciton in (Ni,Cd)PS3., Comment: 40 pages, 4 main figures, 13 supporting figures, accepted by Nano Letters

Language-Conditioned Robotic Grasping (LCRG) aims to develop robots that comprehend and grasp objects based on natural language instructions. While the ability to understand personal objects like my wallet facilitates more natural interaction with human users, current LCRG systems only allow generic language instructions, e.g., the black-colored wallet next to the laptop. To this end, we introduce a task scenario GraspMine alongside a novel dataset aimed at pinpointing and grasping personal objects given personal indicators via learning from a single human-robot interaction, rather than a large labeled dataset. Our proposed method, Personalized Grasping Agent (PGA), addresses GraspMine by leveraging the unlabeled image data of the user's environment, called Reminiscence. Specifically, PGA acquires personal object information by a user presenting a personal object with its associated indicator, followed by PGA inspecting the object by rotating it. Based on the acquired information, PGA pseudo-labels objects in the Reminiscence by our proposed label propagation algorithm. Harnessing the information acquired from the interactions and the pseudo-labeled objects in the Reminiscence, PGA adapts the object grounding model to grasp personal objects. This results in significant efficiency while previous LCRG systems rely on resource-intensive human annotations -- necessitating hundreds of labeled data to learn my wallet. Moreover, PGA outperforms baseline methods across all metrics and even shows comparable performance compared to the fully-supervised method, which learns from 9k annotated data samples. We further validate PGA's real-world applicability by employing a physical robot to execute GrsapMine. Code and data are publicly available at https://github.com/JHKim-snu/PGA., Comment: 8 pages, under review

Interactive Object Grasping (IOG) is the task of identifying and grasping the desired object via human-robot natural language interaction. Current IOG systems assume that a human user initially specifies the target object's category (e.g., bottle). Inspired by pragmatics, where humans often convey their intentions by relying on context to achieve goals, we introduce a new IOG task, Pragmatic-IOG, and the corresponding dataset, Intention-oriented Multi-modal Dialogue (IM-Dial). In our proposed task scenario, an intention-oriented utterance (e.g., "I am thirsty") is initially given to the robot. The robot should then identify the target object by interacting with a human user. Based on the task setup, we propose a new robotic system that can interpret the user's intention and pick up the target object, Pragmatic Object Grasping (PROGrasp). PROGrasp performs Pragmatic-IOG by incorporating modules for visual grounding, question asking, object grasping, and most importantly, answer interpretation for pragmatic inference. Experimental results show that PROGrasp is effective in offline (i.e., target object discovery) and online (i.e., IOG with a physical robot arm) settings. Code and data are available at https://github.com/gicheonkang/prograsp., Comment: ICRA 2024

Language-Guided Robotic Manipulation (LGRM) is a challenging task as it requires a robot to understand human instructions to manipulate everyday objects. Recent approaches in LGRM rely on pre-trained Visual Grounding (VG) models to detect objects without adapting to manipulation environments. This results in a performance drop due to a substantial domain gap between the pre-training and real-world data. A straightforward solution is to collect additional training data, but the cost of human-annotation is extortionate. In this paper, we propose Grounding Vision to Ceaselessly Created Instructions (GVCCI), a lifelong learning framework for LGRM, which continuously learns VG without human supervision. GVCCI iteratively generates synthetic instruction via object detection and trains the VG model with the generated data. We validate our framework in offline and online settings across diverse environments on different VG models. Experimental results show that accumulating synthetic data from GVCCI leads to a steady improvement in VG by up to 56.7% and improves resultant LGRM by up to 29.4%. Furthermore, the qualitative analysis shows that the unadapted VG model often fails to find correct objects due to a strong bias learned from the pre-training data. Finally, we introduce a novel VG dataset for LGRM, consisting of nearly 252k triplets of image-object-instruction from diverse manipulation environments., Comment: Accepted at IROS2023

Strong interactions between different degrees of freedom lead to exotic phases of matter with complex order parameters and emergent collective excitations. Conventional techniques, such as scattering and transport, probe the amplitudes of these excitations, but they are typically insensitive to phase. Therefore, novel methods with phase sensitivity are required to understand ground states with phase modulations and interactions that couple to the phase of collective modes. Here, by performing phase-resolved coherent phonon spectroscopy (CPS), we reveal a hidden spin-lattice coupling in a vdW antiferromagnet FePS$_{3}$ that eluded other phase-insensitive conventional probes, such as Raman and X-ray scattering. With comparative analysis and analytical calculations, we directly show that the magnetic order in FePS$_{3}$ selectively couples to the trigonal distortions through partially filled t$_{2g}$ orbitals. This magnetoelastic coupling is linear in magnetic order and lattice parameters, rendering these distortions inaccessible to inelastic scattering techniques. Our results not only capture the elusive spin-lattice coupling in FePS$_3$, but also establish phase-resolved CPS as a tool to investigate hidden interactions., Comment: 6 pages, 4 figures

Technology forecasts anticipate a new era in which massive numbers of humans, machines, and things are connected to wireless networks to sense, process, act, and communicate with the surrounding environment in a real-time manner. To make the visions come true, the sixth generation (6G) wireless networks should be hyper-connected, implying that there are no constraints on the data rate, coverage, and computing. In this article, we first identify the main challenges for 6G hyper-connectivity, including terabits-per-second (Tbps) data rates for immersive user experiences, zero coverage-hole networks, and pervasive computing for connected intelligence. To overcome these challenges, we highlight key enabling technologies for 6G such as distributed and intelligence-aware cell-free massive multi-input multioutput (MIMO) networks, boundless and fully integrated terrestrial and non-terrestrial networks, and communication-aware distributed computing for computation-intensive applications. We further illustrate and discuss the hyper-connected 6G network architecture along with open issues and future research directions., Comment: 16 pages, 6 figures

In this paper, we propose a novel quantum-secured single-pixel imaging method that utilizes non-classical correlations of a photon pair. Our method can detect any attempts to deceive it by exploiting a non-classical correlation of photon pairs while rejecting strong chaotic light illumination through photon heralding. A security analysis based on polarization-correlation has been conducted, demonstrating that our method has improved security compared to existing quantum-secured imaging. More specifically, a partial deceiving attack, which sends a mixture of a true and a false signal, can be detected with our proposed analysis, while currently employed methods cannot. We also provide proof-of-principle demonstrations of our method and trustworthy images reconstructed using our security analysis. Our method can be developed using matured techniques used in quantum secure communication, thus offering a promising direction for practical applications in secure imaging., Comment: 11 pages, 7 figures

In van der Waals (vdW) materials, strong coupling between different degrees of freedom can hybridize elementary excitations into bound states with mixed character. Correctly identifying the nature and composition of these bound states is key to understanding their ground state properties and excitation spectra. Here, we use ultrafast spectroscopy to reveal bound states of d-orbitals and phonons in 2D vdW antiferromagnet NiPS3. These bound states manifest themselves through equally spaced phonon replicas in frequency domain. These states are optically dark above the N\'eel temperature and become accessible with magnetic order. By launching this phonon and spectrally tracking its amplitude, we establish the electronic origin of bound states as localized d-d excitations. Our data directly yield electron-phonon coupling strength which exceeds the highest known value in 2D systems. These results demonstrate NiPS3 as a platform to study strong interactions between spins, orbitals and lattice, and open pathways to coherent control of 2D magnets.

Combining magnetism with band topology provides various novel phases that are otherwise impossible. Among several cases, noncollinear metallic antiferromagnets can reveal particularly rich topological physics due to their diverse magnetic ground states. However, there are only a few experimental studies due to the lack of suitable materials, especially with triangular lattice antiferromagnets. Here, we report that metallic triangular antiferromagnet Co1/3TaS2 exhibits a substantial anomalous Hall effect (AHE) related to its noncollinear magnetic order. Our first-principles calculations found that hourglass Weyl fermions from the non-symmorphic symmetry trigger AHE. We further show that AHE in Co1/3TaS2 can be characterized by the toroidal moment, a vortex-like multipole component that arises from a combination of chiral lattice and geometrical frustration. Finally, the unusual field-tunability of the toroidal moment puts Co1/3TaS2 as a rare example of a noncollinear metallic antiferromagnet filled with interesting magnetic and topological properties., Comment: 20 pages, 4 figures, Accepted for publication in npj Quantum Materials

We present a method to classically enhance noise-robustness of single-pixel imaging in photon counting regime with a pulsed source. By using time-domain cross-correlations between temporal profiles of a pulsed source and received signals, our scheme classically imitates the noise rejection concept of quantum imaging. Under a strong noise environment in which the background noise intensity is up to 120 times higher than the signal one, we compare three different images obtained by conventional, quantum-enhanced, and classically enhanced schemes. The results show that the classically enhanced scheme can be remarkably robust against noise in image formation, which is comparable to the quantum scheme., Comment: 10 pages, 6 figures

Imaging with non-classically correlated photon-pairs takes advantages over classical limits in terms of sensitivity and signal-to-noise ratio. However, it is still a challenge to achieve a strong resilience to background noise and losses for practical applications. In this work, we present heralded single-pixel imaging that is remarkably robust against bright background noise and severe signal losses. Using a strong temporal correlation between a photon-pair and joint measurement-based imaging method, we achieve the suppression of noise up to 1000 times larger than the signal and also demonstrate the correlation-induced SNR enhancement factor of over 200 against 70 times larger noise and a 90% signal loss compared to non-time-gated classical imaging. Our work enables correlated imaging with a highly scalable photon capacity., Comment: 6 pages, 5 figure, accepted in APL

Ferromagnetism in two-dimensional materials presents a promising platform for the development of ultrathin spintronic devices with advanced functionalities. Recently discovered ferromagnetic van der Waals crystals such as CrI$_{3}$, readily isolated two-dimensional crystals, are highly tunable through external fields or structural modifications. However, there remains a challenge because of material instability under air exposure. Here, we report the observation of an air stable and layer dependent ferromagnetic (FM) van der Waals crystal, CrPS$_{4}$, using magneto-optic Kerr effect microscopy. In contrast to the antiferromagnetic (AFM) bulk, the FM out-of-plane spin orientation is found in the monolayer crystal. Furthermore, alternating AFM and FM properties observed in even and odd layers suggest robust antiferromagnetic exchange interactions between layers. The observed ferromagnetism in these crystals remains resilient even after the air exposure of about a day, providing possibilities for the practical applications of van der Waals spintronics., Comment: 22 pages, 5 figures

Robust multi-level spin memory with the ability to write information electrically is a long-sought capability in spintronics, with great promise for applications. Here we achieve nonvolatile and highly energy-efficient magnetization switching in a single-material device formed of van-der-Waals topological ferromagnet Fe3GeTe2, whose magnetic information can be readily controlled by a tiny current. Furthermore, the switching current density and power dissipation are about 400 and 4000 times smaller than those of the existing spin-orbit-torque magnetic random access memory based on conventional magnet/heavy-metal systems. Most importantly, we also demonstrate multi-level states, switched by electrical current, which can dramatically enhance the information capacity density and reduce computing costs. Thus, our observations combine both high energy efficiency and large information capacity density in one device, showcasing the potential applications of the emerging field of van-der-Waals magnets in the field of spin memory and spintronics., Comment: Accepted by Advanced Functional Materials; 28 pages, 5 main figures, 4 supporting figures

Controlling magnetic states by a small current is essential for the next-generation of energy-efficient spintronic devices. However, it invariably requires considerable energy to change a magnetic ground state of intrinsically quantum nature governed by fundamental Hamiltonian, once stabilized below a phase transition temperature. We report that surprisingly an in-plane current can tune the magnetic state of nm-thin van der Waals ferromagnet Fe3GeTe2 from a hard magnetic state to a soft magnetic state. It is the direct demonstration of the current-induced substantial reduction of the coercive field. This surprising finding is possible because the in-plane current produces a highly unusual type of gigantic spin-orbit torque for Fe3GeTe2. And we further demonstrate a working model of a new nonvolatile magnetic memory based on the principle of our discovery in Fe3GeTe2, controlled by a tiny current. Our findings open up a new window of exciting opportunities for magnetic van der Waals materials with potentially huge impacts on the future development of spintronic and magnetic memory., Comment: 61 pages, 4 main figures, 14 supporting figures

Object-centric world models provide structured representation of the scene and can be an important backbone in reinforcement learning and planning. However, existing approaches suffer in partially-observable environments due to the lack of belief states. In this paper, we propose Structured World Belief, a model for learning and inference of object-centric belief states. Inferred by Sequential Monte Carlo (SMC), our belief states provide multiple object-centric scene hypotheses. To synergize the benefits of SMC particles with object representations, we also propose a new object-centric dynamics model that considers the inductive bias of object permanence. This enables tracking of object states even when they are invisible for a long time. To further facilitate object tracking in this regime, we allow our model to attend flexibly to any spatial location in the image which was restricted in previous models. In experiments, we show that object-centric belief provides a more accurate and robust performance for filtering and generation. Furthermore, we show the efficacy of structured world belief in improving the performance of reinforcement learning, planning and supervised reasoning., Comment: Published in ICML 2021

We propose observable bounds for Gaussian illumination to maximize the signal-to-noise ratio, which minimizes the discrimination error between the presence and absence of a low-reflectivity target using Gaussian states. The observable bounds are achieved with mode-by-mode measurements. In the quantum regime using a two-mode squeezed vacuum state, our observable receiver outperforms the other feasible receivers whereas it cannot approach the quantum Chernoff bound. The corresponding observable cannot be implemented with heterodyne detections due to the additional vacuum noise. In the classical regime using a thermal state, a receiver implemented with a photon number difference measurement approaches its bound regardless of the signal mean photon number, while it asymptotically approaches the classical bound in the limit of a huge idler mean photon number., Comment: 8 pages, 8 figures, close to the published version

We propose Gaussian quantum illumination(QI) protocol exploiting asymmetrically squeezed two-mode(ASTM) state that is generated by applying single-mode squeezing operations on each mode of an initial two-mode squeezed vacuum(TMSV) state, in order to overcome the limited brightness of a TMSV state. We show that the performance of the optimal receiver is enhanced by local squeezing operation on a signal mode whereas the performance of a realistic receiver can be enhanced by local squeezing operations on both input modes. Under a fixed mean photon number of the signal mode, the ASTM state can be close to the TMSV state in the performance of QI while there is a threshold of beating classical illumination in the mean photon number of the initial TMSV state. We also verify that quantum discord cannot be a resource of quantum advantage in the Gaussian QI using the ASTM state, which is a counterexample of a previous claim., Comment: 12 pages, 8 figures; Comments are welcome

With the advanced investigations into low-dimensional systems, it has become essential to find materials having interesting lattices that can be exfoliated down to monolayer. One particular important structure is a kagome lattice with its potentially diverse and vibrant physics. We report a van-der-Waals kagome lattice material, Pd3P2S8, with several unique properties such as an intriguing flat band. The flat band is shown to arise from a possible compact-localized state of all five 4d orbitals of Pd. The diamagnetic susceptibility is precisely measured to support the calculated susceptibility obtained from the band structure. We further demonstrate that Pd3P2S8 can be exfoliated down to monolayer, which ultimately will allow the possible control of the localized states in this two-dimensional kagome lattice using the electric field gating.