Search

In condensed matter systems, interactions between collective modes offer avenues for nonlinear coherent manipulation of coupled excitations and quantum phases. Antiferromagnets, with their inherently coupled magnon modes, provide a promising platform for nonlinear control of microscopic spin waves and macroscopic magnetization. However, nonlinear magnon-magnon interactions have been only partially elaborated, leaving key gaps in the prospects for potential ultrahigh-bandwidth magnonic signal processing. Here, we use a pair of intense terahertz pulses to sequentially excite two distinct coherent magnon modes in an antiferromagnet and find that the magnon mode with a lower frequency undergoes amplification when the higher-frequency mode is driven. We unveil the nonlinear excitation pathways of this stimulated parametric downconversion process by using polarization-selective two-dimensional terahertz spectroscopy. Our work provides fundamental insights into nonlinear magnonics in antiferromagnets, laying the groundwork for forthcoming spintronic and magnonic devices based on nonlinear magnon-magnon interactions.

Achieving realistic animated human avatars requires accurate modeling of pose-dependent clothing deformations. Existing learning-based methods heavily rely on the Linear Blend Skinning (LBS) of minimally-clothed human models like SMPL to model deformation. However, these methods struggle to handle loose clothing, such as long dresses, where the canonicalization process becomes ill-defined when the clothing is far from the body, leading to disjointed and fragmented results. To overcome this limitation, we propose a novel hybrid framework to model challenging clothed humans. Our core idea is to use dedicated strategies to model different regions, depending on whether they are close to or distant from the body. Specifically, we segment the human body into three categories: unclothed, deformed, and generated. We simply replicate unclothed regions that require no deformation. For deformed regions close to the body, we leverage LBS to handle the deformation. As for the generated regions, which correspond to loose clothing areas, we introduce a novel free-form, part-aware generator to model them, as they are less affected by movements. This free-form generation paradigm brings enhanced flexibility and expressiveness to our hybrid framework, enabling it to capture the intricate geometric details of challenging loose clothing, such as skirts and dresses. Experimental results on the benchmark dataset featuring loose clothing demonstrate that our method achieves state-of-the-art performance with superior visual fidelity and realism, particularly in the most challenging cases., Comment: 23 pages, 25 figures

We propose a one-stage framework for real-time multi-person 3D human mesh estimation from a single RGB image. While current one-stage methods, which follow a DETR-style pipeline, achieve state-of-the-art (SOTA) performance with high-resolution inputs, we observe that this particularly benefits the estimation of individuals in smaller scales of the image (e.g., those far from the camera), but at the cost of significantly increased computation overhead. To address this, we introduce scale-adaptive tokens that are dynamically adjusted based on the relative scale of each individual in the image within the DETR framework. Specifically, individuals in smaller scales are processed at higher resolutions, larger ones at lower resolutions, and background regions are further distilled. These scale-adaptive tokens more efficiently encode the image features, facilitating subsequent decoding to regress the human mesh, while allowing the model to allocate computational resources more effectively and focus on more challenging cases. Experiments show that our method preserves the accuracy benefits of high-resolution processing while substantially reducing computational cost, achieving real-time inference with performance comparable to SOTA methods., Comment: 16 pages, 12 figures

Understanding non-human primate behavior is crucial for improving animal welfare, modeling social behavior, and gaining insights into both distinctly human and shared behaviors. Despite recent advances in computer vision, automated analysis of primate behavior remains challenging due to the complexity of their social interactions and the lack of specialized algorithms. Existing methods often struggle with the nuanced behaviors and frequent occlusions characteristic of primate social dynamics. This study aims to develop an effective method for automated detection, tracking, and recognition of chimpanzee behaviors in video footage. Here we show that our proposed method, AlphaChimp, an end-to-end approach that simultaneously detects chimpanzee positions and estimates behavior categories from videos, significantly outperforms existing methods in behavior recognition. AlphaChimp achieves approximately 10% higher tracking accuracy and a 20% improvement in behavior recognition compared to state-of-the-art methods, particularly excelling in the recognition of social behaviors. This superior performance stems from AlphaChimp's innovative architecture, which integrates temporal feature fusion with a Transformer-based self-attention mechanism, enabling more effective capture and interpretation of complex social interactions among chimpanzees. Our approach bridges the gap between computer vision and primatology, enhancing technical capabilities and deepening our understanding of primate communication and sociality. We release our code and models and hope this will facilitate future research in animal social dynamics. This work contributes to ethology, cognitive science, and artificial intelligence, offering new perspectives on social intelligence., Comment: An extension of ChimpACT [arXiv:2310.16447], proposes AlphaChimp for tracking and behavior recognition of chimpanzees. arXiv admin note: substantial text overlap with arXiv:2310.16447

Confronting the challenges of data scarcity and advanced motion synthesis in human-scene interaction modeling, we introduce the TRUMANS dataset alongside a novel HSI motion synthesis method. TRUMANS stands as the most comprehensive motion-captured HSI dataset currently available, encompassing over 15 hours of human interactions across 100 indoor scenes. It intricately captures whole-body human motions and part-level object dynamics, focusing on the realism of contact. This dataset is further scaled up by transforming physical environments into exact virtual models and applying extensive augmentations to appearance and motion for both humans and objects while maintaining interaction fidelity. Utilizing TRUMANS, we devise a diffusion-based autoregressive model that efficiently generates HSI sequences of any length, taking into account both scene context and intended actions. In experiments, our approach shows remarkable zero-shot generalizability on a range of 3D scene datasets (e.g., PROX, Replica, ScanNet, ScanNet++), producing motions that closely mimic original motion-captured sequences, as confirmed by quantitative experiments and human studies.

Temporal repetition counting aims to quantify the repeated action cycles within a video. The majority of existing methods rely on the similarity correlation matrix to characterize the repetitiveness of actions, but their scalability is hindered due to the quadratic computational complexity. In this work, we introduce a novel approach that employs an action query representation to localize repeated action cycles with linear computational complexity. Based on this representation, we further develop two key components to tackle the essential challenges of temporal repetition counting. Firstly, to facilitate open-set action counting, we propose the dynamic update scheme on action queries. Unlike static action queries, this approach dynamically embeds video features into action queries, offering a more flexible and generalizable representation. Secondly, to distinguish between actions of interest and background noise actions, we incorporate inter-query contrastive learning to regularize the video representations corresponding to different action queries. As a result, our method significantly outperforms previous works, particularly in terms of long video sequences, unseen actions, and actions at various speeds. On the challenging RepCountA benchmark, we outperform the state-of-the-art method TransRAC by 26.5% in OBO accuracy, with a 22.7% mean error decrease and 94.1% computational burden reduction. Code is available at https://github.com/lizishi/DeTRC., Comment: code: https://github.com/lizishi/DeTRC, proj page: https://shirleymaxx.github.io/DeTRC/

Estimating robot pose from RGB images is a crucial problem in computer vision and robotics. While previous methods have achieved promising performance, most of them presume full knowledge of robot internal states, e.g. ground-truth robot joint angles. However, this assumption is not always valid in practical situations. In real-world applications such as multi-robot collaboration or human-robot interaction, the robot joint states might not be shared or could be unreliable. On the other hand, existing approaches that estimate robot pose without joint state priors suffer from heavy computation burdens and thus cannot support real-time applications. This work introduces an efficient framework for real-time robot pose estimation from RGB images without requiring known robot states. Our method estimates camera-to-robot rotation, robot state parameters, keypoint locations, and root depth, employing a neural network module for each task to facilitate learning and sim-to-real transfer. Notably, it achieves inference in a single feed-forward pass without iterative optimization. Our approach offers a 12-time speed increase with state-of-the-art accuracy, enabling real-time holistic robot pose estimation for the first time. Code and models are available at https://github.com/Oliverbansk/Holistic-Robot-Pose-Estimation., Comment: Accepted by ECCV 2024

Two-level atoms coupled with single-mode cavity photons are predicted to exhibit a quantum phase transition when the coupling strength exceeds a critical value, entering a phase in which atomic polarization and photonic field are finite even at zero temperature and without external driving. However, this phenomenon, the superradiant phase transition (SRPT), is forbidden by a no-go theorem due to the existence of the diamagnetic term in the Hamiltonian. Here, we present spectroscopic evidence for a magnonic SRPT in ErFeO$_3$, where the role of the photonic mode (two-level atoms) in the photonic SRPT is played by an Fe$^{3+}$ magnon mode (Er$^{3+}$ spins). The absence of the diamagnetic term in the Fe$^{3+}$-Er$^{3+}$ exchange coupling ensures that the no-go theorem does not apply. Terahertz and gigahertz magnetospectroscopy experiments revealed the signatures of the SRPT -- a kink and a softening, respectively, of two spin-magnon hybridized modes at the critical point.

Humans exhibit a remarkable capacity for anticipating the actions of others and planning their own actions accordingly. In this study, we strive to replicate this ability by addressing the social motion prediction problem. We introduce a new benchmark, a novel formulation, and a cognition-inspired framework. We present Wusi, a 3D multi-person motion dataset under the context of team sports, which features intense and strategic human interactions and diverse pose distributions. By reformulating the problem from a multi-agent reinforcement learning perspective, we incorporate behavioral cloning and generative adversarial imitation learning to boost learning efficiency and generalization. Furthermore, we take into account the cognitive aspects of the human social action planning process and develop a cognitive hierarchy framework to predict strategic human social interactions. We conduct comprehensive experiments to validate the effectiveness of our proposed dataset and approach. Code and data are available at https://walter0807.github.io/Social-CH/., Comment: NeurIPS 2023

Due to the complex interactions between rare-earth elements and transition metals, as well as between themselves, rare-earth transition-metal oxides are likely to exhibit highly intriguing and novel magnetic structures and dynamic behaviours. Rare-earth elements in these compounds frequently demonstrate unusual behaviours in their crystal-field (CF) excitations, which necessitate thorough studies for in-depth comprehensions. When cooling from 10 K to 1.5 K through the magnetic ordering temperature of $Er^{3+}$ at 4.1 K, we observed a significant energy shift of the low-lying CF excitation of $Er^{3+}$ in erbium orthoferrite ($ErFeO_3$) from 0.32 meV to 0.75 meV utilizing the inelastic neutron-scattering technique. A sound CF model was proposed for $Er^{3+}$ in $ErFeO_3$ by fitting to the observed CF excitation peaks, which enables to explain all the observed experimental results in a very consistent manner. According to the model, the ground crystal field level of $Er^{3+}$, which corresponds to the lowest Kramers doublet supposed to be at zero energy transfer, has been shifted by the internal magnetic fields induced by both $Er^{3+}$ and $Fe^{3+}$ spin orders below and above the $Er^{3+}$ ordering temperature, respectively. Additional measurements in various magnetic fields offer compelling evidence in favour of this hypothesis. The measured external field dependence of the CF excitation energy led to a derivation of the internal field of $Er^{3+}$ as 0.54 T, which is strongly corroborated by theoretical modelling. Additionally, the g-factor for the $Er^{3+}$ ground state in $ErFeO_3$ shows an exceptionally significant anisotropy., Comment: arXiv admin note: This version has been removed by arXiv administrators as the submitter did not have the right to agree to the license at the time of submission

Understanding the behavior of non-human primates is crucial for improving animal welfare, modeling social behavior, and gaining insights into distinctively human and phylogenetically shared behaviors. However, the lack of datasets on non-human primate behavior hinders in-depth exploration of primate social interactions, posing challenges to research on our closest living relatives. To address these limitations, we present ChimpACT, a comprehensive dataset for quantifying the longitudinal behavior and social relations of chimpanzees within a social group. Spanning from 2015 to 2018, ChimpACT features videos of a group of over 20 chimpanzees residing at the Leipzig Zoo, Germany, with a particular focus on documenting the developmental trajectory of one young male, Azibo. ChimpACT is both comprehensive and challenging, consisting of 163 videos with a cumulative 160,500 frames, each richly annotated with detection, identification, pose estimation, and fine-grained spatiotemporal behavior labels. We benchmark representative methods of three tracks on ChimpACT: (i) tracking and identification, (ii) pose estimation, and (iii) spatiotemporal action detection of the chimpanzees. Our experiments reveal that ChimpACT offers ample opportunities for both devising new methods and adapting existing ones to solve fundamental computer vision tasks applied to chimpanzee groups, such as detection, pose estimation, and behavior analysis, ultimately deepening our comprehension of communication and sociality in non-human primates., Comment: NeurIPS 2023

Human motion generation aims to generate natural human pose sequences and shows immense potential for real-world applications. Substantial progress has been made recently in motion data collection technologies and generation methods, laying the foundation for increasing interest in human motion generation. Most research within this field focuses on generating human motions based on conditional signals, such as text, audio, and scene contexts. While significant advancements have been made in recent years, the task continues to pose challenges due to the intricate nature of human motion and its implicit relationship with conditional signals. In this survey, we present a comprehensive literature review of human motion generation, which, to the best of our knowledge, is the first of its kind in this field. We begin by introducing the background of human motion and generative models, followed by an examination of representative methods for three mainstream sub-tasks: text-conditioned, audio-conditioned, and scene-conditioned human motion generation. Additionally, we provide an overview of common datasets and evaluation metrics. Lastly, we discuss open problems and outline potential future research directions. We hope that this survey could provide the community with a comprehensive glimpse of this rapidly evolving field and inspire novel ideas that address the outstanding challenges., Comment: Accepted to TPAMI

Inspired by the success of volumetric 3D pose estimation, some recent human mesh estimators propose to estimate 3D skeletons as intermediate representations, from which, the dense 3D meshes are regressed by exploiting the mesh topology. However, body shape information is lost in extracting skeletons, leading to mediocre performance. The advanced motion capture systems solve the problem by placing dense physical markers on the body surface, which allows to extract realistic meshes from their non-rigid motions. However, they cannot be applied to wild images without markers. In this work, we present an intermediate representation, named virtual markers, which learns 64 landmark keypoints on the body surface based on the large-scale mocap data in a generative style, mimicking the effects of physical markers. The virtual markers can be accurately detected from wild images and can reconstruct the intact meshes with realistic shapes by simple interpolation. Our approach outperforms the state-of-the-art methods on three datasets. In particular, it surpasses the existing methods by a notable margin on the SURREAL dataset, which has diverse body shapes. Code is available at https://github.com/ShirleyMaxx/VirtualMarker, Comment: CVPR 2023

The Dicke model describes the cooperative interaction of an ensemble of two-level atoms with a single-mode photonic field and exhibits a quantum phase transition as a function of light--matter coupling strength. Extending this model by incorporating short-range atom--atom interactions makes the problem intractable but is expected to produce new phases. Here, we simulate such an extended Dicke model using a crystal of ErFeO$_3$, where the role of atoms (photons) is played by Er$^{3+}$ spins (Fe$^{3+}$ magnons). Through magnetocaloric effect and terahertz magnetospectroscopy measurements, we demonstrated the existence of a novel atomically ordered phase in addition to the superradiant and normal phases that are expected from the standard Dicke model. Further, we elucidated the nature of the phase boundaries in the temperature--magnetic-field phase diagram, identifying both first-order and second-order phase transitions. These results lay the foundation for studying multiatomic quantum optics models using well-characterized many-body condensed matter systems.

Learning 3D human pose prior is essential to human-centered AI. Here, we present GFPose, a versatile framework to model plausible 3D human poses for various applications. At the core of GFPose is a time-dependent score network, which estimates the gradient on each body joint and progressively denoises the perturbed 3D human pose to match a given task specification. During the denoising process, GFPose implicitly incorporates pose priors in gradients and unifies various discriminative and generative tasks in an elegant framework. Despite the simplicity, GFPose demonstrates great potential in several downstream tasks. Our experiments empirically show that 1) as a multi-hypothesis pose estimator, GFPose outperforms existing SOTAs by 20% on Human3.6M dataset. 2) as a single-hypothesis pose estimator, GFPose achieves comparable results to deterministic SOTAs, even with a vanilla backbone. 3) GFPose is able to produce diverse and realistic samples in pose denoising, completion and generation tasks. Project page https://sites.google.com/view/gfpose/

We present a unified perspective on tackling various human-centric video tasks by learning human motion representations from large-scale and heterogeneous data resources. Specifically, we propose a pretraining stage in which a motion encoder is trained to recover the underlying 3D motion from noisy partial 2D observations. The motion representations acquired in this way incorporate geometric, kinematic, and physical knowledge about human motion, which can be easily transferred to multiple downstream tasks. We implement the motion encoder with a Dual-stream Spatio-temporal Transformer (DSTformer) neural network. It could capture long-range spatio-temporal relationships among the skeletal joints comprehensively and adaptively, exemplified by the lowest 3D pose estimation error so far when trained from scratch. Furthermore, our proposed framework achieves state-of-the-art performance on all three downstream tasks by simply finetuning the pretrained motion encoder with a simple regression head (1-2 layers), which demonstrates the versatility of the learned motion representations. Code and models are available at https://motionbert.github.io/, Comment: ICCV 2023 Camera Ready

Depending on the relative rates of coupling and dissipation, a light-matter coupled system is either in the weak- or strong-coupling regime. Here, we present a unique system where the coupling rate continuously increases with an externally applied magnetic field while the dissipation rate remains constant, allowing us to monitor a weak-to-strong coupling transition as a function of magnetic field. We observed a Rabi splitting of a terahertz magnon mode in yttrium orthoferrite above a threshold magnetic field of ~14 T. Based on a microscopic theoretical model, we show that with increasing magnetic field the magnons transition into magnon polaritons through an exceptional point, which will open up new opportunities for in situ control of non-Hermitian systems.

Solving mathematical equations faster and more efficiently has been a Holy Grail for centuries for scientists and engineers across all disciplines. While electronic digital circuits have revolutionized equation solving in recent decades, it has become apparent that performance gains from brute-force approaches of compute-solvers are quickly saturating over time. Instead, paradigms that leverage the universes natural tendency to minimize a systems free energy, such as annealers or Ising Machines, are being sought after due to favorable complexity scaling. Here we introduce a programmable analog solver leveraging the mathematical formal equivalence between Maxwells equations and photonic circuitry. It features a mesh network of nanophotonic beams to find solutions to partial differential equations. As an example, we designed, fabricated, and demonstrated a novel application-specific photonic integrated circuit comprised of electro-optically reconfigurable nodes, and experimentally validated 90% accuracy with respect to a commercial solver. Finally, we tested this photonic integrated chip performance by simulating thermal diffusion on a spacecrafts heat shield during re-entry to a planets atmosphere. The programmable light-circuitry presented herein offers a facile route for solving complex problems and thus will have profound potential applications across many scientific and engineering fields.

While monocular 3D pose estimation seems to have achieved very accurate results on the public datasets, their generalization ability is largely overlooked. In this work, we perform a systematic evaluation of the existing methods and find that they get notably larger errors when tested on different cameras, human poses and appearance. To address the problem, we introduce VirtualPose, a two-stage learning framework to exploit the hidden "free lunch" specific to this task, i.e. generating infinite number of poses and cameras for training models at no cost. To that end, the first stage transforms images to abstract geometry representations (AGR), and then the second maps them to 3D poses. It addresses the generalization issue from two aspects: (1) the first stage can be trained on diverse 2D datasets to reduce the risk of over-fitting to limited appearance; (2) the second stage can be trained on diverse AGR synthesized from a large number of virtual cameras and poses. It outperforms the SOTA methods without using any paired images and 3D poses from the benchmarks, which paves the way for practical applications. Code is available at https://github.com/wkom/VirtualPose.

Tailored light excitation and nonlinear control of lattice vibrations have emerged as powerful strategies to manipulate the properties of quantum materials out of equilibrium. Generalizing the use of coherent phonon-phonon interactions to nonlinear couplings among other types of collective modes would open unprecedented opportunities in the design of novel dynamic functionalities in solids. For example, the collective excitations of magnetic order -- magnons -- can carry information with little energy dissipation, and their coherent and nonlinear control would provide an attractive route to achieve collective-mode-based information processing and storage in forthcoming spintronics and magnonics. Here, we discover that intense terahertz (THz) fields can initiate processes of magnon upconversion mediated by an intermediate magnetic resonance. By using a suite of advanced spectroscopic tools, including a newly demonstrated two-dimensional (2D) THz polarimetry technique enabled by single-shot detection, we unveil the unidirectional nature of coupling between distinct magnon modes of a canted antiferromagnet. Calculations of spin dynamics further suggest that this coupling is a universal feature of antiferromagnets with canted magnetic moments. These results demonstrate a route to inducing desirable energy transfer pathways and THz-induced coupling between coherent magnons in solids and pave the way for a new era in the development of ultrafast control of magnetism.

Photonic Random-Access Memories (P-RAM) are an essential component for the on-chip non-von Neumann photonic computing by eliminating optoelectronic conversion losses in data links. Emerging Phase Change Materials (PCMs) have been showed multilevel memory capability, but demonstrations still yield relatively high optical loss and require cumbersome WRITE-ERASE approaches increasing power consumption and system package challenges. Here we demonstrate a multi-state electrically-programmed low-loss non-volatile photonic memory based on a broadband transparent phase change material (Ge2Sb2Se5, GSSe) with ultra-low absorption in the amorphous state. A zero-static-power and electrically-programmed multi-bit P-RAM is demonstrated on a silicon-on-insulator platform, featuring efficient amplitude modulation up to 0.2 dB/{\mu}m and an ultra-low insertion loss of total 0.12 dB for a 4-bit memory showing a 100x improved signal to loss ratio compared to other phase-change-materials based photonic memories. We further optimize the positioning of dual micro-heaters validating performance tradeoffs. Experimentally we demonstrate a half-a million cyclability test showcasing the robust approach of this material and device. Low-loss photonic retention-of-state adds a key feature for photonic functional and programmable circuits impacting many applications including neural networks, LiDAR, and sensors for example.