Search

We present a framework for generating full-bodied photorealistic avatars that gesture according to the conversational dynamics of a dyadic interaction. Given speech audio, we output multiple possibilities of gestural motion for an individual, including face, body, and hands. The key behind our method is in combining the benefits of sample diversity from vector quantization with the high-frequency details obtained through diffusion to generate more dynamic, expressive motion. We visualize the generated motion using highly photorealistic avatars that can express crucial nuances in gestures (e.g. sneers and smirks). To facilitate this line of research, we introduce a first-of-its-kind multi-view conversational dataset that allows for photorealistic reconstruction. Experiments show our model generates appropriate and diverse gestures, outperforming both diffusion- and VQ-only methods. Furthermore, our perceptual evaluation highlights the importance of photorealism (vs. meshes) in accurately assessing subtle motion details in conversational gestures. Code and dataset available online.

We study the microrheology of active suspensions through direct hydrodynamic simulations using model pusher-like microswimmers. We demonstrate that the friction coefficient of a probe particle is notably reduced by hydrodynamic interactions (HIs) among a moving probe and the swimmers. When a swimmer approaches a probe from the rear (front) side, the repulsive HIs between them are weakened (intensified), which results in a slight front rear asymmetry in swimmer orientation distribution around the probe, creating a significant additional net driving force acting on the probe from the rear side. The present drag-reduction mechanism qualitatively differs from that of the viscosity-reduction observed in sheared bulk systems and depends on probing details. This study provides insights into our fundamental knowledge of hydrodynamic effects in active suspensions and serves as a practical example illuminating distinctions between micro- and macrorheology measurements., Comment: 12 pages, 7 figures

We prove normal approximation bounds for statistics of randomly weighted (simplicial) complexes. In particular, we consider the complete $d$-dimensional complex on $n$ vertices with $d$-simplices equipped with i.i.d. weights. Our normal approximation bounds are quantified in terms of stabilization of difference operators, i.e., the effect on the statistic under addition/deletion of simplices. Our proof is based on Chatterjee's normal approximation bound and is a higher-dimensional analogue of the work of Cao on sparse Erd\H{o}s-R\'enyi random graphs but our bounds are more in the spirit of `quantitative two-scale stabilization' bounds by Lachi\`eze-Rey, Peccati, and Yang. As applications, we prove a CLT for nearest face-weights in randomly weighted $d$-complexes and give a normal approximation bound for local statistics of random $d$-complexes., Comment: Apart from typographical changes, a minor error in the bounds in Corollaries 2.7 and 2.8 has been corrected; 24 pages

In this study, we develop a simple daily assistive robot that controls its own vision according to linguistic instructions. The robot performs several daily tasks such as recording a user's face, hands, or screen, and remotely capturing images of desired locations. To construct such a robot, we combine a pre-trained large-scale vision-language model with a low-cost low-rigidity robot arm. The correlation between the robot's physical and visual information is learned probabilistically using a neural network, and changes in the probability distribution based on changes in time and environment are considered by parametric bias, which is a learnable network input variable. We demonstrate the effectiveness of this learning method by open-vocabulary view control experiments with an actual robot arm, MyCobot., Comment: accepted at Humanoids2023

We present a standard form of master equations (ME) for general one-dimensional non-Markovian (history-dependent) jump processes, complemented by an asymptotic solution derived from an expanded system-size approach. The ME is obtained by developing a general Markovian embedding using a suitable set of auxiliary field variables. This Markovian embedding uses a Laplace-convolution operation applied to the velocity trajectory. We introduce an asymptotic method tailored for this ME standard, generalising the system-size expansion for these jump processes. Under specific stability conditions tied to a single noise source, upon coarse-graining, the Generalized Langevin Equation (GLE) emerges as a universal approximate model for point processes in the weak-coupling limit. This methodology offers a unified analytical toolset for general non-Markovian processes, reinforcing the universal applicability of the GLE founded in microdynamics and the principles of statistical physics., Comment: 29 pages, 8 figures

We present an approach that can reconstruct hands in 3D from monocular input. Our approach for Hand Mesh Recovery, HaMeR, follows a fully transformer-based architecture and can analyze hands with significantly increased accuracy and robustness compared to previous work. The key to HaMeR's success lies in scaling up both the data used for training and the capacity of the deep network for hand reconstruction. For training data, we combine multiple datasets that contain 2D or 3D hand annotations. For the deep model, we use a large scale Vision Transformer architecture. Our final model consistently outperforms the previous baselines on popular 3D hand pose benchmarks. To further evaluate the effect of our design in non-controlled settings, we annotate existing in-the-wild datasets with 2D hand keypoint annotations. On this newly collected dataset of annotations, HInt, we demonstrate significant improvements over existing baselines. We make our code, data and models available on the project website: https://geopavlakos.github.io/hamer/.

We propose NeRFiller, an approach that completes missing portions of a 3D capture via generative 3D inpainting using off-the-shelf 2D visual generative models. Often parts of a captured 3D scene or object are missing due to mesh reconstruction failures or a lack of observations (e.g., contact regions, such as the bottom of objects, or hard-to-reach areas). We approach this challenging 3D inpainting problem by leveraging a 2D inpainting diffusion model. We identify a surprising behavior of these models, where they generate more 3D consistent inpaints when images form a 2$\times$2 grid, and show how to generalize this behavior to more than four images. We then present an iterative framework to distill these inpainted regions into a single consistent 3D scene. In contrast to related works, we focus on completing scenes rather than deleting foreground objects, and our approach does not require tight 2D object masks or text. We compare our approach to relevant baselines adapted to our setting on a variety of scenes, where NeRFiller creates the most 3D consistent and plausible scene completions. Our project page is at https://ethanweber.me/nerfiller., Comment: Project page: https://ethanweber.me/nerfiller

This report provides the mathematical details of the gsplat library, a modular toolbox for efficient differentiable Gaussian splatting, as proposed by Kerbl et al. It provides a self-contained reference for the computations involved in the forward and backward passes of differentiable Gaussian splatting. To facilitate practical usage and development, we provide a user friendly Python API that exposes each component of the forward and backward passes in rasterization at github.com/nerfstudio-project/gsplat ., Comment: Find the library at: https://docs.gsplat.studio/

Engineering of magnetic heterostructures for spintronic applications has entered a new phase, driven by the recent discoveries of topological materials and exfoliated van der Waals materials. Their low-dimensional properties can be dramatically modulated in designer heterostructures via proximity effects from adjacent materials, thus enabling the realization of diverse quantum states and functionalities. Here we investigate spin-orbit coupling (SOC) proximity effects of Pt on the recently discovered quasi-two-dimensional ferromagnetic state at FeSi surface. Skyrmionic bubbles (SkBs) are formed as a result of the enhanced interfacial Dzyloshinskii-Moriya interaction. The strong pinning effects on the SkBs are evidenced from the significant dispersion in size and shape of the SkBs and are further identified as a greatly enhanced threshold current density required for depinning of the SkBs. The robust integrity of the SkB assembly leads to the emergence of higher-order nonlinear Hall effects in the high current density regime, which originate from nontrivial Hall effects due to the noncollinearity of the spin texture, as well as from the current-induced magnetization dynamics via the augmented spin-orbit torque., Comment: 4 figures

Bogoliubov Fermi surfaces of superconducting states arise from point or line nodes by breaking time-reversal symmetry. Because line and point nodes often accompany topologically protected zero-energy surface Andreev bound states (ASBSs) and thereby lead to a characteristic zero-bias conductance peak (ZBCP) in tunneling spectroscopy, we investigate how these properties change when the line and point nodes deform into BFSs. In this paper, we consider spin-quintet $J_{\rm pair}=2$ pairing states of spin-3/2 electrons with BFSs and calculate the surface density of states and the charge conductance. Comparing the obtained results with the cases of spin-singlet $d$-wave pairing states having the same symmetry, we find that the ZBCP associated with point and/or line nodes is blunted or split in accordance with the appearance of the BFSs. On the other hand, when the spin-singlet $d$-wave state has point nodes but does not have SABS on the surface, we obtain a nonzero small electron conductivity at zero bias through the zero-energy states on the BFSs.

Engineering heterostructures with various types of quantum materials can provide an intriguing playground for studying exotic physics induced by the proximity effect. Here, we report the successful synthesis of iron-based superconductor FeSe$_{x}$Te$_{1-x}$ (FST) thin films across the entire composition range of $0 \leq x \leq 1$ and its heterostructure with a magnetic topological insulator by using molecular beam epitaxy. Superconductivity is observed in the FST films with an optimal superconducting transition temperature $T_c$ $\sim$ 12 K at around x = 0.1. We found that superconductivity survives in the very Te-rich films ($x \leq 0.05$), showing stark contrast to bulk crystals with suppression of superconductivity due to an appearance of bicollinear antiferromagnetism accompanied by a monoclinic structural transition. By examining thickness $t$ dependence of magnetic susceptibility and electrical transport properties, we observed a trend where anomalies associated with the first order structural transition broaden in films with below $t \sim$ 100 nm. We infer this observation suggests a suppression of the structural instability near substrates. Furthermore, we fabricated an all chalcogenide-based heterointerface between FST and a magnetic topological insulator (Cr,Bi,Sb)$_{2}$Te$_{3}$ for the first time, observing both superconductivity and a large anomalous Hall conductivity. The anomalous Hall conductivity increases with decreasing temperature, approaching the quantized value of $e^2/h$ down to the measurable minimum temperature at $T_c$. The result suggests coexistence of magnetic and superconducting gaps at low temperatures opening at the top and bottom surfaces, respectively. Our novel magnetic topological insulator/superconductor heterostructure could be an ideal platform to explore chiral Majorana edge mode.

Recognition of the current state is indispensable for the operation of a robot. There are various states to be recognized, such as whether an elevator door is open or closed, whether an object has been grasped correctly, and whether the TV is turned on or off. Until now, these states have been recognized by programmatically describing the state of a point cloud or raw image, by annotating and learning images, by using special sensors, etc. In contrast to these methods, we apply Visual Question Answering (VQA) from a Pre-Trained Vision-Language Model (PTVLM) trained on a large-scale dataset, to such binary state recognition. This idea allows us to intuitively describe state recognition in language without any re-training, thereby improving the recognition ability of robots in a simple and general way. We summarize various techniques in questioning methods and image processing, and clarify their properties through experiments.

QCD axion models have been proposed as a solution to the strong CP problem. QCD instanton effects explicitly violate the global U(1) Peccei-Quinn (PQ) symmetry, and the axion potential is minimized at the CP conserving points. However, it is expected that the global U(1) PQ symmetry is explicitly violated by quantum gravity. This gravitational violation may potentially undermine the PQ solution to the strong CP problem, which is referred to as the axion quality problem. One source of explicit PQ violation is axionic wormholes, such as the Giddings-Strominger model. Generally speaking, the extent to which the U(1) PQ symmetry is violated depends on models. One straightforward extension of the Einstein gravity involves quadratic scalar curvature, which is motivated in cosmological contexts, such as inflation. The aim of this work is to investigate the effects of $R^2$ corrections on axionic wormholes. In this paper, we compute wormhole solutions and evaluate the size of explicit PQ violation in the models with the $R^2$ term. Furthermore, We illustrate the differences between the metric and Palatini formulations., Comment: 9 pages, 10 figures

Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train generalist X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. More details can be found on the project website https://robotics-transformer-x.github.io., Comment: Project website: https://robotics-transformer-x.github.io

The field of visual computing is rapidly advancing due to the emergence of generative artificial intelligence (AI), which unlocks unprecedented capabilities for the generation, editing, and reconstruction of images, videos, and 3D scenes. In these domains, diffusion models are the generative AI architecture of choice. Within the last year alone, the literature on diffusion-based tools and applications has seen exponential growth and relevant papers are published across the computer graphics, computer vision, and AI communities with new works appearing daily on arXiv. This rapid growth of the field makes it difficult to keep up with all recent developments. The goal of this state-of-the-art report (STAR) is to introduce the basic mathematical concepts of diffusion models, implementation details and design choices of the popular Stable Diffusion model, as well as overview important aspects of these generative AI tools, including personalization, conditioning, inversion, among others. Moreover, we give a comprehensive overview of the rapidly growing literature on diffusion-based generation and editing, categorized by the type of generated medium, including 2D images, videos, 3D objects, locomotion, and 4D scenes. Finally, we discuss available datasets, metrics, open challenges, and social implications. This STAR provides an intuitive starting point to explore this exciting topic for researchers, artists, and practitioners alike.

A plethora of correlated and exotic metallic states have been identified on the border of itinerant magnetism, where the long-range spin texture is melted by tuning the magnetic transition temperature (T$_C$) towards zero, referred to as the quantum phase transition (QPT). So far, the study of QPT in itinerant magnets has mainly focused on low-T$_C$ materials (i.e., typically T$_C$ ~ 10 K) where the modification of electronic band structure is subtle, and only makes a small contribution to the QPT. Here we report a distinct example of a magnetic QPT accompanied by a gigantic modification of the electronic structure in FeGe, i.e., a well-studied itinerant chiral magnet hosting near-room-temperature (T$_C$ = 278 K) helical/skyrmion spin texture. The pressure-driven modification of the band structure (e.g., reduction of exchange splitting) is evidenced by magneto-transport study, suggesting a Fermi-surface reconstruction around the magnetic QPT (P ~19 GPa), in stark contrast to the case of typical metallic ferromagnets. Further application of pressure leads to a metal-to-insulator transition above P > 30 GPa, as also corroborated by our density-functional theory (DFT) calculation. Of particular interest is the occurrence of anomalous magneto-transport in the inhomogeneous short-range chiral-spin ground state (P = 20-30 GPa) above the QPT, with longitudinal fluctuations of magnetization. The unexpected observation of spontaneous anomalous Hall effect in this exotic quantum regime suggests macroscopic time-reversal symmetry (TRS) breaking, even in the absence of long-range magnetic order. Our findings mark the large body of unexplored high-T$_C$ itinerant magnets with broken inversion-symmetry as promising candidates of novel ground state formation near QPT.

It is important for daily life support robots to detect changes in their environment and perform tasks. In the field of anomaly detection in computer vision, probabilistic and deep learning methods have been used to calculate the image distance. These methods calculate distances by focusing on image pixels. In contrast, this study aims to detect semantic changes in the daily life environment using the current development of large-scale vision-language models. Using its Visual Question Answering (VQA) model, we propose a method to detect semantic changes by applying multiple questions to a reference image and a current image and obtaining answers in the form of sentences. Unlike deep learning-based methods in anomaly detection, this method does not require any training or fine-tuning, is not affected by noise, and is sensitive to semantic state changes in the real world. In our experiments, we demonstrated the effectiveness of this method by applying it to a patrol task in a real-life environment using a mobile robot, Fetch Mobile Manipulator. In the future, it may be possible to add explanatory power to changes in the daily life environment through spoken language., Comment: Accepted to 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2023)

Qutrit state readout is an important technology not only for execution of qutrit algorithms but also for erasure detection in error correction circuits and leakage error characterization of the gate set. Conventional technique using a specialized IQ discriminator requires memory intensive IQ data for input, and has difficulty in scaling up the system size. In this study, we propose the mid-circuit measurement based discrimination technique which exploits a binary discriminator for qubit readout. Our discriminator shows comparable performance with the IQ discriminator, and readily available for standard quantum processors calibrated for qubit control. We also demonstrate our technique can reimplement typical benchmarking and characterization experiments such as leakage randomized benchmarking and state population decay measurement., Comment: 8 pages, 5 figures

Native multi-qubit parity gates have various potential quantum computing applications, such as entanglement creation, logical state encoding and parity measurement in quantum error correction. Here, using simultaneous cross-resonance drives on two control qubits with a common target, we demonstrate an efficient implementation of a three-qubit parity gate. We have developed a calibration procedure based on the one for the echoed cross-resonance gate. We confirm that our use of simultaneous drives leads to higher interleaved randomized benchmarking fidelities than a naive implementation with two consecutive CNOT gates. We also demonstrate that our simultaneous parity gates can significantly improve the parity measurement error probability for the heavy-hexagon code on an IBM Quantum processor using seven superconducting qubits with all-microwave control.

Grasping objects by a specific part is often crucial for safety and for executing downstream tasks. Yet, learning-based grasp planners lack this behavior unless they are trained on specific object part data, making it a significant challenge to scale object diversity. Instead, we propose LERF-TOGO, Language Embedded Radiance Fields for Task-Oriented Grasping of Objects, which uses vision-language models zero-shot to output a grasp distribution over an object given a natural language query. To accomplish this, we first reconstruct a LERF of the scene, which distills CLIP embeddings into a multi-scale 3D language field queryable with text. However, LERF has no sense of objectness, meaning its relevancy outputs often return incomplete activations over an object which are insufficient for subsequent part queries. LERF-TOGO mitigates this lack of spatial grouping by extracting a 3D object mask via DINO features and then conditionally querying LERF on this mask to obtain a semantic distribution over the object with which to rank grasps from an off-the-shelf grasp planner. We evaluate LERF-TOGO's ability to grasp task-oriented object parts on 31 different physical objects, and find it selects grasps on the correct part in 81% of all trials and grasps successfully in 69%. See the project website at: lerftogo.github.io, Comment: See the project website at: lerftogo.github.io

Cooking tasks are characterized by large changes in the state of the food, which is one of the major challenges in robot execution of cooking tasks. In particular, cooking using a stove to apply heat to the foodstuff causes many special state changes that are not seen in other tasks, making it difficult to design a recognizer. In this study, we propose a unified method for recognizing changes in the cooking state of robots by using the vision-language model that can discriminate open-vocabulary objects in a time-series manner. We collected data on four typical state changes in cooking using a real robot and confirmed the effectiveness of the proposed method. We also compared the conditions and discussed the types of natural language prompts and the image regions that are suitable for recognizing the state changes., Comment: Accepted at IAS18-2023

We study the adjacency matrix of the Linial-Meshulam complex model, which is a higher-dimensional generalization of the Erd\H{o}s-R\'enyi graph model. Recently, Knowles and Rosenthal proved that the empirical spectral distribution of the adjacency matrix is asymptotically given by Wigner's semicircle law in a diluted regime. In this article, we prove a central limit theorem for the linear eigenvalue statistics for test functions of polynomial growth that is of class $C^{2}$ on a closed interval. The proof is based on higher-dimensional combinatorial enumerations and concentration properties of random symmetric matrices. Furthermore, when the test function is a polynomial function, we obtain the explicit formula for the variance of the limiting Gaussian distribution., Comment: 30 pages, 3 figures

We present a framework for generating appropriate facial responses from a listener in dyadic social interactions based on the speaker's words. Given an input transcription of the speaker's words with their timestamps, our approach autoregressively predicts a response of a listener: a sequence of listener facial gestures, quantized using a VQ-VAE. Since gesture is a language component, we propose treating the quantized atomic motion elements as additional language token inputs to a transformer-based large language model. Initializing our transformer with the weights of a language model pre-trained only on text results in significantly higher quality listener responses than training a transformer from scratch. We show that our generated listener motion is fluent and reflective of language semantics through quantitative metrics and a qualitative user study. In our evaluation, we analyze the model's ability to utilize temporal and semantic aspects of spoken text. Project page: https://people.eecs.berkeley.edu/~evonne_ng/projects/text2listen/, Comment: ICCV 2023; Project page: https://people.eecs.berkeley.edu/~evonne_ng/projects/text2listen/

This paper describes a strategy for implementing a robotic system capable of performing General Purpose Service Robot (GPSR) tasks in robocup@home. The GPSR task is that a real robot hears a variety of commands in spoken language and executes a task in a daily life environment. To achieve the task, we integrate foundation models based inference system and a state machine task executable. The foundation models plan the task and detect objects with open vocabulary, and a state machine task executable manages each robot's actions. This system works stable, and we took first place in the RoboCup@home Japan Open 2022's GPSR with 130 points, more than 85 points ahead of the other teams., Comment: In review

In this research, we focus on the order-splitting behavior. The order splitting is a trading strategy to execute their large potential metaorder into small pieces to reduce transaction cost. This strategic behavior is believed to be important because it is a promising candidate for the microscopic origin of the long-range correlation (LRC) in the persistent order flow. Indeed, in 2005, Lillo, Mike, and Farmer (LMF) introduced a microscopic model of the order-splitting traders to predict the asymptotic behavior of the LRC from the microscopic dynamics, even quantitatively. The plausibility of this scenario has been qualitatively investigated by Toth et al. 2015. However, no solid support has been presented yet on the quantitative prediction by the LMF model in the lack of large microscopic datasets. In this report, we have provided the first quantitative statistical analysis of the order-splitting behavior at the level of each trading account. We analyse a large dataset of the Tokyo stock exchange (TSE) market over nine years, including the account data of traders (called virtual servers). The virtual server is a unit of trading accounts in the TSE market, and we can effectively define the trader IDs by an appropriate preprocessing. We apply a strategy clustering to individual traders to identify the order-splitting traders and the random traders. For most of the stocks, we find that the metaorder length distribution obeys power laws with exponent $\alpha$, such that $P(L)\propto L^{-\alpha-1}$ with the metaorder length $L$. By analysing the sign correlation $C(\tau)\propto \tau^{-\gamma}$, we directly confirmed the LMF prediction $\gamma \approx \alpha-1$. Furthermore, we discuss how to estimate the total number of the splitting traders only from public data via the ACF prefactor formula in the LMF model. Our work provides the first quantitative evidence of the LMF model., Comment: 33 pages, 19 figures