Search

Neural networks are seeing rapid adoption in purely inertial odometry, where accelerometer and gyroscope measurements from commodity inertial measurement units (IMU) are used to regress displacements and associated uncertainties. They can learn informative displacement priors, which can be directly fused with the raw data with off-the-shelf non-linear filters. Nevertheless, these networks do not consider the physical roto-reflective symmetries inherent in IMU data, leading to the need to memorize the same priors for every possible motion direction, which hinders generalization. In this work, we characterize these symmetries and show that the IMU data and the resulting displacement and covariance transform equivariantly, when rotated around the gravity vector and reflected with respect to arbitrary planes parallel to gravity. We design a neural network that respects these symmetries by design through equivariant processing in three steps: First, it estimates an equivariant gravity-aligned frame from equivariant vectors and invariant scalars derived from IMU data, leveraging expressive linear and non-linear layers tailored to commute with the underlying symmetry transformation. We then map the IMU data into this frame, thereby achieving an invariant canonicalization that can be directly used with off-the-shelf inertial odometry networks. Finally, we map these network outputs back into the original frame, thereby obtaining equivariant covariances and displacements. We demonstrate the generality of our framework by applying it to the filter-based approach based on TLIO, and the end-to-end RONIN architecture, and show better performance on the TLIO, Aria, RIDI and OxIOD datasets than existing methods., Comment: 27 pages

Current optical flow and point-tracking methods rely heavily on synthetic datasets. Event cameras are novel vision sensors with advantages in challenging visual conditions, but state-of-the-art frame-based methods cannot be easily adapted to event data due to the limitations of current event simulators. We introduce a novel self-supervised loss combining the Contrast Maximization framework with a non-linear motion prior in the form of pixel-level trajectories and propose an efficient solution to solve the high-dimensional assignment problem between non-linear trajectories and events. Their effectiveness is demonstrated in two scenarios: In dense continuous-time motion estimation, our method improves the zero-shot performance of a synthetically trained model on the real-world dataset EVIMO2 by 29%. In optical flow estimation, our method elevates a simple UNet to achieve state-of-the-art performance among self-supervised methods on the DSEC optical flow benchmark. Our code is available at https://github.com/tub-rip/MotionPriorCMax., Comment: 24 pages, 8 figures, 8 tables, Project Page: https://github.com/tub-rip/MotionPriorCMax

We propose a novel test-time optimization approach for efficiently and robustly tracking any pixel at any time in a video. The latest state-of-the-art optimization-based tracking technique, OmniMotion, requires a prohibitively long optimization time, rendering it impractical for downstream applications. OmniMotion is sensitive to the choice of random seeds, leading to unstable convergence. To improve efficiency and robustness, we introduce a novel invertible deformation network, CaDeX++, which factorizes the function representation into a local spatial-temporal feature grid and enhances the expressivity of the coupling blocks with non-linear functions. While CaDeX++ incorporates a stronger geometric bias within its architectural design, it also takes advantage of the inductive bias provided by the vision foundation models. Our system utilizes monocular depth estimation to represent scene geometry and enhances the objective by incorporating DINOv2 long-term semantics to regulate the optimization process. Our experiments demonstrate a substantial improvement in training speed (more than \textbf{10 times} faster), robustness, and accuracy in tracking over the SoTA optimization-based method OmniMotion., Comment: project page: https://timsong412.github.io/FastOmniTrack/

We propose TRAM, a two-stage method to reconstruct a human's global trajectory and motion from in-the-wild videos. TRAM robustifies SLAM to recover the camera motion in the presence of dynamic humans and uses the scene background to derive the motion scale. Using the recovered camera as a metric-scale reference frame, we introduce a video transformer model (VIMO) to regress the kinematic body motion of a human. By composing the two motions, we achieve accurate recovery of 3D humans in the world space, reducing global motion errors by a large margin from prior work. https://yufu-wang.github.io/tram4d/, Comment: The project website: https://yufu-wang.github.io/tram4d/

Event cameras are a novel type of biologically inspired vision sensor known for their high temporal resolution, high dynamic range, and low power consumption. Because of these properties, they are well-suited for processing fast motions that require rapid reactions. Although event cameras have recently shown competitive performance in unsupervised optical flow estimation, performance in detecting independently moving objects (IMOs) is lacking behind, although event-based methods would be suited for this task based on their low latency and HDR properties. Previous approaches to event-based IMO segmentation have been heavily dependent on labeled data. However, biological vision systems have developed the ability to avoid moving objects through daily tasks without being given explicit labels. In this work, we propose the first event framework that generates IMO pseudo-labels using geometric constraints. Due to its unsupervised nature, our method can handle an arbitrary number of not predetermined objects and is easily scalable to datasets where expensive IMO labels are not readily available. We evaluate our approach on the EVIMO dataset and show that it performs competitively with supervised methods, both quantitatively and qualitatively.

We present ContinuityCam, a novel approach to generate a continuous video from a single static RGB image, using an event camera. Conventional cameras struggle with high-speed motion capture due to bandwidth and dynamic range limitations. Event cameras are ideal sensors to solve this problem because they encode compressed change information at high temporal resolution. In this work, we propose a novel task called event-based continuous color video decompression, pairing single static color frames and events to reconstruct temporally continuous videos. Our approach combines continuous long-range motion modeling with a feature-plane-based synthesis neural integration model, enabling frame prediction at arbitrary times within the events. Our method does not rely on additional frames except for the initial image, increasing, thus, the robustness to sudden light changes, minimizing the prediction latency, and decreasing the bandwidth requirement. We introduce a novel single objective beamsplitter setup that acquires aligned images and events and a novel and challenging Event Extreme Decompression Dataset (E2D2) that tests the method in various lighting and motion profiles. We thoroughly evaluate our method through benchmarking reconstruction as well as various downstream tasks. Our approach significantly outperforms the event- and image- based baselines in the proposed task.

With the advantages of large electro-optical coefficient, wide transparency window, and strong optical confinement, thin-film lithium niobate (TFLN) technique has enabled the development of various high-performance optoelectronics devices, ranging from the ultra-wideband electro-optic modulators to the high-efficient quantum sources. However, the TFLN platform does not natively promise lasers and photodiodes. This study presents an InP/InGaAs modified uni-traveling carrier (MUTC) photodiodes heterogeneously integrated on the TFLN platform with a record-high 3-dB bandwidth of 110 GHz and a responsivity of 0.4 A/W at a 1550-nm wavelength. It is implemented on a wafer-level TFLN-InP heterogeneous integration platform and is suitable for the large-scale, multi-function, and high-performance TFLN photonic integrated circuits., Comment: 17 pages, 8 figures

Event-based sensors have recently drawn increasing interest in robotic perception due to their lower latency, higher dynamic range, and lower bandwidth requirements compared to standard CMOS-based imagers. These properties make them ideal tools for real-time perception tasks in highly dynamic environments. In this work, we demonstrate an application where event cameras excel: accurately estimating the impact location of fast-moving objects. We introduce a lightweight event representation called Binary Event History Image (BEHI) to encode event data at low latency, as well as a learning-based approach that allows real-time inference of a confidence-enabled control signal to the robot. To validate our approach, we present an experimental catching system in which we catch fast-flying ping-pong balls. We show that the system is capable of achieving a success rate of 81% in catching balls targeted at different locations, with a velocity of up to 13 m/s even on compute-constrained embedded platforms such as the Nvidia Jetson NX., Comment: 8 pages, 6 figures, IEEE Robotics and Automation Letters ( Volume: 7, Issue: 4, October 2022)

3D reconstruction from multiple views is a successful computer vision field with multiple deployments in applications. State of the art is based on traditional RGB frames that enable optimization of photo-consistency cross views. In this paper, we study the problem of 3D reconstruction from event-cameras, motivated by the advantages of event-based cameras in terms of low power and latency as well as by the biological evidence that eyes in nature capture the same data and still perceive well 3D shape. The foundation of our hypothesis that 3D reconstruction is feasible using events lies in the information contained in the occluding contours and in the continuous scene acquisition with events. We propose Apparent Contour Events (ACE), a novel event-based representation that defines the geometry of the apparent contour of an object. We represent ACE by a spatially and temporally continuous implicit function defined in the event x-y-t space. Furthermore, we design a novel continuous Voxel Carving algorithm enabled by the high temporal resolution of the Apparent Contour Events. To evaluate the performance of the method, we collect MOEC-3D, a 3D event dataset of a set of common real-world objects. We demonstrate the ability of EvAC3D to reconstruct high-fidelity mesh surfaces from real event sequences while allowing the refinement of the 3D reconstruction for each individual event., Comment: 16 pages, 8 figures, European Conference on Computer Vision (ECCV) 2022

Diffractive optical elements (DOEs) have a wide range of applications in optics and photonics, thanks to their capability to perform complex wavefront shaping in a compact form. However, widespread applicability of DOEs is still limited, because existing fabrication methods are cumbersome and expensive. Here, we present a simple and cost-effective fabrication approach for solid, high-performance DOEs. The method is based on conjugating two nearly refractive index-matched solidifiable transparent materials. The index matching allows for extreme scaling up of the elements in the axial dimension, which enables simple fabrication of a template using commercially available 3D printing at tens-of-micrometer resolution. We demonstrated the approach by fabricating and using DOEs serving as microlens arrays, vortex plates, including for highly sensitive applications such as vector beam generation and super-resolution microscopy using MINSTED, and phase-masks for three-dimensional single-molecule localization microscopy. Beyond the advantage of making DOEs widely accessible by drastically simplifying their production, the method also overcomes difficulties faced by existing methods in fabricating highly complex elements, such as high-order vortex plates, and spectrum-encoding phase masks for microscopy.

To enable the accelerated discovery of materials with desirable properties, it is critical to develop accurate and efficient search algorithms. Quantum annealers and similar quantum-inspired optimizers have the potential to provide accelerated computation for certain combinatorial optimization challenges. However, they have not been exploited for materials discovery due to absence of compatible optimization mapping methods. Here we show that by combining cluster expansion with a quantum-inspired superposition technique, we can lever quantum annealers in chemical space exploration for the first time. This approach enables us to accelerate the search of materials with desirable properties order 10-50 times faster than genetic algorithms and bayesian optimizations, with a significant improvement in ground state prediction accuracy. Levering this, we search chemical space for discovery of acidic oxygen evolution reaction (OER) catalysts and find a promising previously unexplored chemical family of Ru-Cr-Mn-Sb-O$_2$. The best catalyst in this chemical family show a mass activity 8 times higher than state-of-art RuO$_2$ and maintain performance for 180 hours while operating at 10mA/cm$^2$ in acidic 0.5 M $H_2SO_4$ electrolyte.

The abrupt outbreak of the COVID-19 pandemic was the most significant event in 2020, which had profound and lasting impacts across the world. Studies on energy markets observed a decline in energy demand and changes in energy consumption behaviors during COVID-19. However, as an essential part of system operation, how the load forecasting performs amid COVID-19 is not well understood. This paper aims to bridge the research gap by systematically evaluating models and features that can be used to improve the load forecasting performance amid COVID-19. Using real-world data from the New York Independent System Operator, our analysis employs three deep learning models and adopts both novel COVID-related features as well as classical weather-related features. We also propose simulating the stay-at-home situation with pre-stay-at-home weekend data and demonstrate its effectiveness in improving load forecasting accuracy during COVID-19., Comment: The 8th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys '21)

Cross-lingual text classification aims at training a classifier on the source language and transferring the knowledge to target languages, which is very useful for low-resource languages. Recent multilingual pretrained language models (mPLM) achieve impressive results in cross-lingual classification tasks, but rarely consider factors beyond semantic similarity, causing performance degradation between some language pairs. In this paper we propose a simple yet effective method to incorporate heterogeneous information within and across languages for cross-lingual text classification using graph convolutional networks (GCN). In particular, we construct a heterogeneous graph by treating documents and words as nodes, and linking nodes with different relations, which include part-of-speech roles, semantic similarity, and document translations. Extensive experiments show that our graph-based method significantly outperforms state-of-the-art models on all tasks, and also achieves consistent performance gain over baselines in low-resource settings where external tools like translators are unavailable., Comment: Accepted by ACL 2021 (short paper)