Search

Neural Fields have emerged as a transformative approach for 3D scene representation in computer vision and robotics, enabling accurate inference of geometry, 3D semantics, and dynamics from posed 2D data. Leveraging differentiable rendering, Neural Fields encompass both continuous implicit and explicit neural representations enabling high-fidelity 3D reconstruction, integration of multi-modal sensor data, and generation of novel viewpoints. This survey explores their applications in robotics, emphasizing their potential to enhance perception, planning, and control. Their compactness, memory efficiency, and differentiability, along with seamless integration with foundation and generative models, make them ideal for real-time applications, improving robot adaptability and decision-making. This paper provides a thorough review of Neural Fields in robotics, categorizing applications across various domains and evaluating their strengths and limitations, based on over 200 papers. First, we present four key Neural Fields frameworks: Occupancy Networks, Signed Distance Fields, Neural Radiance Fields, and Gaussian Splatting. Second, we detail Neural Fields' applications in five major robotics domains: pose estimation, manipulation, navigation, physics, and autonomous driving, highlighting key works and discussing takeaways and open challenges. Finally, we outline the current limitations of Neural Fields in robotics and propose promising directions for future research. Project page: https://robonerf.github.io, Comment: 20 pages, 20 figures. Project Page: https://robonerf.github.io

3D reconstruction from a single image is a long-standing problem in computer vision. Learning-based methods address its inherent scale ambiguity by leveraging increasingly large labeled and unlabeled datasets, to produce geometric priors capable of generating accurate predictions across domains. As a result, state of the art approaches show impressive performance in zero-shot relative and metric depth estimation. Recently, diffusion models have exhibited remarkable scalability and generalizable properties in their learned representations. However, because these models repurpose tools originally designed for image generation, they can only operate on dense ground-truth, which is not available for most depth labels, especially in real-world settings. In this paper we present GRIN, an efficient diffusion model designed to ingest sparse unstructured training data. We use image features with 3D geometric positional encodings to condition the diffusion process both globally and locally, generating depth predictions at a pixel-level. With comprehensive experiments across eight indoor and outdoor datasets, we show that GRIN establishes a new state of the art in zero-shot metric monocular depth estimation even when trained from scratch.

Synthesizing accurate geometry and photo-realistic appearance of small scenes is an active area of research with compelling use cases in gaming, virtual reality, robotic-manipulation, autonomous driving, convenient product capture, and consumer-level photography. When applying scene geometry and appearance estimation techniques to robotics, we found that the narrow cone of possible viewpoints due to the limited range of robot motion and scene clutter caused current estimation techniques to produce poor quality estimates or even fail. On the other hand, in robotic applications, dense metric depth can often be measured directly using stereo and illumination can be controlled. Depth can provide a good initial estimate of the object geometry to improve reconstruction, while multi-illumination images can facilitate relighting. In this work we demonstrate a method to incorporate dense metric depth into the training of neural 3D representations and address an artifact observed while jointly refining geometry and appearance by disambiguating between texture and geometry edges. We also discuss a multi-flash stereo camera system developed to capture the necessary data for our pipeline and show results on relighting and view synthesis with a few training views., Comment: Project webpage: https://stereomfc.github.io

We discuss the effect of rigid rotation on critical temperatures of deconfinement and chiral transitions in the linear sigma model coupled to quarks and the Polyakov loop. We point out the essential role of the causality condition, which requires that any point of the system should rotate slower than the velocity of light. We show that imposing this physical requirement leads to inhibition of the splitting between the chiral and confining transitions, which becomes negligibly small ($\Delta T \sim 1$~MeV or less) for experimentally relevant, slow angular velocities $\Omega \sim 10$~MeV of a $(5-10)$~fm-sized systems. Moreover, the boundedness of the system has a much bigger effect on temperature splitting than the rotation itself: the splitting reaches 10~MeV in a small, one-fermi-sized non-rotating system. The temperature splitting may, however, become enhanced in an academic limit of ultra-relativistic regimes when the boundary of the system rotates at near-to-light velocities., Comment: 16 pages, 8 figures

Stochastic multi-armed bandits (MABs) provide a fundamental reinforcement learning model to study sequential decision making in uncertain environments. The upper confidence bounds (UCB) algorithm gave birth to the renaissance of bandit algorithms, as it achieves near-optimal regret rates under various moment assumptions. Up until recently most UCB methods relied on concentration inequalities leading to confidence bounds which depend on moment parameters, such as the variance proxy, that are usually unknown in practice. In this paper, we propose a new distribution-free, data-driven UCB algorithm for symmetric reward distributions, which needs no moment information. The key idea is to combine a refined, one-sided version of the recently developed resampled median-of-means (RMM) method with UCB. We prove a near-optimal regret bound for the proposed anytime, parameter-free RMM-UCB method, even for heavy-tailed distributions.

The common trade-offs of state-of-the-art methods for multi-shape representation (a single model "packing" multiple objects) involve trading modeling accuracy against memory and storage. We show how to encode multiple shapes represented as continuous neural fields with a higher degree of precision than previously possible and with low memory usage. Key to our approach is a recursive hierarchical formulation that exploits object self-similarity, leading to a highly compressed and efficient shape latent space. Thanks to the recursive formulation, our method supports spatial and global-to-local latent feature fusion without needing to initialize and maintain auxiliary data structures, while still allowing for continuous field queries to enable applications such as raytracing. In experiments on a set of diverse datasets, we provide compelling qualitative results and demonstrate state-of-the-art multi-scene reconstruction and compression results with a single network per dataset., Comment: SIGGRAPH 2024. Project Page: https://zakharos.github.io/projects/refine/

If robots are to work effectively alongside people, they must be able to interpret natural language references to objects in their 3D environment. Understanding 3D referring expressions is challenging -- it requires the ability to both parse the 3D structure of the scene and correctly ground free-form language in the presence of distraction and clutter. We introduce Transcrib3D, an approach that brings together 3D detection methods and the emergent reasoning capabilities of large language models (LLMs). Transcrib3D uses text as the unifying medium, which allows us to sidestep the need to learn shared representations connecting multi-modal inputs, which would require massive amounts of annotated 3D data. As a demonstration of its effectiveness, Transcrib3D achieves state-of-the-art results on 3D reference resolution benchmarks, with a great leap in performance from previous multi-modality baselines. To improve upon zero-shot performance and facilitate local deployment on edge computers and robots, we propose self-correction for fine-tuning that trains smaller models, resulting in performance close to that of large models. We show that our method enables a real robot to perform pick-and-place tasks given queries that contain challenging referring expressions. Project site is at https://ripl.github.io/Transcrib3D., Comment: CORLW 2023

Neural fields excel in computer vision and robotics due to their ability to understand the 3D visual world such as inferring semantics, geometry, and dynamics. Given the capabilities of neural fields in densely representing a 3D scene from 2D images, we ask the question: Can we scale their self-supervised pretraining, specifically using masked autoencoders, to generate effective 3D representations from posed RGB images. Owing to the astounding success of extending transformers to novel data modalities, we employ standard 3D Vision Transformers to suit the unique formulation of NeRFs. We leverage NeRF's volumetric grid as a dense input to the transformer, contrasting it with other 3D representations such as pointclouds where the information density can be uneven, and the representation is irregular. Due to the difficulty of applying masked autoencoders to an implicit representation, such as NeRF, we opt for extracting an explicit representation that canonicalizes scenes across domains by employing the camera trajectory for sampling. Our goal is made possible by masking random patches from NeRF's radiance and density grid and employing a standard 3D Swin Transformer to reconstruct the masked patches. In doing so, the model can learn the semantic and spatial structure of complete scenes. We pretrain this representation at scale on our proposed curated posed-RGB data, totaling over 1.8 million images. Once pretrained, the encoder is used for effective 3D transfer learning. Our novel self-supervised pretraining for NeRFs, NeRF-MAE, scales remarkably well and improves performance on various challenging 3D tasks. Utilizing unlabeled posed 2D data for pretraining, NeRF-MAE significantly outperforms self-supervised 3D pretraining and NeRF scene understanding baselines on Front3D and ScanNet datasets with an absolute performance improvement of over 20% AP50 and 8% AP25 for 3D object detection., Comment: Accepted to ECCV 2024. Project Page: https://nerf-mae.github.io/

Due to the spin-orbit coupling, Dirac fermions, submerged in a thermal bath with finite macroscopic vorticity, exhibit a spin polarisation along the direction parallel to the vorticity vector $\boldsymbol{\Omega}$. Due to the symmetries of the Lagrangian for free massless Dirac particles, there are three independent and classically conserved currents corresponding to the vector, axial, and helical charges. The constitutive relations for the charge currents and the stress-energy tensor at thermal equilibrium, derived in the framework of quantum field theory at finite temperature, reveal vorticity-induced contributions that deviate from the perfect fluid form. In this paper, we consider the mode structure of the corresponding hydrodynamical theory and derive collective excitations associated with coherent fluctuations of all three charges. We show that the chirally imbalanced rotating fluid should possess non-reciprocal gapless waves that propagate with different velocities along and opposite to the vorticity vector. We also uncover a strictly unidirectional mode, which we call the Axial Vortical Wave, propagating in the background of the axial charge density. We point out an unexpected instability in the limit of degenerate matter and discuss possible solutions when helicity and axial charge non-conservation is taken into account., Comment: 41 pages, 6 figures, 1 table. Part 1

In this paper, we consider the effect of interactions on the local, average polarization of a quantum plasma of massless fermion particles characterized by vector, axial, and helical quantum numbers. Due to the helical and axial vortical effects, perturbations in the vector charge in a rotating plasma can lead to chiral and helical charge transfer along the direction of the vorticity vector. At the same time, interactions between the plasma constituents lead to the dissipation of the helical charge through helicity-violating pair annihilation (HVPA) processes and of the axial charge through the axial anomaly. We will discuss separately a QED-like plasma, in which we ignore background electromagnetic fields and thus the axial charge is roughly conserved, as well as a QCD-like plasma, where instanton effects lead to the violation of the axial charge conservation, even in the absence of background chromomagnetic fields. The non-conservation of helicity and chirality leads to a gapping of the Helical, Axial and mixed, Axial-Helical vortical waves that prevents their infrared modes from propagating. On the other hand, usual dissipative effects, such as charge diffusion, lead to significant damping of ultraviolet modes. We end this paper with a discussion of the regimes where these vortical waves may propagate., Comment: 56 pages, 14 figures. Part 2

We present a 3D scene completion method that recovers the complete geometry of multiple unseen objects in complex scenes from a single RGB-D image. Despite notable advancements in single-object 3D shape completion, high-quality reconstructions in highly cluttered real-world multi-object scenes remains a challenge. To address this issue, we propose OctMAE, an architecture that leverages an Octree U-Net and a latent 3D MAE to achieve high-quality and near real-time multi-object scene completion through both local and global geometric reasoning. Because a naive 3D MAE can be computationally intractable and memory intensive even in the latent space, we introduce a novel occlusion masking strategy and adopt 3D rotary embeddings, which significantly improves the runtime and scene completion quality. To generalize to a wide range of objects in diverse scenes, we create a large-scale photorealistic dataset, featuring a diverse set of 12K 3D object models from the Objaverse dataset which are rendered in multi-object scenes with physics-based positioning. Our method outperforms the current state-of-the-art on both synthetic and real-world datasets and demonstrates a strong zero-shot capability., Comment: Published at ECCV 2024, Webpage: https://sh8.io/#/oct_mae

This paper addresses the challenging problem of category-level pose estimation. Current state-of-the-art methods for this task face challenges when dealing with symmetric objects and when attempting to generalize to new environments solely through synthetic data training. In this work, we address these challenges by proposing a probabilistic model that relies on diffusion to estimate dense canonical maps crucial for recovering partial object shapes as well as establishing correspondences essential for pose estimation. Furthermore, we introduce critical components to enhance performance by leveraging the strength of the diffusion models with multi-modal input representations. We demonstrate the effectiveness of our method by testing it on a range of real datasets. Despite being trained solely on our generated synthetic data, our approach achieves state-of-the-art performance and unprecedented generalization qualities, outperforming baselines, even those specifically trained on the target domain., Comment: 8 pages. 9 figures. This work has been submitted to the IEEE for possible publication

In relativistic hydrodynamics with spin, following de Groot--van Leeuwen--van Weert's energy-momentum and spin tensor definitions, we analyze the propagation of spin degrees of freedom. We deduce an analytical formula for spin wave velocity, finding that it approaches half the speed of light in the ultra-relativistic limit. Only transverse degrees of freedom propagate, similar to electromagnetic waves. Additionally, we explore dissipative effects and determine the damping coefficients for Maxwell-J\"uttner statistics.

This paper studies the problem of concept-based interpretability of transformer representations for videos. Concretely, we seek to explain the decision-making process of video transformers based on high-level, spatiotemporal concepts that are automatically discovered. Prior research on concept-based interpretability has concentrated solely on image-level tasks. Comparatively, video models deal with the added temporal dimension, increasing complexity and posing challenges in identifying dynamic concepts over time. In this work, we systematically address these challenges by introducing the first Video Transformer Concept Discovery (VTCD) algorithm. To this end, we propose an efficient approach for unsupervised identification of units of video transformer representations - concepts, and ranking their importance to the output of a model. The resulting concepts are highly interpretable, revealing spatio-temporal reasoning mechanisms and object-centric representations in unstructured video models. Performing this analysis jointly over a diverse set of supervised and self-supervised representations, we discover that some of these mechanism are universal in video transformers. Finally, we show that VTCD can be used for fine-grained action recognition and video object segmentation., Comment: CVPR 2024 (Highlight)

In this paper we present a relativistic Shakhov-type generalization of the Anderson-Witting relaxation time model for the Boltzmann collision integral. The extension is performed by modifying the path on which the distribution function $f_{\mathbf{k}}$ is taken towards local equilibrium $f_{0\mathbf{k}}$, by replacing $f_{\mathbf{k}} - f_{0\mathbf{k}}$ via $f_{\mathbf{k}} - f_{{\rm S}\mathbf{k}}$. The Shakhov-like distribution $f_{{\rm S} \mathbf{k}}$ is constructed using $f_{0\mathbf{k}}$ and the irreducible moments $\rho_r^{\mu_1 \cdots \mu_\ell}$ of $f_\mathbf{k}$ and reduces to $f_{0\mathbf{k}}$ in local equilibrium. Employing the method of moments, we derive systematic high-order Shakhov extensions that allow both the first- and the second-order transport coefficients to be controlled independently of each other. We illustrate the capabilities of the formalism by tweaking the shear-bulk coupling coefficient $\lambda_{\Pi \pi}$ in the frame of the Bjorken flow of massive particles, as well as the diffusion-shear transport coefficients $\ell_{V\pi}$, $\ell_{\pi V}$ in the frame of sound wave propagation in an ultrarelativistic gas. Finally, we illustrate the importance of second-order transport coefficients by comparison with the results of the stochastic BAMPS method in the context of the one-dimensional Riemann problem., Comment: 38 pages, 8 figures, 2 tables

In this paper we revisit the classical method of partitioning classification and study its convergence rate under relaxed conditions, both for observable (non-privatised) and for privatised data. Let the feature vector $X$ take values in $\mathbb{R}^d$ and denote its label by $Y$. Previous results on the partitioning classifier worked with the strong density assumption, which is restrictive, as we demonstrate through simple examples. We assume that the distribution of $X$ is a mixture of an absolutely continuous and a discrete distribution, such that the absolutely continuous component is concentrated to a $d_a$ dimensional subspace. Here, we study the problem under much milder assumptions: in addition to the standard Lipschitz and margin conditions, a novel characteristic of the absolutely continuous component is introduced, by which the exact convergence rate of the classification error probability is calculated, both for the binary and for the multi-label cases. Interestingly, this rate of convergence depends only on the intrinsic dimension $d_a$. The privacy constraints mean that the data $(X_1,Y_1), \dots ,(X_n,Y_n)$ cannot be directly observed, and the classifiers are functions of the randomised outcome of a suitable local differential privacy mechanism. The statistician is free to choose the form of this privacy mechanism, and here we add Laplace distributed noises to the discontinuations of all possible locations of the feature vector $X_i$ and to its label $Y_i$. Again, tight upper bounds on the rate of convergence of the classification error probability are derived, without the strong density assumption, such that this rate depends on $2\,d_a$.

We simulate the space-time dynamics of high-energy collisions based on a microscopic kinetic description, in order to determine the range of applicability of an effective description in relativistic viscous hydrodynamics. We find that hydrodynamics provides a quantitatively accurate description of collective flow when the average inverse Reynolds number is sufficiently small and the early pre-equilibrium stage is properly accounted for. By determining the breakdown of hydrodynamics as a function of system size and energy, we find that it is quantitatively accurate in central lead-lead collisions at LHC energies, but should not be used in typical proton-lead or proton-proton collisions, where the development of collective flow cannot accurately be described within hydrodynamics., Comment: 4 pages, 2 figures. Contribution to the Quark Matter 2023 conference proceedings. v2: Minor changes. Identical to accepted version

We present a relativistic Shakhov-type generalization of the Anderson-Witting relaxation time model for the Boltzmann collision integral to modify the ratio of momentum diffusivity to thermal diffusivity. This is achieved by modifying the path on which the single particle distribution function $f_{\mathbf{k}}$ approaches local equilibrium $f_{0\mathbf{k}}$ by constructing an intermediate Shakhov-type distribution $f_{\mathrm{S}\mathbf{k}}$ similar to the 14-moment approximation of Israel and Stewart. We illustrate the effectiveness of this model in case of the Bjorken expansion of an ideal gas of massive particles and the damping of longitudinal waves through an ultrarelativistic ideal gas., Comment: 7 pages + 3 pages SM; 2 figures

In this paper, we study all transport coefficients of second-order dissipative fluid dynamics derived by V. E. Ambrus et al. [Phys. Rev. D 106, 076005 (2022)] from the relativistic Boltzmann equation in the relaxation-time approximation for the collision integral. These transport coefficients are computed for a classical ideal gas of massive particles, with and without taking into account the conservation of intrinsic quantum numbers. Through rigorous comparison between kinetic theory, second-order dissipative fluid dynamics, and leading-order anisotropic fluid dynamics for a (0+1)--dimensional boost-invariant flow scenario, we show that both fluid-dynamical theories describe the early far-from-equilibrium stage of the expansion reasonably well., Comment: Updated version; 32 pages, 9 figures and 5 tables

In this work, we address the challenging task of 3D object recognition without the reliance on real-world 3D labeled data. Our goal is to predict the 3D shape, size, and 6D pose of objects within a single RGB-D image, operating at the category level and eliminating the need for CAD models during inference. While existing self-supervised methods have made strides in this field, they often suffer from inefficiencies arising from non-end-to-end processing, reliance on separate models for different object categories, and slow surface extraction during the training of implicit reconstruction models; thus hindering both the speed and real-world applicability of the 3D recognition process. Our proposed method leverages a multi-stage training pipeline, designed to efficiently transfer synthetic performance to the real-world domain. This approach is achieved through a combination of 2D and 3D supervised losses during the synthetic domain training, followed by the incorporation of 2D supervised and 3D self-supervised losses on real-world data in two additional learning stages. By adopting this comprehensive strategy, our method successfully overcomes the aforementioned limitations and outperforms existing self-supervised 6D pose and size estimation baselines on the NOCS test-set with a 16.4% absolute improvement in mAP for 6D pose estimation while running in near real-time at 5 Hz., Comment: Project page: https://fsd6d.github.io

3D multi-object tracking (MOT) is essential for an autonomous mobile agent to safely navigate a scene. In order to maximize the perception capabilities of the autonomous agent, we aim to develop a 3D MOT framework that fuses camera and LiDAR sensor information. Building on our prior LiDAR-only work, ShaSTA, which models shape and spatio-temporal affinities for 3D MOT, we propose a novel camera-LiDAR fusion approach for learning affinities. At its core, this work proposes a fusion technique that generates a rich sensory signal incorporating information about depth and distant objects to enhance affinity estimation for improved data association, track lifecycle management, false-positive elimination, false-negative propagation, and track confidence score refinement. Our main contributions include a novel fusion approach for combining camera and LiDAR sensory signals to learn affinities, and a first-of-its-kind multimodal sequential track confidence refinement technique that fuses 2D and 3D detections. Additionally, we perform an ablative analysis on each fusion step to demonstrate the added benefits of incorporating the camera sensor, particular for small, distant objects that tend to suffer from the depth-sensing limits and sparsity of LiDAR sensors. In sum, our technique achieves state-of-the-art performance on the nuScenes benchmark amongst multimodal 3D MOT algorithms using CenterPoint detections., Comment: 8 pages, 1 figure

In this paper we discuss the analytical properties of the binary collision integral for a gas of ultrarelativistic particles interacting via a constant cross-section. Starting from a near-equilibrium expansion over a complete basis of irreducible tensors in momentum space we compute the linearized collision matrices analytically. Using these results we then numerically compute all transport-coefficients of relativistic fluid dynamics with various power-counting schemes that are second-order in Knudsen and/or inverse Reynolds numbers. Furthermore, we also exactly compute the leading-order contribution with respect to the particle mass to the coefficient of bulk viscosity, the relaxation time, and other second-order transport coefficients of the bulk viscous pressure., Comment: Published version

For $k$ a field, we construct a power structure on the Grothendieck--Witt ring of $k$ which has the potential to be compatible with symmetric powers of varieties and the motivic Euler characteristic. We then show this power structure is compatible with the power structure when we restrict to varieties of dimension $0$., Comment: 20 pages, comments welcome! Update is only to fix numbering of theorems cited from other papers which have since been updated

Recent implicit neural representations have shown great results for novel view synthesis. However, existing methods require expensive per-scene optimization from many views hence limiting their application to real-world unbounded urban settings where the objects of interest or backgrounds are observed from very few views. To mitigate this challenge, we introduce a new approach called NeO 360, Neural fields for sparse view synthesis of outdoor scenes. NeO 360 is a generalizable method that reconstructs 360{\deg} scenes from a single or a few posed RGB images. The essence of our approach is in capturing the distribution of complex real-world outdoor 3D scenes and using a hybrid image-conditional triplanar representation that can be queried from any world point. Our representation combines the best of both voxel-based and bird's-eye-view (BEV) representations and is more effective and expressive than each. NeO 360's representation allows us to learn from a large collection of unbounded 3D scenes while offering generalizability to new views and novel scenes from as few as a single image during inference. We demonstrate our approach on the proposed challenging 360{\deg} unbounded dataset, called NeRDS 360, and show that NeO 360 outperforms state-of-the-art generalizable methods for novel view synthesis while also offering editing and composition capabilities. Project page: https://zubair-irshad.github.io/projects/neo360.html, Comment: Accepted to International Conference on Computer Vision (ICCV), 2023. Project page: https://zubair-irshad.github.io/projects/neo360.html

We discuss the imaginary-time formalism for field theories in thermal equilibrium in uniformly accelerating frames. We show that under a Wick rotation of Minkowski spacetime, the Rindler event horizon shrinks to a point in a two-dimensional subspace tangential to the acceleration direction and the imaginary time. We demonstrate that the accelerated version of the Kubo-Martin-Schwinger (KMS) condition implies an identification of all spacetime points related by integer-multiple rotations in the tangential subspace about this Euclidean Rindler event-horizon point, with the rotational quanta defined by the thermal acceleration, $\alpha = a/T$. In the Wick-rotated Rindler hyperbolic coordinates, the KMS relations reduce to standard (anti-)periodic boundary conditions in terms of the imaginary proper time (rapidity) coordinate. Our findings pave the way to study, using first-principle lattice simulations, the Hawking-Unruh radiation in geometries with event horizons, phase transitions in accelerating Early Universe and early stages of quark-gluon plasma created in relativistic heavy-ion collisions., Comment: 9 pages, 2 figures

Autonomous vehicles and robots need to operate over a wide variety of scenarios in order to complete tasks efficiently and safely. Multi-camera self-supervised monocular depth estimation from videos is a promising way to reason about the environment, as it generates metrically scaled geometric predictions from visual data without requiring additional sensors. However, most works assume well-calibrated extrinsics to fully leverage this multi-camera setup, even though accurate and efficient calibration is still a challenging problem. In this work, we introduce a novel method for extrinsic calibration that builds upon the principles of self-supervised monocular depth and ego-motion learning. Our proposed curriculum learning strategy uses monocular depth and pose estimators with velocity supervision to estimate extrinsics, and then jointly learns extrinsic calibration along with depth and pose for a set of overlapping cameras rigidly attached to a moving vehicle. Experiments on a benchmark multi-camera dataset (DDAD) demonstrate that our method enables self-calibration in various scenes robustly and efficiently compared to a traditional vision-based pose estimation pipeline. Furthermore, we demonstrate the benefits of extrinsics self-calibration as a way to improve depth prediction via joint optimization., Comment: Project page: https://sites.google.com/view/tri-sesc

The paper introduces robust independence tests with non-asymptotically guaranteed significance levels for stochastic linear time-invariant systems, assuming that the observed outputs are synchronous, which means that the systems are driven by jointly i.i.d. noises. Our method provides bounds for the type I error probabilities that are distribution-free, i.e., the innovations can have arbitrary distributions. The algorithm combines confidence region estimates with permutation tests and general dependence measures, such as the Hilbert-Schmidt independence criterion and the distance covariance, to detect any nonlinear dependence between the observed systems. We also prove the consistency of our hypothesis tests under mild assumptions and demonstrate the ideas through the example of autoregressive systems.

One of the key objects of binary classification is the regression function, i.e., the conditional expectation of the class labels given the inputs. With the regression function not only a Bayes optimal classifier can be defined, but it also encodes the corresponding misclassification probabilities. The paper presents a resampling framework to construct exact, distribution-free and non-asymptotically guaranteed confidence regions for the true regression function for any user-chosen confidence level. Then, specific algorithms are suggested to demonstrate the framework. It is proved that the constructed confidence regions are strongly consistent, that is, any false model is excluded in the long run with probability one. The exclusion is quantified with probably approximately correct type bounds, as well. Finally, the algorithms are validated via numerical experiments, and the methods are compared to approximate asymptotic confidence ellipsoids.

An ensemble of massless fermions can be characterized by its total helicity charge given by the sum of axial charges of particles minus the sum of axial charges of antiparticles. We show that charged massless fermions develop a dissipationless flow of helicity along the background magnetic field. We dub this transport phenomenon as the Helical Separation Effect (HSE). Contrary to its chiral cousin, the Chiral Separation Effect, the HSE produces the helical current in a neutral plasma in which all chemical potentials vanish. In addition, we uncover the Helical Magnetic Heat Effect which generates a heat flux of Dirac fermions along the magnetic field in the presence of non-vanishing helical charge density. We also discuss possible hydrodynamic modes associated with the HSE in neutral plasma., Comment: 17 pages

We simulate the space-time dynamics of high-energy collisions based on a microscopic kinetic description, in order to determine the range of applicability of an effective description in relativistic viscous hydrodynamics. We find that hydrodynamics provides a quantitatively accurate description of collective flow when the average inverse Reynolds number $\mathrm{Re}^{-1}$ is sufficiently small and the early pre-equilibrium stage is properly accounted for. By determining the breakdown of hydrodynamics as a function of system size and energy, we find that it is quantitatively accurate in central lead-lead collisions at LHC energies, but should not be used in typical proton-lead or proton-proton collisions, where the development of collective flow can not accurately be described within hydrodynamics., Comment: 6 pages, 4 figures. Contribution to the HardProbes2023 conference proceedings

Parametricity is a property of the syntax of type theory implying, e.g., that there is only one function having the type of the polymorphic identity function. Parametricity is usually proven externally, and does not hold internally. Internalising it is difficult because once there is a term witnessing parametricity, it also has to be parametric itself and this results in the appearance of higher dimensional cubes. In previous theories with internal parametricity, either an explicit syntax for higher cubes is present or the theory is extended with a new sort for the interval. In this paper we present a type theory with internal parametricity which is a simple extension of Martin-L\"of type theory: there are a few new type formers, term formers and equations. Geometry is not explicit in this syntax, but emergent: the new operations and equations only refer to objects up to dimension 3. We show that this theory is modelled by presheaves over the BCH cube category. Fibrancy conditions are not needed because we use span-based rather than relational parametricity. We define a gluing model for this theory implying that external parametricity and canonicity hold. The theory can be seen as a special case of a new kind of modal type theory, and it is the simplest setting in which the computational properties of higher observational type theory can be demonstrated.

We study the $(n-1)$-dimensional volume of central hyperplane sections of the $n$-dimensional cube $Q_n$. Our main goal is two-fold: first, we provide an alternative, simpler argument for proving that the volume of the section perpendicular to the main diagonal of the cube is strictly locally maximal for every $n \geq 4$, which was shown before by L. Pournin. Then, we prove that non-diagonal critical central sections of $Q_n$ exist in all dimensions at least $4$. The crux of both proofs is an estimate on the rate of decay of the Laplace-P\'olya integral $J_n(r) = \int_{-\infty}^\infty \mathrm{sinc}^n t \cdot \cos (rt) \mathrm{d} t$ that is achieved by combinatorial means. This also yields improved bounds for Eulerian numbers of the first kind., Comment: 22 pages, 3 figures. Final version, to appear in Advances in Mathematics

Monocular depth estimation is scale-ambiguous, and thus requires scale supervision to produce metric predictions. Even so, the resulting models will be geometry-specific, with learned scales that cannot be directly transferred across domains. Because of that, recent works focus instead on relative depth, eschewing scale in favor of improved up-to-scale zero-shot transfer. In this work we introduce ZeroDepth, a novel monocular depth estimation framework capable of predicting metric scale for arbitrary test images from different domains and camera parameters. This is achieved by (i) the use of input-level geometric embeddings that enable the network to learn a scale prior over objects; and (ii) decoupling the encoder and decoder stages, via a variational latent representation that is conditioned on single frame information. We evaluated ZeroDepth targeting both outdoor (KITTI, DDAD, nuScenes) and indoor (NYUv2) benchmarks, and achieved a new state-of-the-art in both settings using the same pre-trained model, outperforming methods that train on in-domain data and require test-time scaling to produce metric estimates., Comment: Project page: https://sites.google.com/view/tri-zerodepth

A practical benefit of implicit visual representations like Neural Radiance Fields (NeRFs) is their memory efficiency: large scenes can be efficiently stored and shared as small neural nets instead of collections of images. However, operating on these implicit visual data structures requires extending classical image-based vision techniques (e.g., registration, blending) from image sets to neural fields. Towards this goal, we propose NeRFuser, a novel architecture for NeRF registration and blending that assumes only access to pre-generated NeRFs, and not the potentially large sets of images used to generate them. We propose registration from re-rendering, a technique to infer the transformation between NeRFs based on images synthesized from individual NeRFs. For blending, we propose sample-based inverse distance weighting to blend visual information at the ray-sample level. We evaluate NeRFuser on public benchmarks and a self-collected object-centric indoor dataset, showing the robustness of our method, including to views that are challenging to render from the individual source NeRFs., Comment: Code available at https://github.com/ripl/nerfuser

The thermodynamics of rigidly rotating systems experience divergences when the system dimensions transverse to the rotation axis exceed the critical size imposed by the causality constraint. The rotation with imaginary angular frequency, suitable for numerical lattice simulations in Euclidean imaginary-time formalism, experiences fractalization of thermodynamics in the thermodynamic limit, when the system's pressure becomes a fractal function of the rotation frequency. Our work connects two phenomena by studying how thermodynamics fractalizes as the system size grows. We examine an analytically-accessible system of rotating massless scalar matter on a one-dimensional ring and the numerically treatable case of rotation in the cylindrical geometry and show how the ninionic deformation of statistics emerges in these systems. We discuss a no-go theorem on analytical continuation between real- and imaginary-rotating theories. Finally, we compute the moment of inertia and shape deformation coefficients caused by the rotation of the relativistic bosonic gas., Comment: 40 pages, 22 figures; accepted for publication in PRD; fractalization video is available at https://youtu.be/Pk-S_10BM-k

Differentiable volumetric rendering is a powerful paradigm for 3D reconstruction and novel view synthesis. However, standard volume rendering approaches struggle with degenerate geometries in the case of limited viewpoint diversity, a common scenario in robotics applications. In this work, we propose to use the multi-view photometric objective from the self-supervised depth estimation literature as a geometric regularizer for volumetric rendering, significantly improving novel view synthesis without requiring additional information. Building upon this insight, we explore the explicit modeling of scene geometry using a generalist Transformer, jointly learning a radiance field as well as depth and light fields with a set of shared latent codes. We demonstrate that sharing geometric information across tasks is mutually beneficial, leading to improvements over single-task learning without an increase in network complexity. Our DeLiRa architecture achieves state-of-the-art results on the ScanNet benchmark, enabling high quality volumetric rendering as well as real-time novel view and depth synthesis in the limited viewpoint diversity setting., Comment: Project page: https://sites.google.com/view/tri-delira

We present CARTO, a novel approach for reconstructing multiple articulated objects from a single stereo RGB observation. We use implicit object-centric representations and learn a single geometry and articulation decoder for multiple object categories. Despite training on multiple categories, our decoder achieves a comparable reconstruction accuracy to methods that train bespoke decoders separately for each category. Combined with our stereo image encoder we infer the 3D shape, 6D pose, size, joint type, and the joint state of multiple unknown objects in a single forward pass. Our method achieves a 20.4% absolute improvement in mAP 3D IOU50 for novel instances when compared to a two-stage pipeline. Inference time is fast and can run on a NVIDIA TITAN XP GPU at 1 HZ for eight or less objects present. While only trained on simulated data, CARTO transfers to real-world object instances. Code and evaluation data is available at: http://carto.cs.uni-freiburg.de, Comment: 20 pages, 11 figures, accepted at CVPR 2023

3D object detection from visual sensors is a cornerstone capability of robotic systems. State-of-the-art methods focus on reasoning and decoding object bounding boxes from multi-view camera input. In this work we gain intuition from the integral role of multi-view consistency in 3D scene understanding and geometric learning. To this end, we introduce VEDet, a novel 3D object detection framework that exploits 3D multi-view geometry to improve localization through viewpoint awareness and equivariance. VEDet leverages a query-based transformer architecture and encodes the 3D scene by augmenting image features with positional encodings from their 3D perspective geometry. We design view-conditioned queries at the output level, which enables the generation of multiple virtual frames during training to learn viewpoint equivariance by enforcing multi-view consistency. The multi-view geometry injected at the input level as positional encodings and regularized at the loss level provides rich geometric cues for 3D object detection, leading to state-of-the-art performance on the nuScenes benchmark. The code and model are made available at https://github.com/TRI-ML/VEDet., Comment: 11 pages, 4 figures; accepted to CVPR 2023