Search

Ensuring safe, comfortable, and efficient navigation is a critical goal for autonomous driving systems. While end-to-end models trained on large-scale datasets excel in common driving scenarios, they often struggle with rare, long-tail events. Recent progress in large language models (LLMs) has introduced enhanced reasoning capabilities, but their computational demands pose challenges for real-time decision-making and precise planning. This paper presents FASIONAD, a novel dual-system framework inspired by the cognitive model "Thinking, Fast and Slow." The fast system handles routine navigation tasks using rapid, data-driven path planning, while the slow system focuses on complex reasoning and decision-making in challenging or unfamiliar situations. A dynamic switching mechanism based on score distribution and feedback allows seamless transitions between the two systems. Visual prompts generated by the fast system enable human-like reasoning in the slow system, which provides high-quality feedback to enhance the fast system's decision-making. To evaluate FASIONAD, we introduce a new benchmark derived from the nuScenes dataset, specifically designed to differentiate fast and slow scenarios. FASIONAD achieves state-of-the-art performance on this benchmark, establishing a new standard for frameworks integrating fast and slow cognitive processes in autonomous driving. This approach paves the way for more adaptive, human-like autonomous driving systems.

Large Language Models (LLMs) and Reinforcement Learning (RL) are two powerful approaches for building autonomous agents. However, due to limited understanding of the game environment, agents often resort to inefficient exploration and trial-and-error, struggling to develop long-term strategies or make decisions. We propose a method that extracts experience from interaction records to model the underlying laws of the game environment, using these experience as internal motivation to guide agents. These experience, expressed in language, are highly flexible and can either assist agents in reasoning directly or be transformed into rewards for guiding training. Our evaluation results in Crafter demonstrate that both RL and LLM agents benefit from these experience, leading to improved overall performance.

Symmetry is ubiquitous in many real-world phenomena and tasks, such as physics, images, and molecular simulations. Empirical studies have demonstrated that incorporating symmetries into generative models can provide better generalization and sampling efficiency when the underlying data distribution has group symmetry. In this work, we provide the first theoretical analysis and guarantees of score-based generative models (SGMs) for learning distributions that are invariant with respect to some group symmetry and offer the first quantitative comparison between data augmentation and adding equivariant inductive bias. First, building on recent works on the Wasserstein-1 ($\mathbf{d}_1$) guarantees of SGMs and empirical estimations of probability divergences under group symmetry, we provide an improved $\mathbf{d}_1$ generalization bound when the data distribution is group-invariant. Second, we describe the inductive bias of equivariant SGMs using Hamilton-Jacobi-Bellman theory, and rigorously demonstrate that one can learn the score of a symmetrized distribution using equivariant vector fields without data augmentations through the analysis of the optimality and equivalence of score-matching objectives. This also provides practical guidance that one does not have to augment the dataset as long as the vector field or the neural network parametrization is equivariant. Moreover, we quantify the impact of not incorporating equivariant structure into the score parametrization, by showing that non-equivariant vector fields can yield worse generalization bounds. This can be viewed as a type of model-form error that describes the missing structure of non-equivariant vector fields. Numerical simulations corroborate our analysis and highlight that data augmentations cannot replace the role of equivariant vector fields.

Recently, NVS in human-object interaction scenes has received increasing attention. Existing human-object interaction datasets mainly consist of static data with limited views, offering only RGB images or videos, mostly containing interactions between a single person and objects. Moreover, these datasets exhibit complexities in lighting environments, poor synchronization, and low resolution, hindering high-quality human-object interaction studies. In this paper, we introduce a new people-object interaction dataset that comprises 38 series of 30-view multi-person or single-person RGB-D video sequences, accompanied by camera parameters, foreground masks, SMPL models, some point clouds, and mesh files. Video sequences are captured by 30 Kinect Azures, uniformly surrounding the scene, each in 4K resolution 25 FPS, and lasting for 1$\sim$19 seconds. Meanwhile, we evaluate some SOTA NVS models on our dataset to establish the NVS benchmarks. We hope our work can inspire further research in humanobject interaction.

We introduce OmniRe, a holistic approach for efficiently reconstructing high-fidelity dynamic urban scenes from on-device logs. Recent methods for modeling driving sequences using neural radiance fields or Gaussian Splatting have demonstrated the potential of reconstructing challenging dynamic scenes, but often overlook pedestrians and other non-vehicle dynamic actors, hindering a complete pipeline for dynamic urban scene reconstruction. To that end, we propose a comprehensive 3DGS framework for driving scenes, named OmniRe, that allows for accurate, full-length reconstruction of diverse dynamic objects in a driving log. OmniRe builds dynamic neural scene graphs based on Gaussian representations and constructs multiple local canonical spaces that model various dynamic actors, including vehicles, pedestrians, and cyclists, among many others. This capability is unmatched by existing methods. OmniRe allows us to holistically reconstruct different objects present in the scene, subsequently enabling the simulation of reconstructed scenarios with all actors participating in real-time (~60Hz). Extensive evaluations on the Waymo dataset show that our approach outperforms prior state-of-the-art methods quantitatively and qualitatively by a large margin. We believe our work fills a critical gap in driving reconstruction., Comment: See the project page for code, video results and demos: https://ziyc.github.io/omnire/

This paper demonstrates the robustness of Lipschitz-regularized $\alpha$-divergences as objective functionals in generative modeling, showing they enable stable learning across a wide range of target distributions with minimal assumptions. We establish that these divergences remain finite under a mild condition-that the source distribution has a finite first moment-regardless of the properties of the target distribution, making them adaptable to the structure of target distributions. Furthermore, we prove the existence and finiteness of their variational derivatives, which are essential for stable training of generative models such as GANs and gradient flows. For heavy-tailed targets, we derive necessary and sufficient conditions that connect data dimension, $\alpha$, and tail behavior to divergence finiteness, that also provide insights into the selection of suitable $\alpha$'s. We also provide the first sample complexity bounds for empirical estimations of these divergences on unbounded domains. As a byproduct, we obtain the first sample complexity bounds for empirical estimations of these divergences and the Wasserstein-1 metric with group symmetry on unbounded domains. Numerical experiments confirm that generative models leveraging Lipschitz-regularized $\alpha$-divergences can stably learn distributions in various challenging scenarios, including those with heavy tails or complex, low-dimensional, or fractal support, all without any prior knowledge of the structure of target distributions., Comment: 43 pages, 6 figures and 2 tables in the main text

The ever-growing textual records of contemporary social issues, often discussed online with moral rhetoric, present both an opportunity and a challenge for studying how moral concerns are debated in real life. Moral foundations theory is a taxonomy of intuitions widely used in data-driven analyses of online content, but current computational tools to detect moral foundations suffer from the incompleteness and fragility of their lexicons and from poor generalization across data domains. In this paper, we fine-tune a large language model to measure moral foundations in text based on datasets covering news media and long- and short-form online discussions. The resulting model, called Mformer, outperforms existing approaches on the same domains by 4--12% in AUC and further generalizes well to four commonly used moral text datasets, improving by up to 17% in AUC. We present case studies using Mformer to analyze everyday moral dilemmas on Reddit and controversies on Twitter, showing that moral foundations can meaningfully describe people's stance on social issues and such variations are topic-dependent. Pre-trained model and datasets are released publicly. We posit that Mformer will help the research community quantify moral dimensions for a range of tasks and data domains, and eventually contribute to the understanding of moral situations faced by humans and machines., Comment: To be published in ICWSM 2024

360$^\circ$ panoramas are extensively utilized as environmental light sources in computer graphics. However, capturing a 360$^\circ$ $\times$ 180$^\circ$ panorama poses challenges due to the necessity of specialized and costly equipment, and additional human resources. Prior studies develop various learning-based generative methods to synthesize panoramas from a single Narrow Field-of-View (NFoV) image, but they are limited in alterable input patterns, generation quality, and controllability. To address these issues, we propose a novel pipeline called PanoDiff, which efficiently generates complete 360$^\circ$ panoramas using one or more unregistered NFoV images captured from arbitrary angles. Our approach has two primary components to overcome the limitations. Firstly, a two-stage angle prediction module to handle various numbers of NFoV inputs. Secondly, a novel latent diffusion-based panorama generation model uses incomplete panorama and text prompts as control signals and utilizes several geometric augmentation schemes to ensure geometric properties in generated panoramas. Experiments show that PanoDiff achieves state-of-the-art panoramic generation quality and high controllability, making it suitable for applications such as content editing., Comment: Accepted to ACM Multimedia 2023 (MM' 23). Code is available: https://github.com/shanemankiw/Panodiff

Group-invariant generative adversarial networks (GANs) are a type of GANs in which the generators and discriminators are hardwired with group symmetries. Empirical studies have shown that these networks are capable of learning group-invariant distributions with significantly improved data efficiency. In this study, we aim to rigorously quantify this improvement by analyzing the reduction in sample complexity for group-invariant GANs. Our findings indicate that when learning group-invariant distributions, the number of samples required for group-invariant GANs decreases proportionally by a factor of the group size. Importantly, this sample complexity reduction cannot be achieved merely through data augmentation due to the probabilistic dependence of augmented data. Numerical results substantiate our theory and highlight the stark contrast between learning with group-invariant GANs and using data augmentation. This work presents the first statistical performance guarantees for group-invariant generative models, specifically for GANs, and it may shed light on the study of other generative models with group symmetries., Comment: v3: added numerical experiments and comparison with data augmentations; v2: rephrased some paragraphs

Magnetic double perovskite compounds provide a fertile playground to explore interesting electronic and magnetic properties. By complementary macroscopic characterizations, neutron powder diffraction measurements and first-principles calculations, we have performed comprehensive studies on the magnetic ordering in the double perovskite compound Sr$_2$NiWO$_6$. It is found by neutron diffraction to order magnetically in a collinear type-II antiferromagnetic structure in a tetragonal lattice with $k$ = (0.5, 0, 0.5) below $T\rm_N$ = 56 K. In the ground state, the ordered moment of the spin-1 Ni$^{2+}$ ions is determined to be 1.9(2) $\mu\rm_{B}$, indicating a significant quenching of the orbital moment. The Ni$^{2+}$ moments in Sr$_2$NiWO$_6$ are revealed to cant off the $c$ axis by 29.2$^{\circ}$, which is well supported by the first-principles magnetic anisotropy energy calculations. Furthermore, the in-plane and out-of-plane next-nearest-neighbor superexchange couplings ($J\rm_2$ and $J\rm_{2c}$) are found to play a dominant role in the spin Hamiltonian of Sr$_2$NiWO$_6$, which accounts for the stabilization of the type-II AFM structure as its magnetic ground state., Comment: 10 Pages, 6 figures, accepted for publication in Physical Review B

We study the implicit bias of gradient flow on linear equivariant steerable networks in group-invariant binary classification. Our findings reveal that the parameterized predictor converges in direction to the unique group-invariant classifier with a maximum margin defined by the input group action. Under a unitary assumption on the input representation, we establish the equivalence between steerable networks and data augmentation. Furthermore, we demonstrate the improved margin and generalization bound of steerable networks over their non-invariant counterparts.

We rigorously quantify the improvement in the sample complexity of variational divergence estimations for group-invariant distributions. In the cases of the Wasserstein-1 metric and the Lipschitz-regularized $\alpha$-divergences, the reduction of sample complexity is proportional to the group size if the group is finite. In addition to the published version at ICML 2023, our proof indeed has included the case when the group is infinite such as compact Lie groups, the convergence rate can be further improved and depends on the intrinsic dimension of the fundamental domain characterized by the scaling of its covering number. Our approach is different from that in [Tahmasebi & Jegelka, ICML 2024] and our work also applies to asymmetric divergences, such as the Lipschitz-regularized $\alpha$-divergences. For the maximum mean discrepancy (MMD), the improvement of sample complexity is more nuanced, as it depends on not only the group size but also the choice of kernel. Numerical simulations verify our theories., Comment: In addition to our published version at ICML 2023, we include the case when the group is infinite such as compact Lie groups. Our approach is different from that in [Tahmasebi & Jegelka, ICML 2024] and our work also applies to asymmetric divergences, such as the Lipschitz-regularized $\alpha$-divergences

By performing experiment and model studies on a triangular-lattice dipolar magnet KBaGd(BO$_3$)$_2$ (KBGB), we find the highly frustrated magnet with a planar anisotropy hosts a strongly fluctuating dipolar spin liquid (DSL), which originates from the intriguing interplay between dipolar and Heisenberg interactions. The DSL constitutes an extended regime in the field-temperature phase diagram, which gets lowered in temperature as field increases and eventually ends with an unconventional quantum critical point (QCP) at $B_c\simeq 0.75$~T. Based on dipolar Heisenberg model calculations, we identify the DSL as a Berezinskii-Kosterlitz-Thouless (BKT) phase with emergent U(1) symmetry. Due to the tremendous entropy accumulation that can be related to the strong BKT and quantum fluctuations, unprecedented magnetic cooling effects are observed in the DSL regime and particularly near the QCP, making KBGB a superior dipolar coolant to commercial Gd-based refrigerants. We establish the phase diagram for triangular-lattice dipolar quantum magnets where emergent symmetry plays an essential role, and provide a basis and opens an avenue for their applications in sub-Kelvin refrigeration., Comment: 7 pages and 4 figures in main text, 5 pages and 4 figures in Supplementary Materials

The study of higher-order and real topological states as well as the material realization have become a research forefront of topological condensed matter physics in recent years. Twisted bilayer graphene (tbG) is proved to have higher-order and real topology. However whether this conclusion can be extended to other two-dimensional twisted bilayer carbon materials and the mechanism behind it lack explorations. In this paper, we identify the twisted bilayer $\alpha$-graphyne (tbGPY) at large twisting angle as a real Chern insulator (also known as Stiefel-Whitney insulator) and a second-order topological insulator. Our first-principles calculations suggest that the tbGPY at 21.78$^\circ$ is stable at 100 K with a larger bulk gap than the tbG. The non-trivial topological indicators, including the real Chern number and a fractional charge, and the localized in-gap corner states are demonstrated from first-principles and tight-binding calculations. Moreover, with $\mathcal C_{6z}$ symmetry, we prove the equivalence between the two indicators, and explain the existence of the corner states. To decipher the real and higher-order topology inherited from the Moir\'e heterostructure, we construct an effective four band tight-binding model capturing the topology and dispersion of the tbGPY at large twisting angle. A topological phase transition to a trivial insulator is demonstrated by breaking the $\mathcal C_{2y}$ symmetry of the effective model, which gives insights on the trivialization of the tbGPY as reducing the twisting angle to 9.43$^\circ$ suggested by our first-principles calculations.

Spectral methods which represent data points by eigenvectors of kernel matrices or graph Laplacian matrices have been a primary tool in unsupervised data analysis. In many application scenarios, parametrizing the spectral embedding by a neural network that can be trained over batches of data samples gives a promising way to achieve automatic out-of-sample extension as well as computational scalability. Such an approach was taken in the original paper of SpectralNet (Shaham et al. 2018), which we call SpecNet1. The current paper introduces a new neural network approach, named SpecNet2, to compute spectral embedding which optimizes an equivalent objective of the eigen-problem and removes the orthogonalization layer in SpecNet1. SpecNet2 also allows separating the sampling of rows and columns of the graph affinity matrix by tracking the neighbors of each data point through the gradient formula. Theoretically, we show that any local minimizer of the new orthogonalization-free objective reveals the leading eigenvectors. Furthermore, global convergence for this new orthogonalization-free objective using a batch-based gradient descent method is proved. Numerical experiments demonstrate the improved performance and computational efficiency of SpecNet2 on simulated data and image datasets.

The effects of Sr doping on the structural properties of Na$_{2}$BaCo(PO$_{4}$)$_{2}$, a spin-1/2 triangular-lattice antiferromagnet as a quantum spin liquid candidate, are investigated by complementary x-ray and neutron powder diffraction measurements. It is found that in Na$_{2}$Ba$_{1-x}$Sr$_{x}$Co(PO$_{4}$)$_{2}$ (NBSCPO), the trigonal phase (space group $\mathit{P}$$\bar{3}$$\mathit{m}$1) with a perfect triangular lattice of Co$^{2+}$ ions is structurally stable when the doping level of Sr is below 30% ($\mathit{x}$ $\le$ 0.3), while a pure monoclinic phase (space group $\mathit{P}$2$_{1}$/$\mathit{a}$) with slight rotations of CoO$_{6}$ octahedra and displacements of Ba$^{2+}$/Sr$^{2+}$ ions will be established when the Sr doping level is above 60% ($\mathit{x}$ $\ge$ 0.6). Such a doping-induced structural transformation in NBSCPO is supported by first-principles calculations and Raman spectroscopy. Na$_{2}$SrCo(PO$_{4}$)$_{2}$, a novel spin-1/2 triangular-lattice antiferromagnet with glaserite-type structure, although monoclinically distorted, exhibits no long-range magnetic order down to 2 K and a similar negative Curie-Weiss temperature as Na$_{2}$BaCo(PO$_{4}$)$_{2}$ with a perfect triangular lattice, suggesting the robustness of magnetic exchange interaction against the Ba/Sr substitutions., Comment: 26 pages, 7 figures

Na$_{4-\delta}$NiTeO$_{6}$ is a rare example in the transition-metal tellurate family of realizing an $S$ = 1 spin-chain structure. By performing neutron powder diffraction measurements, the ground-state magnetic structure of Na$_{4-\delta}$NiTeO$_{6}$ is determined. These measurements reveal that below $T\rm_{N}$ ${\sim}$ 6.8(2) K, the Ni$^{2+}$ moments form a screwed ferromagnetic (FM) spin-chain structure running along the crystallographic $a$ axis but these FM spin chains are coupled antiferromagnetically along the $b$ and $c$ directions, giving rise to a magnetic propagation vector of $k$ = (0, 1/2, 1/2). This zigzag magnetic order is well supported by first-principles calculations. The moment size of Ni$^{2+}$ spins is determined to be 2.1(1) $\mu$$\rm_{B}$ at 3 K, suggesting a significant quenching of the orbital moment due to the crystalline electric field (CEF) effect. The previously reported metamagnetic transition near $H\rm_{C}$ ${\sim}$ 0.1 T can be understood as a field-induced spin-flip transition. The relatively easy tunability of the dimensionality of its magnetism by external parameters makes Na$_{4-\delta}$NiTeO$_{6}$ a promising candidate for further exploring various types of novel spin-chain physics., Comment: 10 pages, 6 figures

3D shape analysis has been widely explored in the era of deep learning. Numerous models have been developed for various 3D data representation formats, e.g., MeshCNN for meshes, PointNet for point clouds and VoxNet for voxels. In this study, we present Representation-Agnostic Shape Fields (RASF), a generalizable and computation-efficient shape embedding module for 3D deep learning. RASF is implemented with a learnable 3D grid with multiple channels to store local geometry. Based on RASF, shape embeddings for various 3D shape representations (point clouds, meshes and voxels) are retrieved by coordinate indexing. While there are multiple ways to optimize the learnable parameters of RASF, we provide two effective schemes among all in this paper for RASF pre-training: shape reconstruction and normal estimation. Once trained, RASF becomes a plug-and-play performance booster with negligible cost. Extensive experiments on diverse 3D representation formats, networks and applications, validate the universal effectiveness of the proposed RASF. Code and pre-trained models are publicly available https://github.com/seanywang0408/RASF, Comment: The Tenth International Conference on Learning Representations (ICLR 2022). Code is available at https://github.com/seanywang0408/RASF

Dirac semimetals (DSMs) are an important class of topological states of matter. Here, focusing on DSMs of band inversion type, we investigate their boundary modes from the effective model perspective. We show that in order to properly capture the boundary modes, $k$-cubic terms must be included in the effective model, which would drive an evolution of surface degeneracy manifold from a nodal line to a nodal point. Using first-principles calculations, we demonstrate that this feature and the topological hinge modes can be clearly exhibited in $\beta$-CuI. We further extend the discussion to magnetic DSMs and show that the time-reversal symmetry breaking can gap out the surface bands and hence help to expose the hinge modes in the spectrum, which could be beneficial for the experimental detection of hinge modes., Comment: 8 pages, 7 figures

Real topological phases featuring real Chern numbers and second-order boundary modes have been a focus of current research, but finding their material realization remains a challenge. Here, based on first-principles calculations and theoretical analysis, we reveal the already experimentally synthesized three-dimensional (3D) graphdiyne as the first realistic example of the recently proposed second-order real nodal-line semimetal. We show that the material hosts a pair of real nodal rings, each protected by two topological charges: a real Chern number and a 1D winding number. The two charges generate distinct topological boundary modes at distinct boundaries. The real Chern number leads to a pair of hinge Fermi arcs, whereas the winding number protects a double drumhead surface bands. We develop a low-energy model for 3D graphdiyne which captures the essential topological physics. Experimental aspects and possible topological transition to a 3D real Chern insulator phase are discussed., Comment: 8 pages, 4+4 figures. Accepted for publication in Physical Review Letters

Analysis of signals with oscillatory modes with crossover instantaneous frequencies is a challenging problem in time series analysis. One way to handle this problem is lifting the 2-dimensional time-frequency representation to a 3-dimensional representation, called time-frequency-chirp rate (TFC) representation, by adding one extra chirp rate parameter so that crossover frequencies are disentangled in higher dimension. The chirplet transform is an algorithm for this lifting idea, which leads to a TFC representation. However, in practice, we found that it has a strong ``blurring'' effect in the chirp rate axis, which limits its application in real-world data. Moreover, to our knowledge, we have limited mathematical understanding of the chirplet transform in the literature. Motivated by the need for the real-world data analysis, in this paper, we propose the synchrosqueezed chirplet transform (SCT) that enhances the TFC representation given by the chirplet transform. The resulting concentrated TFC representation has high contrast so that one can better distinguish different modes with crossover instantaneous frequencies. The basic idea is to use the phase information in the chirplet transform to determine a reassignment rule that sharpens the TFC representation determined by the chirplet transform. We also analyze the chirplet transform and provide theoretical guarantees of SCT., Comment: Fixed typos and added Remark 2

Background: Static Application Security Testing (SAST) tools purport to assist developers in detecting security issues in source code. These tools typically use rule-based approaches to scan source code for security vulnerabilities. However, due to the significant shortcomings of these tools (i.e., high false positive rates), learning-based approaches for Software Vulnerability Prediction (SVP) are becoming a popular approach. Aims: Despite the similar objectives of these two approaches, their comparative value is unexplored. We provide an empirical analysis of SAST tools and SVP models, to identify their relative capabilities for source code security analysis. Method: We evaluate the detection and assessment performance of several common SAST tools and SVP models on a variety of vulnerability datasets. We further assess the viability and potential benefits of combining the two approaches. Results: SAST tools and SVP models provide similar detection capabilities, but SVP models exhibit better overall performance for both detection and assessment. Unification of the two approaches is difficult due to lacking synergies. Conclusions: Our study generates 12 main findings which provide insights into the capabilities and synergy of these two approaches. Through these observations we provide recommendations for use and improvement., Comment: To be published in ESEM 21; reduced length