Search

Cross-modal transformers have demonstrated superiority in various vision tasks by effectively integrating different modalities. This paper first critiques prior token exchange methods which replace less informative tokens with inter-modal features, and demonstrate exchange based methods underperform cross-attention mechanisms, while the computational demand of the latter inevitably restricts its use with longer sequences. To surmount the computational challenges, we propose GeminiFusion, a pixel-wise fusion approach that capitalizes on aligned cross-modal representations. GeminiFusion elegantly combines intra-modal and inter-modal attentions, dynamically integrating complementary information across modalities. We employ a layer-adaptive noise to adaptively control their interplay on a per-layer basis, thereby achieving a harmonized fusion process. Notably, GeminiFusion maintains linear complexity with respect to the number of input tokens, ensuring this multimodal framework operates with efficiency comparable to unimodal networks. Comprehensive evaluations across multimodal image-to-image translation, 3D object detection and arbitrary-modal semantic segmentation tasks, including RGB, depth, LiDAR, event data, etc. demonstrate the superior performance of our GeminiFusion against leading-edge techniques. The PyTorch code is available at https://github.com/JiaDingCN/GeminiFusion, Comment: Accepted by ICML 2024, code and models are available at https://github.com/JiaDingCN/GeminiFusion

When electrons moving in two-dimensions (2D) are subjected to a strong uniform magnetic field, they form flat bands called Landau levels, which are the basis for the quantum Hall effect. Landau levels can also arise from pseudomagnetic fields (PMFs) induced by lattice distortions; for example, mechanically straining graphene causes its Dirac quasiparticles to form a characteristic set of unequally-spaced Landau levels, including a zeroth Landau level. In three-dimensional (3D) systems, there has thus far been no experimental demonstration of Landau levels or any other type of flat band. For instance, applying a uniform magnetic field to materials hosting Weyl quasiparticles, the 3D generalizations of Dirac quasiparticles, yields bands that are non-flat in the direction of the field. Here, we report on the experimental realization of a flat 3D Landau level in an acoustic crystal. Starting from a lattice whose bandstructure exhibits a nodal ring, we design an inhomogeneous distortion corresponding to a specific pseudomagnetic vector potential (PVP) that causes the nodal ring states to break up into Landau levels, with a zeroth Landau level that is flat along all three directions. These findings point to the possibility of using nodal ring materials to generate 3D flat bands, to access strong interactions and other interesting physical regimes in 3D.

A theory of everything should not only tell us the laws for matter, gravity, and possibly boundary condition for the universe. In addition, it should specify the relation between theory and experience. Here I argue for a minimal prescription in extracting empirical predictions from path integrals by showing that alternative prescriptions are unjustifiable. In this minimal prescription, the relative probability for one experience is obtained by summing over all configurations compatible with that experience, without any further restriction associated with other experiences of the same or other experiential beings. An application to Wigner's friend settings shows that quantum theory admits objective predictions for subjective experiences. Still, quantum theory differs from classical theory in offering individualized as opposed to collective accounts of experiences. This consideration of experience in fundamental theories issues several challenges to popular quantum interpretations, and points to the outstanding need for a theory of experience in understanding physical theories of everything., Comment: 27 pages, 6 figures; contains criticisms of several popular quantum interpretations, comments welcome

The DEtection TRansformer (DETR) algorithm has received considerable attention in the research community and is gradually emerging as a mainstream approach for object detection and other perception tasks. However, the current field lacks a unified and comprehensive benchmark specifically tailored for DETR-based models. To address this issue, we develop a unified, highly modular, and lightweight codebase called detrex, which supports a majority of the mainstream DETR-based instance recognition algorithms, covering various fundamental tasks, including object detection, segmentation, and pose estimation. We conduct extensive experiments under detrex and perform a comprehensive benchmark for DETR-based models. Moreover, we enhance the performance of detection transformers through the refinement of training hyper-parameters, providing strong baselines for supported algorithms.We hope that detrex could offer research communities a standardized and unified platform to evaluate and compare different DETR-based models while fostering a deeper understanding and driving advancements in DETR-based instance recognition. Our code is available at https://github.com/IDEA-Research/detrex. The project is currently being actively developed. We encourage the community to use detrex codebase for further development and contributions., Comment: project link: https://github.com/IDEA-Research/detrex

As a hypothetical topological defect in the geometry of spacetime, vortex strings play a crucial role in shaping the clusters of galaxies that exist today, and their distinct features can provide observable clues about the early universe's evolution. A key feature of vortex strings is that they can interact with Weyl fermionic modes and support topological chiral-anomaly states with massless dispersions at the core of strings. To date, despite many attempts to detect vortex strings in astrophysics or to emulate them in artificially created systems, observation of these topological vortex-string chiral modes remains experimentally elusive. Here we report the experimental observation of such vortex-string chiral modes using a metamaterial system. This is implemented by inhomogeneous perturbation of a Yang-monopole phononic metamaterial. The measured linear dispersion and modal profiles confirm the existence of topological modes bound to and propagating along the vortex string with the chiral anomaly. Our work not only provides a platform for studying diverse cosmic topological defects in astrophysics but also offers intriguing device applications as topological fibres in signal processing and communication techniques., Comment: 3 Figures

Quantum cosmology based on Lorentzian path integrals is a promising avenue. However, many previous works allow non-Lorentzian configurations by integrating the squared scale factor over the whole real line. Here we show that restricting the minisuperspace path integral to Lorentzian configurations with positive squared scale factor can significantly change the expectation values. In addition, this enables the study of causal horizons and their quantum fluctuations, and achieves singularity avoidance trivially by excluding singular minisuperspace geometries as non-Lorentzian. The results indicate that semiclassical saddle point approximation is not always valid in truly Lorentzian quantum cosmology. As a consequence, related works on the tunnelling and no-boundary proposals, bouncing cosmology, and the quantum origin of inflation etc. need to be reexamined., Comment: 28 pages, 10 figures. Comments welcome!

Vision transformers have recently achieved competitive results across various vision tasks but still suffer from heavy computation costs when processing a large number of tokens. Many advanced approaches have been developed to reduce the total number of tokens in large-scale vision transformers, especially for image classification tasks. Typically, they select a small group of essential tokens according to their relevance with the class token, then fine-tune the weights of the vision transformer. Such fine-tuning is less practical for dense prediction due to the much heavier computation and GPU memory cost than image classification. In this paper, we focus on a more challenging problem, i.e., accelerating large-scale vision transformers for dense prediction without any additional re-training or fine-tuning. In response to the fact that high-resolution representations are necessary for dense prediction, we present two non-parametric operators, a token clustering layer to decrease the number of tokens and a token reconstruction layer to increase the number of tokens. The following steps are performed to achieve this: (i) we use the token clustering layer to cluster the neighboring tokens together, resulting in low-resolution representations that maintain the spatial structures; (ii) we apply the following transformer layers only to these low-resolution representations or clustered tokens; and (iii) we use the token reconstruction layer to re-create the high-resolution representations from the refined low-resolution representations. The results obtained by our method are promising on five dense prediction tasks, including object detection, semantic segmentation, panoptic segmentation, instance segmentation, and depth estimation., Comment: Accepted at NeurIPS 2022, camera-ready version, 22 pages, 14 figures

A central open problem of quantum physics is to reconcile theory with experience. In this work I present a framework for studying distinct modes of experience in a superposed world. A modes of experience is characterized by how the world, experiences, and options relate to each other by the perceptions, decisions, and actions, as well as by probabilistic rules encoding probabilistic or deterministic correlations among first person experiences. In a toy model, the life expectancies of beings in different candidate modes of experience are compared. It is found that the quantum mode without macroscopic superposition outlives that with macroscopic superposition and that with real amplitudes. These highlight the prospect to explain a mode of experience by its evolutionary advantages., Comment: 42 pages, 33 figures. Comments welcome!

In several approaches of non-perturbative quantum gravity, a major outstanding problem is to obtain results valid at the infinite lattice refinement limit. Working with Lorentzian simplicial quantum gravity, we compute light ray fluctuation probabilities in 3D and 4D across different lattices. In a simplified refined box model with the Einstein-Hilbert action, numerical results show that lattice refinement does not simply suppress or simply enhance light ray fluctuations, but actually drives very wide and very narrow light probability distributions towards intermediate ones. A comparison across lattices and across couplings reveals numerical hints at a lattice refinement fixed point associated with a universality class of couplings. The results fit the intuition that quantum spacetime fluctuations reflected by light ray fluctuations start wild microscopically and become mild macroscopically. The refined box model is limited by the assumption of a rigid frame at all scales. The present results suggest further studies around the zero-coupling limit to relax the simplifying assumptions of the model., Comment: 29 pages, 21 figures

One-to-one set matching is a key design for DETR to establish its end-to-end capability, so that object detection does not require a hand-crafted NMS (non-maximum suppression) to remove duplicate detections. This end-to-end signature is important for the versatility of DETR, and it has been generalized to broader vision tasks. However, we note that there are few queries assigned as positive samples and the one-to-one set matching significantly reduces the training efficacy of positive samples. We propose a simple yet effective method based on a hybrid matching scheme that combines the original one-to-one matching branch with an auxiliary one-to-many matching branch during training. Our hybrid strategy has been shown to significantly improve accuracy. In inference, only the original one-to-one match branch is used, thus maintaining the end-to-end merit and the same inference efficiency of DETR. The method is named H-DETR, and it shows that a wide range of representative DETR methods can be consistently improved across a wide range of visual tasks, including DeformableDETR, PETRv2, PETR, and TransTrack, among others. The code is available at: https://github.com/HDETR, Comment: CVPR 2023. The code is available at: https://github.com/HDETR

It is often said that in relativistic quantum physics, a fundamental particle ontology is not viable. However, previous works have shown that certain relativistic quantum field theories can be reformulated as path integrals over particle configurations. Drawing on these works, I argue that a fundamental particle ontology is actually viable in relativistic quantum physics., Comment: 14 pages, 5 figures; comments welcome

Many non-trivial ideas have been proposed to resolve singularities in quantum gravity. In this short note I argue that singularity resolution can be trivial in gravitational path integrals, because geodesically incomplete singular spacetimes are usually not included in the sum. For theories where this holds, there is no need to develop non-trivial ideas on singularity resolution. Instead, efforts should better be directed to understand tunneling processes and complex-valued spacetimes., Comment: 6 pages, 2 figures, comments welcome

A non-perturbative study on the quantum fluctuations of light ray propagation through a quantum region of spacetime is long overdue. Within the theory of Lorentzian simplicial quantum gravity, we compute the probabilities for a test light ray to land at different locations after travelling through a symmetry-reduced box region in 2,3 and 4 spacetime dimensions. It is found that for fixed boundary conditions, light ray fluctuations are generically large when all coupling constants are relatively small in absolute value. For fixed coupling constants, as the boundary size is decreased light ray fluctuations first increase and then decrease in a 2D theory with the cosmological constant, Einstein-Hilbert and R-squared terms. While in 3D and 4D theories with the cosmological constant and Einstein-Hilbert terms, as the boundary size is decreased light ray fluctuations just increase. Incidentally, when studying 2D quantum gravity we show that the global time-space duality with the cosmological constant and Einstein-Hilbert terms noted previously also holds when arbitrary even powers of the Ricci scalar are added. We close by discussing how light ray fluctuations can be used in obtaining the continuum limit of non-perturbative Lorentzian quantum gravity., Comment: 41 pages, 42 figures, matches published version well

Evaluating gravitational path integrals in the Lorentzian has been a long-standing challenge due to the numerical sign problem. We show that this challenge can be overcome in simplicial quantum gravity. By deforming the integration contour into the complex, the sign fluctuations can be suppressed, for instance using the holomorphic gradient flow algorithm. Working through simple models, we show that this algorithm enables efficient Monte Carlo simulations for Lorentzian simplicial quantum gravity. In order to allow complex deformations of the integration contour, we provide a manifestly holomorphic formula for Lorentzian simplicial gravity. This leads to a complex version of simplicial gravity that generalizes the Euclidean and Lorentzian cases. Outside the context of numerical computation, complex simplicial gravity is also relevant to studies of singularity resolving processes with complex semi-classical solutions. Along the way, we prove a complex version of the Gauss-Bonnet theorem, which may be of independent interest., Comment: matches well published version

Although the Standard Model of particle physics is usually formulated in terms of fields, it can be equivalently formulated in terms of particles and strings. In this picture particles and open strings are always coupled. This offers an intuitive and graphical explanation for the otherwise mysterious gauge symmetry. In addition, the particle-string formulation avoids introducing redundant path integral configurations that are present in the field formulation. For its explanatory power and economy, the particle-string ontology may be preferred over the field ontology for the Standard Model., Comment: 19 pages, 7 figures, published version

An important task faced by all approaches of quantum gravity is to incorporate superpositions and quantify quantum uncertainties of spacetime causal relations. We address this task in 2D. By identifying a global $Z_2$ symmetry of 1+1D quantum gravity, we show that gravitational path integral configurations come in equal amplitude pairs with timelike and spacelike relations exchanged. As a consequence, any two points are equally probable to be timelike and spacelike separated in a universe without boundary conditions. In the context of simplicial quantum gravity we identify a local symmetry of the action which shows that even with boundary conditions causal uncertainties are generically present. Depending on the boundary conditions, causal uncertainties can still be large and even maximal., Comment: matches well published version

In the past few years, transformers have achieved promising performances on various computer vision tasks. Unfortunately, the immense inference overhead of most existing vision transformers withholds their from being deployed on edge devices such as cell phones and smart watches. Knowledge distillation is a widely used paradigm for compressing cumbersome architectures via transferring information to a compact student. However, most of them are designed for convolutional neural networks (CNNs), which do not fully investigate the character of vision transformer (ViT). In this paper, we utilize the patch-level information and propose a fine-grained manifold distillation method. Specifically, we train a tiny student model to match a pre-trained teacher model in the patch-level manifold space. Then, we decouple the manifold matching loss into three terms with careful design to further reduce the computational costs for the patch relationship. Equipped with the proposed method, a DeiT-Tiny model containing 5M parameters achieves 76.5% top-1 accuracy on ImageNet-1k, which is +2.0% higher than previous distillation approaches. Transfer learning results on other classification benchmarks and downstream vision tasks also demonstrate the superiority of our method over the state-of-the-art algorithms.

The interplay between real-space topological lattice defects and the reciprocal-space topology of energy bands can give rise to novel phenomena, such as one-dimensional topological modes bound to screw dislocations in three-dimensional topological insulators. We obtain direct experimental observations of dislocation-induced helical modes in an acoustic analog of a weak three-dimensional topological insulator. The spatial distribution of the helical modes is found through spin-resolved field mapping, and verified numerically by tight-binding and finite-element calculations. These one-dimensional helical channels can serve as robust waveguides in three-dimensional media. Our experiment paves the way to studying novel physical modes and functionalities enabled by topological lattice defects in three-dimensional classical topological materials., Comment: 6 pages, 4 figures

The recently discovered non-Hermitian skin effect (NHSE) manifests the breakdown of current classification of topological phases in energy-nonconservative systems, and necessitates the introduction of non-Hermitian band topology. So far, all NHSE observations are based on one type of non-Hermitian band topology, in which the complex energy spectrum winds along a closed loop. As recently characterized along a synthetic dimension on a photonic platform, non-Hermitian band topology can exhibit almost arbitrary windings in momentum space, but their actual phenomena in real physical systems remain unclear. Here, we report the experimental realization of NHSE in a one-dimensional (1D) non-reciprocal acoustic crystal. With direct acoustic measurement, we demonstrate that a twisted winding, whose topology consists of two oppositely oriented loops in contact rather than a single loop, will dramatically change the NHSE, following previous predictions of unique features such as the bipolar localization and the Bloch point for a Bloch-wave-like extended state. This work reveals previously unnoticed features of NHSE, and provides the observation of physical phenomena originating from complex non-Hermitian winding topology., Comment: 14 pages, 4 figures

The phenomenon of negative refraction generally requires negative refractive indices or phase discontinuities, which can be realized using metamaterials or metasurfaces. Recent theories have proposed a novel mechanism for negative refraction based on synthetic gauge fields, which affect classical waves as if they were charged particles in electromagnetic fields, but this has not hitherto been demonstrated in experiment. Here, we report on the experimental demonstration of gauge-field-induced negative refraction in a twisted bilayer acoustic metamaterial. The bilayer twisting produces a synthetic gauge field for sound waves propagating within a projected two-dimensional geometry, with the magnitude of the gauge field parameterized by the choice of wavenumber along the third dimension. Waveguiding with backward propagating modes is also demonstrated in a trilayer configuration that implements strong gauge fields. These results provide an alternative route to achieving negative refraction in synthetic materials.

We design and implement a three dimensional acoustic Weyl metamaterial hosting robust modes bound to a one-dimensional topological lattice defect. The modes are related to topological features of the bulk bands, and carry nonzero orbital angular momentum locked to the direction of propagation. They span a range of axial wavenumbers defined by the projections of two bulk Weyl points to a one-dimensional subspace, in a manner analogous to the formation of Fermi arc surface states. We use acoustic experiments to probe their dispersion relation, orbital angular momentum locked waveguiding, and ability to emit acoustic vortices into free space. These results point to new possibilities for creating and exploiting topological modes in three-dimensional structures through the interplay between band topology in momentum space and topological lattice defects in real space.

Indefinite causal structure is generically present in theories of quantum gravity admitting a path integral formulation. We show that summing over causal structures eliminates ultraviolet divergences of matter QFT and resolves spacetime singularities using the non-perturbative World Quantum Gravity approach. Independent information-theoretic and model-independent considerations suggest that the mechanism of ultraviolet regularization by indefinite causal structure also applies to other theories of quantum gravity., Comment: 5 pages, 2 figures. Comments are welcome!

A correlational dialect is introduced within the quantum theory language to give a unified treatment of finite-dimensional informational/operational quantum theories, infinite-dimensional relativistic quantum theories, and quantum gravity. Theories are written in terms of correlation diagrams which specify correlation types and weights. Grouping similar correlation diagrams leads to generalized Feynman diagrams, which in special cases reduce to the familiar Feynman diagrams from quantum field theories. The correlational formalism is applied in a study of correlation constraints, revealing new classes of quantum processes that evade previous characterizations of general quantum processes. The results apply to quantum theories of various kinds, including time-asymmetric theories, time-symmetric theories, theories without predefined time, and theories with indefinite causal structures., Comment: v2 with updated discussions on Feynman diagrams; 6 pages, 2 figures, plus appendix

The Berry phase associated with energy bands in crystals can lead to quantized quantities, such as the quantization of electric dipole polarization in an insulator, known as a one-dimensional (1D) topological insulator (TI) phase. Recent theories have generalized such quantization from dipole to higher multipole moments, giving rise to the discovery of multipole TIs, which exhibit a cascade hierarchy of multipole topology at boundaries of boundaries: A quantized octupole moment in the three-dimensional (3D) bulk can induce quantized quadrupole moments on its two-dimensional (2D) surfaces, which then produce quantized dipole moments along 1D hinges. The model of 2D quadrupole TI has been realized in various classical structures, exhibiting zero-dimensional (0D) in-gap corner states. Here we report on the realization of a quantized octupole TI on the platform of a 3D acoustic metamaterial. By direct acoustic measurement, we observe 0D corner states, 1D hinge states, 2D surface states, and 3D bulk states, as a consequence of the topological hierarchy from octupole moment to quadrupole and dipole moment. The critical conditions of forming a nontrivial octupole moment are further demonstrated by comparing with another two samples possessing a trivial octupole moment. Our work thus establishes the multipole topology and its full cascade hierarchy in 3D geometries.

A new path integral approach of quantum gravity based on relational variables and quantum test objects is presented. We take as a basic variables the squared invariant distance. This invariant quantity is technically simpler to work with than variant quantities such as the metric tensor. It also facilitates the studies of matter coupling and quantum spacetime causal structures. In contrast to approaches based on piecewise linear geometries, here gravity is captured by its effects on quantum test particles and fields. By an observation of Parker, under a Feynman sum a gravitational phase can be traded into a Van Vleck-Morette determinant term. This leads to a new candidate path integral for gravity, which can potentially be computed efficient in the Lorentzian signature. We discuss some ambiguities left in the path integral measure, which invite further clarifications., Comment: mathces well published version

In fundamental theories that accounts for quantum gravitational effects, the spacetime causal structure is expected to be quantum uncertain. Previous studies of quantum causal structure focused on finite-dimensional systems. Here we present an algebraic framework that incorporates both finite- and infinite-dimensional systems including quantum fields. Thanks to the absence of a definite spacetime causal structure, Lagrangian quantum field theories can be studied on a quantum superposition of spacetimes with a point identification structure., Comment: 6 pages, 1 figure

In theories of communication, it is usually presumed that the involved parties perform actions in a fixed causal order. However, practical and fundamental reasons can induce uncertainties in the causal order. Here we show that a maximal uncertainty in the causal order forbids asymptotic quantum communication, while still enabling the noisy transfer of classical information. Therefore causal order, like shared entanglement, is an additional resource for communication. The result is formulated within an asymptotic setting for processes with no fixed causal order, which sets a basis for a quantum information theory in general quantum causal structures., Comment: 5 pages, 1 figure