Search

Modern visual effects (VFX) software has made it possible for skilled artists to create imagery of virtually anything. However, the creation process remains laborious, complex, and largely inaccessible to everyday users. In this work, we present AutoVFX, a framework that automatically creates realistic and dynamic VFX videos from a single video and natural language instructions. By carefully integrating neural scene modeling, LLM-based code generation, and physical simulation, AutoVFX is able to provide physically-grounded, photorealistic editing effects that can be controlled directly using natural language instructions. We conduct extensive experiments to validate AutoVFX's efficacy across a diverse spectrum of videos and instructions. Quantitative and qualitative results suggest that AutoVFX outperforms all competing methods by a large margin in generative quality, instruction alignment, editing versatility, and physical plausibility., Comment: Project page: https://haoyuhsu.github.io/autovfx-website/

The cluster Gutzwiller method is widely used to study the strongly correlated bosonic systems, owing to its ability to provide a more precise description of quantum fluctuations. However, its utility is limited by the exponential increase in computational complexity as the cluster size grows. To overcome this limitation, we propose an artificial intelligence-based method known as $\Delta$-Learning. This approach constructs a predictive model by learning the discrepancies between lower-precision (small cluster sizes) and high-precision (large cluster sizes) implementations of the cluster Gutzwiller method, requiring only a small number of training samples. Using this predictive model, we can effectively forecast the outcomes of high-precision methods with high accuracy. Applied to various Bose-Hubbard models, the $\Delta$-Learning method effectively predicts phase diagrams while significantly reducing the computational resources and time. Furthermore, we have compared the predictive accuracy of $\Delta$-Learning with other direct learning methods and found that $\Delta$-Learning exhibits superior performance in scenarios with limited training data. Therefore, when combined with the cluster Gutzwiller approximation, the $\Delta$-Learning approach offers a computationally efficient and accurate method for studying phase transitions in large, complex bosonic systems.

Materials hosting flat electronic bands are a central focus of condensed matter physics as promising venues for novel electronic ground states. Two-dimensional (2D) geometrically frustrated lattices such as the kagome, dice, and Lieb lattices are attractive targets in this direction, anticipated to realize perfectly flat bands. Synthesizing these special structures, however, poses a formidable challenge, exemplified by the absence of solid-state materials realizing the dice and Lieb lattices. An alternative route leverages atomic orbitals to create the characteristic electron hopping of geometrically frustrated lattices. This strategy promises to expand the list of candidate materials to simpler structures, but is yet to be demonstrated experimentally. Here, we report the realization of frustrated hopping in the van der Waals (vdW) intermetallic Pd$_5$AlI$_2$, emerging from orbital decoration of a primitive square lattice. Using angle-resolved photoemission spectroscopy and quantum oscillations measurements, we demonstrate that the band structure of Pd$_5$AlI$_2$ includes linear Dirac-like bands intersected at their crossing point by a flat band, essential characteristics of frustrated hopping in the Lieb and dice lattices. Moreover, Pd$_5$AlI$_2$ is exceptionally stable, with the unusual bulk band structure and metallicity persisting in ambient conditions down to the monolayer limit. Our ability to realize an electronic structure characteristic of geometrically frustrated lattices establishes orbital decoration of primitive lattices as a new approach towards electronic structures that remain elusive to prevailing lattice-centric searches.

The puckered lattice geometry, along with $p$-orbitals is often overlooked in the study of topological physics. Here, we investigate the higher-order topology of the $p_{x,y}$-orbital bands in acoustic metamaterials using a simplified two-dimensional phosphorene lattice which possesses a puckered structure. Notably, unlike the $s$-orbital bands in planar lattices, the unique higher-order topology observed here is specific to $p$-orbitals and the puckered geometry due to the unusual hopping patterns induced by them. {Using acoustic pump-probe measurements in metamaterials}, we confirm the emergence of the edge and corner states arising due to the unconventional higher-order topology. We reveal the uniqueness of the higher-order topological physics here via complimentary tight-binding calculations, finite-element simulations, and acoustic experiments. We analyze the underlying physics of the special properties of the edge and corner states in the puckered lattice acoustic metamaterials from the picture of Wannier orbitals. Our work sheds light on the intriguing physics of $p$-orbital topological physics in puckered lattices and acoustic metamaterials which lead to unconventional topological boundary states. \end{abstract}, Comment: Accepted by Phys. Rev. B

Non-Hermitian physics has added new ingredients to topological physics, leading to the rising frontier of non-Hermitian topological phases. In this study, we investigate Chern insulator phases emerging from non-Hermitian kagome models with non-reciprocal and pure imaginary next-nearest neighbor hoppings. In the presence or absence of $C_3$ rotation symmetry, hybrid topological-skin effects are explored through the identification of distinct corner skin modes in different energy regions within two band gaps. By employing the dynamical analysis, the underlying physics is revealed from the non-Hermitian skin effects associated with the chiral edge states, leading to diverse non-Hermitian bulk-boundary responses. The simplicity of these kagome models and their rich emergent topological phenomena suggest that they are appealing candidates for studying non-Hermitian topological phases. We further discuss the possible realizations of these models in non-Hermitian metamaterials.

Fragile topology, akin to twisted bilayer graphene and the exotic phases therein, is a notable topological class with intriguing properties. However, due to its unique nature and the lack of bulk-edge correspondence, the experimental signature of fragile topology has been under debated since its birth. Here, we demonstrate experimentally that fragile topological phases with filling anomaly can be probed via screw dislocations, despite that they do not support gapless edge states. Using a designer hexagonal phononic crystal with a fragile topological band gap, we find that 1D gapless bound modes can emerge at a screw dislocation due to the bulk fragile topology. We then establish a connection between our system and the twisted boundary condition via the gauge invariance principle and illustrate that such an emergent phenomenon is an intrinsic property of fragile topological phases with filling anomaly. We observe experimentally the 1D topological bound states using the pump-probe measurements of their dispersion and wavefunctions, which unveils a novel bulk-defect correspondence of fragile topology and a powerful tool for probing fragile topological phases and materials., Comment: Submitted to Science Bulletin

The interplay between substrates and superconducting thin films has attracted increasing attention. Here, we report an in-depth investigation on superconducting properties of the epitaxial TiN thin films grown on three different substrates by dc reactive magnetron sputtering. The TiN films grown on (0001) sapphire exhibit (111) crystal orientation, while that grown on (100) Si and MgO substrates exhibit (100) orientation. Moreover, the samples grown on Si reveal a relatively lower level of disorder, accompanied by the higher critical transition temperature $T_c$ and smaller magnitude of upper critical field slope near $T_c$. Remarkably, we uncovered a rather high value of superconducting gap (with $\Delta_0/k_BT_c$ = 3.05) in TiN film on Si indicating a very strong coupling superconductivity, in sharp contrast to the case using sapphires and MgO as the substrate which reveals a weak-coupling feature. The comprehensive analysis considering the scenarios of the three substrate shows that the grain size of the thin films may be an important factor influencing the superconductivity., Comment: 8 pages, 6 figures

We propose an end-to-end approach for gaze target detection: predicting a head-target connection between individuals and the target image regions they are looking at. Most of the existing methods use independent components such as off-the-shelf head detectors or have problems in establishing associations between heads and gaze targets. In contrast, we investigate an end-to-end multi-person Gaze target detection framework with Heads and Targets Association (GazeHTA), which predicts multiple head-target instances based solely on input scene image. GazeHTA addresses challenges in gaze target detection by (1) leveraging a pre-trained diffusion model to extract scene features for rich semantic understanding, (2) re-injecting a head feature to enhance the head priors for improved head understanding, and (3) learning a connection map as the explicit visual associations between heads and gaze targets. Our extensive experimental results demonstrate that GazeHTA outperforms state-of-the-art gaze target detection methods and two adapted diffusion-based baselines on two standard datasets.

Creating high-quality and interactive virtual environments, such as games and simulators, often involves complex and costly manual modeling processes. In this paper, we present Video2Game, a novel approach that automatically converts videos of real-world scenes into realistic and interactive game environments. At the heart of our system are three core components:(i) a neural radiance fields (NeRF) module that effectively captures the geometry and visual appearance of the scene; (ii) a mesh module that distills the knowledge from NeRF for faster rendering; and (iii) a physics module that models the interactions and physical dynamics among the objects. By following the carefully designed pipeline, one can construct an interactable and actionable digital replica of the real world. We benchmark our system on both indoor and large-scale outdoor scenes. We show that we can not only produce highly-realistic renderings in real-time, but also build interactive games on top., Comment: CVPR 2024. Project page (with code): https://video2game.github.io/

Accurate 3D kinematics estimation of human body is crucial in various applications for human health and mobility, such as rehabilitation, injury prevention, and diagnosis, as it helps to understand the biomechanical loading experienced during movement. Conventional marker-based motion capture is expensive in terms of financial investment, time, and the expertise required. Moreover, due to the scarcity of datasets with accurate annotations, existing markerless motion capture methods suffer from challenges including unreliable 2D keypoint detection, limited anatomic accuracy, and low generalization capability. In this work, we propose a novel biomechanics-aware network that directly outputs 3D kinematics from two input views with consideration of biomechanical prior and spatio-temporal information. To train the model, we create synthetic dataset ODAH with accurate kinematics annotations generated by aligning the body mesh from the SMPL-X model and a full-body OpenSim skeletal model. Our extensive experiments demonstrate that the proposed approach, only trained on synthetic data, outperforms previous state-of-the-art methods when evaluated across multiple datasets, revealing a promising direction for enhancing video-based human motion capture

While numerous 3D reconstruction and novel-view synthesis methods allow for photorealistic rendering of a scene from multi-view images easily captured with consumer cameras, they bake illumination in their representations and fall short of supporting advanced applications like material editing, relighting, and virtual object insertion. The reconstruction of physically based material properties and lighting via inverse rendering promises to enable such applications. However, most inverse rendering techniques require high dynamic range (HDR) images as input, a setting that is inaccessible to most users. We present a method that recovers the physically based material properties and spatially-varying HDR lighting of a scene from multi-view, low-dynamic-range (LDR) images. We model the LDR image formation process in our inverse rendering pipeline and propose a novel optimization strategy for material, lighting, and a camera response model. We evaluate our approach with synthetic and real scenes compared to the state-of-the-art inverse rendering methods that take either LDR or HDR input. Our method outperforms existing methods taking LDR images as input, and allows for highly realistic relighting and object insertion., Comment: Project Website: https://irisldr.github.io/

Multiscale metrics such as negative Sobolev norms are effective for quantifying the degree of mixedness of a passive scalar field advected by an incompressible flow in the absence of diffusion. In this paper we introduce a mix norm that is motivated by Sobolev norm $H^{-1}$ for a general domain with a no-flux boundary. We then derive an explicit expression for the optimal flow that maximizes the instantaneous decay rate of the mix norm under fixed energy and enstrophy constraints. Numerical simulations indicate that the mix norm decays exponentially or faster for various initial conditions and geometries and the rate is closely related to the smallest non-zero eigenvalue of the Laplace operator. These results generalize previous findings restricted for a periodic domain for its analytical and numerical simplicity. Additionally, we observe that periodic boundaries tend to induce a faster decay in mix norm compared to no-flux conditions under the fixed energy constraint, while the comparison is reversed for the fixed enstrophy constraint. In the special case of even initial distributions, two types of boundary conditions yield the same optimal flow and mix norm decay.

Entanglement entropy is a fundamental concept with rising importance in different fields ranging from quantum information science, black holes to materials science. In complex materials and systems, entanglement entropy provides insight into the collective degrees of freedom that underlie the systems' complex behaviours. As well-known predictions, the entanglement entropy exhibits area laws for systems with gapped excitations, whereas it follows the Gioev-Klich-Widom scaling law in gapless fermion systems. Furthermore, the entanglement spectrum provides salient characterizations of topological phases and phase transitions beyond the conventional paradigms. However, many of these fundamental predictions have not yet been confirmed in experiments due to the difficulties in measuring entanglement entropy in physical systems. Here, we report the experimental verification of the above predictions by probing the nonlocal correlations in phononic systems. From the pump-probe responses in phononic crystals, we obtain the entanglement entropy and entanglement spectrum for phononic systems with the fermion filling analog. With these measurements, we verify the Gioev-Klich-Widom scaling law of entanglement entropy for various quasiparticle dispersions in one- and two-dimensions. Moreover, we observe the salient signatures of topological phases in the entanglement spectrum and entanglement entropy which unveil an unprecedented probe of topological phases without relying on the bulk-boundary correspondence. The progress here opens a frontier where entanglement entropy serves as an important experimental tool in the study of emergent phases and phase transitions which can be generalized to non-Hermitian and other unconventional regimes.

The stacking of intrinsically magnetic van der Waals materials provides a fertile platform to explore tunable transport effects of magnons, presenting significant prospects for spintronic applications. The possibility of having topologically nontrivial magnons in these systems can further expand the scope of exploration. In this work, we consider a bilayer system with intralayer ferromagnetic exchange and a weak interlayer antiferromagnetic exchange, and study the topological magnon-polaron excitations induced by magnetoelastic couplings. Under an applied magnetic field, the system features a metamagnetic transition, where the magnetic ground state changes from antiparallel layers to parallel. We show that the metamagnetic transition is accompanied by a transition of the topological structure of the magnon polarons, which results in discernible changes in the topology induced transport effects. The magnetic-field dependence of the thermal Hall conductivity and spin Nernst coefficient is analyzed with linear response theories., Comment: published with open access as part of the APL Special Collection on Magnonics

Non-Hermitian effects have emerged as a new paradigm for the manipulation of phases of matter that profoundly changes our understanding of non-equilibrium systems, introducing novel concepts such as exceptional points and spectral topology, as well as exotic phenomena such as non-Hermitian skin effects (NHSEs). Most existing studies, however, focus on non-Hermitian eigenstates, whereas dynamic properties of non-Hermitian systems have been discussed only very recently, predicting unexpected phenomena such as wave self-healing, chiral Zener tunneling, and the dynamic NHSEs that are not yet confirmed in experiments. Here, we report the first experimental observation of rich non-Hermitian skin dynamics using tunable one-dimensional nonreciprocal double-chain mechanical systems with glide-time symmetry. Remarkably, dynamic NHSEs are observed with various dynamic behaviors in different dynamic phases, revealing the intriguing nature of these phases that can be understood via the generalized Brillouin zone and the related concepts. Moreover, the observed tunable non-Hermitian skin dynamics and amplifications, the bulk unidirectional wave propagation, and the boundary wave trapping provide promising ways to guide, trap, and amplify waves in a controllable and robust way. Our findings unveil the fundamental aspects and open a new pathway toward non-Hermitian dynamics, which will fertilize the study of non-equilibrium phases of matter and give rise to novel applications in information processing.

The effects of different substrates on the off state leakage current in gallium nitride planar diodes are experimentally demonstrated and studied by analyzing temperature-dependent current voltage characteristics.

Domain generalization (DG) is about learning models that generalize well to new domains that are related to, but different from, the training domain(s). It is a fundamental problem in machine learning and has attracted much attention in recent years. A large number of approaches have been proposed. Different approaches are motivated from different perspectives, making it difficult to gain an overall understanding of the area. In this paper, we propose a causal framework for domain generalization and present an understanding of common DG approaches in the framework. Our work sheds new lights on the following questions: (1) What are the key ideas behind each DG method? (2) Why is it expected to improve generalization to new domains theoretically? (3) How are different DG methods related to each other and what are relative advantages and limitations? By providing a unified perspective on DG, we hope to help researchers better understand the underlying principles and develop more effective approaches for this critical problem in machine learning.

This paper is devoted to studying the weak-field gravitational lensing properties of a 4D ESTGB black hole, which is surrounded by the plasma medium. The effects of the magnetic charges and the three plasma distribution models in the deflection of light around a 4D ESTGB black hole are investigated in detail. We find that the uniform plasma leads to a larger deflection of light rays in comparison with the singular isothermal sphere (SIS), the non-singular isothermal sphere (NSIS) models. Moreover, the deflection angle increases slightly as the absolute value of the magnetic charge decreases. Finally, we analyze the total magnification of image due to weak gravitational lensing around the black hole. The result shows that the presence of a uniform plasma medium remarkably enhances the total magnification whereas the non-uniform plasma reduces the total magnification., Comment: 24 pages, 15 figures

This paper studies the black hole shadow, absorption cross section, and Hawking radiation of a massless scalar field in the background of a static spherically symmetric black hole spacetime that is surrounded by a cloud of strings in Rastall gravity. Specifically, the effects of the parameters $a$ and $\beta$ on the photon sphere and shadow radii are investigated. The results show that as the negative parameter $\beta$ decreases, the photon sphere and shadow radii change in an N-shape. In addition, the absorption cross section obtained after solving the massless Klein-Gordon equation is calculated using the sinc approximation and the partial waves method. We compare the absorption cross section obtained by the sinc approximation and the partial waves method, and find it to be exceptionally consistent in the mid-to-high frequency region. Furthermore, the effects of parameters $a$ and $\beta$ on absorption are examined in detail. Finally, we study in detail the effects of the parameters $a$, $\beta$ and $l$ on the Hawking radiation power emission spectrum of the considered black hole. It turns out that the string parameter $a$ always suppresses the power emission spectrum, indicating that such black holes live longer when the string parameter $a$ is increased while other parameters are fixed., Comment: 27 pages, 10 figures, 1 Table

The gray-body factor and the absorption cross section of the 4D ESTGB gravity with a mode of nonlinear electrodynamics for the massless Dirac field are studied in this paper. The magnetic charge value varies between $-2^{(\frac{5}{3})}/3$ and $0$ as well as the ADM mass is set to $1$, which corresponds to a non-extreme black hole. The gray-body factor is obtained using the semi-analytic WKB method after solving the massless Dirac equation. When the absolute value of magnetic charge is increasing, the gray-body factor $\gamma(\omega)$ is decreasing. In addition, the partial absorption cross section and the total absorption cross section are calculated by using the partial wave method. We find that the maximum value of partial absorption cross section decreases as $\kappa$ increases. And the existence of magnetic charge causes the diminishing of the total absorption cross section. Finally, we find that the absorption cross section of the Dirac field is more sensitive to electric charge than magnetic charge by comparing the absorption cross section of the Reissner-Nordstr$\rm\ddot{o}$m and ESTGB-NLED black holes., Comment: 11 pages, 7 figures

We present UrbanIR (Urban Scene Inverse Rendering), a new inverse graphics model that enables realistic, free-viewpoint renderings of scenes under various lighting conditions with a single video. It accurately infers shape, albedo, visibility, and sun and sky illumination from wide-baseline videos, such as those from car-mounted cameras, differing from NeRF's dense view settings. In this context, standard methods often yield subpar geometry and material estimates, such as inaccurate roof representations and numerous 'floaters'. UrbanIR addresses these issues with novel losses that reduce errors in inverse graphics inference and rendering artifacts. Its techniques allow for precise shadow volume estimation in the original scene. The model's outputs support controllable editing, enabling photorealistic free-viewpoint renderings of night simulations, relit scenes, and inserted objects, marking a significant improvement over existing state-of-the-art methods., Comment: https://urbaninverserendering.github.io/

We present Sim-on-Wheels, a safe, realistic, and vehicle-in-loop framework to test autonomous vehicles' performance in the real world under safety-critical scenarios. Sim-on-wheels runs on a self-driving vehicle operating in the physical world. It creates virtual traffic participants with risky behaviors and seamlessly inserts the virtual events into images perceived from the physical world in real-time. The manipulated images are fed into autonomy, allowing the self-driving vehicle to react to such virtual events. The full pipeline runs on the actual vehicle and interacts with the physical world, but the safety-critical events it sees are virtual. Sim-on-Wheels is safe, interactive, realistic, and easy to use. The experiments demonstrate the potential of Sim-on-Wheels to facilitate the process of testing autonomous driving in challenging real-world scenes with high fidelity and low risk.

The need to explain the output of a deep neural network classifier is now widely recognized. While previous methods typically explain a single class in the output, we advocate explaining the whole output, which is a probability distribution over multiple classes. A whole-output explanation can help a human user gain an overall understanding of model behaviour instead of only one aspect of it. It can also provide a natural framework where one can examine the evidence used to discriminate between competing classes, and thereby obtain contrastive explanations. In this paper, we propose a contrastive whole-output explanation (CWOX) method for image classification, and evaluate it using quantitative metrics and through human subject studies. The source code of CWOX is available at https://github.com/vaynexie/CWOX., Comment: To appear at UAI 2023