Search

Video representation is a long-standing problem that is crucial for various down-stream tasks, such as tracking,depth prediction,segmentation,view synthesis,and editing. However, current methods either struggle to model complex motions due to the absence of 3D structure or rely on implicit 3D representations that are ill-suited for manipulation tasks. To address these challenges, we introduce a novel explicit 3D representation-video Gaussian representation -- that embeds a video into 3D Gaussians. Our proposed representation models video appearance in a 3D canonical space using explicit Gaussians as proxies and associates each Gaussian with 3D motions for video motion. This approach offers a more intrinsic and explicit representation than layered atlas or volumetric pixel matrices. To obtain such a representation, we distill 2D priors, such as optical flow and depth, from foundation models to regularize learning in this ill-posed setting. Extensive applications demonstrate the versatility of our new video representation. It has been proven effective in numerous video processing tasks, including tracking, consistent video depth and feature refinement, motion and appearance editing, and stereoscopic video generation. Project page: https://sunyangtian.github.io/spatter_a_video_web/

In this paper, we present an implicit surface reconstruction method with 3D Gaussian Splatting (3DGS), namely 3DGSR, that allows for accurate 3D reconstruction with intricate details while inheriting the high efficiency and rendering quality of 3DGS. The key insight is incorporating an implicit signed distance field (SDF) within 3D Gaussians to enable them to be aligned and jointly optimized. First, we introduce a differentiable SDF-to-opacity transformation function that converts SDF values into corresponding Gaussians' opacities. This function connects the SDF and 3D Gaussians, allowing for unified optimization and enforcing surface constraints on the 3D Gaussians. During learning, optimizing the 3D Gaussians provides supervisory signals for SDF learning, enabling the reconstruction of intricate details. However, this only provides sparse supervisory signals to the SDF at locations occupied by Gaussians, which is insufficient for learning a continuous SDF. Then, to address this limitation, we incorporate volumetric rendering and align the rendered geometric attributes (depth, normal) with those derived from 3D Gaussians. This consistency regularization introduces supervisory signals to locations not covered by discrete 3D Gaussians, effectively eliminating redundant surfaces outside the Gaussian sampling range. Our extensive experimental results demonstrate that our 3DGSR method enables high-quality 3D surface reconstruction while preserving the efficiency and rendering quality of 3DGS. Besides, our method competes favorably with leading surface reconstruction techniques while offering a more efficient learning process and much better rendering qualities. The code will be available at https://github.com/CVMI-Lab/3DGSR.

Scene reconstruction from multi-view images is a fundamental problem in computer vision and graphics. Recent neural implicit surface reconstruction methods have achieved high-quality results; however, editing and manipulating the 3D geometry of reconstructed scenes remains challenging due to the absence of naturally decomposed object entities and complex object/background compositions. In this paper, we present Total-Decom, a novel method for decomposed 3D reconstruction with minimal human interaction. Our approach seamlessly integrates the Segment Anything Model (SAM) with hybrid implicit-explicit neural surface representations and a mesh-based region-growing technique for accurate 3D object decomposition. Total-Decom requires minimal human annotations while providing users with real-time control over the granularity and quality of decomposition. We extensively evaluate our method on benchmark datasets and demonstrate its potential for downstream applications, such as animation and scene editing. The code is available at https://github.com/CVMI-Lab/Total-Decom.git., Comment: 8 pages, 7 figures, accepted by CVPR 2024

An updated catalog consisting of 1092 6.7-GHz methanol maser sources was reported in this work. Additionally, the NH3 (1, 1), NH3 (2, 2), and NH3 (3, 3) transitions were observed towards 214 star forming regions using the Shanghai Tianma radio telescope (TMRT) in order to examine the differences in physical environments, such as excitation temperature and column density of molecular clouds associated with methanol masers on the Galactic scale. Statistical results reveal that the number of 6.7 GHz methanol masers in the Perseus arm is significantly lower than that in the other three main spiral arms. In addition, the Perseus arm also has the lowest gas column density among the main spiral arms traced by the NH3 observations. Both of these findings suggest that the Perseus arm has the lowest rate of high-mass star formation compared to the other three main spiral arms. We also observed a trend in which both the luminosity of 6.7 GHz methanol masers and the ammonia gas column density decreased as the galactocentric distances. This finding indicates that the density of material in the inner Milky Way is generally higher than that in the outer Milky Way. It further suggests that high-mass stars are more easily formed at the head of spiral arms. Furthermore, we found that the column density of ammonia gas is higher in the regions on the arms than that in the inter-arm regions, supporting that the former is more likely to be the birthplace of high-mass stars.

LiNGAM determines the variable order from cause to effect using additive noise models, but it faces challenges with confounding. Previous methods maintained LiNGAM's fundamental structure while trying to identify and address variables affected by confounding. As a result, these methods required significant computational resources regardless of the presence of confounding, and they did not ensure the detection of all confounding types. In contrast, this paper enhances LiNGAM by introducing LiNGAM-MMI, a method that quantifies the magnitude of confounding using KL divergence and arranges the variables to minimize its impact. This method efficiently achieves a globally optimal variable order through the shortest path problem formulation. LiNGAM-MMI processes data as efficiently as traditional LiNGAM in scenarios without confounding while effectively addressing confounding situations. Our experimental results suggest that LiNGAM-MMI more accurately determines the correct variable order, both in the presence and absence of confounding., Comment: arXiv admin note: text overlap with arXiv:2007.11131 by other authors

Novel view synthesis for dynamic scenes is still a challenging problem in computer vision and graphics. Recently, Gaussian splatting has emerged as a robust technique to represent static scenes and enable high-quality and real-time novel view synthesis. Building upon this technique, we propose a new representation that explicitly decomposes the motion and appearance of dynamic scenes into sparse control points and dense Gaussians, respectively. Our key idea is to use sparse control points, significantly fewer in number than the Gaussians, to learn compact 6 DoF transformation bases, which can be locally interpolated through learned interpolation weights to yield the motion field of 3D Gaussians. We employ a deformation MLP to predict time-varying 6 DoF transformations for each control point, which reduces learning complexities, enhances learning abilities, and facilitates obtaining temporal and spatial coherent motion patterns. Then, we jointly learn the 3D Gaussians, the canonical space locations of control points, and the deformation MLP to reconstruct the appearance, geometry, and dynamics of 3D scenes. During learning, the location and number of control points are adaptively adjusted to accommodate varying motion complexities in different regions, and an ARAP loss following the principle of as rigid as possible is developed to enforce spatial continuity and local rigidity of learned motions. Finally, thanks to the explicit sparse motion representation and its decomposition from appearance, our method can enable user-controlled motion editing while retaining high-fidelity appearances. Extensive experiments demonstrate that our approach outperforms existing approaches on novel view synthesis with a high rendering speed and enables novel appearance-preserved motion editing applications. Project page: https://yihua7.github.io/SC-GS-web/, Comment: Code link: https://github.com/yihua7/SC-GS

Stereo matching aims to estimate the disparity between matching pixels in a stereo image pair, which is important to robotics, autonomous driving, and other computer vision tasks. Despite the development of numerous impressive methods in recent years, determining the most suitable architecture for practical application remains challenging. Addressing this gap, our paper introduces a comprehensive benchmark focusing on practical applicability rather than solely on individual models for optimized performance. Specifically, we develop a flexible and efficient stereo matching codebase, called OpenStereo. OpenStereo includes training and inference codes of more than 10 network models, making it, to our knowledge, the most complete stereo matching toolbox available. Based on OpenStereo, we conducted experiments and have achieved or surpassed the performance metrics reported in the original paper. Additionally, we conduct an exhaustive analysis and deconstruction of recent developments in stereo matching through comprehensive ablative experiments. These investigations inspired the creation of StereoBase, a strong baseline model. Our StereoBase ranks 1st on SceneFlow, KITTI 2015, 2012 (Reflective) among published methods and achieves the best performance across all metrics. In addition, StereoBase has strong cross-dataset generalization. Code is available at \url{https://github.com/XiandaGuo/OpenStereo}., Comment: Code is available at: https://github.com/XiandaGuo/OpenStereo

Gaussian Boson sampling (GBS) plays a crucially important role in demonstrating quantum advantage. As a major imperfection, the limited connectivity of the linear optical network weakens the quantum advantage result in recent experiments. Here we present a faster classical algorithm to simulate the GBS process with limited connectivity. In this work, we introduce an enhanced classical algorithm for simulating GBS processes with limited connectivity. It computes the loop Hafnian of an $n \times n$ symmetric matrix with bandwidth $w$ in $O(nw2^w)$ time which is better than the previous fastest algorithm which runs in $O(nw^2 2^w)$ time. This classical algorithm is helpful on clarifying how limited connectivity affects the computational complexity of GBS and tightening the boundary of quantum advantage in the GBS problem., Comment: 2 figures

We propose an experimental scheme to realize non-Abelian dynamical gauge field for ultracold fermions, which induces a novel pairing mechanism of topological superfluidity. The dynamical gauge fields arise from nontrivial interplay effect between the strong Zeeman splitting and Hubbard interaction in a two-dimensional (2D) optical Raman lattice. The spin-flip transitions are forbidden by the large Zeeman detuning, but are restored when the Zeeman splitting is compensated by Hubbard interaction. This scheme allows to generate a dynamical non-Abelian gauge field that leads to a Dirac type correlated 2D spin-orbit interaction depending on local state configurations. The topological superfluid from a novel pairing driven by 2D dynamical gauge fields is reached, with analytic and numerical results being obtained. Our work may open up a door to emulate non-Abelian dynamical gauge fields and correlated topological phases with experimental feasibility., Comment: 5+7 pages, 4+6 figures

We establish the geometry behind the quantum $6j$-symbols under only the admissibility conditions as in the definition of the Turaev-Viro invariants of $3$-manifolds. As a classification, we show that the $6$-tuples in the quantum $6j$-symbols give in a precise way to the dihedral angles of (1) a spherical tetrahedron, (2) a generalized Euclidean tetrahedron, (3) a generalized hyperbolic tetrahedron or (4) in the degenerate case the angles between four oriented straight lines in the Euclidean plane. We also show that for a large proportion of the cases, the $6$-tuples always give the dihedral angles of a generalized hyperbolic tetrahedron and the exponential growth rate of the corresponding quantum $6j$-symbols equals the suitably defined volume of this generalized hyperbolic tetrahedron. It is worth mentioning that the volume of a generalized hyperbolic tetrahedron can be negative, hence the corresponding sequence of the quantum $6j$-symbols could decay exponentially. This is a phenomenon that has never been aware of before., Comment: 55 pages, 16 figures

This study demonstrates that double descent can be mitigated by adding a dropout layer adjacent to the fully connected linear layer. The unexpected double-descent phenomenon garnered substantial attention in recent years, resulting in fluctuating prediction error rates as either sample size or model size increases. Our paper posits that the optimal test error, in terms of the dropout rate, shows a monotonic decrease in linear regression with increasing sample size. Although we do not provide a precise mathematical proof of this statement, we empirically validate through experiments that the test error decreases for each dropout rate. The statement we prove is that the expected test error for each dropout rate within a certain range decreases when the dropout rate is fixed. Our experimental results substantiate our claim, showing that dropout with an optimal dropout rate can yield a monotonic test error curve in nonlinear neural networks. These experiments were conducted using the Fashion-MNIST and CIFAR-10 datasets. These findings imply the potential benefit of incorporating dropout into risk curve scaling to address the peak phenomenon. To our knowledge, this study represents the first investigation into the relationship between dropout and double descent.