Search

We introduce Temporal Residual Jacobians as a novel representation to enable data-driven motion transfer. Our approach does not assume access to any rigging or intermediate shape keyframes, produces geometrically and temporally consistent motions, and can be used to transfer long motion sequences. Central to our approach are two coupled neural networks that individually predict local geometric and temporal changes that are subsequently integrated, spatially and temporally, to produce the final animated meshes. The two networks are jointly trained, complement each other in producing spatial and temporal signals, and are supervised directly with 3D positional information. During inference, in the absence of keyframes, our method essentially solves a motion extrapolation problem. We test our setup on diverse meshes (synthetic and scanned shapes) to demonstrate its superiority in generating realistic and natural-looking animations on unseen body shapes against SoTA alternatives. Supplemental video and code are available at https://temporaljacobians.github.io/ ., Comment: 15 pages, 6 figures

Effective representation of 2D images is fundamental in digital image processing, where traditional methods like raster and vector graphics struggle with sharpness and textural complexity respectively. Current neural fields offer high-fidelity and resolution independence but require predefined meshes with known discontinuities, restricting their utility. We observe that by treating all mesh edges as potential discontinuities, we can represent the magnitude of discontinuities with continuous variables and optimize. Based on this observation, we introduce a novel discontinuous neural field model that jointly approximate the target image and recovers discontinuities. Through systematic evaluations, our neural field demonstrates superior performance in denoising and super-resolution tasks compared to InstantNGP, achieving improvements of over 5dB and 10dB, respectively. Our model also outperforms Mumford-Shah-based methods in accurately capturing discontinuities, with Chamfer distances 3.5x closer to the ground truth. Additionally, our approach shows remarkable capability in handling complex artistic drawings and natural images.

Measurements can detect errors in a decohered quantum memory allowing active error correction to increase the memory time. Previous understanding of this mechanism has focused on evaluating the performance of error correction algorithms based on measurement results. In this work, we instead intrinsically characterize the information dynamics in a quantum memory under repeated measurements, using coherent information and relative entropy. We consider the dynamics of a $d$-dimensional stabilizer code subject to Pauli errors and noisy stabilizer measurements and develop a $(d+1)$-dimensional statistical mechanics model for the information-theoretic diagnostics. Our model is dual to the model previously obtained for the optimal decoding algorithm, and the potential decoding transition in the quantum memory again manifests as a thermal phase transition in the statistical mechanics model. We explicitly derive the model and study the phase transition in information encoding in three examples: surface codes, repetition codes, and the XZZX code., Comment: 27 pages, 9 figures

The success of denoising diffusion models in representing rich data distributions over 2D raster images has prompted research on extending them to other data representations, such as vector graphics. Unfortunately due to their variable structure and scarcity of vector training data, directly applying diffusion models on this domain remains a challenging problem. Using workarounds like optimization via Score Distillation Sampling (SDS) is also fraught with difficulty, as vector representations are non trivial to directly optimize and tend to result in implausible geometries such as redundant or self-intersecting shapes. NIVeL addresses these challenges by reinterpreting the problem on an alternative, intermediate domain which preserves the desirable properties of vector graphics -- mainly sparsity of representation and resolution-independence. This alternative domain is based on neural implicit fields expressed in a set of decomposable, editable layers. Based on our experiments, NIVeL produces text-to-vector graphics results of significantly better quality than the state-of-the-art.

We present personalized residuals and localized attention-guided sampling for efficient concept-driven generation using text-to-image diffusion models. Our method first represents concepts by freezing the weights of a pretrained text-conditioned diffusion model and learning low-rank residuals for a small subset of the model's layers. The residual-based approach then directly enables application of our proposed sampling technique, which applies the learned residuals only in areas where the concept is localized via cross-attention and applies the original diffusion weights in all other regions. Localized sampling therefore combines the learned identity of the concept with the existing generative prior of the underlying diffusion model. We show that personalized residuals effectively capture the identity of a concept in ~3 minutes on a single GPU without the use of regularization images and with fewer parameters than previous models, and localized sampling allows using the original model as strong prior for large parts of the image., Comment: CVPR 2024. Project page at https://cusuh.github.io/personalized-residuals

Animated logos are a compelling and ubiquitous way individuals and brands represent themselves online. Manually authoring these logos can require significant artistic skill and effort. To help novice designers animate logos, design tools currently offer templates and animation presets. However, these solutions can be limited in their expressive range. Large language models have the potential to help novice designers create animated logos by generating animation code that is tailored to their content. In this paper, we introduce LogoMotion, an LLM-based system that takes in a layered document and generates animated logos through visually-grounded program synthesis. We introduce techniques to create an HTML representation of a canvas, identify primary and secondary elements, synthesize animation code, and visually debug animation errors. When compared with an industry standard tool, we find that LogoMotion produces animations that are more content-aware and are on par in terms of quality. We conclude with a discussion of the implications of LLM-generated animation for motion design.

Procedural noise is a fundamental component of computer graphics pipelines, offering a flexible way to generate textures that exhibit "natural" random variation. Many different types of noise exist, each produced by a separate algorithm. In this paper, we present a single generative model which can learn to generate multiple types of noise as well as blend between them. In addition, it is capable of producing spatially-varying noise blends despite not having access to such data for training. These features are enabled by training a denoising diffusion model using a novel combination of data augmentation and network conditioning techniques. Like procedural noise generators, the model's behavior is controllable via interpretable parameters and a source of randomness. We use our model to produce a variety of visually compelling noise textures. We also present an application of our model to improving inverse procedural material design; using our model in place of fixed-type noise nodes in a procedural material graph results in higher-fidelity material reconstructions without needing to know the type of noise in advance., Comment: In ACM Transactions on Graphics (Proceedings of SIGGRAPH) 2024, 21 pages

Quantum systems subject to random unitary evolution and measurements at random points in spacetime exhibit entanglement phase transitions which depend on the frequency of these measurements. Past work has experimentally observed entanglement phase transitions on near-term quantum computers, but the characterization approach using entanglement entropy is not scalable due to exponential overhead of quantum state tomography and post selection. Recently, an alternative protocol to detect entanglement phase transitions using linear cross-entropy was proposed which eliminates both bottlenecks. Here, we report the demonstration of this protocol in systems with one-dimensional and all-to-all connectivities on IBM's quantum hardware on up to 22 qubits, a regime which is presently inaccessible if post-selection is required. We demonstrate a collapse of the data into a scale-invariant form with critical exponents agreeing with theory within uncertainty. Our demonstration paves the way for studies of measurement-induced entanglement phase transitions and associated critical phenomena on larger near-term quantum systems., Comment: 23 pages, 15 figures

Current controls over diffusion models (e.g., through text or ControlNet) for image generation fall short in recognizing abstract, continuous attributes like illumination direction or non-rigid shape change. In this paper, we present an approach for allowing users of text-to-image models to have fine-grained control of several attributes in an image. We do this by engineering special sets of input tokens that can be transformed in a continuous manner -- we call them Continuous 3D Words. These attributes can, for example, be represented as sliders and applied jointly with text prompts for fine-grained control over image generation. Given only a single mesh and a rendering engine, we show that our approach can be adopted to provide continuous user control over several 3D-aware attributes, including time-of-day illumination, bird wing orientation, dollyzoom effect, and object poses. Our method is capable of conditioning image creation with multiple Continuous 3D Words and text descriptions simultaneously while adding no overhead to the generative process. Project Page: https://ttchengab.github.io/continuous_3d_words, Comment: Project Page: https://ttchengab.github.io/continuous_3d_words

How to frame (or crop) a photo often depends on the image subject and its context; e.g., a human portrait. Recent works have defined the subject-aware image cropping task as a nuanced and practical version of image cropping. We propose a weakly-supervised approach (GenCrop) to learn what makes a high-quality, subject-aware crop from professional stock images. Unlike supervised prior work, GenCrop requires no new manual annotations beyond the existing stock image collection. The key challenge in learning from this data, however, is that the images are already cropped and we do not know what regions were removed. Our insight is to combine a library of stock images with a modern, pre-trained text-to-image diffusion model. The stock image collection provides diversity and its images serve as pseudo-labels for a good crop, while the text-image diffusion model is used to out-paint (i.e., outward inpainting) realistic uncropped images. Using this procedure, we are able to automatically generate a large dataset of cropped-uncropped training pairs to train a cropping model. Despite being weakly-supervised, GenCrop is competitive with state-of-the-art supervised methods and significantly better than comparable weakly-supervised baselines on quantitative and qualitative evaluation metrics., Comment: AAAI 24. Extended version with supplemental materials

We present VecFusion, a new neural architecture that can generate vector fonts with varying topological structures and precise control point positions. Our approach is a cascaded diffusion model which consists of a raster diffusion model followed by a vector diffusion model. The raster model generates low-resolution, rasterized fonts with auxiliary control point information, capturing the global style and shape of the font, while the vector model synthesizes vector fonts conditioned on the low-resolution raster fonts from the first stage. To synthesize long and complex curves, our vector diffusion model uses a transformer architecture and a novel vector representation that enables the modeling of diverse vector geometry and the precise prediction of control points. Our experiments show that, in contrast to previous generative models for vector graphics, our new cascaded vector diffusion model generates higher quality vector fonts, with complex structures and diverse styles.

We introduce a novel image-to-painting method that facilitates the creation of large-scale, high-fidelity paintings with human-like quality and stylistic variation. To process large images and gain control over the painting process, we introduce a segmentation-based painting process and a dynamic attention map approach inspired by human painting strategies, allowing optimization of brush strokes to proceed in batches over different image regions, thereby capturing both large-scale structure and fine details, while also allowing stylistic control over detail. Our optimized batch processing and patch-based loss framework enable efficient handling of large canvases, ensuring our painted outputs are both aesthetically compelling and functionally superior as compared to previous methods, as confirmed by rigorous evaluations. Code available at: https://github.com/manuelladron/semantic\_based\_painting.git, Comment: 8 pages

Exploring variations of 3D shapes is a time-consuming process in traditional 3D modeling tools. Deep generative models of 3D shapes often feature continuous latent spaces that can, in principle, be used to explore potential variations starting from a set of input shapes. In practice, doing so can be problematic: latent spaces are high dimensional and hard to visualize, contain shapes that are not relevant to the input shapes, and linear paths through them often lead to sub-optimal shape transitions. Furthermore, one would ideally be able to explore variations in the original high-quality meshes used to train the generative model, not its lower-quality output geometry. In this paper, we present a method to explore variations among a given set of landmark shapes by constructing a mapping from an easily-navigable 2D exploration space to a subspace of a pre-trained generative model. We first describe how to find a mapping that spans the set of input landmark shapes and exhibits smooth variations between them. We then show how to turn the variations in this subspace into deformation fields, to transfer those variations to high-quality meshes for the landmark shapes. Our results show that our method can produce visually-pleasing and easily-navigable 2D exploration spaces for several different shape categories, especially as compared to prior work on learning deformation spaces for 3D shapes., Comment: SIGGRAPH Asia 2023, 15 pages

The linear cross-entropy (LXE) has been recently proposed as a scalable probe of the measurement-driven phase transition between volume- and area-law-entangled phases of pure-state trajectories in certain monitored quantum circuits. Here, we demonstrate that the LXE can distinguish distinct area-law-entangled phases of monitored circuits with symmetries, and extract universal behavior at the critical points separating these phases. We focus on (1+1)-dimensional monitored circuits with an on-site $\mathbb{Z}_{2}$ symmetry. For an appropriate choice of initial states, the LXE distinguishes the area-law-entangled spin glass and paramagnetic phases of the monitored trajectories. At the critical point, described by two-dimensional percolation, the LXE exhibits universal behavior which depends sensitively on boundary conditions, and the choice of initial states. With open boundary conditions, we show that the LXE relates to crossing probabilities in critical percolation, and is thus given by a known universal function of the aspect ratio of the dynamics, which quantitatively agrees with numerical studies of the LXE at criticality. The LXE probes correlations of other operators in percolation with periodic boundary conditions. We show that the LXE is sensitive to the richer phase diagram of the circuit model in the presence of symmmetric unitary gates. Lastly, we consider the effect of noise during the circuit evolution, and propose potential solutions to counter it., Comment: 12+6 pages, 16 figures. V2: References added

Automatic generation of fonts can be an important aid to typeface design. Many current approaches regard glyphs as pixelated images, which present artifacts when scaling and inevitable quality losses after vectorization. On the other hand, existing vector font synthesis methods either fail to represent the shape concisely or require vector supervision during training. To push the quality of vector font synthesis to the next level, we propose a novel dual-part representation for vector glyphs, where each glyph is modeled as a collection of closed "positive" and "negative" path pairs. The glyph contour is then obtained by boolean operations on these paths. We first learn such a representation only from glyph images and devise a subsequent contour refinement step to align the contour with an image representation to further enhance details. Our method, named DualVector, outperforms state-of-the-art methods in vector font synthesis both quantitatively and qualitatively. Our synthesized vector fonts can be easily converted to common digital font formats like TrueType Font for practical use. The code is released at https://github.com/thuliu-yt16/dualvector., Comment: CVPR 2023

Adaptive quantum circuits, in which unitary operations, measurements, and feedback are used to steer quantum many-body systems, provide an exciting opportunity to generate new dynamical steady states. We introduce an adaptive quantum dynamics with continuous symmetry where unitary operations, measurements, and local unitary feedback are used to drive ordering. In this setting, we find a pure steady state hosting symmetry-breaking order, which is the ground state of a gapless, local Hamiltonian. We explore the dynamical properties of the approach to this steady state. We find that this steady-state order is fragile to perturbations, even those that respect the continuous symmetry., Comment: 17 pages, 10 figures

This report surveys advances in deep learning-based modeling techniques that address four different 3D indoor scene analysis tasks, as well as synthesis of 3D indoor scenes. We describe different kinds of representations for indoor scenes, various indoor scene datasets available for research in the aforementioned areas, and discuss notable works employing machine learning models for such scene modeling tasks based on these representations. Specifically, we focus on the analysis and synthesis of 3D indoor scenes. With respect to analysis, we focus on four basic scene understanding tasks -- 3D object detection, 3D scene segmentation, 3D scene reconstruction and 3D scene similarity. And for synthesis, we mainly discuss neural scene synthesis works, though also highlighting model-driven methods that allow for human-centric, progressive scene synthesis. We identify the challenges involved in modeling scenes for these tasks and the kind of machinery that needs to be developed to adapt to the data representation, and the task setting in general. For each of these tasks, we provide a comprehensive summary of the state-of-the-art works across different axes such as the choice of data representation, backbone, evaluation metric, input, output, etc., providing an organized review of the literature. Towards the end, we discuss some interesting research directions that have the potential to make a direct impact on the way users interact and engage with these virtual scene models, making them an integral part of the metaverse., Comment: Published in Computer Graphics Forum, Aug 2023

Representing a 3D shape with a set of primitives can aid perception of structure, improve robotic object manipulation, and enable editing, stylization, and compression of 3D shapes. Existing methods either use simple parametric primitives or learn a generative shape space of parts. Both have limitations: parametric primitives lead to coarse approximations, while learned parts offer too little control over the decomposition. We instead propose to decompose shapes using a library of 3D parts provided by the user, giving full control over the choice of parts. The library can contain parts with high-quality geometry that are suitable for a given category, resulting in meaningful decompositions with clean geometry. The type of decomposition can also be controlled through the choice of parts in the library. Our method works via a self-supervised approach that iteratively retrieves parts from the library and refines their placements. We show that this approach gives higher reconstruction accuracy and more desirable decompositions than existing approaches. Additionally, we show how the decomposition can be controlled through the part library by using different part libraries to reconstruct the same shapes., Comment: CVPR 2023

The Floquet code utilizes a periodic sequence of two-qubit measurements to realize the topological order. After each measurement round, the instantaneous stabilizer group can be mapped to a honeycomb toric code, explaining the topological feature. The code also possesses a time-crystal order - the $e-m$ transmutation after every cycle, breaking the Floquet symmetry of the measurement schedule. This behavior is distinct from the stationary topological order realized in either random circuits or time-independent Hamiltonian. Therefore, the resultant phase belongs to the overlap between the classes of Floquet enriched topological orders and measurement-induced phases. In this work, we construct a continuous path interpolating between the Floquet and toric codes, focusing on the transition between the time-crystal and stationary topological phases. We show that this transition is characterized by a divergent length scale. We also add single-qubit perturbations to the model and obtain a richer two-dimensional parametric phase diagram of the Floquet code, showing the stability of the Floquet enriched topological order., Comment: 6+8 pages, 13 figures

Diffusion models currently achieve state-of-the-art performance for both conditional and unconditional image generation. However, so far, image diffusion models do not support tasks required for 3D understanding, such as view-consistent 3D generation or single-view object reconstruction. In this paper, we present RenderDiffusion, the first diffusion model for 3D generation and inference, trained using only monocular 2D supervision. Central to our method is a novel image denoising architecture that generates and renders an intermediate three-dimensional representation of a scene in each denoising step. This enforces a strong inductive structure within the diffusion process, providing a 3D consistent representation while only requiring 2D supervision. The resulting 3D representation can be rendered from any view. We evaluate RenderDiffusion on FFHQ, AFHQ, ShapeNet and CLEVR datasets, showing competitive performance for generation of 3D scenes and inference of 3D scenes from 2D images. Additionally, our diffusion-based approach allows us to use 2D inpainting to edit 3D scenes., Comment: Accepted at CVPR 2023. Project page: https://github.com/Anciukevicius/RenderDiffusion

The coherent superposition of quantum states is an important resource for quantum information processing which distinguishes quantum dynamics and information from their classical counterparts. In this article we determine the coherence requirements to communicate quantum information in a broad setting encompassing monitored quantum dynamics and quantum error correction codes. We determine these requirements by considering hybrid circuits that are generated by a quantum information game played between two opponents, Alice and Eve, who compete by applying unitaries and measurements on a fixed number of qubits. Alice applies unitaries in an attempt to maintain quantum channel capacity, while Eve applies measurements in an attempt to destroy it. By limiting the coherence generating or destroying operations available to each opponent, we determine Alice's coherence requirements. When Alice plays a random strategy aimed at mimicking generic monitored quantum dynamics, we discover a coherence-tuned phase transitions in entanglement and quantum channel capacity. We then derive a theorem giving the minimum coherence required by Alice in any successful strategy, and conclude by proving that coherence sets an upper bound on the code distance in any stabelizer quantum error correction codes. Such bounds provide a rigorous quantification of the coherence resource requirements for quantum communication and error correction., Comment: 19+11 pages, 12+3 figures

We investigate the prospects of employing the linear cross-entropy to experimentally access measurement-induced phase transitions (MIPT) without requiring any postselection of quantum trajectories. For two random circuits that are identical in the bulk but with different initial states, the linear cross-entropy $\chi$ between the bulk measurement outcome distributions in the two circuits acts as a boundary order parameter, and can be used to distinguish the volume law from area law phases. In the volume law phase (and in the thermodynamic limit) the bulk measurements cannot distinguish between the two different initial states, and $\chi = 1$. In the area law phase $\chi < 1$. For circuits with Clifford gates, we provide numerical evidence that $\chi$ can be sampled to accuracy $\epsilon$ from $O(1/\epsilon^2)$ trajectories, by running the first circuit on a quantum simulator without postselection, aided by a classical simulation of the second. We also find that for weak depolarizing noise the signature of the MIPT is still present for intermediate system sizes. In our protocol we have the freedom of choosing initial states such that the "classical" side can be simulated efficiently, while simulating the "quantum" side is still classically hard., Comment: 7+8 pages, 6 figures. v2: 7+9 pages, 3+3 figures. Updated discussions on sample size (Fig. 2d, 2e), and new results from random Haar circuits (Fig. 3b). Accepted version

Measurements allow efficient preparation of interesting quantum many-body states with long-range entanglement, conditioned on additional transformations based on measurement outcomes. Here, we demonstrate that the so-called conformal quantum critical points (CQCP) can be obtained by performing general single-site measurements in an appropriate basis on the cluster states in $d\geq2$. The equal-time correlators of the said states are described by correlation functions of certain $d$-dimensional classical models at finite temperatures and feature spatial conformal invariance. This establishes an exact correspondence between the measurement-prepared critical states and conformal field theories of a range of critical spin models, including familiar Ising models and gauge theories. Furthermore, by mapping the long-range entanglement structure of measured quantum states into the correlations of the corresponding thermal spin model, we rigorously establish the stability condition of the long-range entanglement in the measurement-prepared quantum states deviating from the ideal setting. Most importantly, we describe protocols to decode the resulting quantum phases and transitions without post-selection, thus transferring the exponential measurement complexity to a polynomial classical computation. Therefore, our findings suggest a novel mechanism in which a quantum critical wavefunction emerges, providing new practical ways to study quantum phases and conformal quantum critical points., Comment: 37 pages, 11 figures

Quantum circuits -- built from local unitary gates and local measurements -- are a new playground for quantum many-body physics and a tractable setting to explore universal collective phenomena far-from-equilibrium. These models have shed light on longstanding questions about thermalization and chaos, and on the underlying universal dynamics of quantum information and entanglement. In addition, such models generate new sets of questions and give rise to phenomena with no traditional analog, such as new dynamical phases in quantum systems that are monitored by an external observer. Quantum circuit dynamics is also topical in view of experimental progress in building digital quantum simulators that allow control of precisely these ingredients. Randomness in the circuit elements allows a high level of theoretical control, with a key theme being mappings between real-time quantum dynamics and effective classical lattice models or dynamical processes. Many of the universal phenomena that can be identified in this tractable setting apply to much wider classes of more structured many-body dynamics., Comment: Review article for Annual Review of Condensed Matter Physics; comments welcome

We introduce the task of spotting temporally precise, fine-grained events in video (detecting the precise moment in time events occur). Precise spotting requires models to reason globally about the full-time scale of actions and locally to identify subtle frame-to-frame appearance and motion differences that identify events during these actions. Surprisingly, we find that top performing solutions to prior video understanding tasks such as action detection and segmentation do not simultaneously meet both requirements. In response, we propose E2E-Spot, a compact, end-to-end model that performs well on the precise spotting task and can be trained quickly on a single GPU. We demonstrate that E2E-Spot significantly outperforms recent baselines adapted from the video action detection, segmentation, and spotting literature to the precise spotting task. Finally, we contribute new annotations and splits to several fine-grained sports action datasets to make these datasets suitable for future work on precise spotting., Comment: ECCV 2022; Website URL: https://jhong93.github.io/projects/spot.html

Stein discrepancies have emerged as a powerful statistical tool, being applied to fundamental statistical problems including parameter inference, goodness-of-fit testing, and sampling. The canonical Stein discrepancies require the derivatives of a statistical model to be computed, and in return provide theoretical guarantees of convergence detection and control. However, for complex statistical models, the stable numerical computation of derivatives can require bespoke algorithmic development and render Stein discrepancies impractical. This paper focuses on posterior approximation using Stein discrepancies, and introduces a collection of non-canonical Stein discrepancies that are gradient free, meaning that derivatives of the statistical model are not required. Sufficient conditions for convergence detection and control are established, and applications to sampling and variational inference are presented.

Rigged puppets are one of the most prevalent representations to create 2D character animations. Creating these puppets requires partitioning characters into independently moving parts. In this work, we present a method to automatically identify such articulated parts from a small set of character poses shown in a sprite sheet, which is an illustration of the character that artists often draw before puppet creation. Our method is trained to infer articulated parts, e.g. head, torso and limbs, that can be re-assembled to best reconstruct the given poses. Our results demonstrate significantly better performance than alternatives qualitatively and quantitatively.Our project page https://zhan-xu.github.io/parts/ includes our code and data.

We present ASSET, a neural architecture for automatically modifying an input high-resolution image according to a user's edits on its semantic segmentation map. Our architecture is based on a transformer with a novel attention mechanism. Our key idea is to sparsify the transformer's attention matrix at high resolutions, guided by dense attention extracted at lower image resolutions. While previous attention mechanisms are computationally too expensive for handling high-resolution images or are overly constrained within specific image regions hampering long-range interactions, our novel attention mechanism is both computationally efficient and effective. Our sparsified attention mechanism is able to capture long-range interactions and context, leading to synthesizing interesting phenomena in scenes, such as reflections of landscapes onto water or flora consistent with the rest of the landscape, that were not possible to generate reliably with previous convnets and transformer approaches. We present qualitative and quantitative results, along with user studies, demonstrating the effectiveness of our method., Comment: SIGGRAPH 2022 - Journal Track

Approaches for single-view reconstruction typically rely on viewpoint annotations, silhouettes, the absence of background, multiple views of the same instance, a template shape, or symmetry. We avoid all such supervision and assumptions by explicitly leveraging the consistency between images of different object instances. As a result, our method can learn from large collections of unlabelled images depicting the same object category. Our main contributions are two ways for leveraging cross-instance consistency: (i) progressive conditioning, a training strategy to gradually specialize the model from category to instances in a curriculum learning fashion; and (ii) neighbor reconstruction, a loss enforcing consistency between instances having similar shape or texture. Also critical to the success of our method are: our structured autoencoding architecture decomposing an image into explicit shape, texture, pose, and background; an adapted formulation of differential rendering; and a new optimization scheme alternating between 3D and pose learning. We compare our approach, UNICORN, both on the diverse synthetic ShapeNet dataset - the classical benchmark for methods requiring multiple views as supervision - and on standard real-image benchmarks (Pascal3D+ Car, CUB) for which most methods require known templates and silhouette annotations. We also showcase applicability to more challenging real-world collections (CompCars, LSUN), where silhouettes are not available and images are not cropped around the object., Comment: ECCV 2022. Project webpage with code and videos: http://imagine.enpc.fr/~monniert/UNICORN/

Sequential algorithms are popular for experimental design, enabling emulation, optimisation and inference to be efficiently performed. For most of these applications bespoke software has been developed, but the approach is general and many of the actual computations performed in such software are identical. Motivated by the diverse problems that can in principle be solved with common code, this paper presents GaussED, a simple probabilistic programming language coupled to a powerful experimental design engine, which together automate sequential experimental design for approximating a (possibly nonlinear) quantity of interest in Gaussian processes models. Using a handful of commands, GaussED can be used to: solve linear partial differential equations, perform tomographic reconstruction from integral data and implement Bayesian optimisation with gradient data.

This paper introduces a model for producing stylized line drawings from 3D shapes. The model takes a 3D shape and a viewpoint as input, and outputs a drawing with textured strokes, with variations in stroke thickness, deformation, and color learned from an artist's style. The model is fully differentiable. We train its parameters from a single training drawing of another 3D shape. We show that, in contrast to previous image-based methods, the use of a geometric representation of 3D shape and 2D strokes allows the model to transfer important aspects of shape and texture style while preserving contours. Our method outputs the resulting drawing in a vector representation, enabling richer downstream analysis or editing in interactive applications., Comment: Accepted to ICCV 2021

We introduce an exact mapping of Clifford (stabilizer) random tensor networks (RTNs) and monitored quantum circuits, onto a statistical mechanics model. With Haar unitaries, the fundamental degrees of freedom ('spins') are permutations because all operators commuting with the action of the unitaries on a tensor product arise from permutations of the tensor factors ('Schur-Weyl duality'). For unitaries restricted to the smaller Clifford group, the set of commuting operators, the 'commutant', forming the new 'spin' degrees of freedom, will be larger. We use the recent full characterization of this commutant by Gross et al., Comm. Math. Phys. 385, 1325 (2021), to construct the Clifford statistical mechanics models for on-site Hilbert space dimensions which are powers of a prime number $p$. We show that the Boltzmann weights are invariant under a symmetry group involving orthogonal matrices with entries in the finite number field ${\bf F}_p$. This implies that the symmetry group, and consequently all universal properties of entanglement transitions in Clifford circuits and RTNs will in general depend on, and only on the prime $p$. We show that Clifford monitored circuits with on-site Hilbert space dimension $d=p^M$ are described by percolation in the limits $d \to \infty$ at (a) $p=$ fixed but $M\to \infty$, and at (b) $M= 1$ but $p \to \infty$. In the limit (a) we calculate the effective central charge, and in the limit (b) we derive the following universal minimal cut entanglement entropy $S_A =(\sqrt{3}/\pi)\ln p \ln L_A$ for $d=p$ large at the transition. We verify those predictions numerically, and present extensive numerical results for critical exponents at the transition in monitored Clifford circuits for prime number on-site Hilbert space dimension $d=p$ for a variety of different values of $p$, and find that they approach percolation values at large $p$., Comment: 23 pages, 5 figures. Abstract shortened to meet arxiv requirements, see pdf for full abstract

Human pose is a useful feature for fine-grained sports action understanding. However, pose estimators are often unreliable when run on sports video due to domain shift and factors such as motion blur and occlusions. This leads to poor accuracy when downstream tasks, such as action recognition, depend on pose. End-to-end learning circumvents pose, but requires more labels to generalize. We introduce Video Pose Distillation (VPD), a weakly-supervised technique to learn features for new video domains, such as individual sports that challenge pose estimation. Under VPD, a student network learns to extract robust pose features from RGB frames in the sports video, such that, whenever pose is considered reliable, the features match the output of a pretrained teacher pose detector. Our strategy retains the best of both pose and end-to-end worlds, exploiting the rich visual patterns in raw video frames, while learning features that agree with the athletes' pose and motion in the target video domain to avoid over-fitting to patterns unrelated to athletes' motion. VPD features improve performance on few-shot, fine-grained action recognition, retrieval, and detection tasks in four real-world sports video datasets, without requiring additional ground-truth pose annotations., Comment: ICCV 2021 (poster)

Generating font glyphs of consistent style from one or a few reference glyphs, i.e., font completion, is an important task in topographical design. As the problem is more well-defined than general image style transfer tasks, thus it has received interest from both vision and machine learning communities. Existing approaches address this problem as a direct image-to-image translation task. In this work, we innovate to explore the generation of font glyphs as 2D graphic objects with the graph as an intermediate representation, so that more intrinsic graphic properties of font styles can be captured. Specifically, we formulate a cross-modality cycled image-to-image model structure with a graph constructor between an image encoder and an image renderer. The novel graph constructor maps a glyph's latent code to its graph representation that matches expert knowledge, which is trained to help the translation task. Our model generates improved results than both image-to-image baseline and previous state-of-the-art methods for glyph completion. Furthermore, the graph representation output by our model also provides an intuitive interface for users to do local editing and manipulation. Our proposed cross-modality cycled representation learning has the potential to be applied to other domains with prior knowledge from different data modalities. Our code is available at https://github.com/VITA-Group/Font_Completion_Graph., Comment: submitted to IEEE Transactions on Multimedia (TMM)

We explore a class of "open" quantum circuit models with local decoherence ("noise") and local projective measurements, each respecting a global Z_2 symmetry. The model supports a spin glass phase where the Z_2 symmetry is spontaneously broken (not possible in an equilibrium 1d system), a paramagnetic phase characterized by a divergent susceptibility, and an intermediate "trivial" phase. All three phases are also stable to Z_2-symmetric local unitary gates, and the dynamical phase transitions between the phases are in the percolation universality class. The open circuit dynamics can be purified by explicitly introducing a bath with its own "scrambling" dynamics, as in [Bao, Choi, Altman, arXiv:2102.09164], which does not change any of the universal physics. Within the spin glass phase the circuit dynamics can be interpreted as a quantum repetition code, with each stabilizer of the code measured stochastically at a finite rate, and the decoherences as effective bit-flip errors. Motivated by the geometry of the spin glass phase, we devise a novel decoding algorithm for recovering an arbitrary initial qubit state in the code space, assuming knowledge of the history of the measurement outcomes, and the ability of performing local Pauli measurements and gates on the final state. For a circuit with L^d qubits running for time T, the time needed to execute the decoder scales as O(L^d T) (with dimensionality d). With this decoder in hand, we find that the information of the initial encoded qubit state can be retained (and then recovered) for a time logarithmic in L for a 1d circuit, and for a time at least linear in L in 2d below a finite error threshold. For both the repetition and toric codes, we compare and contrast our decoding algorithm with earlier algorithms that map the error model to the random bond Ising model., Comment: 32+epsilon pages, 22 figures. Published version

We investigate the problem of training generative models on a very sparse collection of 3D models. We use geometrically motivated energies to augment and thus boost a sparse collection of example (training) models. We analyze the Hessian of the as-rigid-as-possible (ARAP) energy to sample from and project to the underlying (local) shape space, and use the augmented dataset to train a variational autoencoder (VAE). We iterate the process of building latent spaces of VAE and augmenting the associated dataset, to progressively reveal a richer and more expressive generative space for creating geometrically and semantically valid samples. Our framework allows us to train generative 3D models even with a small set of good quality 3D models, which are typically hard to curate. We extensively evaluate our method against a set of strong baselines, provide ablation studies and demonstrate application towards establishing shape correspondences. We present multiple examples of interesting and meaningful shape variations even when starting from as few as 3-10 training shapes., Comment: CVPR 2022

We explore a class of random tensor network models with "stabilizer" local tensors which we name Random Stabilizer Tensor Networks (RSTNs). For RSTNs defined on a two-dimensional square lattice, we perform extensive numerical studies of entanglement phase transitions between volume-law and area-law entangled phases of the one-dimensional boundary states. These transitions occur when either (a) the bond dimension $D$ of the constituent tensors is varied, or (b) the tensor network is subject to random breaking of bulk bonds, implemented by forced measurements. In the absence of broken bonds, we find that the RSTN supports a volume-law entangled boundary state with bond dimension $D\geq3$ where $D$ is a prime number, and an area-law entangled boundary state for $D=2$. Upon breaking bonds at random in the bulk with probability $p$, there exists a critical measurement rate $p_c$ for each $D\geq 3$ above which the boundary state becomes area-law entangled. To explore the conformal invariance at these entanglement transitions for different prime $D$, we consider tensor networks on a finite rectangular geometry with a variety of boundary conditions, and extract universal operator scaling dimensions via extensive numerical calculations of the entanglement entropy, mutual information and mutual negativity at their respective critical points. Our results at large $D$ approach known universal data of percolation conformal field theory, while showing clear discrepancies at smaller $D$, suggesting a distinct entanglement transition universality class for each prime $D$. We further study universal entanglement properties in the volume-law phase and demonstrate quantitative agreement with the recently proposed description in terms of a directed polymer in a random environment., Comment: Updated to published version

Fonts are ubiquitous across documents and come in a variety of styles. They are either represented in a native vector format or rasterized to produce fixed resolution images. In the first case, the non-standard representation prevents benefiting from latest network architectures for neural representations; while, in the latter case, the rasterized representation, when encoded via networks, results in loss of data fidelity, as font-specific discontinuities like edges and corners are difficult to represent using neural networks. Based on the observation that complex fonts can be represented by a superposition of a set of simpler occupancy functions, we introduce \textit{multi-implicits} to represent fonts as a permutation-invariant set of learned implict functions, without losing features (e.g., edges and corners). However, while multi-implicits locally preserve font features, obtaining supervision in the form of ground truth multi-channel signals is a problem in itself. Instead, we propose how to train such a representation with only local supervision, while the proposed neural architecture directly finds globally consistent multi-implicits for font families. We extensively evaluate the proposed representation for various tasks including reconstruction, interpolation, and synthesis to demonstrate clear advantages with existing alternatives. Additionally, the representation naturally enables glyph completion, wherein a single characteristic font is used to synthesize a whole font family in the target style.

Monitored quantum dynamics reveal quantum state trajectories which exhibit a rich phenomenology of entanglement structures, including a transition from a weakly-monitored volume law entangled phase to a strongly-monitored area law phase. For one-dimensional hybrid circuits with both random unitary dynamics and interspersed measurements, we combine analytic mappings to an effective statistical mechanics model with extensive numerical simulations on hybrid Clifford circuits to demonstrate that the universal entanglement properties of the volume law phase can be quantitatively described by a fluctuating entanglement domain wall that is equivalent to a "directed polymer in a random environment" (DPRE). This relationship improves upon a qualitative "mean-field" statistical mechanics of the volume-law-entangled phase [1, 2]. For the Clifford circuit in various geometries, we obtain agreement between the subleading entanglement entropies and error correcting properties of the volume-law phase (which quantify its stability to projective measurements) with predictions of the DPRE. We further demonstrate that depolarizing noise in the hybrid dynamics near the final circuit time can drive a continuous phase transition to a non-error correcting volume law phase that is not immune to the disentangling action of projective measurements. We observe this transition in hybrid Clifford dynamics, and obtain quantitative agreement with critical exponents for a "pinning" phase transition of the DPRE in the presence of an attractive interface., Comment: 22 pages, 18 figures

Artists and video game designers often construct 2D animations using libraries of sprites -- textured patches of objects and characters. We propose a deep learning approach that decomposes sprite-based video animations into a disentangled representation of recurring graphic elements in a self-supervised manner. By jointly learning a dictionary of possibly transparent patches and training a network that places them onto a canvas, we deconstruct sprite-based content into a sparse, consistent, and explicit representation that can be easily used in downstream tasks, like editing or analysis. Our framework offers a promising approach for discovering recurring visual patterns in image collections without supervision., Comment: Accepted to NeurIPS 2021

We introduce CharacterGAN, a generative model that can be trained on only a few samples (8 - 15) of a given character. Our model generates novel poses based on keypoint locations, which can be modified in real time while providing interactive feedback, allowing for intuitive reposing and animation. Since we only have very limited training samples, one of the key challenges lies in how to address (dis)occlusions, e.g. when a hand moves behind or in front of a body. To address this, we introduce a novel layering approach which explicitly splits the input keypoints into different layers which are processed independently. These layers represent different parts of the character and provide a strong implicit bias that helps to obtain realistic results even with strong (dis)occlusions. To combine the features of individual layers we use an adaptive scaling approach conditioned on all keypoints. Finally, we introduce a mask connectivity constraint to reduce distortion artifacts that occur with extreme out-of-distribution poses at test time. We show that our approach outperforms recent baselines and creates realistic animations for diverse characters. We also show that our model can handle discrete state changes, for example a profile facing left or right, that the different layers do indeed learn features specific for the respective keypoints in those layers, and that our model scales to larger datasets when more data is available., Comment: Best Paper WACV 2022. Code available at https://github.com/tohinz/CharacterGAN

We introduce a deep generative network for 3D shape detailization, akin to stylization with the style being geometric details. We address the challenge of creating large varieties of high-resolution and detailed 3D geometry from a small set of exemplars by treating the problem as that of geometric detail transfer. Given a low-resolution coarse voxel shape, our network refines it, via voxel upsampling, into a higher-resolution shape enriched with geometric details. The output shape preserves the overall structure (or content) of the input, while its detail generation is conditioned on an input "style code" corresponding to a detailed exemplar. Our 3D detailization via conditional refinement is realized by a generative adversarial network, coined DECOR-GAN. The network utilizes a 3D CNN generator for upsampling coarse voxels and a 3D PatchGAN discriminator to enforce local patches of the generated model to be similar to those in the training detailed shapes. During testing, a style code is fed into the generator to condition the refinement. We demonstrate that our method can refine a coarse shape into a variety of detailed shapes with different styles. The generated results are evaluated in terms of content preservation, plausibility, and diversity. Comprehensive ablation studies are conducted to validate our network designs. Code is available at https://github.com/czq142857/DECOR-GAN., Comment: CVPR 2021 (oral). Code: https://github.com/czq142857/DECOR-GAN