Search

We introduce Temporal Residual Jacobians as a novel representation to enable data-driven motion transfer. Our approach does not assume access to any rigging or intermediate shape keyframes, produces geometrically and temporally consistent motions, and can be used to transfer long motion sequences. Central to our approach are two coupled neural networks that individually predict local geometric and temporal changes that are subsequently integrated, spatially and temporally, to produce the final animated meshes. The two networks are jointly trained, complement each other in producing spatial and temporal signals, and are supervised directly with 3D positional information. During inference, in the absence of keyframes, our method essentially solves a motion extrapolation problem. We test our setup on diverse meshes (synthetic and scanned shapes) to demonstrate its superiority in generating realistic and natural-looking animations on unseen body shapes against SoTA alternatives. Supplemental video and code are available at https://temporaljacobians.github.io/ ., Comment: 15 pages, 6 figures

Effective representation of 2D images is fundamental in digital image processing, where traditional methods like raster and vector graphics struggle with sharpness and textural complexity respectively. Current neural fields offer high-fidelity and resolution independence but require predefined meshes with known discontinuities, restricting their utility. We observe that by treating all mesh edges as potential discontinuities, we can represent the magnitude of discontinuities with continuous variables and optimize. Based on this observation, we introduce a novel discontinuous neural field model that jointly approximate the target image and recovers discontinuities. Through systematic evaluations, our neural field demonstrates superior performance in denoising and super-resolution tasks compared to InstantNGP, achieving improvements of over 5dB and 10dB, respectively. Our model also outperforms Mumford-Shah-based methods in accurately capturing discontinuities, with Chamfer distances 3.5x closer to the ground truth. Additionally, our approach shows remarkable capability in handling complex artistic drawings and natural images.

Measurements can detect errors in a decohered quantum memory allowing active error correction to increase the memory time. Previous understanding of this mechanism has focused on evaluating the performance of error correction algorithms based on measurement results. In this work, we instead intrinsically characterize the information dynamics in a quantum memory under repeated measurements, using coherent information and relative entropy. We consider the dynamics of a $d$-dimensional stabilizer code subject to Pauli errors and noisy stabilizer measurements and develop a $(d+1)$-dimensional statistical mechanics model for the information-theoretic diagnostics. Our model is dual to the model previously obtained for the optimal decoding algorithm, and the potential decoding transition in the quantum memory again manifests as a thermal phase transition in the statistical mechanics model. We explicitly derive the model and study the phase transition in information encoding in three examples: surface codes, repetition codes, and the XZZX code., Comment: 27 pages, 9 figures

The success of denoising diffusion models in representing rich data distributions over 2D raster images has prompted research on extending them to other data representations, such as vector graphics. Unfortunately due to their variable structure and scarcity of vector training data, directly applying diffusion models on this domain remains a challenging problem. Using workarounds like optimization via Score Distillation Sampling (SDS) is also fraught with difficulty, as vector representations are non trivial to directly optimize and tend to result in implausible geometries such as redundant or self-intersecting shapes. NIVeL addresses these challenges by reinterpreting the problem on an alternative, intermediate domain which preserves the desirable properties of vector graphics -- mainly sparsity of representation and resolution-independence. This alternative domain is based on neural implicit fields expressed in a set of decomposable, editable layers. Based on our experiments, NIVeL produces text-to-vector graphics results of significantly better quality than the state-of-the-art.

We present personalized residuals and localized attention-guided sampling for efficient concept-driven generation using text-to-image diffusion models. Our method first represents concepts by freezing the weights of a pretrained text-conditioned diffusion model and learning low-rank residuals for a small subset of the model's layers. The residual-based approach then directly enables application of our proposed sampling technique, which applies the learned residuals only in areas where the concept is localized via cross-attention and applies the original diffusion weights in all other regions. Localized sampling therefore combines the learned identity of the concept with the existing generative prior of the underlying diffusion model. We show that personalized residuals effectively capture the identity of a concept in ~3 minutes on a single GPU without the use of regularization images and with fewer parameters than previous models, and localized sampling allows using the original model as strong prior for large parts of the image., Comment: CVPR 2024. Project page at https://cusuh.github.io/personalized-residuals

Animated logos are a compelling and ubiquitous way individuals and brands represent themselves online. Manually authoring these logos can require significant artistic skill and effort. To help novice designers animate logos, design tools currently offer templates and animation presets. However, these solutions can be limited in their expressive range. Large language models have the potential to help novice designers create animated logos by generating animation code that is tailored to their content. In this paper, we introduce LogoMotion, an LLM-based system that takes in a layered document and generates animated logos through visually-grounded program synthesis. We introduce techniques to create an HTML representation of a canvas, identify primary and secondary elements, synthesize animation code, and visually debug animation errors. When compared with an industry standard tool, we find that LogoMotion produces animations that are more content-aware and are on par in terms of quality. We conclude with a discussion of the implications of LLM-generated animation for motion design.

Procedural noise is a fundamental component of computer graphics pipelines, offering a flexible way to generate textures that exhibit "natural" random variation. Many different types of noise exist, each produced by a separate algorithm. In this paper, we present a single generative model which can learn to generate multiple types of noise as well as blend between them. In addition, it is capable of producing spatially-varying noise blends despite not having access to such data for training. These features are enabled by training a denoising diffusion model using a novel combination of data augmentation and network conditioning techniques. Like procedural noise generators, the model's behavior is controllable via interpretable parameters and a source of randomness. We use our model to produce a variety of visually compelling noise textures. We also present an application of our model to improving inverse procedural material design; using our model in place of fixed-type noise nodes in a procedural material graph results in higher-fidelity material reconstructions without needing to know the type of noise in advance., Comment: In ACM Transactions on Graphics (Proceedings of SIGGRAPH) 2024, 21 pages

Quantum systems subject to random unitary evolution and measurements at random points in spacetime exhibit entanglement phase transitions which depend on the frequency of these measurements. Past work has experimentally observed entanglement phase transitions on near-term quantum computers, but the characterization approach using entanglement entropy is not scalable due to exponential overhead of quantum state tomography and post selection. Recently, an alternative protocol to detect entanglement phase transitions using linear cross-entropy was proposed which eliminates both bottlenecks. Here, we report the demonstration of this protocol in systems with one-dimensional and all-to-all connectivities on IBM's quantum hardware on up to 22 qubits, a regime which is presently inaccessible if post-selection is required. We demonstrate a collapse of the data into a scale-invariant form with critical exponents agreeing with theory within uncertainty. Our demonstration paves the way for studies of measurement-induced entanglement phase transitions and associated critical phenomena on larger near-term quantum systems., Comment: 23 pages, 15 figures

Current controls over diffusion models (e.g., through text or ControlNet) for image generation fall short in recognizing abstract, continuous attributes like illumination direction or non-rigid shape change. In this paper, we present an approach for allowing users of text-to-image models to have fine-grained control of several attributes in an image. We do this by engineering special sets of input tokens that can be transformed in a continuous manner -- we call them Continuous 3D Words. These attributes can, for example, be represented as sliders and applied jointly with text prompts for fine-grained control over image generation. Given only a single mesh and a rendering engine, we show that our approach can be adopted to provide continuous user control over several 3D-aware attributes, including time-of-day illumination, bird wing orientation, dollyzoom effect, and object poses. Our method is capable of conditioning image creation with multiple Continuous 3D Words and text descriptions simultaneously while adding no overhead to the generative process. Project Page: https://ttchengab.github.io/continuous_3d_words, Comment: Project Page: https://ttchengab.github.io/continuous_3d_words

How to frame (or crop) a photo often depends on the image subject and its context; e.g., a human portrait. Recent works have defined the subject-aware image cropping task as a nuanced and practical version of image cropping. We propose a weakly-supervised approach (GenCrop) to learn what makes a high-quality, subject-aware crop from professional stock images. Unlike supervised prior work, GenCrop requires no new manual annotations beyond the existing stock image collection. The key challenge in learning from this data, however, is that the images are already cropped and we do not know what regions were removed. Our insight is to combine a library of stock images with a modern, pre-trained text-to-image diffusion model. The stock image collection provides diversity and its images serve as pseudo-labels for a good crop, while the text-image diffusion model is used to out-paint (i.e., outward inpainting) realistic uncropped images. Using this procedure, we are able to automatically generate a large dataset of cropped-uncropped training pairs to train a cropping model. Despite being weakly-supervised, GenCrop is competitive with state-of-the-art supervised methods and significantly better than comparable weakly-supervised baselines on quantitative and qualitative evaluation metrics., Comment: AAAI 24. Extended version with supplemental materials

We present VecFusion, a new neural architecture that can generate vector fonts with varying topological structures and precise control point positions. Our approach is a cascaded diffusion model which consists of a raster diffusion model followed by a vector diffusion model. The raster model generates low-resolution, rasterized fonts with auxiliary control point information, capturing the global style and shape of the font, while the vector model synthesizes vector fonts conditioned on the low-resolution raster fonts from the first stage. To synthesize long and complex curves, our vector diffusion model uses a transformer architecture and a novel vector representation that enables the modeling of diverse vector geometry and the precise prediction of control points. Our experiments show that, in contrast to previous generative models for vector graphics, our new cascaded vector diffusion model generates higher quality vector fonts, with complex structures and diverse styles.

We introduce a novel image-to-painting method that facilitates the creation of large-scale, high-fidelity paintings with human-like quality and stylistic variation. To process large images and gain control over the painting process, we introduce a segmentation-based painting process and a dynamic attention map approach inspired by human painting strategies, allowing optimization of brush strokes to proceed in batches over different image regions, thereby capturing both large-scale structure and fine details, while also allowing stylistic control over detail. Our optimized batch processing and patch-based loss framework enable efficient handling of large canvases, ensuring our painted outputs are both aesthetically compelling and functionally superior as compared to previous methods, as confirmed by rigorous evaluations. Code available at: https://github.com/manuelladron/semantic\_based\_painting.git, Comment: 8 pages

Exploring variations of 3D shapes is a time-consuming process in traditional 3D modeling tools. Deep generative models of 3D shapes often feature continuous latent spaces that can, in principle, be used to explore potential variations starting from a set of input shapes. In practice, doing so can be problematic: latent spaces are high dimensional and hard to visualize, contain shapes that are not relevant to the input shapes, and linear paths through them often lead to sub-optimal shape transitions. Furthermore, one would ideally be able to explore variations in the original high-quality meshes used to train the generative model, not its lower-quality output geometry. In this paper, we present a method to explore variations among a given set of landmark shapes by constructing a mapping from an easily-navigable 2D exploration space to a subspace of a pre-trained generative model. We first describe how to find a mapping that spans the set of input landmark shapes and exhibits smooth variations between them. We then show how to turn the variations in this subspace into deformation fields, to transfer those variations to high-quality meshes for the landmark shapes. Our results show that our method can produce visually-pleasing and easily-navigable 2D exploration spaces for several different shape categories, especially as compared to prior work on learning deformation spaces for 3D shapes., Comment: SIGGRAPH Asia 2023, 15 pages

The linear cross-entropy (LXE) has been recently proposed as a scalable probe of the measurement-driven phase transition between volume- and area-law-entangled phases of pure-state trajectories in certain monitored quantum circuits. Here, we demonstrate that the LXE can distinguish distinct area-law-entangled phases of monitored circuits with symmetries, and extract universal behavior at the critical points separating these phases. We focus on (1+1)-dimensional monitored circuits with an on-site $\mathbb{Z}_{2}$ symmetry. For an appropriate choice of initial states, the LXE distinguishes the area-law-entangled spin glass and paramagnetic phases of the monitored trajectories. At the critical point, described by two-dimensional percolation, the LXE exhibits universal behavior which depends sensitively on boundary conditions, and the choice of initial states. With open boundary conditions, we show that the LXE relates to crossing probabilities in critical percolation, and is thus given by a known universal function of the aspect ratio of the dynamics, which quantitatively agrees with numerical studies of the LXE at criticality. The LXE probes correlations of other operators in percolation with periodic boundary conditions. We show that the LXE is sensitive to the richer phase diagram of the circuit model in the presence of symmmetric unitary gates. Lastly, we consider the effect of noise during the circuit evolution, and propose potential solutions to counter it., Comment: 12+6 pages, 16 figures. V2: References added

Automatic generation of fonts can be an important aid to typeface design. Many current approaches regard glyphs as pixelated images, which present artifacts when scaling and inevitable quality losses after vectorization. On the other hand, existing vector font synthesis methods either fail to represent the shape concisely or require vector supervision during training. To push the quality of vector font synthesis to the next level, we propose a novel dual-part representation for vector glyphs, where each glyph is modeled as a collection of closed "positive" and "negative" path pairs. The glyph contour is then obtained by boolean operations on these paths. We first learn such a representation only from glyph images and devise a subsequent contour refinement step to align the contour with an image representation to further enhance details. Our method, named DualVector, outperforms state-of-the-art methods in vector font synthesis both quantitatively and qualitatively. Our synthesized vector fonts can be easily converted to common digital font formats like TrueType Font for practical use. The code is released at https://github.com/thuliu-yt16/dualvector., Comment: CVPR 2023

Adaptive quantum circuits, in which unitary operations, measurements, and feedback are used to steer quantum many-body systems, provide an exciting opportunity to generate new dynamical steady states. We introduce an adaptive quantum dynamics with continuous symmetry where unitary operations, measurements, and local unitary feedback are used to drive ordering. In this setting, we find a pure steady state hosting symmetry-breaking order, which is the ground state of a gapless, local Hamiltonian. We explore the dynamical properties of the approach to this steady state. We find that this steady-state order is fragile to perturbations, even those that respect the continuous symmetry., Comment: 17 pages, 10 figures

This report surveys advances in deep learning-based modeling techniques that address four different 3D indoor scene analysis tasks, as well as synthesis of 3D indoor scenes. We describe different kinds of representations for indoor scenes, various indoor scene datasets available for research in the aforementioned areas, and discuss notable works employing machine learning models for such scene modeling tasks based on these representations. Specifically, we focus on the analysis and synthesis of 3D indoor scenes. With respect to analysis, we focus on four basic scene understanding tasks -- 3D object detection, 3D scene segmentation, 3D scene reconstruction and 3D scene similarity. And for synthesis, we mainly discuss neural scene synthesis works, though also highlighting model-driven methods that allow for human-centric, progressive scene synthesis. We identify the challenges involved in modeling scenes for these tasks and the kind of machinery that needs to be developed to adapt to the data representation, and the task setting in general. For each of these tasks, we provide a comprehensive summary of the state-of-the-art works across different axes such as the choice of data representation, backbone, evaluation metric, input, output, etc., providing an organized review of the literature. Towards the end, we discuss some interesting research directions that have the potential to make a direct impact on the way users interact and engage with these virtual scene models, making them an integral part of the metaverse., Comment: Published in Computer Graphics Forum, Aug 2023

Representing a 3D shape with a set of primitives can aid perception of structure, improve robotic object manipulation, and enable editing, stylization, and compression of 3D shapes. Existing methods either use simple parametric primitives or learn a generative shape space of parts. Both have limitations: parametric primitives lead to coarse approximations, while learned parts offer too little control over the decomposition. We instead propose to decompose shapes using a library of 3D parts provided by the user, giving full control over the choice of parts. The library can contain parts with high-quality geometry that are suitable for a given category, resulting in meaningful decompositions with clean geometry. The type of decomposition can also be controlled through the choice of parts in the library. Our method works via a self-supervised approach that iteratively retrieves parts from the library and refines their placements. We show that this approach gives higher reconstruction accuracy and more desirable decompositions than existing approaches. Additionally, we show how the decomposition can be controlled through the part library by using different part libraries to reconstruct the same shapes., Comment: CVPR 2023

The Floquet code utilizes a periodic sequence of two-qubit measurements to realize the topological order. After each measurement round, the instantaneous stabilizer group can be mapped to a honeycomb toric code, explaining the topological feature. The code also possesses a time-crystal order - the $e-m$ transmutation after every cycle, breaking the Floquet symmetry of the measurement schedule. This behavior is distinct from the stationary topological order realized in either random circuits or time-independent Hamiltonian. Therefore, the resultant phase belongs to the overlap between the classes of Floquet enriched topological orders and measurement-induced phases. In this work, we construct a continuous path interpolating between the Floquet and toric codes, focusing on the transition between the time-crystal and stationary topological phases. We show that this transition is characterized by a divergent length scale. We also add single-qubit perturbations to the model and obtain a richer two-dimensional parametric phase diagram of the Floquet code, showing the stability of the Floquet enriched topological order., Comment: 6+8 pages, 13 figures

Diffusion models currently achieve state-of-the-art performance for both conditional and unconditional image generation. However, so far, image diffusion models do not support tasks required for 3D understanding, such as view-consistent 3D generation or single-view object reconstruction. In this paper, we present RenderDiffusion, the first diffusion model for 3D generation and inference, trained using only monocular 2D supervision. Central to our method is a novel image denoising architecture that generates and renders an intermediate three-dimensional representation of a scene in each denoising step. This enforces a strong inductive structure within the diffusion process, providing a 3D consistent representation while only requiring 2D supervision. The resulting 3D representation can be rendered from any view. We evaluate RenderDiffusion on FFHQ, AFHQ, ShapeNet and CLEVR datasets, showing competitive performance for generation of 3D scenes and inference of 3D scenes from 2D images. Additionally, our diffusion-based approach allows us to use 2D inpainting to edit 3D scenes., Comment: Accepted at CVPR 2023. Project page: https://github.com/Anciukevicius/RenderDiffusion

The coherent superposition of quantum states is an important resource for quantum information processing which distinguishes quantum dynamics and information from their classical counterparts. In this article we determine the coherence requirements to communicate quantum information in a broad setting encompassing monitored quantum dynamics and quantum error correction codes. We determine these requirements by considering hybrid circuits that are generated by a quantum information game played between two opponents, Alice and Eve, who compete by applying unitaries and measurements on a fixed number of qubits. Alice applies unitaries in an attempt to maintain quantum channel capacity, while Eve applies measurements in an attempt to destroy it. By limiting the coherence generating or destroying operations available to each opponent, we determine Alice's coherence requirements. When Alice plays a random strategy aimed at mimicking generic monitored quantum dynamics, we discover a coherence-tuned phase transitions in entanglement and quantum channel capacity. We then derive a theorem giving the minimum coherence required by Alice in any successful strategy, and conclude by proving that coherence sets an upper bound on the code distance in any stabelizer quantum error correction codes. Such bounds provide a rigorous quantification of the coherence resource requirements for quantum communication and error correction., Comment: 19+11 pages, 12+3 figures

We investigate the prospects of employing the linear cross-entropy to experimentally access measurement-induced phase transitions (MIPT) without requiring any postselection of quantum trajectories. For two random circuits that are identical in the bulk but with different initial states, the linear cross-entropy $\chi$ between the bulk measurement outcome distributions in the two circuits acts as a boundary order parameter, and can be used to distinguish the volume law from area law phases. In the volume law phase (and in the thermodynamic limit) the bulk measurements cannot distinguish between the two different initial states, and $\chi = 1$. In the area law phase $\chi < 1$. For circuits with Clifford gates, we provide numerical evidence that $\chi$ can be sampled to accuracy $\epsilon$ from $O(1/\epsilon^2)$ trajectories, by running the first circuit on a quantum simulator without postselection, aided by a classical simulation of the second. We also find that for weak depolarizing noise the signature of the MIPT is still present for intermediate system sizes. In our protocol we have the freedom of choosing initial states such that the "classical" side can be simulated efficiently, while simulating the "quantum" side is still classically hard., Comment: 7+8 pages, 6 figures. v2: 7+9 pages, 3+3 figures. Updated discussions on sample size (Fig. 2d, 2e), and new results from random Haar circuits (Fig. 3b). Accepted version

Measurements allow efficient preparation of interesting quantum many-body states with long-range entanglement, conditioned on additional transformations based on measurement outcomes. Here, we demonstrate that the so-called conformal quantum critical points (CQCP) can be obtained by performing general single-site measurements in an appropriate basis on the cluster states in $d\geq2$. The equal-time correlators of the said states are described by correlation functions of certain $d$-dimensional classical models at finite temperatures and feature spatial conformal invariance. This establishes an exact correspondence between the measurement-prepared critical states and conformal field theories of a range of critical spin models, including familiar Ising models and gauge theories. Furthermore, by mapping the long-range entanglement structure of measured quantum states into the correlations of the corresponding thermal spin model, we rigorously establish the stability condition of the long-range entanglement in the measurement-prepared quantum states deviating from the ideal setting. Most importantly, we describe protocols to decode the resulting quantum phases and transitions without post-selection, thus transferring the exponential measurement complexity to a polynomial classical computation. Therefore, our findings suggest a novel mechanism in which a quantum critical wavefunction emerges, providing new practical ways to study quantum phases and conformal quantum critical points., Comment: 37 pages, 11 figures

Quantum circuits -- built from local unitary gates and local measurements -- are a new playground for quantum many-body physics and a tractable setting to explore universal collective phenomena far-from-equilibrium. These models have shed light on longstanding questions about thermalization and chaos, and on the underlying universal dynamics of quantum information and entanglement. In addition, such models generate new sets of questions and give rise to phenomena with no traditional analog, such as new dynamical phases in quantum systems that are monitored by an external observer. Quantum circuit dynamics is also topical in view of experimental progress in building digital quantum simulators that allow control of precisely these ingredients. Randomness in the circuit elements allows a high level of theoretical control, with a key theme being mappings between real-time quantum dynamics and effective classical lattice models or dynamical processes. Many of the universal phenomena that can be identified in this tractable setting apply to much wider classes of more structured many-body dynamics., Comment: Review article for Annual Review of Condensed Matter Physics; comments welcome