Search

Image editing involves a variety of complex tasks and requires efficient and precise manipulation techniques. In this paper, we present MagicQuill, an integrated image editing system that enables swift actualization of creative ideas. Our system features a streamlined yet functionally robust interface, allowing for the articulation of editing operations (e.g., inserting elements, erasing objects, altering color) with minimal input. These interactions are monitored by a multimodal large language model (MLLM) to anticipate editing intentions in real time, bypassing the need for explicit prompt entry. Finally, we apply a powerful diffusion prior, enhanced by a carefully learned two-branch plug-in module, to process editing requests with precise control. Experimental results demonstrate the effectiveness of MagicQuill in achieving high-quality image edits. Please visit https://magic-quill.github.io to try out our system., Comment: Code and demo available at https://magic-quill.github.io

We propose Framer for interactive frame interpolation, which targets producing smoothly transitioning frames between two images as per user creativity. Concretely, besides taking the start and end frames as inputs, our approach supports customizing the transition process by tailoring the trajectory of some selected keypoints. Such a design enjoys two clear benefits. First, incorporating human interaction mitigates the issue arising from numerous possibilities of transforming one image to another, and in turn enables finer control of local motions. Second, as the most basic form of interaction, keypoints help establish the correspondence across frames, enhancing the model to handle challenging cases (e.g., objects on the start and end frames are of different shapes and styles). It is noteworthy that our system also offers an "autopilot" mode, where we introduce a module to estimate the keypoints and refine the trajectory automatically, to simplify the usage in practice. Extensive experimental results demonstrate the appealing performance of Framer on various applications, such as image morphing, time-lapse video generation, cartoon interpolation, etc. The code, the model, and the interface will be released to facilitate further research., Comment: Project page: https://aim-uofa.github.io/Framer/

This study introduces a high-order perturbation methodology to categorize two primary solution types within duality-invariant nonlinear electrodynamic theories, adhering to the differential self-duality criterion. The first solution type aligns with irrelevant stress tensor flows, resembling $T\bar{T}$ dynamics, and the second involves a blend of irrelevant $T\bar{T}$-like and marginal root-$T\bar{T}$-like deformations. Our approach facilitates the investigation of diverse duality-invariant nonlinear electrodynamics theories and their stress tensor flows and confirms the commutativity of flows initiated by irrelevant and marginal operators., Comment: 25 pages, 1 figure

We develop a generic geometric formalism that incorporates both $T\bar{T}$-like and root-$T\bar{T}$-like deformations in arbitrary dimensions. This framework applies to a wide family of stress-energy tensor perturbations and encompasses various well-known field theories. Building upon the recently proposed correspondence between Ricci-based gravity and $T\bar{T}$-like deformations, we further extend this duality to include root-$T\bar{T}$-like perturbations. This refinement extends the potential applications of our approach and contributes to a deeper exploration of the interplay between stress tensor perturbations and gravitational dynamics. Among the various original outcomes detailed in this article, we have also obtained a deformation of the flat Jackiw-Teitelboim gravity action., Comment: 11 pages, no figures; v2: references added

Text animation serves as an expressive medium, transforming static communication into dynamic experiences by infusing words with motion to evoke emotions, emphasize meanings, and construct compelling narratives. Crafting animations that are semantically aware poses significant challenges, demanding expertise in graphic design and animation. We present an automated text animation scheme, termed "Dynamic Typography", which combines two challenging tasks. It deforms letters to convey semantic meaning and infuses them with vibrant movements based on user prompts. Our technique harnesses vector graphics representations and an end-to-end optimization-based framework. This framework employs neural displacement fields to convert letters into base shapes and applies per-frame motion, encouraging coherence with the intended textual concept. Shape preservation techniques and perceptual loss regularization are employed to maintain legibility and structural integrity throughout the animation process. We demonstrate the generalizability of our approach across various text-to-video models and highlight the superiority of our end-to-end methodology over baseline methods, which might comprise separate tasks. Through quantitative and qualitative evaluations, we demonstrate the effectiveness of our framework in generating coherent text animations that faithfully interpret user prompts while maintaining readability. Our code is available at: https://animate-your-word.github.io/demo/., Comment: Our demo and code is available at: https://animate-your-word.github.io/demo/

3D Gaussians have recently emerged as an efficient representation for novel view synthesis. This work studies its editability with a particular focus on the inpainting task, which aims to supplement an incomplete set of 3D Gaussians with additional points for visually harmonious rendering. Compared to 2D inpainting, the crux of inpainting 3D Gaussians is to figure out the rendering-relevant properties of the introduced points, whose optimization largely benefits from their initial 3D positions. To this end, we propose to guide the point initialization with an image-conditioned depth completion model, which learns to directly restore the depth map based on the observed image. Such a design allows our model to fill in depth values at an aligned scale with the original depth, and also to harness strong generalizability from largescale diffusion prior. Thanks to the more accurate depth completion, our approach, dubbed InFusion, surpasses existing alternatives with sufficiently better fidelity and efficiency under various complex scenarios. We further demonstrate the effectiveness of InFusion with several practical applications, such as inpainting with user-specific texture or with novel object insertion., Comment: Project page: https://johanan528.github.io/Infusion

This work presents 3DPE, a practical method that can efficiently edit a face image following given prompts, like reference images or text descriptions, in a 3D-aware manner. To this end, a lightweight module is distilled from a 3D portrait generator and a text-to-image model, which provide prior knowledge of face geometry and superior editing capability, respectively. Such a design brings two compelling advantages over existing approaches. First, our method achieves real-time editing with a feedforward network (i.e., ~0.04s per image), over 100x faster than the second competitor. Second, thanks to the powerful priors, our module could focus on the learning of editing-related variations, such that it manages to handle various types of editing simultaneously in the training phase and further supports fast adaptation to user-specified customized types of editing during inference (e.g., with ~5min fine-tuning per style)., Comment: ECCV 2024 camera-ready version. Project page: https://github.com/EzioBy/3dpe

We introduce Text2Immersion, an elegant method for producing high-quality 3D immersive scenes from text prompts. Our proposed pipeline initiates by progressively generating a Gaussian cloud using pre-trained 2D diffusion and depth estimation models. This is followed by a refining stage on the Gaussian cloud, interpolating and refining it to enhance the details of the generated scene. Distinct from prevalent methods that focus on single object or indoor scenes, or employ zoom-out trajectories, our approach generates diverse scenes with various objects, even extending to the creation of imaginary scenes. Consequently, Text2Immersion can have wide-ranging implications for various applications such as virtual reality, game development, and automated content creation. Extensive evaluations demonstrate that our system surpasses other methods in rendering quality and diversity, further progressing towards text-driven 3D scene generation. We will make the source code publicly accessible at the project page., Comment: Project page: https://ken-ouyang.github.io/text2immersion/index.html

Neural radiance fields, which represent a 3D scene as a color field and a density field, have demonstrated great progress in novel view synthesis yet are unfavorable for editing due to the implicitness. In view of such a deficiency, we propose to replace the color field with an explicit 2D appearance aggregation, also called canonical image, with which users can easily customize their 3D editing via 2D image processing. To avoid the distortion effect and facilitate convenient editing, we complement the canonical image with a projection field that maps 3D points onto 2D pixels for texture lookup. This field is carefully initialized with a pseudo canonical camera model and optimized with offset regularity to ensure naturalness of the aggregated appearance. Extensive experimental results on three datasets suggest that our representation, dubbed AGAP, well supports various ways of 3D editing (e.g., stylization, interactive drawing, and content extraction) with no need of re-optimization for each case, demonstrating its generalizability and efficiency. Project page is available at https://felixcheng97.github.io/AGAP/., Comment: Project Webpage: https://felixcheng97.github.io/AGAP/, Code: https://github.com/felixcheng97/AGAP

We construct supersymmetric fermionic Wilson loops along general curves in four-dimensional $\mathcal{N}=4$ super Yang-Mills theory and along general planar curves in $\mathcal{N}=2$ superconformal $SU(N)\times SU(N)$ quiver theory. These loops are generalizations of the Zarembo loops and are cohomologically equivalent to them. In $\mathcal{N}=4$ super Yang-Mills theory, we compute their expectation values and verify the cohomological equivalence relation up to the order $g^4$ in perturbation theory., Comment: v1, 18 pages, 4 figures; v2, minor changes, 18 pages, 4 figures; v3, minor changes, 18 pages, 4 figures

We present the content deformation field CoDeF as a new type of video representation, which consists of a canonical content field aggregating the static contents in the entire video and a temporal deformation field recording the transformations from the canonical image (i.e., rendered from the canonical content field) to each individual frame along the time axis.Given a target video, these two fields are jointly optimized to reconstruct it through a carefully tailored rendering pipeline.We advisedly introduce some regularizations into the optimization process, urging the canonical content field to inherit semantics (e.g., the object shape) from the video.With such a design, CoDeF naturally supports lifting image algorithms for video processing, in the sense that one can apply an image algorithm to the canonical image and effortlessly propagate the outcomes to the entire video with the aid of the temporal deformation field.We experimentally show that CoDeF is able to lift image-to-image translation to video-to-video translation and lift keypoint detection to keypoint tracking without any training.More importantly, thanks to our lifting strategy that deploys the algorithms on only one image, we achieve superior cross-frame consistency in processed videos compared to existing video-to-video translation approaches, and even manage to track non-rigid objects like water and smog.Project page can be found at https://qiuyu96.github.io/CoDeF/., Comment: Project Webpage: https://qiuyu96.github.io/CoDeF/, Code: https://github.com/qiuyu96/CoDeF

We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views while preserving specific details of the input image. High-fidelity 3D GAN inversion is inherently challenging due to the geometry-texture trade-off in 3D inversion, where overfitting to a single view input image often damages the estimated geometry during the latent optimization. To solve this challenge, we propose a novel pipeline that builds on the pseudo-multi-view estimation with visibility analysis. We keep the original textures for the visible parts and utilize generative priors for the occluded parts. Extensive experiments show that our approach achieves advantageous reconstruction and novel view synthesis quality over state-of-the-art methods, even for images with out-of-distribution textures. The proposed pipeline also enables image attribute editing with the inverted latent code and 3D-aware texture modification. Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content., Comment: Project website: https://ken-ouyang.github.io/HFGI3D/index.html ; Github link: https://github.com/jiaxinxie97/HFGI3D

We propose to use pretraining to boost general image-to-image translation. Prior image-to-image translation methods usually need dedicated architectural design and train individual translation models from scratch, struggling for high-quality generation of complex scenes, especially when paired training data are not abundant. In this paper, we regard each image-to-image translation problem as a downstream task and introduce a simple and generic framework that adapts a pretrained diffusion model to accommodate various kinds of image-to-image translation. We also propose adversarial training to enhance the texture synthesis in the diffusion model training, in conjunction with normalized guidance sampling to improve the generation quality. We present extensive empirical comparison across various tasks on challenging benchmarks such as ADE20K, COCO-Stuff, and DIODE, showing the proposed pretraining-based image-to-image translation (PITI) is capable of synthesizing images of unprecedented realism and faithfulness., Comment: Project Page: https://tengfei-wang.github.io/PITI/index.html

We construct for the first time Drukker-Trancanelli (DT) type fermionic Bogomol\`nyi-Prasad-Sommerfield (BPS) Wilson loops in four-dimensional $\mathcal{N}=2$ superconformal $SU(N)\times SU(N)$ quiver theory and $\mathcal{N}=4$ super Yang-Mills theory. The connections of these fermionic BPS Wilson loops have a supermatrix structure. We construct timelike BPS Wilson lines in Minkowski spacetime and circular BPS Wilson loops in Euclidean space. These Wilson loops involve dimensionful parameters. For generic values of parameters, they preserve one real (complex) supercharge in Lorentzian (Euclidean) signature. Supersymmetry enhancement for Wilson loops happens when the parameters satisfy certain constraints. The nature of such loops is quite different from the Wilson loop operators involving fermions constructed previously in the literature on four-dimensional gauge theories. We hope that further investigations of such new Wilson loops will explore deep structures in both the gauge theories and gauge/gravity dualities., Comment: 26 pages, no figures; v2, 27 pages, no figures, typos fixed, references added, minor changes; v3 29 pages, no figures, minor revision, typos fixed, references added; v3, 29 pages, no figures,minor reversion

In this work we calculate the partition functions of $\mathcal{N}=1$ type 0A and 0B JT supergravity (SJT) on 2D surfaces of arbitrary genus with multiple finite cut-off boundaries, based on the $T\bar{T}$ deformed super-Schwarzian theories. In terms of SJT/matrix model duality, we compute the corresponding correlation functions in the $T\bar{T}$ deformed matrix model side by using topological recursion relations as well as the transformation properties of topological recursion relations under $T\bar{T}$ deformation. We check that the partition functions finite cut-off 0A and 0B SJT on generic 2D surfaces match the associated correlation functions in $T\bar{T}$ deformed matrix models respectively., Comment: 27 pages. A replacement of v1 whose PDF cannot be opened. References added

We propose pose-guided multiplane image (MPI) synthesis which can render an animatable character in real scenes with photorealistic quality. We use a portable camera rig to capture the multi-view images along with the driving signal for the moving subject. Our method generalizes the image-to-image translation paradigm, which translates the human pose to a 3D scene representation -- MPIs that can be rendered in free viewpoints, using the multi-views captures as supervision. To fully cultivate the potential of MPI, we propose depth-adaptive MPI which can be learned using variable exposure images while being robust to inaccurate camera registration. Our method demonstrates advantageous novel-view synthesis quality over the state-of-the-art approaches for characters with challenging motions. Moreover, the proposed method is generalizable to novel combinations of training poses and can be explicitly controlled. Our method achieves such expressive and animatable character rendering all in real time, serving as a promising solution for practical applications., Comment: Project website: https://ken-ouyang.github.io/cmpi/index.html

We compute the minimal area of a string worldsheet ending on two infinite periodic light-like Wilson lines in the AdS$_3$ boundary, which is dual to the first non-planar correction to the gluon scattering amplitude in $\mathcal{N}=4$ SYM at the strong coupling. Using the connection between the Hitchin system and the thermodynamic Bethe ansatz (TBA) equations, we present an analytic method to compute the minimal area surface and express the non-trivial part of the minimal area in terms of the free energy of the TBA-like equations. Given the cross ratios as inputs, the area computed from the TBA-like equations matches that calculated using the numerical integration., Comment: 31 pages, 8 figures

Applying an image processing algorithm independently to each video frame often leads to temporal inconsistency in the resulting video. To address this issue, we present a novel and general approach for blind video temporal consistency. Our method is only trained on a pair of original and processed videos directly instead of a large dataset. Unlike most previous methods that enforce temporal consistency with optical flow, we show that temporal consistency can be achieved by training a convolutional neural network on a video with Deep Video Prior (DVP). Moreover, a carefully designed iteratively reweighted training strategy is proposed to address the challenging multimodal inconsistency problem. We demonstrate the effectiveness of our approach on 7 computer vision tasks on videos. Extensive quantitative and perceptual experiments show that our approach obtains superior performance than state-of-the-art methods on blind video temporal consistency. We further extend DVP to video propagation and demonstrate its effectiveness in propagating three different types of information (color, artistic style, and object segmentation). A progressive propagation strategy with pseudo labels is also proposed to enhance DVP's performance on video propagation. Our source codes are publicly available at https://github.com/ChenyangLEI/deep-video-prior., Comment: Accepted by TPAMI in Dec 2021; extension of NeurIPS2020 Blind Video Temporal Consistency via Deep Video Prior. arXiv admin note: substantial text overlap with arXiv:2010.11838

We propose a novel framework for video inpainting by adopting an internal learning strategy. Unlike previous methods that use optical flow for cross-frame context propagation to inpaint unknown regions, we show that this can be achieved implicitly by fitting a convolutional neural network to known regions. Moreover, to handle challenging sequences with ambiguous backgrounds or long-term occlusion, we design two regularization terms to preserve high-frequency details and long-term temporal consistency. Extensive experiments on the DAVIS dataset demonstrate that the proposed method achieves state-of-the-art inpainting quality quantitatively and qualitatively. We further extend the proposed method to another challenging task: learning to remove an object from a video giving a single object mask in only one frame in a 4K video., Comment: ICCV 2021

Although recent inpainting approaches have demonstrated significant improvements with deep neural networks, they still suffer from artifacts such as blunt structures and abrupt colors when filling in the missing regions. To address these issues, we propose an external-internal inpainting scheme with a monochromic bottleneck that helps image inpainting models remove these artifacts. In the external learning stage, we reconstruct missing structures and details in the monochromic space to reduce the learning dimension. In the internal learning stage, we propose a novel internal color propagation method with progressive learning strategies for consistent color restoration. Extensive experiments demonstrate that our proposed scheme helps image inpainting models produce more structure-preserved and visually compelling results., Comment: CVPR 2021

We present a controllable camera simulator based on deep neural networks to synthesize raw image data under different camera settings, including exposure time, ISO, and aperture. The proposed simulator includes an exposure module that utilizes the principle of modern lens designs for correcting the luminance level. It also contains a noise module using the noise level function and an aperture module with adaptive attention to simulate the side effects on noise and defocus blur. To facilitate the learning of a simulator model, we collect a dataset of the 10,000 raw images of 450 scenes with different exposure settings. Quantitative experiments and qualitative comparisons show that our approach outperforms relevant baselines in raw data synthesize on multiple cameras. Furthermore, the camera simulator enables various applications, including large-aperture enhancement, HDR, auto exposure, and data augmentation for training local feature detectors. Our work represents the first attempt to simulate a camera sensor's behavior leveraging both the advantage of traditional raw sensor features and the power of data-driven deep learning., Comment: Accepted to CVPR2021

We study circular BPS Wilson loops in the $\mathcal{N}=2$ superconformal $n$-node quiver theories at large $N$ and strong 't Hooft coupling by using localization. We compute the expectation values of Wilson loops in the limit when the 't Hooft couplings are hierarchically different and when they are nearly equal. Based on these results, we make a conjecture for arbitrary strong couplings., Comment: 27 pages, 5 figures; v2: references added. v3: matches published version

We study the $T\bar{T}$ deformation of the chiral bosons and show the equivalence between the chiral bosons of opposite chiralities and the scalar fields at the Hamiltonian level under the deformation. We also derive the deformed Lagrangian of more generic theories which contain an arbitrary number of chiral bosons to all orders. By using these results, we derive the $T\bar{T}$ deformed boundary action of the AdS$_3$ gravity theory in the Chern-Simons formulation. We compute the deformed one-loop torus partition function, which satisfies the $T\bar{T}$ flow equation up to the one-loop order. Finally, we calculate the deformed stress tensor of a solution describing a BTZ black hole in the boundary theory, which coincides with the boundary stress tensor derived from the BTZ black hole with a finite cutoff., Comment: 29 pages, references added

We consider Toda field theories in a classical Euclidean $AdS_2$ background. We compute the four-point functions of boundary operators in the $a_1$, $a_2$ and $b_2$ Toda field theories. They take the same form as the four-point functions of generators in the corresponding $\mathcal{W}$-algebras. Therefore we conjecture that the boundary operators are in one-to-one correspondence with the generators in the $\mathcal{W}$-algebras., Comment: 14 pages

We prove the integrability of the two-loop open spin chain Hamiltonian from ABJM determinant like operators given in arXiv:1809.09941. By explicitly constructing R-matrices and K-matrices, we successfully obtain the two-loop Hamiltonian from the double row transfer matrices. This proves the integrability of our two-loop Hamiltonian. Based on the vacuum eigenvalues of the transfer matrices, we make a conjecture on the eigenvalues of the transfer matrices for general excited states. Bethe ansatz equations are simply obtained from the analytic conditions at the superficial poles of the eigenvalues., Comment: v3, appendix B added, notation for indices of R and K matrices unified, typoes fixed, published version, 22 pages; v2, 21 pages, no figures, typoes fixed, minor changes

We explore the importance of spatial contextual information in human pose estimation. Most state-of-the-art pose networks are trained in a multi-stage manner and produce several auxiliary predictions for deep supervision. With this principle, we present two conceptually simple and yet computational efficient modules, namely Cascade Prediction Fusion (CPF) and Pose Graph Neural Network (PGNN), to exploit underlying contextual information. Cascade prediction fusion accumulates prediction maps from previous stages to extract informative signals. The resulting maps also function as a prior to guide prediction at following stages. To promote spatial correlation among joints, our PGNN learns a structured representation of human pose as a graph. Direct message passing between different joints is enabled and spatial relation is captured. These two modules require very limited computational complexity. Experimental results demonstrate that our method consistently outperforms previous methods on MPII and LSP benchmark.