Search

Generating 3D visual scenes is at the forefront of visual generative AI, but current 3D generation techniques struggle with generating scenes with multiple high-resolution objects. Here we introduce Lay-A-Scene, which solves the task of Open-set 3D Object Arrangement, effectively arranging unseen objects. Given a set of 3D objects, the task is to find a plausible arrangement of these objects in a scene. We address this task by leveraging pre-trained text-to-image models. We personalize the model and explain how to generate images of a scene that contains multiple predefined objects without neglecting any of them. Then, we describe how to infer the 3D poses and arrangement of objects from a 2D generated image by finding a consistent projection of objects onto the 2D scene. We evaluate the quality of Lay-A-Scene using 3D objects from Objaverse and human raters and find that it often generates coherent and feasible 3D object arrangements.

Metallic transition metal dichalcogenides (TMDs), consisting of H-NbSe$_2$, H-NbS$_2$, H-TaSe$_2$ and H-TaS$_2$, remain superconducting down to a thickness of a single layer. In these materials, thickness affects a variety of properties, including Ising protection, two-band superconductivity, and the critical temperature $T_C$, which decreases for the Nb-based, and increases for the Ta-based materials. This contradicting trend is puzzling, and has precluded the development of a unified theory. We approach the question of thickness-evolution of $T_C$ and the superconducting gap $\Delta$ by measuring high-resolution tunneling spectra in TaS$_2$-based stacked devices. Our measurements allow for simultaneous evaluation of $\Delta$, $T_C$, and the upper critical field $H_{C2}$. The latter, we find, is strongly enhanced towards the single-layer limit, following a $H_{C2} \propto \Delta^2$ proportionality ratio. Our main finding is that the same ratio holds for the entire family of metallic TMDs: TaS$_2$ and NbSe$_2$ of all thicknesses, bulk TaSe$_2$ and bulk NbS$_2$, extending over 4 orders of magnitude in $H_{C2}$ and covering both clean and dirty limits. We propose that this tunability across the TMD family is controlled by the competing charge density wave (CDW) phase. Using Gor'kov's theory, we calculate how a CDW order affects the quasiparticle density of states and the resulting $T_C$ and $H_{C2}$. Our results suggest that CDW is the key determinant factor limiting $T_C$ in the TMD family. They also show that $H_{C2}$ is universally enhanced by a factor of two orders of magnitude above the expected value, an effect that remains an open question., Comment: 28 pages, 16 figures

Integrating a notion of symmetry into point cloud neural networks is a provably effective way to improve their generalization capability. Of particular interest are $E(3)$ equivariant point cloud networks where Euclidean transformations applied to the inputs are preserved in the outputs. Recent efforts aim to extend networks that are $E(3)$ equivariant, to accommodate inputs made of multiple parts, each of which exhibits local $E(3)$ symmetry. In practical settings, however, the partitioning into individually transforming regions is unknown a priori. Errors in the partition prediction would unavoidably map to errors in respecting the true input symmetry. Past works have proposed different ways to predict the partition, which may exhibit uncontrolled errors in their ability to maintain equivariance to the actual partition. To this end, we introduce APEN: a general framework for constructing approximate piecewise-$E(3)$ equivariant point networks. Our primary insight is that functions that are equivariant with respect to a finer partition will also maintain equivariance in relation to the true partition. Leveraging this observation, we propose a design where the equivariance approximation error at each layers can be bounded solely in terms of (i) uncertainty quantification of the partition prediction, and (ii) bounds on the probability of failing to suggest a proper subpartition of the ground truth one. We demonstrate the effectiveness of APEN using two data types exemplifying part-based symmetry: (i) real-world scans of room scenes containing multiple furniture-type objects; and, (ii) human motions, characterized by articulated parts exhibiting rigid movement. Our empirical results demonstrate the advantage of integrating piecewise $E(3)$ symmetry into network design, showing a distinct improvement in generalization compared to prior works for both classification and segmentation tasks., Comment: ICLR 2024

Text-to-image models offer a new level of creative flexibility by allowing users to guide the image generation process through natural language. However, using these models to consistently portray the same subject across diverse prompts remains challenging. Existing approaches fine-tune the model to teach it new words that describe specific user-provided subjects or add image conditioning to the model. These methods require lengthy per-subject optimization or large-scale pre-training. Moreover, they struggle to align generated images with text prompts and face difficulties in portraying multiple subjects. Here, we present ConsiStory, a training-free approach that enables consistent subject generation by sharing the internal activations of the pretrained model. We introduce a subject-driven shared attention block and correspondence-based feature injection to promote subject consistency between images. Additionally, we develop strategies to encourage layout diversity while maintaining subject consistency. We compare ConsiStory to a range of baselines, and demonstrate state-of-the-art performance on subject consistency and text alignment, without requiring a single optimization step. Finally, ConsiStory can naturally extend to multi-subject scenarios, and even enable training-free personalization for common objects., Comment: Accepted to journal track of SIGGRAPH 2024 (TOG). Project page is at https://consistory-paper.github.io

Text-to-image (T2I) personalization allows users to guide the creative image generation process by combining their own visual concepts in natural language prompts. Recently, encoder-based techniques have emerged as a new effective approach for T2I personalization, reducing the need for multiple images and long training times. However, most existing encoders are limited to a single-class domain, which hinders their ability to handle diverse concepts. In this work, we propose a domain-agnostic method that does not require any specialized dataset or prior information about the personalized concepts. We introduce a novel contrastive-based regularization technique to maintain high fidelity to the target concept characteristics while keeping the predicted embeddings close to editable regions of the latent space, by pushing the predicted tokens toward their nearest existing CLIP tokens. Our experimental results demonstrate the effectiveness of our approach and show how the learned tokens are more semantic than tokens predicted by unregularized models. This leads to a better representation that achieves state-of-the-art performance while being more flexible than previous methods., Comment: Project page at https://datencoder.github.io

We present a novel method for reconstructing a 3D implicit surface from a large-scale, sparse, and noisy point cloud. Our approach builds upon the recently introduced Neural Kernel Fields (NKF) representation. It enjoys similar generalization capabilities to NKF, while simultaneously addressing its main limitations: (a) We can scale to large scenes through compactly supported kernel functions, which enable the use of memory-efficient sparse linear solvers. (b) We are robust to noise, through a gradient fitting solve. (c) We minimize training requirements, enabling us to learn from any dataset of dense oriented points, and even mix training data consisting of objects and scenes at different scales. Our method is capable of reconstructing millions of points in a few seconds, and handling very large scenes in an out-of-core fashion. We achieve state-of-the-art results on reconstruction benchmarks consisting of single objects, indoor scenes, and outdoor scenes., Comment: CVPR 2023

The synthesis of two-dimensional van der Waals magnets has paved the way for both technological applications and fundamental research on magnetism confined to ultra-small length scales. Edge magnetic moments in ferromagnets are expected to be less magnetized than in the sample interior because of the reduced amount of neighboring ferromagnetic spins at the sample edge. We recently demonstrated that CrGeTe3 (CGT) flakes thinner than 10 nm are hard ferromagnets; i.e., they exhibit an open hysteresis loop. In contrast, thicker flakes exhibit zero net remnant field in the interior, with hard ferromagnetism present only at the cleaved edges. This experimental observation suggests that a nontrivial interaction exists between the sample edge and the interior. Here, we demonstrate that artificial edges fabricated by focus ion beam etching also display hard ferromagnetism. This enables us to write magnetic nanowires in CGT directly and use this method to characterize the magnetic interaction between the interior and edge. The results indicate that the interior saturation and depolarization fields depend on the lateral dimensions of the sample. Most notably, the interior region between the edges of a sample narrower than 300 nm becomes a hard ferromagnet, suggesting an enhancement of the magnetic exchange induced by the proximity of the edges. Last, we find that the CGT regions amorphized by the gallium beam are nonmagnetic, which introduces a novel method to tune the local magnetic properties of CGT films, potentially enabling integration into spintronic devices., Comment: Main text: 13 pages, 4 figures. Supplementary information: 8 pages, 6 figures

Colloidal semiconductor quantum dots are robust emitters implemented in numerous prototype and commercial optoelectronic devices. However, active fluorescence color tuning, achieved so far by electric-field induced Stark effect, has been limited to a small spectral range, and accompanied by intensity reduction due to the electron-hole charge separation effect. Utilizing quantum dot molecules that manifest two coupled emission centers, we present a novel electric-field induced instantaneous color switching effect. Reversible emission color switching without intensity loss is achieved on a single particle level, as corroborated by correlated electron microscopy imaging. Simulations establish that this is due to the electron wavefunction toggling between the two centers dictated by the electric-field and affected by the coupling strength. The quantum dot molecules manifesting two coupled emission centers may be tailored to emit distinct colors, opening the path for sensitive field sensing and color switchable devices such as a novel pixel design for displays or an electric field color tunable single photon source., Comment: Manuscript: 27 pages, 6 figures, Supplementary: 38 pages

Text-to-image models (T2I) offer a new level of flexibility by allowing users to guide the creative process through natural language. However, personalizing these models to align with user-provided visual concepts remains a challenging problem. The task of T2I personalization poses multiple hard challenges, such as maintaining high visual fidelity while allowing creative control, combining multiple personalized concepts in a single image, and keeping a small model size. We present Perfusion, a T2I personalization method that addresses these challenges using dynamic rank-1 updates to the underlying T2I model. Perfusion avoids overfitting by introducing a new mechanism that "locks" new concepts' cross-attention Keys to their superordinate category. Additionally, we develop a gated rank-1 approach that enables us to control the influence of a learned concept during inference time and to combine multiple concepts. This allows runtime-efficient balancing of visual-fidelity and textual-alignment with a single 100KB trained model, which is five orders of magnitude smaller than the current state of the art. Moreover, it can span different operating points across the Pareto front without additional training. Finally, we show that Perfusion outperforms strong baselines in both qualitative and quantitative terms. Importantly, key-locking leads to novel results compared to traditional approaches, allowing to portray personalized object interactions in unprecedented ways, even in one-shot settings., Comment: Accepted to SIGGRAPH 2023. Project page is in https://research.nvidia.com/labs/par/Perfusion/

Text-to-image personalization aims to teach a pre-trained diffusion model to reason about novel, user provided concepts, embedding them into new scenes guided by natural language prompts. However, current personalization approaches struggle with lengthy training times, high storage requirements or loss of identity. To overcome these limitations, we propose an encoder-based domain-tuning approach. Our key insight is that by underfitting on a large set of concepts from a given domain, we can improve generalization and create a model that is more amenable to quickly adding novel concepts from the same domain. Specifically, we employ two components: First, an encoder that takes as an input a single image of a target concept from a given domain, e.g. a specific face, and learns to map it into a word-embedding representing the concept. Second, a set of regularized weight-offsets for the text-to-image model that learn how to effectively ingest additional concepts. Together, these components are used to guide the learning of unseen concepts, allowing us to personalize a model using only a single image and as few as 5 training steps - accelerating personalization from dozens of minutes to seconds, while preserving quality., Comment: Project page at https://tuning-encoder.github.io/

Text-to-image models offer unprecedented freedom to guide creation through natural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them in new roles and novel scenes. In other words, we ask: how can we use language-guided models to turn our cat into a painting, or imagine a new product based on our favorite toy? Here we present a simple approach that allows such creative freedom. Using only 3-5 images of a user-provided concept, like an object or a style, we learn to represent it through new "words" in the embedding space of a frozen text-to-image model. These "words" can be composed into natural language sentences, guiding personalized creation in an intuitive way. Notably, we find evidence that a single word embedding is sufficient for capturing unique and varied concepts. We compare our approach to a wide range of baselines, and demonstrate that it can more faithfully portray the concepts across a range of applications and tasks. Our code, data and new words will be available at: https://textual-inversion.github.io, Comment: Project page: https://textual-inversion.github.io

CrGeTe3 (CGT) is a semiconducting vdW ferromagnet shown to possess magnetism down to a two-layer thick sample. Although CGT is one of the leading candidates for spintronics devices, a comprehensive analysis of CGT thickness dependent magnetization is currently lacking. In this work, we employ scanning SQUID-on-tip (SOT) microscopy to resolve the magnetic properties of exfoliated CGT flakes at 4.2 K. Combining transport measurements of CGT/NbSe2 samples with SOT images, we present the magnetic texture and hysteretic magnetism of CGT, thereby matching the global behavior of CGT to the domain structure extracted from local SOT magnetic imaging. Using this method, we provide a thickness dependent magnetization state diagram of bare CGT films. No zero-field magnetic memory was found for films thicker than 10 nm and hard ferromagnetism was found below that critical thickness. Using scanning SOT microscopy, we identify a unique edge magnetism, contrasting the results attained in the CGT interior., Comment: Main text: 15 pages, 5 figures. Supplementary information: 9 pages, 10 figures. Supplementary videos: 5

Large Vision & Language models pretrained on web-scale data provide representations that are invaluable for numerous V&L problems. However, it is unclear how they can be used for reasoning about user-specific visual concepts in unstructured language. This problem arises in multiple domains, from personalized image retrieval to personalized interaction with smart devices. We introduce a new learning setup called Personalized Vision & Language (PerVL) with two new benchmark datasets for retrieving and segmenting user-specific "personalized" concepts "in the wild". In PerVL, one should learn personalized concepts (1) independently of the downstream task (2) allowing a pretrained model to reason about them with free language, and (3) does not require personalized negative examples. We propose an architecture for solving PerVL that operates by extending the input vocabulary of a pretrained model with new word embeddings for the new personalized concepts. The model can then reason about them by simply using them in a sentence. We demonstrate that our approach learns personalized visual concepts from a few examples and can effectively apply them in image retrieval and semantic segmentation using rich textual queries., Comment: Accepted to ECCV (Oral). Compared to the ECCV camera ready version, we moved the ablation study to the main text, and updated the related work

Reasoning and interacting with dynamic environments is a fundamental problem in AI, but it becomes extremely challenging when actions can trigger cascades of cross-dependent events. We introduce a new supervised learning setup called {\em Cascade} where an agent is shown a video of a physically simulated dynamic scene, and is asked to intervene and trigger a cascade of events, such that the system reaches a "counterfactual" goal. For instance, the agent may be asked to "Make the blue ball hit the red one, by pushing the green ball". The agent intervention is drawn from a continuous space, and cascades of events makes the dynamics highly non-linear. We combine semantic tree search with an event-driven forward model and devise an algorithm that learns to search in semantic trees in continuous spaces. We demonstrate that our approach learns to effectively follow instructions to intervene in previously unseen complex scenes. It can also reason about alternative outcomes, when provided an observed cascade of events.

The task of shape space learning involves mapping a train set of shapes to and from a latent representation space with good generalization properties. Often, real-world collections of shapes have symmetries, which can be defined as transformations that do not change the essence of the shape. A natural way to incorporate symmetries in shape space learning is to ask that the mapping to the shape space (encoder) and mapping from the shape space (decoder) are equivariant to the relevant symmetries. In this paper, we present a framework for incorporating equivariance in encoders and decoders by introducing two contributions: (i) adapting the recent Frame Averaging (FA) framework for building generic, efficient, and maximally expressive Equivariant autoencoders; and (ii) constructing autoencoders equivariant to piecewise Euclidean motions applied to different parts of the shape. To the best of our knowledge, this is the first fully piecewise Euclidean equivariant autoencoder construction. Training our framework is simple: it uses standard reconstruction losses and does not require the introduction of new losses. Our architectures are built of standard (backbone) architectures with the appropriate frame averaging to make them equivariant. Testing our framework on both rigid shapes dataset using implicit neural representations, and articulated shape datasets using mesh-based neural networks show state-of-the-art generalization to unseen test shapes, improving relevant baselines by a large margin. In particular, our method demonstrates significant improvement in generalizing to unseen articulated poses., Comment: Accepted to CVPR 2022

Many machine learning tasks involve learning functions that are known to be invariant or equivariant to certain symmetries of the input data. However, it is often challenging to design neural network architectures that respect these symmetries while being expressive and computationally efficient. For example, Euclidean motion invariant/equivariant graph or point cloud neural networks. We introduce Frame Averaging (FA), a general purpose and systematic framework for adapting known (backbone) architectures to become invariant or equivariant to new symmetry types. Our framework builds on the well known group averaging operator that guarantees invariance or equivariance but is intractable. In contrast, we observe that for many important classes of symmetries, this operator can be replaced with an averaging operator over a small subset of the group elements, called a frame. We show that averaging over a frame guarantees exact invariance or equivariance while often being much simpler to compute than averaging over the entire group. Furthermore, we prove that FA-based models have maximal expressive power in a broad setting and in general preserve the expressive power of their backbone architectures. Using frame averaging, we propose a new class of universal Graph Neural Networks (GNNs), universal Euclidean motion invariant point cloud networks, and Euclidean motion invariant Message Passing (MP) GNNs. We demonstrate the practical effectiveness of FA on several applications including point cloud normal estimation, beyond $2$-WL graph separation, and $n$-body dynamics prediction, achieving state-of-the-art results in all of these benchmarks.

Implicit neural representation is a recent approach to learn shape collections as zero level-sets of neural networks, where each shape is represented by a latent code. So far, the focus has been shape reconstruction, while shape generalization was mostly left to generic encoder-decoder or auto-decoder regularization. In this paper we advocate deformation-aware regularization for implicit neural representations, aiming at producing plausible deformations as latent code changes. The challenge is that implicit representations do not capture correspondences between different shapes, which makes it difficult to represent and regularize their deformations. Thus, we propose to pair the implicit representation of the shapes with an explicit, piecewise linear deformation field, learned as an auxiliary function. We demonstrate that, by regularizing these deformation fields, we can encourage the implicit neural representation to induce natural deformations in the learned shape space, such as as-rigid-as-possible deformations.