Search

Scaling up autoregressive models in vision has not proven as beneficial as in large language models. In this work, we investigate this scaling problem in the context of text-to-image generation, focusing on two critical factors: whether models use discrete or continuous tokens, and whether tokens are generated in a random or fixed raster order using BERT- or GPT-like transformer architectures. Our empirical results show that, while all models scale effectively in terms of validation loss, their evaluation performance -- measured by FID, GenEval score, and visual quality -- follows different trends. Models based on continuous tokens achieve significantly better visual quality than those using discrete tokens. Furthermore, the generation order and attention mechanisms significantly affect the GenEval score: random-order models achieve notably better GenEval scores compared to raster-order models. Inspired by these findings, we train Fluid, a random-order autoregressive model on continuous tokens. Fluid 10.5B model achieves a new state-of-the-art zero-shot FID of 6.16 on MS-COCO 30K, and 0.69 overall score on the GenEval benchmark. We hope our findings and results will encourage future efforts to further bridge the scaling gap between vision and language models., Comment: Tech report

Conventional wisdom holds that autoregressive models for image generation are typically accompanied by vector-quantized tokens. We observe that while a discrete-valued space can facilitate representing a categorical distribution, it is not a necessity for autoregressive modeling. In this work, we propose to model the per-token probability distribution using a diffusion procedure, which allows us to apply autoregressive models in a continuous-valued space. Rather than using categorical cross-entropy loss, we define a Diffusion Loss function to model the per-token probability. This approach eliminates the need for discrete-valued tokenizers. We evaluate its effectiveness across a wide range of cases, including standard autoregressive models and generalized masked autoregressive (MAR) variants. By removing vector quantization, our image generator achieves strong results while enjoying the speed advantage of sequence modeling. We hope this work will motivate the use of autoregressive generation in other continuous-valued domains and applications. Code is available at: https://github.com/LTH14/mar., Comment: Neurips 2024 (Spotlight). Code: https://github.com/LTH14/mar

Fractals are ubiquitous in the natural world, and their connection with phase transitions has been widely observed. This study investigates mechanisms of fractal formation from the perspective of phase transitions. A novel set of probability calculation methods is introduced to establish a direct link between fractals and phase transitions. Notably, in the Ising model, a specific category of boundary lattice points undergoes a phase transition when the associated weight reaches approximately 0.4. The identified correlation between phase transitions and fractals suggests the emergence of fractal structures at this critical weight. The paper offers supporting evidence for this conclusion through the deliberate manipulation of the proposed probability-based method. This research contributes to a deeper understanding of the interplay between fractals and phase transitions, providing valuable insights for further exploration in diverse scientific domains.

Lattice models exhibit significant potential in investigating phase transitions, yet they encounter numerous computational challenges. To address these issues, this study introduces a Monte Carlo-based approach that transforms lattice models into a network model with intricate inter-node correlations. This framework enables a profound analysis of Ising, JQ, and XY models. By decomposing the network into a maximum entropy and a conservative component, under the constraint of detailed balance, this work derive an estimation formula for the temperature-dependent magnetic induction in Ising models. Notably, the critical exponent $\beta$ in the Ising model aligns well with established results, and the predicted phase transition point in the three-dimensional Ising model exhibits a mere $0.7 \%$ deviation from numerical simulations., Comment: 9 pages, 4 figures

In multilayer moir\'e heterostructures, the interference of multiple twist angles ubiquitously leads to tunable ultra-long-wavelength patterns known as supermoir\'e lattices. However, their impact on the system's many-body electronic phase diagram remains largely unexplored. We present local compressibility measurements revealing numerous incompressible states resulting from supermoir\'e-lattice-scale isospin symmetry breaking driven by strong interactions. By using the supermoir\'e lattice occupancy as a probe of isospin symmetry, we observe an unexpected doubling of the miniband filling near $\nu=-2$, possibly indicating a hidden phase transition or normal-state pairing proximal to the superconducting phase. Our work establishes supermoir\'e lattices as a tunable parameter for designing novel quantum phases and an effective tool for unraveling correlated phenomena in moir\'e materials.

As synthetic data becomes higher quality and proliferates on the internet, machine learning models are increasingly trained on a mix of human- and machine-generated data. Despite the successful stories of using synthetic data for representation learning, using synthetic data for generative model training creates "self-consuming loops" which may lead to training instability or even collapse, unless certain conditions are met. Our paper aims to stabilize self-consuming generative model training. Our theoretical results demonstrate that by introducing an idealized correction function, which maps a data point to be more likely under the true data distribution, self-consuming loops can be made exponentially more stable. We then propose self-correction functions, which rely on expert knowledge (e.g. the laws of physics programmed in a simulator), and aim to approximate the idealized corrector automatically and at scale. We empirically validate the effectiveness of self-correcting self-consuming loops on the challenging human motion synthesis task, and observe that it successfully avoids model collapse, even when the ratio of synthetic data to real data is as high as 100%., Comment: Camera ready version (ICML 2024). Code at https://nategillman.com/sc-sc.html

In strongly interacting systems with multiple energy bands, the interplay between electrons with different effective masses and the enlarged Hilbert space drives intricate correlated phenomena that do not occur in single-band systems. Recently, magic-angle twisted trilayer graphene (MATTG) has emerged as a promising tunable platform for such investigations: the system hosts both slowly dispersing, "heavy" electrons inhabiting its flat bands as well as delocalized "light" bands that disperse as free Dirac fermions. Most remarkably, superconductivity in twisted trilayer graphene and multilayer analogues with additional dispersive bands exhibits Pauli limit violation and spans a wider range of phase space compared to that in twisted bilayer graphene, where the dispersive bands are absent. This suggests that the interactions between different bands may play a fundamental role in stabilizing correlated phases in twisted graphene multilayers. Here, we elucidate the interplay between the light and heavy electrons in MATTG as a function of doping and magnetic field by performing local compressibility measurements with a scanning single-electron-transistor microscope. We establish that commonly observed resistive features near moir\'e band fillings $\nu$=-2, 1, 2 and 3 host a finite population of light Dirac electrons at the Fermi level despite a gap opening in the flat band sector. At higher magnetic field and near charge neutrality, we discover a new type of phase transition sequence that is robust over nearly 10 micrometers but exhibits complex spatial dependence. Mean-field calculations establish that these transitions arise from the competing population of the two subsystems and that the Dirac sector can be viewed as a new flavor analogous to the spin and valley degrees of freedom.

We study a crucial yet often overlooked issue inherent to Vision Transformers (ViTs): feature maps of these models exhibit grid-like artifacts, which hurt the performance of ViTs in downstream dense prediction tasks such as semantic segmentation, depth prediction, and object discovery. We trace this issue down to the positional embeddings at the input stage. To mitigate this, we propose a two-stage denoising approach, termed Denoising Vision Transformers (DVT). In the first stage, we separate the clean features from those contaminated by positional artifacts by enforcing cross-view feature consistency with neural fields on a per-image basis. This per-image optimization process extracts artifact-free features from raw ViT outputs, providing clean feature estimates for offline applications. In the second stage, we train a lightweight transformer block to predict clean features from raw ViT outputs, leveraging the derived estimates of the clean features as supervision. Our method, DVT, does not require re-training the existing pre-trained ViTs, and is immediately applicable to any Vision Transformer architecture. We evaluate our method on a variety of representative ViTs (DINO, DeiT-III, EVA02, CLIP, DINOv2, DINOv2-reg) and demonstrate that DVT consistently improves existing state-of-the-art general-purpose models in semantic and geometric tasks across multiple datasets. We hope our study will encourage a re-evaluation of ViT design, especially regarding the naive use of positional embeddings. Our code and checkpoints are publicly available., Comment: Accepted to ECCV2024. Project website: https://jiawei-yang.github.io/DenoisingViT/

We introduce SynCLR, a novel approach for learning visual representations exclusively from synthetic images and synthetic captions, without any real data. We synthesize a large dataset of image captions using LLMs, then use an off-the-shelf text-to-image model to generate multiple images corresponding to each synthetic caption. We perform visual representation learning on these synthetic images via contrastive learning, treating images sharing the same caption as positive pairs. The resulting representations transfer well to many downstream tasks, competing favorably with other general-purpose visual representation learners such as CLIP and DINO v2 in image classification tasks. Furthermore, in dense prediction tasks such as semantic segmentation, SynCLR outperforms previous self-supervised methods by a significant margin, e.g., improving over MAE and iBOT by 6.2 and 4.3 mIoU on ADE20k for ViT-B/16., Comment: Code is available at https://github.com/google-research/syn-rep-learn

Recent significant advances in text-to-image models unlock the possibility of training vision systems using synthetic images, potentially overcoming the difficulty of collecting curated data at scale. It is unclear, however, how these models behave at scale, as more synthetic data is added to the training set. In this paper we study the scaling laws of synthetic images generated by state of the art text-to-image models, for the training of supervised models: image classifiers with label supervision, and CLIP with language supervision. We identify several factors, including text prompts, classifier-free guidance scale, and types of text-to-image models, that significantly affect scaling behavior. After tuning these factors, we observe that synthetic images demonstrate a scaling trend similar to, but slightly less effective than, real images in CLIP training, while they significantly underperform in scaling when training supervised image classifiers. Our analysis indicates that the main reason for this underperformance is the inability of off-the-shelf text-to-image models to generate certain concepts, a limitation that significantly impairs the training of image classifiers. Our findings also suggest that scaling synthetic data can be particularly effective in scenarios such as: (1) when there is a limited supply of real images for a supervised problem (e.g., fewer than 0.5 million images in ImageNet), (2) when the evaluation dataset diverges significantly from the training data, indicating the out-of-distribution scenario, or (3) when synthetic data is used in conjunction with real images, as demonstrated in the training of CLIP models.

Quantum advantage requires overcoming noise-induced degradation of quantum systems. Conventional methods for reducing noise such as error mitigation face scalability issues in deep circuits. Specifically, noise hampers the extraction of amplitude and observable information from quantum systems. In this work, we present a novel algorithm that enhances the estimation of amplitude and observable under noise. Remarkably, our algorithm exhibits robustness against noise that varies across different depths of the quantum circuits. We assess the accuracy of amplitude and observable using numerical analysis and theoretically analyze the impact of gate-dependent noise on the results. This algorithm is a potential candidate for noise-resilient approaches that have high computational accuracy., Comment: 12 pages,18 figures

Current vision-language generative models rely on expansive corpora of paired image-text data to attain optimal performance and generalization capabilities. However, automatically collecting such data (e.g. via large-scale web scraping) leads to low quality and poor image-text correlation, while human annotation is more accurate but requires significant manual effort and expense. We introduce $\textbf{ITIT}$ ($\textbf{I}$n$\textbf{T}$egrating $\textbf{I}$mage $\textbf{T}$ext): an innovative training paradigm grounded in the concept of cycle consistency which allows vision-language training on unpaired image and text data. ITIT is comprised of a joint image-text encoder with disjoint image and text decoders that enable bidirectional image-to-text and text-to-image generation in a single framework. During training, ITIT leverages a small set of paired image-text data to ensure its output matches the input reasonably well in both directions. Simultaneously, the model is also trained on much larger datasets containing only images or texts. This is achieved by enforcing cycle consistency between the original unpaired samples and the cycle-generated counterparts. For instance, it generates a caption for a given input image and then uses the caption to create an output image, and enforces similarity between the input and output images. Our experiments show that ITIT with unpaired datasets exhibits similar scaling behavior as using high-quality paired data. We demonstrate image generation and captioning performance on par with state-of-the-art text-to-image and image-to-text models with orders of magnitude fewer (only 3M) paired image-text data.

Generative processes that involve solving differential equations, such as diffusion models, frequently necessitate balancing speed and quality. ODE-based samplers are fast but plateau in performance while SDE-based samplers deliver higher sample quality at the cost of increased sampling time. We attribute this difference to sampling errors: ODE-samplers involve smaller discretization errors while stochasticity in SDE contracts accumulated errors. Based on these findings, we propose a novel sampling algorithm called Restart in order to better balance discretization errors and contraction. The sampling method alternates between adding substantial noise in additional forward steps and strictly following a backward ODE. Empirically, Restart sampler surpasses previous SDE and ODE samplers in both speed and accuracy. Restart not only outperforms the previous best SDE results, but also accelerates the sampling speed by 10-fold / 2-fold on CIFAR-10 / ImageNet $64 \times 64$. In addition, it attains significantly better sample quality than ODE samplers within comparable sampling times. Moreover, Restart better balances text-image alignment/visual quality versus diversity than previous samplers in the large-scale text-to-image Stable Diffusion model pre-trained on LAION $512 \times 512$. Code is available at https://github.com/Newbeeer/diffusion_restart_sampling, Comment: Code is available at https://github.com/Newbeeer/diffusion_restart_sampling

Light carries both longitudinal and transverse spin angular momentum. The spin can couple with its orbital counterpart via the Berry phase, known as the spin-orbit interaction (SOI) of light. The SOI of light discovered previously belongs to the longitudinal one, which relies on the Berry phase in momentum space, such as the optical Magnus effect and the spin Hall effect. Here, we show that transverse SOI, relying on the Berry phase in real space, is inherent in the Helmholtz equation when transverse spinning light propagates in curved paths. The transverse SOI lifts the degeneracy of dispersion relations of light for opposite transverse spin states, analogous to the Dresselhaus effect. Transverse SOI is ubiquitous in nanophotonic systems where transverse spin and optical path bending are inevitable. It can also explain anomalous effects like the dispersion relation of surface plasmon polariton on curved paths and the energy level of whispering gallery modes. Our results reveal the analogies of spin photonics and spintronics and offer a new degree of freedom for integrated photonics, spin photonics, and astrophysics.

We investigate the potential of learning visual representations using synthetic images generated by text-to-image models. This is a natural question in the light of the excellent performance of such models in generating high-quality images. We consider specifically the Stable Diffusion, one of the leading open source text-to-image models. We show that (1) when the generative model is configured with proper classifier-free guidance scale, training self-supervised methods on synthetic images can match or beat the real image counterpart; (2) by treating the multiple images generated from the same text prompt as positives for each other, we develop a multi-positive contrastive learning method, which we call StableRep. With solely synthetic images, the representations learned by StableRep surpass the performance of representations learned by SimCLR and CLIP using the same set of text prompts and corresponding real images, on large scale datasets. When we further add language supervision, StableRep trained with 20M synthetic images achieves better accuracy than CLIP trained with 50M real images., Comment: code is available at: https://github.com/google-research/syn-rep-learn

Contrastive Language-Image Pre-training (CLIP) stands as one of the most effective and scalable methods for training transferable vision models using paired image and text data. CLIP models are trained using contrastive loss, which typically relies on data augmentations to prevent overfitting and shortcuts. However, in the CLIP training paradigm, data augmentations are exclusively applied to image inputs, while language inputs remain unchanged throughout the entire training process, limiting the exposure of diverse texts to the same image. In this paper, we introduce Language augmented CLIP (LaCLIP), a simple yet highly effective approach to enhance CLIP training through language rewrites. Leveraging the in-context learning capability of large language models, we rewrite the text descriptions associated with each image. These rewritten texts exhibit diversity in sentence structure and vocabulary while preserving the original key concepts and meanings. During training, LaCLIP randomly selects either the original texts or the rewritten versions as text augmentations for each image. Extensive experiments on CC3M, CC12M, RedCaps and LAION-400M datasets show that CLIP pre-training with language rewrites significantly improves the transfer performance without computation or memory overhead during training. Specifically for ImageNet zero-shot accuracy, LaCLIP outperforms CLIP by 8.2% on CC12M and 2.4% on LAION-400M. Code is available at https://github.com/LijieFan/LaCLIP., Comment: NeurIPS 2023

In order to gain a deeper understanding of complex systems and infer key information using minimal data, I classify all configurations based on classical probability, starting from the dimensions of energy and different categories of configurations. By utilizing the principle of maximum entropy, it is concluded that all possible configurations with the same energy have equal probabilities of occurrence. By using different representations of high and low energy, the emergence of a transition point has been inferred. Finally, I take the Ising model as an example and calculate the transition point of the thermodynamic phase transition, which is determined to be 2.25. This value is very close to the simulation value obtained by Monte Carlo method, but I have only consumed a small amount of computational resources in the process.

The magnetic correlations at the superconductor/ferromagnet (S/F) interfaces play a crucial role in realizing dissipation-less spin-based logic and memory technologies, such as triplet-supercurrent spin-valves and "{\pi}" Josephson junctions. Here we report the coexistence of an induced large magnetic moment and a crypto ferromagnetic state at high-quality nitride S/F interfaces. Using polarized neutron reflectometry and d. c. SQUID measurements, we quantitatively determined the magnetization profile of S/F bilayer and confirmed the induced magnetic moment in the adjacent superconductor only exists below TC. Interestingly, the direction of the induced moment in the superconductors was unexpectedly parallel to that in the ferromagnet, which contrasts with earlier findings in S/F heterostructures based on metals or oxides. The first-principles calculations verify the observed unusual interfacial spin texture is caused by the Heisenberg direct exchange coupling through d orbital overlapping and severe charge transfer across the interfaces. Our work establishes an incisive experimental probe for understanding the magnetic proximity behavior at S/F interfaces and provides a prototype epitaxial building block for superconducting spintronics., Comment: 21 pages, 5 figures, supplementary file with 14 figures

We introduce a new family of physics-inspired generative models termed PFGM++ that unifies diffusion models and Poisson Flow Generative Models (PFGM). These models realize generative trajectories for $N$ dimensional data by embedding paths in $N{+}D$ dimensional space while still controlling the progression with a simple scalar norm of the $D$ additional variables. The new models reduce to PFGM when $D{=}1$ and to diffusion models when $D{\to}\infty$. The flexibility of choosing $D$ allows us to trade off robustness against rigidity as increasing $D$ results in more concentrated coupling between the data and the additional variable norms. We dispense with the biased large batch field targets used in PFGM and instead provide an unbiased perturbation-based objective similar to diffusion models. To explore different choices of $D$, we provide a direct alignment method for transferring well-tuned hyperparameters from diffusion models ($D{\to} \infty$) to any finite $D$ values. Our experiments show that models with finite $D$ can be superior to previous state-of-the-art diffusion models on CIFAR-10/FFHQ $64{\times}64$ datasets, with FID scores of $1.91/2.43$ when $D{=}2048/128$. In class-conditional setting, $D{=}2048$ yields current state-of-the-art FID of $1.74$ on CIFAR-10. In addition, we demonstrate that models with smaller $D$ exhibit improved robustness against modeling errors. Code is available at https://github.com/Newbeeer/pfgmpp, Comment: Code is available at https://github.com/Newbeeer/pfgmpp

The iterative problem of solving nonlinear equations is studied. A new Newton like iterative method with adjustable parameters is designed based on the dynamic system theory. In order to avoid the derivative function in the iterative scheme, the difference quotient is used instead of the derivative. Different from the existing methods, the difference quotient scheme in this paper has higher accuracy. Thus, the new iterative method is suitable for a wider range of initial values. Finally, several numerical examples are given to verify the practicability and superiority of the method., Comment: 7pages,under review

We consider the sequential quantum channel discrimination problem using adaptive and non-adaptive strategies. In this setting the number of uses of the underlying quantum channel is not fixed but a random variable that is either bounded in expectation or with high probability. We show that both types of error probabilities decrease to zero exponentially fast and, when using adaptive strategies, the rates are characterized by the measured relative entropy between two quantum channels, yielding a strictly larger region than that achievable by non-adaptive strategies. Allowing for quantum memory, we see that the optimal rates are given by the regularized channel relative entropy. Finally, we discuss achievable rates when allowing for repeated measurements via quantum instruments and conjecture that the achievable rate region is not larger than that achievable with POVMs by connecting the result to the strong converse for the quantum channel Stein's Lemma.