Search

We consider the task of Image-to-Video (I2V) generation, which involves transforming static images into realistic video sequences based on a textual description. While recent advancements produce photorealistic outputs, they frequently struggle to create videos with accurate and consistent object motion, especially in multi-object scenarios. To address these limitations, we propose a two-stage compositional framework that decomposes I2V generation into: (i) An explicit intermediate representation generation stage, followed by (ii) A video generation stage that is conditioned on this representation. Our key innovation is the introduction of a mask-based motion trajectory as an intermediate representation, that captures both semantic object information and motion, enabling an expressive but compact representation of motion and semantics. To incorporate the learned representation in the second stage, we utilize object-level attention objectives. Specifically, we consider a spatial, per-object, masked-cross attention objective, integrating object-specific prompts into corresponding latent space regions and a masked spatio-temporal self-attention objective, ensuring frame-to-frame consistency for each object. We evaluate our method on challenging benchmarks with multi-object and high-motion scenarios and empirically demonstrate that the proposed method achieves state-of-the-art results in temporal coherence, motion realism, and text-prompt faithfulness. Additionally, we introduce \benchmark, a new challenging benchmark for single-object and multi-object I2V generation, and demonstrate our method's superiority on this benchmark. Project page is available at https://guyyariv.github.io/TTM/.

We study a monolayer of metal balls under periodic chiral driving in the horizontal plane. Energy dissipation occurs in this system via (i) inelastic collisions and (ii) frictional interaction with the substrate. We show that below a density-dependent critical drive, the system phase separates into a fluid phase coexisting with a solid phase. Unlike ordinary coexistence, however, the system does not obey the lever rule, as the fluid-phase density depends on the overall particle density. Additionally, the pressure is discontinuous across the fluid-solid interface, accompanied by a chiral edge current at the interface.

Commonsense reasoning is fundamentally based on multimodal knowledge. However, existing large language models (LLMs) are primarily trained using textual data only, limiting their ability to incorporate essential visual information. In contrast, Visual Language Models, which excel at visually-oriented tasks, often fail at non-visual tasks such as basic commonsense reasoning. This divergence highlights a critical challenge - the integration of robust visual understanding with foundational text-based language reasoning. To this end, we introduce a method aimed at enhancing LLMs' visual commonsense. Specifically, our method generates multiple images based on the input text prompt and integrates these into the model's decision-making process by mixing their prediction probabilities. To facilitate multimodal grounded language modeling, we employ a late-fusion layer that combines the projected visual features with the output of a pre-trained LLM conditioned on text only. This late-fusion layer enables predictions based on comprehensive image-text knowledge as well as text only when this is required. We evaluate our approach using several visual commonsense reasoning tasks together with traditional NLP tasks, including common sense reasoning and reading comprehension. Our experimental results demonstrate significant superiority over existing baselines. When applied to recent state-of-the-art LLMs (e.g., Llama3), we observe improvements not only in visual common sense but also in traditional NLP benchmarks. Code and models are available under https://github.com/guyyariv/vLMIG.

The Young-Dupr\'e equation is a cornerstone of the equilibrium theory of capillary and wetting phenomena. In the biological world, interfacial phenomena are ubiquitous, from the spreading of bacterial colonies to tissue growth and flocking of birds, but the description of such active systems escapes the realm of equilibrium physics. Here we show how a microscopic, mechanical definition of surface tension allows us to build an Active Young-Dupr\'e equation able to account for the partial wetting observed in simulations of active particles interacting via pairwise forces. Remarkably, the equation shows that the corresponding steady interfaces do not result from a simple balance between the surface tensions at play but instead emerge from a complex feedback mechanism. The interfaces are indeed stabilized by a drag force due to the emergence of steady currents, which are themselves a by-product of the symmetry breaking induced by the interfaces. These currents also lead to new physics by selecting the sizes and shapes of adsorbed droplets, breaking the equilibrium scale-free nature of the problem. Finally, we demonstrate a spectacular consequence of the negative value of the liquid-gas surface tensions in systems undergoing motility-induced phase separation: partially-immersed objects are expelled from the liquid phase, in stark contrast with what is observed in passive systems. All in all, our results lay the foundations for a theory of wetting in active systems., Comment: 31 pages, 6 figures

In quantum mechanics, measurements are dynamical processes and thus they should be capable of inducing currents. The symmetries of the Hamiltonian and measurement operator provide an organizing principle for understanding the conditions for such currents to emerge. The central role is played by the inversion and time-reversal symmetries. We classify the distinct behaviors that emerge from single and repeated measurements, with and without coupling to a dissipative bath. While the breaking of inversion symmetry alone is sufficient to generate currents through measurements, the breaking of time-reversal symmetry by the measurement operator leads to a dramatic increase in the magnitude of the currents. We consider the dependence on the measurement rate and find that the current is non-monotonic. Furthermore, nondegenerate measurements can lead to current loops within the steady state even in the Zeno limit., Comment: 20 pages, 11 figures

Starting from Robbins (1952), the literature on experimentation via multi-armed bandits has wed exploration and exploitation. Nonetheless, in many applications, agents' exploration and exploitation need not be intertwined: a policymaker may assess new policies different than the status quo; an investor may evaluate projects outside her portfolio. We characterize the optimal experimentation policy when exploration and exploitation are disentangled in the case of Poisson bandits, allowing for general news structures. The optimal policy features complete learning asymptotically, exhibits lots of persistence, but cannot be identified by an index a la Gittins. Disentanglement is particularly valuable for intermediate parameter values.

"Everyone knows the moon is made of cheese..." This line, famously uttered by Wallace to his canine sidekick Gromit in the 80s classic, may be one of the most cruelly underappreciated movie quotations of our time. Indeed, while most scientists today would simply reject Wallace's claim as preposterous, we aim to revisit his theory on the composition of our natural satellite, revealing that it may not be as implausible as the scientific consensus would have it. Through a revelatory novel analysis of existing data, we will show that very simple cheese-based models can provide a convincing explanation of the Lunar surface's spectral characteristics in the near-infrared. Using the tried and tested PLS (Partial Least Squares) method, we efficiently and reliably retrieve the concentrations of various cheese types in different locations of the Lunar surface. Our results bring to light a bold and flavourful prediction about the Moon's composition, which lays the groundwork for an important paradigm shift in planetary sciences. We urge the scientific community to take a serious notice of this piquant novel interpretation, and strongly consider it in their future models of planetary composition and formation in our solar system and beyond.

We illuminate effects of surface-charge convection intrinsic to leaky-dielectric electrohydrodynamics by analyzing the symmetric steady state of a circular drop in an external field at arbitrary electric Reynolds number $\mathrm{Re}_E$. In formulating the problem, we identify an exact factorization that reduces the number of dimensionless parameters from four -- $\mathrm{Re}_E$ and the conductivity, permittivity and viscosity ratios -- to two: a modified electric Reynolds number $\widetilde{\mathrm{Re}}$ and a charging parameter $\varpi$. In the case $\varpi<0$, where charge relaxation in the drop phase is slower than in the suspending phase, and, as a consequence, the interface polarizes antiparallel to the external field, we find that above a critical $\widetilde{\mathrm{Re}}$ value the solution exhibits a blowup singularity such that the surface-charge density diverges antisymmetrically with the $-1/3$ power of distance from the equator. We use local analysis to uncover the structure of that blowup singularity, wherein surface charges are convected by a locally induced flow towards the equator where they annihilate. To study the blowup regime, we devise a numerical scheme encoding that local structure where the blowup prefactor is determined by a global charging -- annihilation balance. We also employ asymptotic analysis to construct a universal problem governing the blowup solutions in the regime $\widetilde{\mathrm{Re}}\gg1$, far beyond the blowup threshold. In the case $\varpi>0$, where charge relaxation is faster in the drop phase and the interface polarizes parallel to the external field, we numerically observe and asymptotically characterize the formation at large $\widetilde{\mathrm{Re}}$ of stagnant, perfectly conducting surface-charge caps about the drop poles.

We show that an inclusion placed inside a dilute Stokesian suspension of microswimmers induces power-law number-density modulations and flows. These take a different form depending on whether the inclusion is held fixed by an external force, for example an optical tweezer, or if it is free. When the inclusion is held in place, the far-field fluid flow is a Stokeslet, while the microswimmer density decays as $1/r^{2+\epsilon}$, with $r$ the distance from the inclusion, and $\epsilon$ an anomalous exponent which depends on the symmetry of the inclusion and varies continuously as a function of a dimensionless number characterizing the relative amplitudes of the convective and diffusive effects. The angular dependence takes a non-trivial form which depends on the same dimensionless number. When the inclusion is free to move, the far-field fluid flow is a stresslet and the microswimmer density decays as $1/r^2$ with a simple angular dependence. These long-range modulations mediate long-range interactions between inclusions that we characterize., Comment: To be published in PRX

Monitored random circuits, consisting of alternating layers of entangling two-qubit gates and projective single-qubit measurements applied to some fraction $p$ of the qubits, have been a topic of recent interest. In particular, the resulting steady state exhibits a phase transition from highly correlated states with "volume-law" entanglement at $pp_{c}$. It is hard to access this transition experimentally, as it cannot be seen at the ensemble level. Naively, to observe it one must repeat the experiment until the set of measurement results repeats itself, with likelihood that is exponentially small in the number of measurements. To overcome this issue, we present a hybrid quantum-classical algorithm which creates a matrix product state (MPS) based "unitary mirror" of the projected circuit. Polynomial-sized tensor networks can represent quantum states with area-law entanglement, and so the unitary mirror can well-approximate the experimental state above $p_{c}$ but fails exponentially below it. The breaking of this mirror can thus pinpoint the critical point. We outline the algorithm and how such results would be obtained. We present a bound on the maximum entanglement entropy of any given state that is well-represented by an MPS, and from the bound suggest how the volume-law phase can be bounded. We consider whether the entanglement could similarly be bounded from below where the MPS fails. Finally, we present numerical results for small qubit numbers and for monitored circuits with random Clifford gates., Comment: 5 pages, 4 figures, supplemental material 8 pages

We present an experimental study of decentralized two-sided matching markets with no transfers. Experimental participants are informed of everyone's preferences and can make arbitrary non-binding match offers that get finalized when a period of market inactivity has elapsed. Several insights emerge. First, stable outcomes are prevalent. Second, while centralized clearinghouses commonly aim at implementing extremal stable matchings, our decentralized markets most frequently culminate in the median stable matching. Third, preferences' cardinal representations impact the stable partners participants match with. Last, the dynamics underlying our results exhibit strategic sophistication, with agents successfully avoiding cycles of blocking pairs.

Current diffusion or flow-based generative models for 3D shapes divide to two: distilling pre-trained 2D image diffusion models, and training directly on 3D shapes. When training a diffusion or flow models on 3D shapes a crucial design choice is the shape representation. An effective shape representation needs to adhere three design principles: it should allow an efficient conversion of large 3D datasets to the representation form; it should provide a good tradeoff of approximation power versus number of parameters; and it should have a simple tensorial form that is compatible with existing powerful neural architectures. While standard 3D shape representations such as volumetric grids and point clouds do not adhere to all these principles simultaneously, we advocate in this paper a new representation that does. We introduce Mosaic-SDF (M-SDF): a simple 3D shape representation that approximates the Signed Distance Function (SDF) of a given shape by using a set of local grids spread near the shape's boundary. The M-SDF representation is fast to compute for each shape individually making it readily parallelizable; it is parameter efficient as it only covers the space around the shape's boundary; and it has a simple matrix form, compatible with Transformer-based architectures. We demonstrate the efficacy of the M-SDF representation by using it to train a 3D generative flow model including class-conditioned generation with the 3D Warehouse dataset, and text-to-3D generation using a dataset of about 600k caption-shape pairs., Comment: More results and details can be found at https://lioryariv.github.io/msdf

We consider phoretic self-propulsion of a chemically active colloid where solute is consumed at both the colloid boundary and within the bulk solution. Assuming first-order kinetics, the dimensionless transport problem is governed by the surface Damk\"ohler number ${\mathcal{S}}$ and the bulk Damk\"ohler number ${\mathcal B}$. The dimensionless colloid velocity $U$, normalized by a self-phoretic scale, is a nonlinear function of these two parameters. We identify two scenarios where these numbers are linked. When the controlling physical parameter is colloid size, ${\mathcal{S}}$ is proportional to ${\mathcal B}^{1/2}$; when the controlling parameter is solute diffusivity, ${\mathcal{S}}$ is proportional to ${\mathcal B}$. In the limit of small Damk\"ohler numbers, $U$ adopts the same asymptotic limit in both scenarios, proportional to ${\mathcal{S}}$. In the limit of large Damk\"ohler numbers, the deviations of solute concentration from the equilibrium value are restricted to a narrow layer about the active portion of the colloid boundary. The asymptotic predictions of the associated boundary-layer problem are corroborated by an eigenfunction solution of the exact problem. The boundary-layer structure breaks down near the transition between the active and inactive portions of the boundary. The transport problem in that local region partially resembles the classical Sommerfeld problem of wave diffraction from an edge.

Active systems are driven out of equilibrium by exchanging energy and momentum with their environment. This endows them with anomalous mechanical properties that we review in this colloquium for the case of dry scalar active matter, which has attracted considerable attention. These unusual properties lead to a rich physics when active fluids are in contact with boundaries, inclusions, tracers, or disordered potentials. Indeed, studies of the mechanical pressure of active fluids and of the dynamics of passive tracers have shown that active systems impact their environment in non-trivial ways, for example, by propelling and rotating anisotropic inclusions. Conversely, the long-ranged density and current modulations induced by localized obstacles show how the environment can have a far-reaching impact on active fluids. This is best exemplified by the propensity of bulk and boundary disorder to destroy bulk phase separation in active matter, showing active systems to be much more sensitive to their surroundings than passive ones. This colloquium aims at providing a unifying perspective on the rich interplay between active systems and their environments., Comment: Review article, feedback is welcome

We consider the task of generating diverse and realistic videos guided by natural audio samples from a wide variety of semantic classes. For this task, the videos are required to be aligned both globally and temporally with the input audio: globally, the input audio is semantically associated with the entire output video, and temporally, each segment of the input audio is associated with a corresponding segment of that video. We utilize an existing text-conditioned video generation model and a pre-trained audio encoder model. The proposed method is based on a lightweight adaptor network, which learns to map the audio-based representation to the input representation expected by the text-to-video generation model. As such, it also enables video generation conditioned on text, audio, and, for the first time as far as we can ascertain, on both text and audio. We validate our method extensively on three datasets demonstrating significant semantic diversity of audio-video samples and further propose a novel evaluation metric (AV-Align) to assess the alignment of generated videos with input audio samples. AV-Align is based on the detection and comparison of energy peaks in both modalities. In comparison to recent state-of-the-art approaches, our method generates videos that are better aligned with the input sound, both with respect to content and temporal axis. We also show that videos produced by our method present higher visual quality and are more diverse., Comment: 9 pages, 6 figures

Entanglement and its propagation are central to understanding a multitude of physical properties of quantum systems. Notably, within closed quantum many-body systems, entanglement is believed to yield emergent thermodynamic behavior. However, a universal understanding remains challenging due to the non-integrability and computational intractability of most large-scale quantum systems. Quantum hardware platforms provide a means to study the formation and scaling of entanglement in interacting many-body systems. Here, we use a controllable $4 \times 4$ array of superconducting qubits to emulate a two-dimensional hard-core Bose-Hubbard lattice. We generate superposition states by simultaneously driving all lattice sites and extract correlation lengths and entanglement entropy across its many-body energy spectrum. We observe volume-law entanglement scaling for states at the center of the spectrum and a crossover to the onset of area-law scaling near its edges.

We study the stability of the ordered phase of flocking models with a scalar order parameter. Using both the active Ising model and a hydrodynamic description, we show that droplets of particles moving in the direction opposite to that of the ordered phase nucleate and grow. We characterize analytically this self-similar growth and demonstrate that droplets spread ballistically in all directions. Our results imply that, in the thermodynamic limit, discrete-symmetry flocks -- and, by extension, continuous-symmetry flocks with rotational anisotropy -- are metastable in all dimensions.

In recent years, image generation has shown a great leap in performance, where diffusion models play a central role. Although generating high-quality images, such models are mainly conditioned on textual descriptions. This begs the question: "how can we adopt such models to be conditioned on other modalities?". In this paper, we propose a novel method utilizing latent diffusion models trained for text-to-image-generation to generate images conditioned on audio recordings. Using a pre-trained audio encoding model, the proposed method encodes audio into a new token, which can be considered as an adaptation layer between the audio and text representations. Such a modeling paradigm requires a small number of trainable parameters, making the proposed approach appealing for lightweight optimization. Results suggest the proposed method is superior to the evaluated baseline methods, considering objective and subjective metrics. Code and samples are available at: https://pages.cs.huji.ac.il/adiyoss-lab/AudioToken., Comment: Accepted to INTERSPEECH 2023

The recently introduced Quantum Lego framework provides a powerful method for generating complex quantum error correcting codes (QECCs) out of simple ones. We gamify this process and unlock a new avenue for code design and discovery using reinforcement learning (RL). One benefit of RL is that we can specify \textit{arbitrary} properties of the code to be optimized. We train on two such properties, maximizing the code distance, and minimizing the probability of logical error under biased Pauli noise. For the first, we show that the trained agent identifies ways to increase code distance beyond naive concatenation, saturating the linear programming bound for CSS codes on 13 qubits. With a learning objective to minimize the logical error probability under biased Pauli noise, we find the best known CSS code at this task for $\lesssim 20$ qubits. Compared to other (locally deformed) CSS codes, including Surface, XZZX, and 2D Color codes, our $[[17,1,3]]$ code construction actually has \textit{lower} adversarial distance, yet better protects the logical information, highlighting the importance of QECC desiderata. Lastly, we comment on how this RL framework can be used in conjunction with physical quantum devices to tailor a code without explicit characterization of the noise model., Comment: 10 pages + appendices; v2 figure updated and note added

Surface reconstruction has been seeing a lot of progress lately by utilizing Implicit Neural Representations (INRs). Despite their success, INRs often introduce hard to control inductive bias (i.e., the solution surface can exhibit unexplainable behaviours), have costly inference, and are slow to train. The goal of this work is to show that replacing neural networks with simple grid functions, along with two novel geometric priors achieve comparable results to INRs, with instant inference, and improved training times. To that end we introduce VisCo Grids: a grid-based surface reconstruction method incorporating Viscosity and Coarea priors. Intuitively, the Viscosity prior replaces the smoothness inductive bias of INRs, while the Coarea favors a minimal area solution. Experimenting with VisCo Grids on a standard reconstruction baseline provided comparable results to the best performing INRs on this dataset., Comment: Published in NeurIPS 2022

We present a method for reconstructing high-quality meshes of large unbounded real-world scenes suitable for photorealistic novel view synthesis. We first optimize a hybrid neural volume-surface scene representation designed to have well-behaved level sets that correspond to surfaces in the scene. We then bake this representation into a high-quality triangle mesh, which we equip with a simple and fast view-dependent appearance model based on spherical Gaussians. Finally, we optimize this baked representation to best reproduce the captured viewpoints, resulting in a model that can leverage accelerated polygon rasterization pipelines for real-time view synthesis on commodity hardware. Our approach outperforms previous scene representations for real-time rendering in terms of accuracy, speed, and power consumption, and produces high quality meshes that enable applications such as appearance editing and physical simulation., Comment: Video and interactive web demo available at https://bakedsdf.github.io/

Recent advances in text-to-image generation with diffusion models present transformative capabilities in image quality. However, user controllability of the generated image, and fast adaptation to new tasks still remains an open challenge, currently mostly addressed by costly and long re-training and fine-tuning or ad-hoc adaptations to specific image generation tasks. In this work, we present MultiDiffusion, a unified framework that enables versatile and controllable image generation, using a pre-trained text-to-image diffusion model, without any further training or finetuning. At the center of our approach is a new generation process, based on an optimization task that binds together multiple diffusion generation processes with a shared set of parameters or constraints. We show that MultiDiffusion can be readily applied to generate high quality and diverse images that adhere to user-provided controls, such as desired aspect ratio (e.g., panorama), and spatial guiding signals, ranging from tight segmentation masks to bounding boxes. Project webpage: https://multidiffusion.github.io

The design of easy to operate high-fidelity two qubit gates remains an area of ongoing research. Many of the common schemes require dedicated controls lines, while others are vulnerable to issues of frequency crowding. Here, we propose a scheme for coupling a chain of transmons acting as logical qubits via a quantum bus of auxiliary qubits. The auxiliary array is made of floating transmons, and through the use of mediated interactions we generate effective capacitance between them beyond the nearest neighbor. Logical qubits are not directly coupled to each other, but they can be coupled by bringing them closer in frequency to the far-detuned auxiliary arrays. This allows for tunable coupling between non-neighboring logical qubits, and for the application of entangling gates to three or more qubits at once., Comment: 11 pages, 6 figures