Search

Vision-Language (VL) models have gained significant research focus, enabling remarkable advances in multimodal reasoning. These architectures typically comprise a vision encoder, a Large Language Model (LLM), and a projection module that aligns visual features with the LLM's representation space. Despite their success, a critical limitation persists: the vision encoding process remains decoupled from user queries, often in the form of image-related questions. Consequently, the resulting visual features may not be optimally attuned to the query-specific elements of the image. To address this, we introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal reasoning, which embeds question awareness directly within the vision encoder. This integration results in dynamic visual features focusing on relevant image aspects to the posed question. QA-ViT is model-agnostic and can be incorporated efficiently into any VL architecture. Extensive experiments demonstrate the effectiveness of applying our method to various multimodal architectures, leading to consistent improvement across diverse tasks and showcasing its potential for enhancing visual and scene-text understanding.

The usual dictionary between geometry and commutative algebra is not appropriate for Arithmetic geometry because addition is a singular operation at the "Real prime". We replace Rings, with addition and multiplication, by Props (=strict symmetric monoidal category generated by one object), or by Bioperad (=two closed symmetric operads acting on each other): to a ring we associate the prop of all matrices over it, with matrix multiplication and block direct sums as the basic operations, or the bioperad consisting of all raw and column vectors over it. We define the "commutative" props and bioperads, and using them we develop a generalized algebraic geometry, following Grothendieck footsteps closely. This new geometry is appropriate for Arithmetic (and potentially also for Physics).

Many problems in robotics involve creating or breaking multiple contacts nearly simultaneously or in an indeterminate order. We present a novel general purpose numerical integrator based on the theory of Event Selected Systems (ESS). Many multicontact models are ESS, which has recently been shown to imply that despite a discontinuous vector field, the flow of these systems is continuous, piecewise smooth, and has a well defined orbital derivative for all trajectories, which can be rapidly computed. We provide an elementary proof that our integrator is first-order accurate and verify numerically that it is in fact second-order accurate as its construction anticipated. We also compare our integrator, implemented in NumPy, to a MuJoCo simulation on models with 2 to 100 contacts, and confirm that the increase in simulation time per contact is nearly identical. The results suggest that this novel integrator can be invaluable for modelling and control in many robotics applications.

For a composition $f=f_1\circ\cdots \circ f_r$ of polynomials $f_i\in \mathbb Q[x]$ of degrees $d_i\geq 5$ with alternating or symmetric monodromy group, we show that the monodromy group of $f$ contains the iterated wreath product $A_{d_r}\wr \cdots\wr A_{d_1}$. A similar property holds more generally for polynomials that do not factor through $x^d$ or Chebyshev. We derive consequences to arithmetic dynamics regarding arboreal representations, and forward and backward orbits of such $f$. In particular, given an orbit $(a_n)_{n=0}^\infty$ of $f$ as above, we show that for "almost all" $a\in \mathbb Z$, the set of primes $p$ for which some $a_n$ is congruent to $a$ mod $p$ is "small".

The Clifford+T gate set is a topological generating set for PU(2), which has been well-studied from the perspective of quantum computation on a single qubit. The discovery that it generates a full S-arithmetic subgroup of PU(2) has led to a fruitful interaction between quantum computation and number theory, resulting in a proof that words in these gates cover PU(2) in an almost-optimal manner. In this paper we study the analogue gate set for PU(3). We show that in PU(3) the group generated by the Clifford+T gates is not arithmetic - in fact, it is a thin matrix group, namely a Zariski-dense group of infinite index in its ambient S-arithmetic group. On the other hand, we study a recently proposed extension of the Clifford+T gates, called Clifford+D, and show that these do generate a full S-arithmetic subgroup of PU(3), and satisfy a slightly weaker almost-optimal covering property than that of Clifford+T in PU(2). The proofs are different from those for PU(2): while both gate sets act naturally on a (Bruhat-Tits) tree, in PU(2) the generated group acts transitively on the vertices of the tree, and this is a main ingredient in proving both arithmeticity and efficiency. In the PU(3) Clifford+D case the action on the tree is far from being transitive. This makes the proof of arithmeticity considerably harder, and the study of efficiency by automorphic representation theory becomes more involved, and results in a covering rate which differs from the optimal one by a factor of $log_3(105)\approx 4.236$.

Finding accurate solutions to the electronic Schr\"odinger equation plays an important role in discovering important molecular and material energies and characteristics. Consequently, solving systems with large numbers of electrons has become increasingly important. Variational Monte Carlo (VMC) methods, especially those approximated through deep neural networks, are promising in this regard. In this paper, we aim to integrate one such model called the FermiNet, a post-Hartree-Fock (HF) Deep Neural Network (DNN) model, into a standard and widely used open source library, DeepChem. We also propose novel initialization techniques to overcome the difficulties associated with the assignment of excess or lack of electrons for ions., Comment: Accepted at 3rd Annual AAAI Workshop on AI to Accelerate Science and Engineering (AI2ASE)

We analyze correlation functions of $SU(k) \times SU(2)_F$ flavor currents in a family of three-dimensional ${\cal N}=4$ superconformal field theories, combining analytic bootstrap methods with input from supersymmetric localization. Via holographic duality, we extract gluon and graviton scattering amplitudes of M-theory on ${\rm AdS}_4\times S^7/\mathbb{Z}_k$ which contains a $\mathbb{C}^2/\mathbb{Z}_{k}$ orbifold singularity. From these results, we derive aspects of the effective description of M-theory on the orbifold singularity beyond its leading low energy limit. We also determine a threshold correction to the holographic correlator from the combined contribution of two-loop gluon and tree-level bulk graviton exchange., Comment: 44 pages plus appendices, v2 typos corrected

We combine supersymmetric localization with the numerical conformal bootstrap to bound the scaling dimension and OPE coefficient of the lowest-dimension operator in $\mathcal{N} = 4$ $\text{SU}(N)$ super-Yang-Mills theory for a wide range of $N$ and Yang-Mills couplings $g_\text{YM}$. We find that our bounds are approximately saturated by weak coupling results at small $g_\text{YM}$. Furthermore, at large $N$ our bounds interpolate between integrability results for the Konishi operator at small $g_\text{YM}$ and strong-coupling results, including the first few stringy corrections, for the lowest-dimension double-trace operator at large $g_\text{YM}$. In particular, our scaling dimension bounds describe the level splitting between the single- and double-trace operators at intermediate coupling., Comment: 42 pages, 13 figures

Diffusion inversion is the problem of taking an image and a text prompt that describes it and finding a noise latent that would generate the exact same image. Most current deterministic inversion techniques operate by approximately solving an implicit equation and may converge slowly or yield poor reconstructed images. We formulate the problem by finding the roots of an implicit equation and devlop a method to solve it efficiently. Our solution is based on Newton-Raphson (NR), a well-known technique in numerical analysis. We show that a vanilla application of NR is computationally infeasible while naively transforming it to a computationally tractable alternative tends to converge to out-of-distribution solutions, resulting in poor reconstruction and editing. We therefore derive an efficient guided formulation that fastly converges and provides high-quality reconstructions and editing. We showcase our method on real image editing with three popular open-sourced diffusion models: Stable Diffusion, SDXL-Turbo, and Flux with different deterministic schedulers. Our solution, Guided Newton-Raphson Inversion, inverts an image within 0.4 sec (on an A100 GPU) for few-step models (SDXL-Turbo and Flux.1), opening the door for interactive image editing. We further show improved results in image interpolation and generation of rare objects.

Vector image representation is a popular choice when editability and flexibility in resolution are desired. However, most images are only available in raster form, making raster-to-vector image conversion (vectorization) an important task. Classical methods for vectorization are either domain-specific or yield an abundance of shapes which limits editability and interpretability. Learning-based methods, that use differentiable rendering, have revolutionized vectorization, at the cost of poor generalization to out-of-training distribution domains, and optimization-based counterparts are either slow or produce non-editable and redundant shapes. In this work, we propose Optimize & Reduce (O&R), a top-down approach to vectorization that is both fast and domain-agnostic. O&R aims to attain a compact representation of input images by iteratively optimizing B\'ezier curve parameters and significantly reducing the number of shapes, using a devised importance measure. We contribute a benchmark of five datasets comprising images from a broad spectrum of image complexities - from emojis to natural-like images. Through extensive experiments on hundreds of images, we demonstrate that our method is domain agnostic and outperforms existing works in both reconstruction and perceptual quality for a fixed number of shapes. Moreover, we show that our algorithm is $\times 10$ faster than the state-of-the-art optimization-based method.

In their seminal paper, Lubotzky, Phillips and Sarnak (LPS) defined the notion of regular Ramanujan graphs and gave an explicit construction of infinite families of $(p+1)$-regular Ramanujan Cayley graphs, for infinitely many $p$. In this paper we extend the work of LPS and its successors to bigraphs (biregular bipartite graphs), in several aspects: we investigate the combinatorial properties of various generalizations of the notion of Ramanujan graphs, define a notion of Cayley bigraphs, and give explicit constructions of infinite families of $(p^3+1,p+1)$-regular Ramanujan Cayley bigraphs, for infinitely many $p$. Both the LPS graphs and our ones are arithmetic, arising as quotients of Bruhat-Tits trees by congruence subgroups of arithmetic lattices in a $p$-adic group, $PGL_2(\mathbb{Q}_p)$ for LPS and $PU_3(\mathbb{Q}_p)$ for us. In both cases the Ramanujan property relates to the Generalized Ramanujan Conjecture (GRC), on the respective groups. But while for $PGL_2$ the GRC holds unconditionally, this is not so in the case of $PU_3$. We find explicit cases where the GRC does and does not hold, and use this to construct arithmetic non-Ramanujan Cayley bigraphs as well, and prove that nevertheless they satisfy the Sarnak-Xue density hypothesis. On the combinatorial side, we present a pseudorandomness characterization of Ramanujan bigraphs, and a more general notion of biexpanders. We also show that the graphs we construct exhibit the cutoff phenomenon with bounded window size for the mixing time of non-backtracking random walks, either as a consequence of the Ramanujan property, or the density hypothesis. Finally, we present some other applications of our work: super golden gates for $PU_3$, Ramanujan and non-Ramanujan complexes of type $\tilde{A}_2$, optimal strong approximation for $p$-arithmetic subgroups of $PSU_3$ and vanishing of Betti numbers of Picard modular surfaces.

Traditional 2D pose estimation models are limited by their category-specific design, making them suitable only for predefined object categories. This restriction becomes particularly challenging when dealing with novel objects due to the lack of relevant training data. To address this limitation, category-agnostic pose estimation (CAPE) was introduced. CAPE aims to enable keypoint localization for arbitrary object categories using a few-shot single model, requiring minimal support images with annotated keypoints. We present a significant departure from conventional CAPE techniques, which treat keypoints as isolated entities, by treating the input pose data as a graph. We leverage the inherent geometrical relations between keypoints through a graph-based network to break symmetry, preserve structure, and better handle occlusions. We validate our approach on the MP-100 benchmark, a comprehensive dataset comprising over 20,000 images spanning over 100 categories. Our solution boosts performance by 0.98% under a 1-shot setting, achieving a new state-of-the-art for CAPE. Additionally, we enhance the dataset with skeleton annotations. Our code and data are publicly available.

We present a method for semantically transferring the visual appearance of one natural image to another. Specifically, our goal is to generate an image in which objects in a source structure image are "painted" with the visual appearance of their semantically related objects in a target appearance image. To integrate semantic information into our framework, our key idea is to leverage a pre-trained and fixed Vision Transformer (ViT) model. Specifically, we derive novel disentangled representations of structure and appearance extracted from deep ViT features. We then establish an objective function that splices the desired structure and appearance representations, interweaving them together in the space of ViT features. Based on our objective function, we propose two frameworks of semantic appearance transfer -- "Splice", which works by training a generator on a single and arbitrary pair of structure-appearance images, and "SpliceNet", a feed-forward real-time appearance transfer model trained on a dataset of images from a specific domain. Our frameworks do not involve adversarial training, nor do they require any additional input information such as semantic segmentation or correspondences. We demonstrate high-resolution results on a variety of in-the-wild image pairs, under significant variations in the number of objects, pose, and appearance. Code and supplementary material are available in our project page: splice-vit.github.io., Comment: Accepted to ACM Transactions on Graphics. arXiv admin note: substantial text overlap with arXiv:2201.00424

Continual learning is a subfield of machine learning, which aims to allow machine learning models to continuously learn on new data, by accumulating knowledge without forgetting what was learned in the past. In this work, we take a step back, and ask: "Why should one care about continual learning in the first place?". We set the stage by examining recent continual learning papers published at four major machine learning conferences, and show that memory-constrained settings dominate the field. Then, we discuss five open problems in machine learning, and even though they might seem unrelated to continual learning at first sight, we show that continual learning will inevitably be part of their solution. These problems are model editing, personalization and specialization, on-device learning, faster (re-)training and reinforcement learning. Finally, by comparing the desiderata from these unsolved problems and the current assumptions in continual learning, we highlight and discuss four future directions for continual learning research. We hope that this work offers an interesting perspective on the future of continual learning, while displaying its potential value and the paths we have to pursue in order to make it successful. This work is the result of the many discussions the authors had at the Dagstuhl seminar on Deep Continual Learning, in March 2023.

Advances in optical and electrophysiological recording technologies have made it possible to record the dynamics of thousands of neurons, opening up new possibilities for interpreting and controlling large neural populations in behaving animals. A promising way to extract computational principles from these large datasets is to train data-constrained recurrent neural networks (dRNNs). Performing this training in real-time could open doors for research techniques and medical applications to model and control interventions at single-cell resolution and drive desired forms of animal behavior. However, existing training algorithms for dRNNs are inefficient and have limited scalability, making it a challenge to analyze large neural recordings even in offline scenarios. To address these issues, we introduce a training method termed Convex Optimization of Recurrent Neural Networks (CORNN). In studies of simulated recordings, CORNN attained training speeds ~100-fold faster than traditional optimization approaches while maintaining or enhancing modeling accuracy. We further validated CORNN on simulations with thousands of cells that performed simple computations such as those of a 3-bit flip-flop or the execution of a timed response. Finally, we showed that CORNN can robustly reproduce network dynamics and underlying attractor structures despite mismatches between generator and inference models, severe subsampling of observed neurons, or mismatches in neural time-scales. Overall, by training dRNNs with millions of parameters in subminute processing times on a standard computer, CORNN constitutes a first step towards real-time network reproduction constrained on large-scale neural recordings and a powerful computational tool for advancing the understanding of neural computation., Comment: Accepted at NeurIPS 2023