Search

Two-photon states generated through degenerate spontaneous parametric down-conversion (SPDC) can exhibit sharp correlations in the transverse spatial coordinates. This effect, associated with the separability of the biphoton wavefunction into functions of the centroid and the difference coordinates, leads to unique free space propagation features. Here, we show that if a phase object is placed in the image plane of the source, only the propagation of the function of the centroid coordinates is altered. This propagation can be experimentally reconstructed in coincidence imaging and via post-selection on spatially correlated events, which will reveal the diffraction pattern from the object with a phase enhancement due to the probes' two-photon nature. Our findings offer applications for non-interferometric, quantum-enhanced phase imaging.

Identifying an appropriate task space that simplifies control solutions is important for solving robotic manipulation problems. One approach to this problem is learning an appropriate low-dimensional action space. Linear and nonlinear action mapping methods have trade-offs between simplicity on the one hand and the ability to express motor commands outside of a single low-dimensional subspace on the other. We propose that learning local linear action representations that adapt based on the current configuration of the robot achieves both of these benefits. Our state-conditioned linear maps ensure that for any given state, the high-dimensional robotic actuations are linear in the low-dimensional action. As the robot state evolves, so do the action mappings, ensuring the ability to represent motions that are immediately necessary. These local linear representations guarantee desirable theoretical properties by design, and we validate these findings empirically through two user studies. Results suggest state-conditioned linear maps outperform conditional autoencoder and PCA baselines on a pick-and-place task and perform comparably to mode switching in a more complex pouring task., Comment: 7 Pages, 8 Figures, Presented at the 2023 IEEE International Conference on Robotics and Automation (ICRA)

We present MM1.5, a new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning. Building upon the MM1 architecture, MM1.5 adopts a data-centric approach to model training, systematically exploring the impact of diverse data mixtures across the entire model training lifecycle. This includes high-quality OCR data and synthetic captions for continual pre-training, as well as an optimized visual instruction-tuning data mixture for supervised fine-tuning. Our models range from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE) variants, and demonstrate that careful data curation and training strategies can yield strong performance even at small scales (1B and 3B). Additionally, we introduce two specialized variants: MM1.5-Video, designed for video understanding, and MM1.5-UI, tailored for mobile UI understanding. Through extensive empirical studies and ablations, we provide detailed insights into the training processes and decisions that inform our final designs, offering valuable guidance for future research in MLLM development.

[Context] Large Language Models (LLMs) have shown good performance in several software development-related tasks such as program repair, documentation, code refactoring, debugging, and testing. Adapters are specialized, small modules designed for parameter efficient fine-tuning of LLMs for specific tasks, domains, or applications without requiring extensive retraining of the entire model. These adapters offer a more efficient way to customize LLMs for particular needs, leveraging the pre-existing capabilities of the large model. Merging LLMs and adapters has shown promising results for various natural language domains and tasks, enabling the use of the learned models and adapters without additional training for a new task. [Objective] This research proposes continual merging and empirically studies the capabilities of merged adapters in Code LLMs, specially for the Automated Program Repair (APR) task. The goal is to gain insights into whether and how merging task-specific adapters can affect the performance of APR. [Method] In our framework, MergeRepair, we plan to merge multiple task-specific adapters using three different merging methods and evaluate the performance of the merged adapter for the APR task. Particularly, we will employ two main merging scenarios for all three techniques, (i) merging using equal-weight averaging applied on parameters of different adapters, where all adapters are of equal importance; and (ii) our proposed approach, continual merging, in which we sequentially merge the task-specific adapters and the order and weight of merged adapters matter. By exploratory study of merging techniques, we will investigate the improvement and generalizability of merged adapters for APR. Through continual merging, we will explore the capability of merged adapters and the effect of task order, as it occurs in real-world software projects.

The growing complexity of networks and the variety of future scenarios with diverse and often stringent performance requirements call for a higher level of automation. Intent-based management emerges as a solution to attain high level of automation, enabling human operators to solely communicate with the network through high-level intents. The intents consist of the targets in the form of expectations (i.e., latency expectation) from a service and based on the expectations the required network configurations should be done accordingly. It is almost inevitable that when a network action is taken to fulfill one intent, it can cause negative impacts on the performance of another intent, which results in a conflict. In this paper, we aim to address the conflict issue and autonomous management of intent-based networking, and propose an online learning method based on the hierarchical multi-armed bandits approach for an effective management. Thanks to this hierarchical structure, it performs an efficient exploration and exploitation of network configurations with respect to the dynamic network conditions. We show that our algorithm is an effective approach regarding resource allocation and satisfaction of intent expectations.

We propose SlowFast-LLaVA (or SF-LLaVA for short), a training-free video large language model (LLM) that can jointly capture detailed spatial semantics and long-range temporal context without exceeding the token budget of commonly used LLMs. This is realized by using a two-stream SlowFast design of inputs for Video LLMs to aggregate features from sampled frames in an effective way. Specifically, the Slow pathway extracts features at a low frame rate while keeping as much spatial detail as possible (e.g., with 12x24 tokens), and the Fast pathway operates on a high frame rate but uses a larger spatial pooling stride (e.g., downsampling 6x) to focus on the motion cues. As a result, this design allows us to adequately capture both spatial and temporal features that are beneficial for detailed video understanding. Experimental results show that SF-LLaVA outperforms existing training-free methods on a wide range of video tasks. On some benchmarks, it achieves comparable or even better performance compared to state-of-the-art Video LLMs that are fine-tuned on video datasets. Code has been made available at: https://github.com/apple/ml-slowfast-llava., Comment: Technical report

This article introduces the analytical approach of practice mapping, using vector embeddings of network actions and interactions to map commonalities and disjunctures in the practices of social media users, as a framework for methodological advancement beyond the limitations of conventional network analysis and visualisation. In particular, the methodological framework we outline here has the potential to incorporate multiple distinct modes of interaction into a single practice map, can be further enriched with account-level attributes such as information gleaned from textual analysis, profile information, available demographic details, and other features, and can be applied even to a cross-platform analysis of communicative patterns and practices. The article presents practice mapping as an analytical framework and outlines its key methodological considerations. Given its prominence in past social media research, we draw on examples and data from the platform formerly known as Twitter in order to enable experienced scholars to translate their approaches to a practice mapping paradigm more easily, but point out how data from other platforms may be used in equivalent ways in practice mapping studies. We illustrate the utility of the approach by applying it to a dataset where the application of conventional network analysis and visualisation approaches has produced few meaningful insights.

Preference alignment has become a crucial component in enhancing the performance of Large Language Models (LLMs), yet its impact in Multimodal Large Language Models (MLLMs) remains comparatively underexplored. Similar to language models, MLLMs for image understanding tasks encounter challenges like hallucination. In MLLMs, hallucination can occur not only by stating incorrect facts but also by producing responses that are inconsistent with the image content. A primary objective of alignment for MLLMs is to encourage these models to align responses more closely with image information. Recently, multiple works have introduced preference datasets for MLLMs and examined different alignment methods, including Direct Preference Optimization (DPO) and Proximal Policy Optimization (PPO). However, due to variations in datasets, base model types, and alignment methods, it remains unclear which specific elements contribute most significantly to the reported improvements in these works. In this paper, we independently analyze each aspect of preference alignment in MLLMs. We start by categorizing the alignment algorithms into two groups, offline (such as DPO), and online (such as online-DPO), and show that combining offline and online methods can improve the performance of the model in certain scenarios. We review a variety of published multimodal preference datasets and discuss how the details of their construction impact model performance. Based on these insights, we introduce a novel way of creating multimodal preference data called Bias-Driven Hallucination Sampling (BDHS) that needs neither additional annotation nor external models, and show that it can achieve competitive performance to previously published alignment work for multimodal models across a range of benchmarks.

The emerging citation-based QA systems are gaining more attention especially in generative AI search applications. The importance of extracted knowledge provided to these systems is vital from both accuracy (completeness of information) and efficiency (extracting the information in a timely manner). In this regard, citation-based QA systems are suffering from two shortcomings. First, they usually rely only on web as a source of extracted knowledge and adding other external knowledge sources can hamper the efficiency of the system. Second, web-retrieved contents are usually obtained by some simple heuristics such as fixed length or breakpoints which might lead to splitting information into pieces. To mitigate these issues, we propose our enhanced web and efficient knowledge graph (KG) retrieval solution (EWEK-QA) to enrich the content of the extracted knowledge fed to the system. This has been done through designing an adaptive web retriever and incorporating KGs triples in an efficient manner. We demonstrate the effectiveness of EWEK-QA over the open-source state-of-the-art (SoTA) web-based and KG baseline models using a comprehensive set of quantitative and human evaluation experiments. Our model is able to: first, improve the web-retriever baseline in terms of extracting more relevant passages (>20\%), the coverage of answer span (>25\%) and self containment (>35\%); second, obtain and integrate KG triples into its pipeline very efficiently (by avoiding any LLM calls) to outperform the web-only and KG-only SoTA baselines significantly in 7 quantitative QA tasks and our human evaluation.

Current multimodal and multitask foundation models like 4M or UnifiedIO show promising results, but in practice their out-of-the-box abilities to accept diverse inputs and perform diverse tasks are limited by the (usually rather small) number of modalities and tasks they are trained on. In this paper, we expand upon the capabilities of them by training a single model on tens of highly diverse modalities and by performing co-training on large-scale multimodal datasets and text corpora. This includes training on several semantic and geometric modalities, feature maps from recent state of the art models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM and 4DHumans, and a range of new modalities that allow for novel ways to interact with the model and steer the generation, for example image metadata or color palettes. A crucial step in this process is performing discrete tokenization on various modalities, whether they are image-like, neural network feature maps, vectors, structured data like instance segmentation or human poses, or data that can be represented as text. Through this, we expand on the out-of-the-box capabilities of multimodal models and specifically show the possibility of training one model to solve at least 3x more tasks/modalities than existing ones and doing so without a loss in performance. This enables more fine-grained and controllable multimodal generation capabilities and allows us to study the distillation of models trained on diverse data and objectives into a unified model. We successfully scale the training to a three billion parameter model using tens of modalities and different datasets. The resulting models and training code are open sourced at 4m.epfl.ch., Comment: Project page at 4m.epfl.ch

Recently, frame multipliers, pair frames, and controlled frames have been investigated to improve the numerical efficiency of iterative algorithms for inverting the frame operator and other applications of frames. In this paper, the concept of biframe is introduced for a Hilbert space. A biframe is a pair of sequences in a Hilbert space that applies to an inequality similar to frame inequality. Also, it can be regarded as a generalization of controlled frames and a special kind of pair frames. The basic properties of biframes are investigated based on the biframe operator. Then, biframes are classified based on the type of their constituent sequences. In particular, biframes for which one of the constituent sequences is an orthonormal basis $\{e_k\}_{k=1}^\infty$ are studied. Then, a new class of Riesz bases denoted by $[\{e_k\}]$, is introduced and is called b-Riesz bases. An interesting result is also proved, showing that the set of all b-Riesz bases is a proper subset of the set of all Riesz bases. More precisely, b-Riesz bases induce an equivalence relation on $[\{e_k\}]$., Comment: 29 PAGES

Maximal Center Gauge (MCG) aims to detect center vortices by maximizing a gauge functional and then projecting onto the center elements of the respective group. The requirement for unrestricted maximization of the gauge functional has proven to be untenable because it was shown that it leads to an underestimation of the string tension. To counter this problem, the ensemble of local gauge maxima is investigated and it is found that the unrestricted maximization can be replaced by a maximization restricted to the Gaussian distributed part of the ensemble. Such restricted maximization weakens the problem of an underestimated string tension.

The complete measurement of the quantum state of two correlated photons requires reconstructing the amplitude and phase of the biphoton wavefunction. We show how, by means of spatially resolved single photon detection, one can infer the spatial structure of bi-photons generated by spontaneous parametric down conversion. In particular, a spatially resolved analysis of the second-order correlations allows us to isolate the moduli of the pump and phasematching contributions to the two-photon states. When carrying this analysis on different propagation planes, the free space propagation of pump and phasematching is observed. This result allows, in principle, to gain enough information to reconstruct also the phase of pump and phasematching, and thus the full biphoton wavefunction. We show this in different examples where the pump is shaped as a superposition of orbital angular momentum modes or as a smooth amplitude with a phase structure with no singularities. The corresponding phase structure is retrieved employing maximum likelihood or genetic algorithms. These findings have potential applications in fast, efficient quantum state characterisation that does not require any control over the source.

The present article introduces, mathematically analyzes, and numerically validates a new weak Galerkin (WG) mixed-FEM based on Banach spaces for the stationary Navier--Stokes equation in pseudostress-velocity formulation. More precisely, a modified pseudostress tensor, called $ \boldsymbol{\sigma} $, depending on the pressure, and the diffusive and convective terms has been introduced in the proposed technique, and a dual-mixed variational formulation has been derived where the aforementioned pseudostress tensor and the velocity, are the main unknowns of the system, whereas the pressure is computed via a post-processing formula. Thus, it is sufficient to provide a WG space for the tensor variable and a space of piecewise polynomial vectors of total degree at most 'k' for the velocity. Moreover, in order to define the weak discrete bilinear form, whose continuous version involves the classical divergence operator, the weak divergence operator as a well-known alternative for the classical divergence operator in a suitable discrete subspace is proposed. The well-posedness of the numerical solution is proven using a fixed-point approach and the discrete versions of the Babu\v{s}ka-Brezzi theory and the Banach-Ne\v{c}as-Babu\v{s}ka theorem. Additionally, an a priori error estimate is derived for the proposed method. Finally, several numerical results illustrating the method's good performance and confirming the theoretical rates of convergence are presented.

This paper explores the integration of Automated Guided Vehicles (AGVs) in warehouse order picking, a crucial and cost-intensive aspect of warehouse operations. The booming AGV industry, accelerated by the COVID-19 pandemic, is witnessing widespread adoption due to its efficiency, reliability, and cost-effectiveness in automating warehouse tasks. This paper focuses on enhancing the picker-to-parts system, prevalent in small to medium-sized warehouses, through the strategic use of AGVs. We discuss the benefits and applications of AGVs in various warehouse tasks, highlighting their transformative potential in improving operational efficiency. We examine the deployment of AGVs by leading companies in the industry, showcasing their varied functionalities in warehouse management. Addressing the gap in research on optimizing operational performance in hybrid environments where humans and AGVs coexist, our study delves into a dynamic picker-to-parts warehouse scenario. We propose a novel approach Neural Approximate Dynamic Programming approach for coordinating a mixed team of human and AGV workers, aiming to maximize order throughput and operational efficiency. This involves innovative solutions for non-myopic decision making, order batching, and battery management. We also discuss the integration of advanced robotics technology in automating the complete order-picking process. Through a comprehensive numerical study, our work offers valuable insights for managing a heterogeneous workforce in a hybrid warehouse setting, contributing significantly to the field of warehouse automation and logistics.

The characterization of a quantum device is a crucial step in the development of quantum experiments. This is accomplished via Quantum Process Tomography, which combines the outcomes of different projective measurements to deliver a possible reconstruction of the underlying process. The tomography is typically performed by processing an overcomplete set of measurements and extracting the process matrix from maximum-likelihood estimation. Here, we introduce a new technique, referred to as Fourier Quantum Process Tomography, which requires a reduced number of measurements, and benchmark its performance against the standard maximum-likelihood approach. Fourier Quantum Process Tomography is based on measuring probability distributions in two conjugate spaces for different state preparations and projections. Exploiting the concept of phase retrieval, our scheme achieves a complete and robust characterization of the setup by processing a near-minimal set of measurements. We experimentally test the technique on different space-dependent polarization transformations, reporting average fidelities higher than 90% and significant computational advantage.

Current machine learning models for vision are often highly specialized and limited to a single modality and task. In contrast, recent large language models exhibit a wide range of capabilities, hinting at a possibility for similarly versatile models in computer vision. In this paper, we take a step in this direction and propose a multimodal training scheme called 4M. It consists of training a single unified Transformer encoder-decoder using a masked modeling objective across a wide range of input/output modalities - including text, images, geometric, and semantic modalities, as well as neural network feature maps. 4M achieves scalability by unifying the representation space of all modalities through mapping them into discrete tokens and performing multimodal masked modeling on a small randomized subset of tokens. 4M leads to models that exhibit several key capabilities: (1) they can perform a diverse set of vision tasks out of the box, (2) they excel when fine-tuned for unseen downstream tasks or new input modalities, and (3) they can function as a generative model that can be conditioned on arbitrary modalities, enabling a wide variety of expressive multimodal editing capabilities with remarkable flexibility. Through experimental analyses, we demonstrate the potential of 4M for training versatile and scalable foundation models for vision tasks, setting the stage for further exploration in multimodal learning for vision and other domains., Comment: NeurIPS 2023 Spotlight. Project page at https://4m.epfl.ch/