Search

Effectively aligning with human judgment when evaluating machine-generated image captions represents a complex yet intriguing challenge. Existing evaluation metrics like CIDEr or CLIP-Score fall short in this regard as they do not take into account the corresponding image or lack the capability of encoding fine-grained details and penalizing hallucinations. To overcome these issues, in this paper, we propose BRIDGE, a new learnable and reference-free image captioning metric that employs a novel module to map visual features into dense vectors and integrates them into multi-modal pseudo-captions which are built during the evaluation process. This approach results in a multimodal metric that properly incorporates information from the input image without relying on reference captions, bridging the gap between human judgment and machine-generated image captions. Experiments spanning several datasets demonstrate that our proposal achieves state-of-the-art results compared to existing reference-free evaluation scores. Our source code and trained models are publicly available at: https://github.com/aimagelab/bridge-score., Comment: ECCV 2024

We develop a model based on mean-field games of competitive firms producing similar goods according to a standard AK model with a depreciation rate of capital generating pollution as a byproduct. Our analysis focuses on the widely-used cap-and-trade pollution regulation. Under this regulation, firms have the flexibility to respond by implementing pollution abatement, reducing output, and participating in emission trading, while a regulator dynamically allocates emission allowances to each firm. The resulting mean-field game is of linear quadratic type and equivalent to a mean-field type control problem, i.e., it is a potential game. We find explicit solutions to this problem through the solutions to differential equations of Riccati type. Further, we investigate the carbon emission equilibrium price that satisfies the market clearing condition and find a specific form of FBSDE of McKean-Vlasov type with common noise. The solution to this equation provides an approximate equilibrium price. Additionally, we demonstrate that the degree of competition is vital in determining the economic consequences of pollution regulation.

We develop a three-timescale framework for modelling climate change and introduce a space-heterogeneous one-dimensional energy balance model. This model, addressing temperature fluctuations from rising carbon dioxide levels and the super-greenhouse effect in tropical regions, fits within the setting of stochastic reaction-diffusion equations. Our results show how both mean and variance of temperature increase, without the system going through a bifurcation point. This study aims to advance the conceptual understanding of the extreme weather events frequency increase due to climate change.

The objective of image captioning models is to bridge the gap between the visual and linguistic modalities by generating natural language descriptions that accurately reflect the content of input images. In recent years, researchers have leveraged deep learning-based models and made advances in the extraction of visual features and the design of multimodal connections to tackle this task. This work presents a novel approach towards developing image captioning models that utilize an external kNN memory to improve the generation process. Specifically, we propose two model variants that incorporate a knowledge retriever component that is based on visual similarities, a differentiable encoder to represent input images, and a kNN-augmented language model to predict tokens based on contextual cues and text retrieved from the external memory. We experimentally validate our approach on COCO and nocaps datasets and demonstrate that incorporating an explicit external memory can significantly enhance the quality of captions, especially with a larger retrieval corpus. This work provides valuable insights into retrieval-augmented captioning models and opens up new avenues for improving image captioning at a larger scale., Comment: ACM Transactions on Multimedia Computing, Communications and Applications (2024)

Multimodal LLMs are the natural evolution of LLMs, and enlarge their capabilities so as to work beyond the pure textual modality. As research is being carried out to design novel architectures and vision-and-language adapters, in this paper we concentrate on endowing such models with the capability of answering questions that require external knowledge. Our approach, termed Wiki-LLaVA, aims at integrating an external knowledge source of multimodal documents, which is accessed through a hierarchical retrieval pipeline. Relevant passages, using this approach, are retrieved from the external knowledge source and employed as additional context for the LLM, augmenting the effectiveness and precision of generated dialogues. We conduct extensive experiments on datasets tailored for visual question answering with external data and demonstrate the appropriateness of our approach., Comment: CVPR 2024 Workshop on What is Next in Multimodal Foundation Models

Connecting text and visual modalities plays an essential role in generative intelligence. For this reason, inspired by the success of large language models, significant research efforts are being devoted to the development of Multimodal Large Language Models (MLLMs). These models can seamlessly integrate visual and textual modalities, while providing a dialogue-based interface and instruction-following capabilities. In this paper, we provide a comprehensive review of recent visual-based MLLMs, analyzing their architectural choices, multimodal alignment strategies, and training techniques. We also conduct a detailed analysis of these models across a wide range of tasks, including visual grounding, image generation and editing, visual understanding, and domain-specific applications. Additionally, we compile and describe training datasets and evaluation benchmarks, conducting comparisons among existing models in terms of performance and computational requirements. Overall, this survey offers a comprehensive overview of the current state of the art, laying the groundwork for future MLLMs., Comment: ACL 2024 (Findings)

In this work, we use the global electromagnetic and electrostatic gyro kinetic approaches to investigate the effects of zonal flows forced-driven by Alfv\'en modes due to their excitation by energetic particles (EPs), on the dynamics of ITG (Ion temperature gradient) instabilities. The equilibrium of the 92416 JET tokamak shot is considered. The linear and nonlinear Alfv\'en modes dynamics, as well as the zonal flow dynamics, are investigated and their respective radial structures and saturation levels are reported. ITG dynamics in the presence of the zonal flows excited by these Alfv\'en modes are also investigated. We find that, the zonal flows forced-driven by Alfv\'en modes can significantly impact the ITG dynamics. A zonal flow amplitude scan reveals the existence of an inverse relation between the zonal flow amplitude and the ITG growth rate. These results show that, forced-driven zonal flows can be an important indirect part of turbulence mitigation due to the injection of energetic particles., Comment: 13 pages, 10 figures

The Agenda 2030 recognises corruption as a major obstacle to sustainable development and integrates its reduction among SDG targets, in view of developing peaceful, just and strong institutions. In this paper, we propose a method to assess the validity of corruption indicators within an Item Response Theory framework, which explicitly accounts for the latent and multidimensional facet of corruption. Towards this main aim, a set of fifteen red flag indicators of corruption risk in public procurement is computed on data included in the Italian National Database of Public Contracts. Results show a multidimensional structure composed of sub-groups of red flag indicators i. measuring distinct corruption risk categories, which differ in nature, type and entity, and are generally non-superimposable; ii. mirroring distinct dynamics related to specific SDG principles and targets.

Image captioning, like many tasks involving vision and language, currently relies on Transformer-based architectures for extracting the semantics in an image and translating it into linguistically coherent descriptions. Although successful, the attention operator only considers a weighted summation of projections of the current input sample, therefore ignoring the relevant semantic information which can come from the joint observation of other samples. In this paper, we devise a network which can perform attention over activations obtained while processing other training samples, through a prototypical memory model. Our memory models the distribution of past keys and values through the definition of prototype vectors which are both discriminative and compact. Experimentally, we assess the performance of the proposed model on the COCO dataset, in comparison with carefully designed baselines and state-of-the-art approaches, and by investigating the role of each of the proposed components. We demonstrate that our proposal can increase the performance of an encoder-decoder Transformer by 3.7 CIDEr points both when training in cross-entropy only and when fine-tuning with self-critical sequence training. Source code and trained models are available at: https://github.com/aimagelab/PMA-Net., Comment: ICCV 2023

Nonlinear simulations of Alfv\'en modes (AM) driven by energetic particles (EP) in the presence of turbulence are performed with the gyrokinetic particle-in-cell code ORB5. The AMs carry a heat flux, and consequently they nonlinearly modify the plasma temperature profiles. The isolated effect of this modification on the dynamics of turbulence is studied, by means of electrostatic simulations. We find that turbulence is reduced when the profiles relaxed by the AM are used, with respect to the simulation where the unperturbed profiles are used. This is an example of indirect interaction of EPs and turbulence. First, an analytic magnetic equilibrium with circular concentric flux surfaces is considered as a simplified example for this study. Then, an application to an experimentally relevant case of ASDEX Upgrade is discussed.

Machine Unlearning is an emerging paradigm for selectively removing the impact of training datapoints from a network. Unlike existing methods that target a limited subset or a single class, our framework unlearns all classes in a single round. We achieve this by modulating the network's components using memory matrices, enabling the network to demonstrate selective unlearning behavior for any class after training. By discovering weights that are specific to each class, our approach also recovers a representation of the classes which is explainable by design. We test the proposed framework on small- and medium-scale image classification datasets, with both convolution- and Transformer-based backbones, showcasing the potential for explainable solutions through unlearning., Comment: IEEE Intelligent Systems (2024)

The CLIP model has been recently proven to be very effective for a variety of cross-modal tasks, including the evaluation of captions generated from vision-and-language architectures. In this paper, we propose a new recipe for a contrastive-based evaluation metric for image captioning, namely Positive-Augmented Contrastive learning Score (PAC-S), that in a novel way unifies the learning of a contrastive visual-semantic space with the addition of generated images and text on curated data. Experiments spanning several datasets demonstrate that our new metric achieves the highest correlation with human judgments on both images and videos, outperforming existing reference-based metrics like CIDEr and SPICE and reference-free metrics like CLIP-Score. Finally, we test the system-level correlation of the proposed metric when considering popular image captioning approaches, and assess the impact of employing different cross-modal features. Our source code and trained models are publicly available at: https://github.com/aimagelab/pacscore., Comment: CVPR 2023 (highlight paper)

In this work, we revisit the linear gyro-kinetic theory of geodesic acoustic modes (GAMs) and derive a general dispersion relation for an arbitrary equilibrium distribution function of ions. A bi-Maxwellian distribution of ions is then used to study the effects of ion temperature anisotropy on GAM frequency and growth rate. We find that ion temperature anisotropy yields sensible modifications to both the GAM frequency and growth rate as both tend to increase with anisotropy and these results are strongly affected by the electron to ion temperature ratio.

Image captioning models aim at connecting Vision and Language by providing natural language descriptions of input images. In the past few years, the task has been tackled by learning parametric models and proposing visual feature extraction advancements or by modeling better multi-modal connections. In this paper, we investigate the development of an image captioning approach with a kNN memory, with which knowledge can be retrieved from an external corpus to aid the generation process. Our architecture combines a knowledge retriever based on visual similarities, a differentiable encoder, and a kNN-augmented attention layer to predict tokens based on the past context and on text retrieved from the external memory. Experimental results, conducted on the COCO dataset, demonstrate that employing an explicit external memory can aid the generation process and increase caption quality. Our work opens up new avenues for improving image captioning models at larger scale., Comment: CBMI 2022

We perform a high-resolution two-dimensional fully-kinetic numerical simulation of a turbulent plasma system with observation-driven conditions, in order to investigate the interplay between turbulence, magnetic reconnection, and particle heating from ion to sub-electron scales in the near-Sun solar wind. We find that the power spectra of the turbulent plasma and electromagnetic fluctuations show multiple power-law intervals down to scales smaller than the electron gyroradius. Magnetic reconnection is observed to occur in correspondence of current sheets with a thickness of the order of the electron inertial length, which form and shrink due to interacting ion-scale vortexes. In some cases, both ion and electron outflows are observed (the classic reconnection scenario), while in others -- typically for the shortest current sheets -- only electron jets are presents ("electron-only reconnection"). At the onset of reconnection, the electron temperature starts to increase and a strong parallel temperature anisotropy develops. This suggests that in strong turbulence electron-scale coherent structures may play a significant role for electron heating, as impulsive and localized phenomena such as magnetic reconnection may transfer energy from the electromagnetic fields to particles more efficiently than damping mechanisms related to interactions with wave-like fluctuations., Comment: 18 pages, 6 figures

A GNSS signal authentication concept named semi-assisted authentication is proposed. It is based on the re-encryption and publication of keystream sequences of some milliseconds from an already existing encrypted signal. Some seconds after the keystreams are transmitted in the signal-in-space, the signal broadcasts the key allowing to decrypt the sequences and the a-posteriori correlation at the receiver. The concept is particularized as Galileo Assisted Commercial Authentication Service, or ACAS, for Galileo E1-B, with OSNMA used for the decryption keys, and E6C, assumed to be encrypted in the near future. This work proposes the ACAS cryptographic operations and a model for signal processing and authentication verification. Semi-assisted authentication can be provided without any modification to the signal plan of an existing GNSS, without the disclosure of signal encryption keys, and for several days of receiver autonomy, depending on its storage capabilities.

How the turbulent energy cascade develops below the magnetohydrodynamic scales in space and astrophysical plasmas is a major open question. Here, we measure the power spectrum of magnetic fluctuations in Parker Solar Probe's observations close to the Sun and in state-of-the-art numerical simulations of plasma turbulence. Both reveal a power-law behavior with a slope compatible with $-11/3$ at scales smaller than the ion characteristic scales, steeper than what is typically observed in the solar wind and in the Earth's magnetosheath. We explain such behavior by developing a simple two-fluid model which does not require any kinetic processes nor electron-inertia effects. This is characterized by a significant contribution of the ion kinetic energy to the total turbulent energy cascade at sub-ion scales, although the dynamics is driven by the magnetic field through the current density. We expect that this regime may be relevant for a broad class of low-beta plasmas, e.g. the solar corona, non-relativistic magnetized jets and disks, and laboratory plasmas.

Recent large cancer studies have measured somatic alterations in an unprecedented number of tumours. These large datasets allow the identification of cancer-related sets of genetic alterations by identifying relevant combinatorial patterns. Among such patterns, mutual exclusivity has been employed by several recent methods that have shown its effectivenes in characterizing gene sets associated to cancer. Mutual exclusivity arises because of the complementarity, at the functional level, of alterations in genes which are part of a group (e.g., a pathway) performing a given function. The availability of quantitative target profiles, from genetic perturbations or from clinical phenotypes, provides additional information that can be leveraged to improve the identification of cancer related gene sets by discovering groups with complementary functional associations with such targets. In this work we study the problem of finding groups of mutually exclusive alterations associated with a quantitative (functional) target. We propose a combinatorial formulation for the problem, and prove that the associated computation problem is computationally hard. We design two algorithms to solve the problem and implement them in our tool UNCOVER. We provide analytic evidence of the effectiveness of UNCOVER in finding high-quality solutions and show experimentally that UNCOVER finds sets of alterations significantly associated with functional targets in a variety of scenarios. In addition, our algorithms are much faster than the state-of-the-art, allowing the analysis of large datasets of thousands of target profiles from cancer cell lines. We show that on one such dataset from project Achilles our methods identify several significant gene sets with complementary functional associations with targets., Comment: Accepted at RECOMB 2018

In this paper we study the scaling relations for the triggering of the fast, or "ideal", tearing instability starting from equilibrium configurations relevant to astrophysical as well as laboratory plasmas that differ from the simple Harris current sheet configuration. We present the linear tearing instability analysis for equilibrium magnetic fields which a) go to zero at the boundary of the domain and b) contain a double current sheet system (the latter previously studied as a cartesian proxy for the m=1 kink mode in cylindrical plasmas). More generally, we discuss the critical aspect ratio scalings at which the growth rates become independent of the Lundquist number $S$, in terms of the dependence of the $\Delta'$ parameter on the wavenumber $k$ of unstable modes. The scaling $\Delta'(k)$ with $k$ at small $k$ is found to categorize different equilibria broadly: the critical aspect ratios may be even smaller than $L/a \sim S^{\alpha}$ with $\alpha=1/3$ originally found for the Harris current sheet, but there exists a general lower bound $\alpha\ge1/4$., Comment: 7 pages, 5 figures