Search

Hallucinations in Large Vision-Language Models (LVLMs) significantly undermine their reliability, motivating researchers to explore the causes of hallucination. However, most studies primarily focus on the language aspect rather than the visual. In this paper, we address how LVLMs process visual information and whether this process causes hallucination. Firstly, we use the attention lens to identify the stages at which LVLMs handle visual data, discovering that the middle layers are crucial. Moreover, we find that these layers can be further divided into two stages: "visual information enrichment" and "semantic refinement" which respectively propagate visual data to object tokens and interpret it through text. By analyzing attention patterns during the visual information enrichment stage, we find that real tokens consistently receive higher attention weights than hallucinated ones, serving as a strong indicator of hallucination. Further examination of multi-head attention maps reveals that hallucination tokens often result from heads interacting with inconsistent objects. Based on these insights, we propose a simple inference-time method that adjusts visual attention by integrating information across various heads. Extensive experiments demonstrate that this approach effectively mitigates hallucinations in mainstream LVLMs without additional training costs.

Uncertainty persists over how and why some countries become democratic and others do not, or why some countries remain democratic and others 'backslide' toward autocracy. Furthermore, while scholars generally agree on the nature of 'democracy' and 'autocracy', the nature of regimes in-between, and changes between them, are much less clear. By applying the spectral dimensionality-reduction technique Diffusion Map to political-science data from the V-Dem project for the period 1900 to 2021, we identify a low-dimensional non-linear manifold on which all electoral regimes move. Using the diffusion equation from statistical physics, we measure the time scale on which countries change their degree of electoral quality, freedom of association, and freedom of expression depending on their position on the manifold. By quantifying the coefficients of the diffusion equation for each country and over time, we show that democracies behave like sub-diffusive (i.e. slow spreading) particles and that autocracies on the verge of collapse behave like super-diffusive (i.e. fast spreading) particles. We show that regimes in-between exhibit diffusion dynamics distinct from autocracies and democracies, and an overall higher instability. Furthermore, we show that a country's position on the manifold and its dynamics are linked to its propensity for civil conflict. Our study pioneers the use of statistical physics in the analysis of political regimes. Our results provide a quantitative foundation for developing theories about what changes during democratization and democratic backsliding, as well as a new framework for regime-transformation and risk-of-conflict assessment., Comment: 15 pages, 5 figures

When fine-tuning zero-shot models like CLIP, our desideratum is for the fine-tuned model to excel in both in-distribution (ID) and out-of-distribution (OOD). Recently, ensemble-based models (ESM) have been shown to offer significant robustness improvement, while preserving high ID accuracy. However, our study finds that ESMs do not solve the ID-OOD trade-offs: they achieve peak performance for ID and OOD accuracy at different mixing coefficients. When optimized for OOD accuracy, the ensemble model exhibits a noticeable decline in ID accuracy, and vice versa. In contrast, we propose a sample-wise ensembling technique that can simultaneously attain the best ID and OOD accuracy without the trade-offs. Specifically, we construct a Zero-Shot Failure (ZSF) set containing training samples incorrectly predicted by the zero-shot model. For each test sample, we calculate its distance to the ZSF set and assign a higher weight to the fine-tuned model in the ensemble if the distance is small. We term our method Variance Reduction Fine-tuning (VRF), as it effectively reduces the variance in ensemble predictions, thereby decreasing residual error. On ImageNet and five derived distribution shifts, our VRF further improves the OOD accuracy by 1.5 - 2.0 pp over the ensemble baselines while maintaining or increasing ID accuracy. VRF achieves similar large robustness gains (0.9 - 3.1 pp) on other distribution shifts benchmarks. Codes are available in https://github.com/BeierZhu/VRF., Comment: Accepted to NeurIPS 2024

Vision-language models, such as CLIP, have shown impressive generalization capacities when using appropriate text descriptions. While optimizing prompts on downstream labeled data has proven effective in improving performance, these methods entail labor costs for annotations and are limited by their quality. Additionally, since CLIP is pre-trained on highly imbalanced Web-scale data, it suffers from inherent label bias that leads to suboptimal performance. To tackle the above challenges, we propose a label-Free prompt distribution learning and bias correction framework, dubbed as **Frolic**, which boosts zero-shot performance without the need for labeled data. Specifically, our Frolic learns distributions over prompt prototypes to capture diverse visual representations and adaptively fuses these with the original CLIP through confidence matching. This fused model is further enhanced by correcting label bias via a label-free logit adjustment. Notably, our method is not only training-free but also circumvents the necessity for hyper-parameter tuning. Extensive experimental results across 16 datasets demonstrate the efficacy of our approach, particularly outperforming the state-of-the-art by an average of $2.6\%$ on 10 datasets with CLIP ViT-B/16 and achieving an average margin of $1.5\%$ on ImageNet and its five distribution shifts with CLIP ViT-B/16. Codes are available in https://github.com/zhuhsingyuu/Frolic., Comment: NeurIPS 2024 Spotlight

The Robot Operating System (ROS) has become the de facto standard middleware in robotics, widely adopted across domains ranging from education to industrial applications. The RoboStack distribution has extended ROS's accessibility by facilitating installation across all major operating systems and architectures, integrating seamlessly with scientific tools such as PyTorch and Open3D. This paper presents ROS2WASM, a novel integration of RoboStack with WebAssembly, enabling the execution of ROS 2 and its associated software directly within web browsers, without requiring local installations. This approach significantly enhances reproducibility and shareability of research, lowers barriers to robotics education, and leverages WebAssembly's robust security framework to protect against malicious code. We detail our methodology for cross-compiling ROS 2 packages into WebAssembly, the development of a specialized middleware for ROS 2 communication within browsers, and the implementation of a web platform available at www.ros2wasm.dev that allows users to interact with ROS 2 environments. Additionally, we extend support to the Robotics Toolbox for Python and adapt its Swift simulator for browser compatibility. Our work paves the way for unprecedented accessibility in robotics, offering scalable, secure, and reproducible environments that have the potential to transform educational and research paradigms., Comment: 7 pages, 8 figures, under review

The ubiquitous occurrence of toroidal vortices or vortex rings in fluid-dynamic scenarios in nature has garnered significant attention of scientific frontier, whilst, the electromagnetic counterparts of which were only proposed recently with two distinct manifestations: vector toroidal pulses [Nat. Photon. 16, 523 (2022)] and scalar phase toroidal vortices [Nat. Photon. 16, 519 (2022)]. This dichotomy in the understanding of toroidal vortex phenomena has prompted a reassessment of their fundamental nature. Herein, we theoretically propose a novel form of electromagnetic toroidal vortex solutions, that uniquely integrate both scalar and vector characteristics, challenging the prevailing notion of their mutual exclusivity. We also present the experimental generation of the hybrid toroidal vortex pulses by a compact coaxial horn emitter augmented with a metasurface. This methodology not only demonstrates the feasibility of creating such complex vortex structures but also endows the resulting pulses with unique properties, including the coexistence of transverse orbital angular momentum, electromagnetic vortex streets, and topological skyrmion textures. These attributes introduce new dimensions in topologically complex structured waves, opening avenues for enhanced free-space information transmission, topologically nontrivial light-matter interaction and microscopy techniques.

While bacterial swimming has been well characterized in uniform liquid environments, only little is known about how bacteria propagate through complex environments, such as gel-like matrices or porous media that are typically encountered in tissue or soil. Here, we study swimming motility of the soil bacterium Pseudomonas putida (P. putida) in polysaccharide matrices formed by different concentrations of agar. P. putida cells display intermittent run-motility in the gel, where run times are exponentially distributed and intermittently occurring dwell times follow a waiting-time distribution with a power-law decay. An analysis of the turn angle distribution suggests that both, flagella mediated turning as well as mechanical trapping in the agar matrix play a role in the overall swimming pattern. Based on the experimentally observed motility pattern and measured waiting-time distributions, we propose a minimal active particle model which correctly describes the observed time dependence of the mean square displacement of the bacterial swimmers., Comment: 6 pages, 4 figures

The SNO+ detector operated initially as a water Cherenkov detector. The implementation of a sealed covergas system midway through water data taking resulted in a significant reduction in the activity of $^{222}$Rn daughters in the detector and allowed the lowest background to the solar electron scattering signal above 5 MeV achieved to date. This paper reports an updated SNO+ water phase $^8$B solar neutrino analysis with a total livetime of 282.4 days and an analysis threshold of 3.5 MeV. The $^8$B solar neutrino flux is found to be $\left(2.32^{+0.18}_{-0.17}\text{(stat.)}^{+0.07}_{-0.05}\text{(syst.)}\right)\times10^{6}$ cm$^{-2}$s$^{-1}$ assuming no neutrino oscillations, or $\left(5.36^{+0.41}_{-0.39}\text{(stat.)}^{+0.17}_{-0.16}\text{(syst.)} \right)\times10^{6}$ cm$^{-2}$s$^{-1}$ assuming standard neutrino oscillation parameters, in good agreement with both previous measurements and Standard Solar Model Calculations. The electron recoil spectrum is presented above 3.5 MeV.

Vision-language models such as CLIP are capable of mapping the different modality data into a unified feature space, enabling zero/few-shot inference by measuring the similarity of given images and texts. However, most existing methods overlook modality gaps in CLIP's encoded features, which is shown as the text and image features lie far apart from each other, resulting in limited classification performance. To tackle this issue, we introduce a method called Selective Vision-Language Subspace Projection (SSP), which incorporates local image features and utilizes them as a bridge to enhance the alignment between image-text pairs. Specifically, our SSP framework comprises two parallel modules: a vision projector and a language projector. Both projectors utilize local image features to span the respective subspaces for image and texts, thereby projecting the image and text features into their respective subspaces to achieve alignment. Moreover, our approach entails only training-free matrix calculations and can be seamlessly integrated into advanced CLIP-based few-shot learning frameworks. Extensive experiments on 11 datasets have demonstrated SSP's superior text-image alignment capabilities, outperforming the state-of-the-art alignment methods. The code is available at https://github.com/zhuhsingyuu/SSP, Comment: Accepted as an Oral Paper at ACM Multimedia 2024

The SNO+ collaboration reports its first spectral analysis of long-baseline reactor antineutrino oscillation using 114 tonne-years of data. Fitting the neutrino oscillation probability to the observed energy spectrum yields constraints on the neutrino mass-squared difference $\Delta m^2_{21}$. In the ranges allowed by previous measurements, the best-fit $\Delta m^2_{21}$ is (8.85$^{+1.10}_{-1.33}$) $\times$ 10$^{-5}$ eV$^2$. This measurement is continuing in the next phases of SNO+ and is expected to surpass the present global precision on $\Delta m^2_{21}$ with about three years of data.

The development of safety-critical aerospace systems is traditionally dominated by the C language. Its language characteristics make it trivial to accidentally introduce memory safety issues resulting in undefined behavior or security vulnerabilities. The Rust language aims to drastically reduce the chance of introducing bugs and consequently produces overall more secure and safer code. However, due to its relatively short lifespan, industry adaption in safety-critical environments is still lacking. This work provides a set of recommendations for the development of safety-critical space systems in Rust. Our recommendations are based on insights from our multi-fold contributions towards safer and more secure aerospace systems: We provide a comprehensive overview of ongoing efforts to adapt Rust for safety-critical system programming, highlighting its potential to enhance system robustness. Next, we introduce a procedure for partially rewriting C-based systems in Rust, offering a pragmatic pathway to improving safety without necessitating a full system overhaul. During the execution of our rewriting case study, we identify and fix three previously undiscovered vulnerabilities in a popular open-source satellite communication protocol. Finally, we introduce a new Rust compiler target configuration for bare metal PowerPC. With this, we aim to broaden Rust's applicability in space-oriented projects, as the architecture is commonly encountered in the domain, e.g., in the James Webb Space Telescope., Comment: IEEE Security for Space Systems (3S) 2024

The United States national forest inventory (NFI) serves as the foundation for forest aboveground biomass (AGB) and carbon accounting across the nation. These data enable design-based estimates of forest carbon stocks and stock-changes at state and regional levels, but also serve as inputs to model-based approaches for characterizing forest carbon stocks and stock-changes at finer resolutions. Although NFI tree and plot-level data are often treated as truth in these models, they are in fact estimates based on regional species-group models known collectively as the Component Ratio Method (CRM). In late 2023 the Forest Inventory and Analysis (FIA) program introduced a new National Scale Volume and Biomass Estimators (NSVB) system to replace CRM nationwide and offer more precise and accurate representations of forest AGB and carbon. Given the prevalence of model-based AGB studies relying on FIA, there is concern about the transferability of methods from CRM to NSVB models, as well as the comparability of existing CRM AGB products (e.g. maps) to new and forthcoming NSVB AGB products. To begin addressing these concerns we compared previously published CRM AGB maps to new maps produced using identical methods with NSVB AGB reference data. Our results suggest that models relying on passive satellite imagery (e.g. Landsat) provide acceptable estimates of point-in-time NSVB AGB and carbon stocks, but fail to accurately quantify growth in mature closed-canopy forests. We highlight that existing estimates, models, and maps based on FIA reference data are no longer compatible with NSVB, and recommend new methods as well as updated models and maps for accommodating this step-change. Our collective ability to adopt NSVB in our modeling and mapping workflows will help us provide the most accurate spatial forest carbon data possible in order to better inform local management and decision making., Comment: Manuscript: 16 pages, 7 figures; Supplements: 3 pages, 2 figures; Submitted to: Remote Sensing of Environment

In this study, we utilize the Kalman-Filter analysis to assess market efficiency in major stock markets. The Kalman-Filter operates in two stages, assuming that the data contains a consistent trendline representing the true market value prior to being affected by noise. Unlike traditional methods, it can forecast stock price movements effectively. Our findings reveal significant portfolio returns in emerging markets such as Korea, Vietnam, and Malaysia, as well as positive returns in developed markets like the UK, Europe, Japan, and Hong Kong. This suggests that the Kalman-Filter-based price reversal indicator yields promising results across various market types.

The Gromov-Wasserstein (GW) transport problem is a relaxation of classic optimal transport, which seeks a transport between two measures while preserving their internal geometry. Due to meeting this theoretical underpinning, it is a valuable tool for the analysis of objects that do not possess a natural embedding or should be studied independently of it. Prime applications can thus be found in e.g. shape matching, classification and interpolation tasks. To tackle the latter, one theoretically justified approach is the employment of multi-marginal GW transport and GW barycenters, which are Fr\'echet means with respect to the GW distance. However, because the computation of GW itself already poses a quadratic and non-convex optimization problem, the determination of GW barycenters is a hard task and algorithms for their computation are scarce. In this paper, we revisit a known procedure for the determination of Fr\'echet means in Riemannian manifolds via tangential approximations in the context of GW. We provide a characterization of barycenters in the GW tangent space, which ultimately gives rise to a fixpoint iteration for approximating GW barycenters using multi-marginal plans. We propose a relaxation of this fixpoint iteration and show that it monotonously decreases the barycenter loss. In certain cases our proposed method naturally provides us with barycentric embeddings. The resulting algorithm is capable of producing qualitative shape interpolations between multiple 3d shapes with support sizes of over thousands of points in reasonable time. In addition, we verify our method on shape classification and multi-graph matching tasks.

In this paper, we present an empirical study on image recognition fairness, i.e., extreme class accuracy disparity on balanced data like ImageNet. We experimentally demonstrate that classes are not equal and the fairness issue is prevalent for image classification models across various datasets, network architectures, and model capacities. Moreover, several intriguing properties of fairness are identified. First, the unfairness lies in problematic representation rather than classifier bias. Second, with the proposed concept of Model Prediction Bias, we investigate the origins of problematic representation during optimization. Our findings reveal that models tend to exhibit greater prediction biases for classes that are more challenging to recognize. It means that more other classes will be confused with harder classes. Then the False Positives (FPs) will dominate the learning in optimization, thus leading to their poor accuracy. Further, we conclude that data augmentation and representation learning algorithms improve overall performance by promoting fairness to some degree in image classification. The Code is available at https://github.com/dvlab-research/Parametric-Contrastive-Learning., Comment: CVPR 2024

In this paper, we are concerned with estimating the joint probability of random variables $X$ and $Y$, given $N$ independent observation blocks $(\boldsymbol{x}^i,\boldsymbol{y}^i)$, $i=1,\ldots,N$, each of $M$ samples $(\boldsymbol{x}^i,\boldsymbol{y}^i) = \bigl((x^i_j, y^i_{\sigma^i(j)}) \bigr)_{j=1}^M$, where $\sigma^i$ denotes an unknown permutation of i.i.d. sampled pairs $(x^i_j,y_j^i)$, $j=1,\ldots,M$. This means that the internal ordering of the $M$ samples within an observation block is not known. We derive a maximum-likelihood inference functional, propose a computationally tractable approximation and analyze their properties. In particular, we prove a $\Gamma$-convergence result showing that we can recover the true density from empirical approximations as the number $N$ of blocks goes to infinity. Using entropic optimal transport kernels, we model a class of hypothesis spaces of density functions over which the inference functional can be minimized. This hypothesis class is particularly suited for approximate inference of transfer operators from data. We solve the resulting discrete minimization problem by a modification of the EMML algorithm to take addional transition probability constraints into account and prove the convergence of this algorithm. Proof-of-concept examples demonstrate the potential of our method.

Coronary anatomy governs local haemodynamics associated with atherosclerotic development, progression and ultimately adverse clinical outcomes. However, lack of large sample size studies and methods to link adverse haemodynamics to anatomical information has hindered meaningful insights to date. The Left Main coronary bifurcations of 127 patients with suspected coronary artery disease in the absence of significant stenosis were segmented from CTCA images before computing the local haemodynamics. We correlated 11 coronary anatomical characteristics with the normalised lumen area exposed to adverse haemodynamics linked with atherosclerotic processes. These include mean curvatures and diameters of branches, bifurcation and inflow angles, and Finet's ratio as the anatomical parameters, and low Time-Averaged Endothelial Shear Stress (lowTAESS<0.5Pa), high Oscillatory Shear Index (highOSI>0.1), and high Relative Residence Time (highRRT>4.17 1/Pa) for the haemodynamic consideration. We separately tested if the geometric measures and haemodynamics indicators differed between subgroups (sex, smokers, and those with hypertension). We then use a step-down multiple linear regression model to find the best model for predicting lowTAESS, highOSI and highRRT. Finet's Ratio (FR) significantly correlated to lowTAESS (p<0.001). Vessel diameters and curvature correlated to highOSI (both p<0.001). Finet's ratio, vessel diameters and daughter branch curvature independently correlated to RRT (all p<0.01). Our results indicate that specific anatomical vessel characteristics may be used as a surrogate of adverse haemodynamic environment associated with clinically adverse mechanisms of disease. This is especially powerful with the latest computing resources and may unlock clinical integration via standard imaging modalities as biomarkers without further computationally expensive simulations., Comment: 14 pages, 2 figures

The link between atherosclerosis and blood flow-induced haemodynamic luminal shear stresses is well established. However, this understanding has not been translated into clinical practice because of the interdependent effects of the complex coronary anatomy and a multitude of potential haemodynamic metrics, which have been challenging to delineate. Thus, this study aims to identify anatomical and haemodynamic differences in coronary trees at different stages of stenoses. A total of 39 left coronary trees were considered, which are publicly available. Each coronary tree was dissected into bifurcations and non-bifurcating segments for comparisons. We calculated a full set of anatomical metrics and performed transient flow simulations to solve the normalised luminal area exposed to Low Time-Average Endothelial Shear Stress (%LowTAESS), High Oscillatory Shear Index (%HighOSI), and High Relative Residence Time (%HighRRT). We statistically investigated the differences between non-stenosed (n=20, Diameter Stenosis DS=0%), moderately (n=12, 0%=70%) stenosed cases, whereby p<0.05* is considered significant. Only the average curvature and %HighOSI differed between the non-stenosed, and moderately or severely stenosed for the coronary trees (p=0.024* and p<0.001*), and non-bifurcating segments (p=0.027* and p<0.001*). %LowTAESS@0.5Pa and %HighRRT@2.5Pa-1 significantly differed between moderately (0%=70\%) stenosed trees (p=0.009* and p=0.012*). Our findings suggest curvature and potentially %HighOSI being critical factors in coronary plaque onset in non-bifurcating segments, whereas %LowTAESS and %HighRRT affect plaque progression after onset.

Coronary artery diseases are among the leading causes of mortality worldwide. Timely and accurate diagnosis, facilitated by precise coronary artery segmentation, is pivotal in changing patient outcomes. In the realm of biomedical imaging, convolutional neural networks, especially the U-Net architecture, have revolutionised segmentation processes. However, one of the primary challenges remains the lack of benchmarking datasets specific to coronary arteries. However through the use of the recently published public dataset ASOCA, the potential of deep learning for accurate coronary segmentation can be improved. This paper delves deep into examining the performance of 25 distinct encoder-decoder combinations. Through analysis of the 40 cases provided to ASOCA participants, it is revealed that the EfficientNet-LinkNet combination, serving as encoder and decoder, stands out. It achieves a Dice coefficient of 0.882 and a 95th percentile Hausdorff distance of 4.753. These findings not only underscore the superiority of our model in comparison to those presented at the MICCAI 2020 challenge but also set the stage for future advancements in coronary artery segmentation, opening doors to enhanced diagnostic and treatment strategies., Comment: 7 pages. This article has been accepted for publication in IEEE 38th International Conference on Image and Vision Computing New Zealand (ICVNZ 2023). This is the author's version which has not been fully edited and content may change prior to final publication

Foundation models like CLIP allow zero-shot transfer on various tasks without additional training data. Yet, the zero-shot performance is less competitive than a fully supervised one. Thus, to enhance the performance, fine-tuning and ensembling are also commonly adopted to better fit the downstream tasks. However, we argue that such prior work has overlooked the inherent biases in foundation models. Due to the highly imbalanced Web-scale training set, these foundation models are inevitably skewed toward frequent semantics, and thus the subsequent fine-tuning or ensembling is still biased. In this study, we systematically examine the biases in foundation models and demonstrate the efficacy of our proposed Generalized Logit Adjustment (GLA) method. Note that bias estimation in foundation models is challenging, as most pre-train data cannot be explicitly accessed like in traditional long-tailed classification tasks. To this end, GLA has an optimization-based bias estimation approach for debiasing foundation models. As our work resolves a fundamental flaw in the pre-training, the proposed GLA demonstrates significant improvements across a diverse range of tasks: it achieves 1.5 pp accuracy gains on ImageNet, an large average improvement (1.4-4.6 pp) on 11 few-shot datasets, 2.4 pp gains on long-tailed classification. Codes are in \url{https://github.com/BeierZhu/GLA}., Comment: V2 proposed a more effective method for label distribution estimation. V1 fixed a typo in abstract; Accepted by NeurIPS2023