Search

Evaluating large vision-language models (LVLMs) is very expensive, due to the high computational costs and the wide variety of tasks. The good news is that if we already have some observed performance scores, we may be able to infer unknown ones. In this study, we propose a new framework for predicting unknown performance scores based on observed ones from other LVLMs or tasks. We first formulate the performance prediction as a matrix completion task. Specifically, we construct a sparse performance matrix $\boldsymbol{R}$, where each entry $R_{mn}$ represents the performance score of the $m$-th model on the $n$-th dataset. By applying probabilistic matrix factorization (PMF) with Markov chain Monte Carlo (MCMC), we can complete the performance matrix, that is, predict unknown scores. Additionally, we estimate the uncertainty of performance prediction based on MCMC. Practitioners can evaluate their models on untested tasks with higher uncertainty first, quickly reducing errors in performance prediction. We further introduce several improvements to enhance PMF for scenarios with sparse observed performance scores. In experiments, we systematically evaluate 108 LVLMs on 176 datasets from 36 benchmarks, constructing training and testing sets for validating our framework. Our experiments demonstrate the accuracy of PMF in predicting unknown scores, the reliability of uncertainty estimates in ordering evaluations, and the effectiveness of our enhancements for handling sparse data., Comment: Under Review. Project page: https://github.com/Qinyu-Allen-Zhao/CrossPred-LVLM

The advancement of large language models (LLMs) has led to a greater challenge of having a rigorous and systematic evaluation of complex tasks performed, especially in enterprise applications. Therefore, LLMs need to be able to benchmark enterprise datasets for various tasks. This work presents a systematic exploration of benchmarking strategies tailored to LLM evaluation, focusing on the utilization of domain-specific datasets and consisting of a variety of NLP tasks. The proposed evaluation framework encompasses 25 publicly available datasets from diverse enterprise domains like financial services, legal, cyber security, and climate and sustainability. The diverse performance of 13 models across different enterprise tasks highlights the importance of selecting the right model based on the specific requirements of each task. Code and prompts are available on GitHub.

The relative contribution of emission from stellar sources and accretion onto supermassive black holes to reionization has been brought into focus again by the apparent high abundance of faint AGN at $4\lesssim z\lesssim11$ uncovered by JWST. We investigate the contribution of these faint AGN to hydrogen and the early stages of helium reionization using the GPU-based radiative transfer code ATON-HE by post-processing a cosmological hydrodynamical simulation from the SHERWOOD-RELICS suite of simulations. We study four models of reionization: two previously studied galaxy-only late-end reionization models and two new models -- a QSO-assisted model and a QSO-only model. In the QSO-assisted model, 1\% of the haloes host AGN with a 10~Myr lifetime, and the AGN luminosities are scaled such that the AGN contribution to the hydrogen-ionizing emissivity is 20\% of that contributed by galaxies. In the QSO-only model, quasars account for all the hydrogen-ionizing emissivity, with 10\% of the haloes hosting AGN, each with a 10 Myr lifetime. All models are calibrated to the observed mean Lyman-$\alpha$ forest transmission at $5\lesssim z\lesssim6.2$. We find that the QSO-assisted model requires an emissivity factor of $1.8$ lower than the galaxy-only models towards the end of reionization and fits the observed distribution of the Lyman-$\alpha$ optical depths well. Our QSO-only model is inconsistent with the observed Lyman-$\alpha$ optical depths distribution. It also results in too high IGM temperatures at $z\lesssim 5$ due to an early onset of HeII reionization unless the escape fraction of HeII-ionizing photons is assumed to be low. Our results suggest that a modest contribution to reionization by faint AGN is in good agreement with the Lyman-$\alpha$ forest data. In contrast, a scenario dominated by faint AGN appears difficult to reconcile with these observations., Comment: 19 pages, 11 figures

We propose in this paper an analytically new construct of a diffusion model whose drift and diffusion parameters yield an exponentially time-decaying Signal to Noise Ratio in the forward process. In reverse, the construct cleverly carries out the learning of the diffusion coefficients on the structure of clean images using an autoencoder. The proposed methodology significantly accelerates the diffusion process, reducing the required diffusion time steps from around 1000 seen in conventional models to 200-500 without compromising image quality in the reverse-time diffusion. In a departure from conventional models which typically use time-consuming multiple runs, we introduce a parallel data-driven model to generate a reverse-time diffusion trajectory in a single run of the model. The resulting collective block-sequential generative model eliminates the need for MCMC-based sub-sampling correction for safeguarding and improving image quality, to further improve the acceleration of image generation. Collectively, these advancements yield a generative model that is an order of magnitude faster than conventional approaches, while maintaining high fidelity and diversity in generated images, hence promising widespread applicability in rapid image synthesis tasks.

We present a new suite of late-end reionization simulations performed with ATON-HE, a revised version of the GPU-based radiative transfer code ATON that includes helium. The simulations are able to reproduce the Ly$\alpha$ flux distribution of the E-XQR-30 sample of QSO absorption spectra at $5 \lesssim z \lesssim 6.2$, and show that a large variety of reionization models are consistent with these data. We explore a range of variations in source models and in the early-stage evolution of reionization. Our fiducial reionization history has a midpoint of reionization at $z = 6.5$, but we also explore an `Early' reionization history with a midpoint at $z = 7.5$ and an `Extremely Early' reionization history with a midpoint at $z = 9.5$. Haloes massive enough to host observed Ly$\alpha$ emitters are highly biased. The fraction of such haloes embedded in ionized bubbles that are large enough to allow high Ly$\alpha$ transmission becomes close to unity much before the volume filling factor of ionized regions. For our fiducial reionization history this happens at $z = 8$, probably too late to be consistent with the detection by JWST of abundant Ly$\alpha$ emission out to $z = 11$. A reionization history in our `Early' model or perhaps even our `Extremely Early' model may be required, suggesting a Thomson scattering optical depth in tension with that reported by Planck, but consistent with recent suggestions of a significantly higher value., Comment: 25 pages, 15 figures

Large vision-language models (LVLMs), designed to interpret and respond to human instructions, occasionally generate hallucinated or harmful content due to inappropriate instructions. This study uses linear probing to shed light on the hidden knowledge at the output layers of LVLMs. We demonstrate that the logit distributions of the first tokens contain sufficient information to determine whether to respond to the instructions, including recognizing unanswerable visual questions, defending against jailbreaking attacks, and identifying deceptive questions. Such hidden knowledge is gradually lost in logits of subsequent tokens during response generation. Then, we illustrate a simple decoding strategy at the generation of the first token, effectively improving the generated content. In experiments, we find a few interesting insights: First, the CLIP model already contains a strong signal for solving these tasks, which indicates potential bias in the existing datasets. Second, we observe performance improvement by utilizing the first logit distributions on three additional tasks, including indicating uncertainty in math solving, mitigating hallucination, and image classification. Last, with the same training data, simply finetuning LVLMs improves models' performance but is still inferior to linear probing on these tasks., Comment: ECCV 2024. Project page: https://github.com/Qinyu-Allen-Zhao/LVLM-LP

Neural architecture search automates the design of neural network architectures usually by exploring a large and thus complex architecture search space. To advance the architecture search, we present a graph diffusion-based NAS approach that uses discrete conditional graph diffusion processes to generate high-performing neural network architectures. We then propose a multi-conditioned classifier-free guidance approach applied to graph diffusion networks to jointly impose constraints such as high accuracy and low hardware latency. Unlike the related work, our method is completely differentiable and requires only a single model training. In our evaluations, we show promising results on six standard benchmarks, yielding novel and unique architectures at a fast speed, i.e. less than 0.2 seconds per architecture. Furthermore, we demonstrate the generalisability and efficiency of our method through experiments on ImageNet dataset., Comment: Accepted at Transactions on Machine Learning Research (TMLR)

Feature shaping refers to a family of methods that exhibit state-of-the-art performance for out-of-distribution (OOD) detection. These approaches manipulate the feature representation, typically from the penultimate layer of a pre-trained deep learning model, so as to better differentiate between in-distribution (ID) and OOD samples. However, existing feature-shaping methods usually employ rules manually designed for specific model architectures and OOD datasets, which consequently limit their generalization ability. To address this gap, we first formulate an abstract optimization framework for studying feature-shaping methods. We then propose a concrete reduction of the framework with a simple piecewise constant shaping function and show that existing feature-shaping methods approximate the optimal solution to the concrete optimization problem. Further, assuming that OOD data is inaccessible, we propose a formulation that yields a closed-form solution for the piecewise constant shaping function, utilizing solely the ID data. Through extensive experiments, we show that the feature-shaping function optimized by our method improves the generalization ability of OOD detection across a large variety of datasets and model architectures., Comment: ICLR 2024. Project page: https://github.com/Qinyu-Allen-Zhao/OptFSOOD

Quantized networks use less computational and memory resources and are suitable for deployment on edge devices. While quantization-aware training QAT is the well-studied approach to quantize the networks at low precision, most research focuses on over-parameterized networks for classification with limited studies on popular and edge device friendly single-shot object detection and semantic segmentation methods like YOLO. Moreover, majority of QAT methods rely on Straight-through Estimator (STE) approximation which suffers from an oscillation phenomenon resulting in sub-optimal network quantization. In this paper, we show that it is difficult to achieve extremely low precision (4-bit and lower) for efficient YOLO models even with SOTA QAT methods due to oscillation issue and existing methods to overcome this problem are not effective on these models. To mitigate the effect of oscillation, we first propose Exponentially Moving Average (EMA) based update to the QAT model. Further, we propose a simple QAT correction method, namely QC, that takes only a single epoch of training after standard QAT procedure to correct the error induced by oscillating weights and activations resulting in a more accurate quantized model. With extensive evaluation on COCO dataset using various YOLO5 and YOLO7 variants, we show that our correction method improves quantized YOLO networks consistently on both object detection and segmentation tasks at low-precision (4-bit and 3-bit)., Comment: WACV 2024

Let $G$ be a connected multigraph with $n$ vertices, and suppose $G$ has been edge-colored with $n-1$ colors so that each color class induces a spanning tree. Rota's Basis Conjecture for graphic matroids posits that one can find $n-1$ mutually edge-disjoint rainbow spanning trees. In a recent paper, Maezawa and Yazawa have shown that the conjecture holds if one assumes that the color classes induce spanning stars. We delve further into the star case to explore some extreme subcases including: all stars with different centers, the same center, or one of two centers. In addition, we identify the cases in which a graph composed of monochromatic stars can be decomposed into rainbow stars. We also show that the statement is false if one replaces `stars' with `paths'., Comment: 15 pages, 8 figures

Confronting measurements of the Lyman-$\alpha$ forest with cosmological hydrodynamical simulations has produced stringent constraints on models of particle dark matter and the thermal and ionization state of the intergalactic medium. We investigate the robustness of such models of the Lyman-$\alpha$ forest, focussing on the effect of particle initial conditions on the Lyman-$\alpha$ forest statistics in cosmological SPH simulations. We study multiple particle initialization algorithms in simulations that are designed to be identical in other respects. In agreement with the literature, we find that the correct linear theory evolution is obtained when a glass-like configuration is used for initial unperturbed gas particle positions alongside a regular grid configuration for dark matter particles and the use of non-identical initial density perturbations for gas and dark matter. However, we report that this introduces a large scale-dependent distortion in the one-dimensional Lyman-$\alpha$ transmission power spectrum at small scales ($k > 0.05$ s/km). The effect is close to $50\%$ at $k\sim 0.1$ s/km, and persists at higher resolution. This can severely bias inferences in parameters such as the dark matter particle mass. By considering multiple initial conditions codes and their variations, we also study the impact of a variety of other assumptions and algorithmic choices, such as adaptive softening, background radiation density, particle staggering, and perturbation theory accuracy, on the matter power spectrum, the Lyman-$\alpha$ flux power spectrum, and the Lyman-$\alpha$ flux PDF. This work reveals possible pathways towards more accurate theoretical models of the Lyman-$\alpha$ forest to match the quality of upcoming measurements., Comment: 18 pages, 16 figures, accepted for publication in MNRAS, comments welcome

Quantum key distribution protocols have been designed for layered networks employing multidimensional entangled and separable orbital angular momentum states [Phys. Rev. A 97, 032312 (2018), Int. J. Theor. Phys. 62, 104 (2023)]. This paper seeks an answer to the overarching question -- in the context of secure quantum communication tasks, do orbital angular momentum states act merely as an alternative or do they act as an indispensable resource? We start by showing that the task of quantum key distribution in layered networks can also be accomplished with several copies of lower-dimensional states such as polarization qubits. For this reason, orbital angular momentum states do not offer any intrinsic advantage in layered quantum key distribution. The potential of OAM states unveils itself in the enhancement of key generation rates and integrated quantum communication tasks, which we present in this paper. These tasks can be implemented exclusively with high-dimensional OAM entangled states. In fact, we show that the employment of orbital angular momentum states eliminates the need for entangled state measurements, whose implementation is resource-intensive. We believe that this study opens up a possibility for designing several quantum information processing tasks in which multidimensional OAM states act as an indispensable resource.

Whilst deep learning techniques have achieved excellent emotion prediction, they still require large amounts of labelled training data, which are (a) onerous and tedious to compile, and (b) prone to errors and biases. We propose Multi-Task Contrastive Learning for Affect Representation (\textbf{MT-CLAR}) for few-shot affect inference. MT-CLAR combines multi-task learning with a Siamese network trained via contrastive learning to infer from a pair of expressive facial images (a) the (dis)similarity between the facial expressions, and (b) the difference in valence and arousal levels of the two faces. We further extend the image-based MT-CLAR framework for automated video labelling where, given one or a few labelled video frames (termed \textit{support-set}), MT-CLAR labels the remainder of the video for valence and arousal. Experiments are performed on the AFEW-VA dataset with multiple support-set configurations; moreover, supervised learning on representations learnt via MT-CLAR are used for valence, arousal and categorical emotion prediction on the AffectNet and AFEW-VA datasets. The results show that valence and arousal predictions via MT-CLAR are very comparable to the state-of-the-art (SOTA), and we significantly outperform SOTA with a support-set $\approx$6\% the size of the video dataset., Comment: 10 pages, 6 figures, to be published in Proceedings of the 31st ACM International Conference on Multimedia (MM '23)

Meetings play a critical infrastructural role in the coordination of work. In recent years, due to shift to hybrid and remote work, more meetings are moving to online Computer Mediated Spaces. This has led to new problems (e.g. more time spent in less engaging meetings) and new opportunities (e.g. automated transcription/captioning and recap support). Recent advances in large language models (LLMs) for dialog summarization have the potential to improve the experience of meetings by reducing individuals' meeting load and increasing the clarity and alignment of meeting outputs. Despite this potential, they face technological limitation due to long transcripts and inability to capture diverse recap needs based on user's context. To address these gaps, we design, implement and evaluate in-context a meeting recap system. We first conceptualize two salient recap representations -- important highlights, and a structured, hierarchical minutes view. We develop a system to operationalize the representations with dialogue summarization as its building blocks. Finally, we evaluate the effectiveness of the system with seven users in the context of their work meetings. Our findings show promise in using LLM-based dialogue summarization for meeting recap and the need for both representations in different contexts. However, we find that LLM-based recap still lacks an understanding of whats personally relevant to participants, can miss important details, and mis-attributions can be detrimental to group dynamics. We identify collaboration opportunities such as a shared recap document that a high quality recap enables. We report on implications for designing AI systems to partner with users to learn and improve from natural interactions to overcome the limitations related to personal relevance and summarization quality., Comment: in review for CSCW 24

Scalable modern-time fault-tolerant quantum computation and quantum communication in a network employ a large number of physical qubits. For example, IBM is reported to have made a 127-qubit quantum computer. Unlike classical computation, quantum computation employs different types of logical qubits and qudits in terms of physical multiqubit and multiqudit systems respectively. Given this, of particular interest to us is to enquire on how quantum coherence in logical qubits is a manifestation of underlying quantum correlations in constituent physical multiqubit systems and vice-versa. In a recent work [Asthana, Sooryansh. New J Phys 24.5 (2022): 053026], we have shown that there is reciprocity in nonclassical correlations in physical multiqubit systems and coherence in a single logical qubit system. Subsequently, we have generalised the framework to higher dimensional quantum systems []. The crux of this study is that a single nonclassicality condition derived for quantum coherence in a logical system detects more than one type of nonclassicality in Hilbert spaces of nonidentical dimensions., Comment: Accepted in QIP 2023 (held in Ghent, Belgium) as a poster presentation

We propose a quantum secure direct communication protocol using SU (2)- invariant $2\times N$ separable states, identified as separable equivalents of two-qubit entangled Werner states in [Bharath & Ravishankar, Phys. Rev. A 89, 062110}]. These states can be experimentally realized as incoherent superpositions of separable polarisation-OAM states., Comment: Accepted for oral presentation in International Conference on Quantum Computing and Communications held at Bathinda, Punjab, India

Secure communication in layered networks having differently preferred participants has attracted a lot of research attention. Protocols for key distribution in a layered network have been recently proposed in [M. Pivoluska et al., Phys. Rev. A 97, 032312] by employing asymmetrically entangled multiqudit states. Due to the employment of asymmetrically entangled multiqudit states, the yield of these protocols is very low. To address this issue, in this work, we have proposed semi-quantum secure communication protocols by employing separable states only which give a better yield and a higher key generation rate. As illustrations, we present two representative protocols. The first protocol allows sharing of two keys simultaneously in a network of two layers. The second protocol facilitates direct communication in one layer and key distribution in the other. The separable states, i.e., coherent pulses of orbital angular momentum required in the protocols are easily realizable with current technologies., Comment: Accepted in QIP 2023 (held at Ghent, Belgium) for a poster presentation

We present a protocol for collaborative error-immune information through atmospheric turbulence. The protocol allows transferring of information in a network of three participants., Comment: Accepted for oral presentation in International Conference on Quantum Computing and Communications, 2023 held at Bathinda, Punjab, India

Whilst a majority of affective computing research focuses on inferring emotions, examining mood or understanding the \textit{mood-emotion interplay} has received significantly less attention. Building on prior work, we (a) deduce and incorporate emotion-change ($\Delta$) information for inferring mood, without resorting to annotated labels, and (b) attempt mood prediction for long duration video clips, in alignment with the characterisation of mood. We generate the emotion-change ($\Delta$) labels via metric learning from a pre-trained Siamese Network, and use these in addition to mood labels for mood classification. Experiments evaluating \textit{unimodal} (training only using mood labels) vs \textit{multimodal} (training using mood plus $\Delta$ labels) models show that mood prediction benefits from the incorporation of emotion-change information, emphasising the importance of modelling the mood-emotion interplay for effective mood inference., Comment: 9 pages, 3 figures, 6 tables, published in IEEE International Conference on Affective Computing and Intelligent Interaction

High mass accretion rate onto strongly magnetised neutron stars results in the appearance of accretion columns supported by the radiation pressure and confined by the strong magnetic field of a star. At mass accretion rates above $\sim 10^{19}\,{\rm g\,s^{-1}}$, accretion columns are expected to be advective. Under such conditions, a noticeable part of the total energy release can be carried away by neutrinos of a MeV energy range. Relying on a simple model of the neutrino luminosity of accreting strongly magnetised neutron stars, we estimate the neutrino energy fluxes expected from six ULX pulsars known up to date and three brightest Be X-ray transits hosting magnetised neutron stars. Despite the large neutrino luminosity expected in ULX pulsars, the neutrino energy flux from the Be X-ray transients of our Galaxy, SMC and LMC is dominant. However, the neutrino flux from the brightest X-ray transients is estimated to be below the isotropic background by two orders of magnitude at least, which makes impossible direct registration of neutrino emission from accreting strongly magnetised neutron stars nowadays., Comment: 7 pages, 6 figures, submitted to MNRAS

We measure the mean free path ($\lambda_{\rm mfp,HI}$), photo-ionization rate ($\langle \Gamma_{\rm HI} \rangle$) and neutral fraction ($\langle f_{\rm HI} \rangle$) of hydrogen in 12 redshift bins at $4.85

Secure quantum communication protocols based on a prepare-and-measure scheme employ mutually unbiased bases. In these protocols, many runs, in which different participants measure in different bases, simply go wasted. In this paper, we show that it is possible to reduce the number of such runs by a suitable design of the key generation rule. This results in a significant increase in the key generation rate (KGR). We illustrate this advantage by proposing quantum and semi-quantum key distribution protocols by employing effective qubits encoded in higher dimensional quantum systems. None of them demands the preparation of entangled states as resources and a relatively large amount of information can be transferred. For this reason, we believe that our proposals are worth pursuing experimentally.