Search

Verifying the violation of Bell's inequality is one of the most representative methods to demonstrate that entangled photon pairs prepared in a quantum optics-based system exhibit quantum properties. While experiments on Bell inequality violations have been theoretically well-established and extensively conducted to implement various quantum information technologies in laboratory settings, mathematical modeling for accurately predicting the distribution of high-intensity entangled photon pairs in high-loss environments remains an issue that requires further research. As the brightness of the entangled photon pairs increases, the influence of multi-photon effects becomes more significant, leading to a decrease in the CHSH value $S$ and also a reduction in the standard deviation of the CHSH value $\Delta S$. Therefore, a new analysis of the $(S-2)/\Delta S$ value is required to more precisely confirm the degree of CHSH inequality violation including the reliability of $S$. In this paper, we propose a mathematical model to predict the $(S-2)/\Delta S$ value as a function of the brightness of the entangled photon pair source, and we also suggest the need to optimize the brightness of this source. Additionally, we provide experimental evidence supporting this model. The experiment confirms that when the mean photon number is $\mu=0.026$ in an entanglement distribution setup with a total loss of $-19.03$ dB, the CHSH value drops to 2.69, while the $(S-2)/\Delta S$ value increases to 60.95., Comment: 11 pages, 1 table, and 4 figures

Galaxy interactions can significantly affect the star formation in galaxies, but it remains a challenge to achieve a consensus on the star formation rate (SFR) enhancement in galaxy pairs. Here, we investigate the SFR enhancement of gas-rich galaxy pairs detected by the Widefield ASKAP L-band Legacy All-sky Blind surveY (WALLABY). We construct a sample of 278 paired galaxies spanning a stellar mass ($M_\ast$) range from $10^{7.6}$ to $10^{11.2}M_\odot$. We obtain individual masses of atomic hydrogen (HI) for these paired galaxies, using a novel deblending algorithm for HI data cubes. Quantifying the interaction stages and strengths with parameters motivated by first principles, we find that at fixed stellar and HI mass, the alteration in SFR of galaxy pairs starts when their dark matter halos encounter. For galaxies with stellar mass lower than $10^9M_\odot$, their SFRs show tentative suppression of 1.4 sigma after the halo encounter, and then become enhanced when their HI disks overlap, regardless of mass ratios. In contrast, the SFRs of galaxies with $M_\ast > 10^9M_\odot$ increase monotonically toward smaller projected distances and radial velocity offsets. When a close companion is present, a pronounced SFR enhancement is found for the most HI-poor high-mass galaxies in our sample. Collecting the observational evidence, we provide a coherent picture of the evolution of galaxy pairs, and discuss how the tidal effects and hydrodynamic processes shape the SFR enhancement. Our results provide a coherent picture of gas-rich galaxy interactions and impose constraints on the underlying physical processes., Comment: 25 pages, 15 figures, accepted for publication in ApJ

The large deviations theory for heavy-tailed processes has seen significant advances in the recent past. In particular, Rhee et al. (2019) and Bazhba et al. (2020) established large deviation asymptotics at the sample-path level for L\'evy processes and random walks with regularly varying and (heavy-tailed) Weibull-type increments. This leaves the lognormal case -- one of the three most prominent classes of heavy-tailed distributions, alongside regular variation and Weibull -- open. This article establishes the \emph{extended large deviation principle} (extended LDP) at the sample-path level for one-dimensional L\'evy processes and random walks with lognormal-type increments. Building on these results, we also establish the extended LDPs for multi-dimensional processes with independent coordinates. We demonstrate the sharpness of these results by constructing counterexamples, thereby proving that our results cannot be strengthened to a standard LDP under $J_1$ topology and $M_1'$ topology.

Mutual Information (MI) is a fundamental metric for quantifying dependency between two random variables. When we can access only the samples, but not the underlying distribution functions, we can evaluate MI using sample-based estimators. Assessment of such MI estimators, however, has almost always relied on analytical datasets including Gaussian multivariates. Such datasets allow analytical calculations of the true MI values, but they are limited in that they do not reflect the complexities of real-world datasets. This study introduces a comprehensive benchmark suite for evaluating neural MI estimators on unstructured datasets, specifically focusing on images and texts. By leveraging same-class sampling for positive pairing and introducing a binary symmetric channel trick, we show that we can accurately manipulate true MI values of real-world datasets. Using the benchmark suite, we investigate seven challenging scenarios, shedding light on the reliability of neural MI estimators for unstructured datasets., Comment: NeurIPS 2024

Recent advances in audio understanding tasks leverage the reasoning capabilities of LLMs. However, adapting LLMs to learn audio concepts requires massive training data and substantial computational resources. To address these challenges, Retrieval-Augmented Generation (RAG) retrieves audio-text pairs from a knowledge base (KB) and augments them with query audio to generate accurate textual responses. In RAG, the relevance of the retrieved information plays a crucial role in effectively processing the input. In this paper, we analyze how different retrieval methods and knowledge bases impact the relevance of audio-text pairs and the performance of audio captioning with RAG. We propose generative pair-to-pair retrieval, which uses the generated caption as a text query to accurately find relevant audio-text pairs to the query audio, thereby improving the relevance and accuracy of retrieved information. Additionally, we refine the large-scale knowledge base to retain only audio-text pairs that align with the contextualized intents. Our approach achieves state-of-the-art results on benchmarks including AudioCaps, Clotho, and Auto-ACD, with detailed ablation studies validating the effectiveness of our retrieval and KB construction methods.

Binary Code Similarity Analysis (BCSA) has a wide spectrum of applications, including plagiarism detection, vulnerability discovery, and malware analysis, thus drawing significant attention from the security community. However, conventional techniques often face challenges in balancing both accuracy and scalability simultaneously. To overcome these existing problems, a surge of deep learning-based work has been recently proposed. Unfortunately, many researchers still find it extremely difficult to conduct relevant studies or extend existing approaches. First, prior work typically relies on proprietary benchmark without making the entire dataset publicly accessible. Consequently, a large-scale, well-labeled dataset for binary code similarity analysis remains precious and scarce. Moreover, previous work has primarily focused on comparing at the function level, rather than exploring other finer granularities. Therefore, we argue that the lack of a fine-grained dataset for BCSA leaves a critical gap in current research. To address these challenges, we construct a benchmark dataset for fine-grained binary code similarity analysis called BinSimDB, which contains equivalent pairs of smaller binary code snippets, such as basic blocks. Specifically, we propose BMerge and BPair algorithms to bridge the discrepancies between two binary code snippets caused by different optimization levels or platforms. Furthermore, we empirically study the properties of our dataset and evaluate its effectiveness for the BCSA research. The experimental results demonstrate that BinSimDB significantly improves the performance of binary code similarity comparison., Comment: Published by International Conference on Security and Privacy in Communication Networks (SecureComm 2024)

The next generation of radio astronomy telescopes are challenging existing data analysis paradigms, as they have an order of magnitude larger collecting area and bandwidth. The two primary problems encountered when processing this data are the need for storage and that processing is primarily I/O limited. An example of this is the data deluge expected from the SKA-Low Telescope of about 300 PB per year. To remedy these issues, we have demonstrated lossy and lossless compression of data on an existing precursor telescope, the Australian Square Kilometre Array Pathfinder (ASKAP), using MGARD and ADIOS2 libraries. We find data processing is faster by a factor of 7 and give compression ratios from a factor of 7 (lossless) up to 37 (lossy with an absolute error bound of 1e-3). We discuss the effectiveness of lossy MGARD compression and its adherence to the designated error bounds, the trade-off between these error bounds and the corresponding compression ratios, as well as the potential consequences of these I/O and storage improvements on the science quality of the data products., Comment: 9 pages, 10 figures. Included in the conference proceedings of Cray User Group Conference 2024

We present the Pilot Survey Phase 2 data release for the Wide-field ASKAP L-band Legacy All-sky Blind surveY (WALLABY), carried-out using the Australian SKA Pathfinder (ASKAP). We present 1760 HI detections (with a default spatial resolution of 30") from three pilot fields including the NGC 5044 and NGC 4808 groups as well as the Vela field, covering a total of ~180 deg$^2$ of the sky and spanning a redshift up to $z \simeq 0.09$. This release also includes kinematic models for over 126 spatially resolved galaxies. The observed median rms noise in the image cubes is 1.7 mJy per 30" beam and 18.5 kHz channel. This corresponds to a 5$\sigma$ HI column density sensitivity of $\sim 9.1\times10^{19}(1 + z)^4$ cm$^{-2}$ per 30" beam and $\sim 20$ km/s channel, and a 5$\sigma$ HI mass sensitivity of $\sim 5.5\times10^8 (D/100$ Mpc)$^{2}$ M$_{\odot}$ for point sources. Furthermore, we also present for the first time 12" high-resolution images ("cut-outs") and catalogues for a sub-sample of 80 sources from the Pilot Survey Phase 2 fields. While we are able to recover sources with lower signal-to-noise ratio compared to sources in the Public Data Release 1, we do note that some data quality issues still persist, notably, flux discrepancies that are linked to the impact of side lobes associated with the dirty beams due to inadequate deconvolution. However, in spite of these limitations, the WALLABY Pilot Survey Phase 2 has already produced roughly a third of the number of HIPASS sources, making this the largest spatially resolved HI sample from a single survey to date., Comment: 25 pages, 16 figures, 6 tables, accepted for publication in Publications of the Astronomical Society of Australia (PASA)

The data volumes generated by the WALLABY atomic Hydrogen (HI) survey using the Australiian Square Kilometre Array Pathfinder (ASKAP) necessitate greater automation and reliable automation in the task of source-finding and cataloguing. To this end, we introduce and explore a novel deep learning framework for detecting low Signal-to-Noise Ratio (SNR) HI sources in an automated fashion. Specfically, our proposed method provides an automated process for separating true HI detections from false positives when used in combination with the Source Finding Application (SoFiA) output candidate catalogues. Leveraging the spatial and depth capabilities of 3D Convolutional Neural Networks (CNNs), our method is specifically designed to recognise patterns and features in three-dimensional space, making it uniquely suited for rejecting false positive sources in low SNR scenarios generated by conventional linear methods. As a result, our approach is significantly more accurate in source detection and results in considerably fewer false detections compared to previous linear statistics-based source finding algorithms. Performance tests using mock galaxies injected into real ASKAP data cubes reveal our method's capability to achieve near-100% completeness and reliability at a relatively low integrated SNR~3-5. An at-scale version of this tool will greatly maximise the science output from the upcoming widefield HI surveys., Comment: 14 pages, 12 figures, accepted for publication in the Publications of the Astronomical Society of Australia

The Global Positioning System (GPS), one of the Global Navigation Satellite Systems (GNSS), provides accurate position, navigation and time (PNT) information to various applications. One of the application that is highly receiving attention is satellite vehicles, especially Low Earth Orbit (LEO) satellites. Due to their limited ways to get PNT information and low performance of their onboard clocks, GPS system time (GPST) provided by GPS is a good reference clock to synchronize. However, GPS is well-known for its vulnerability to intentional or unintentional interference. This study aims to maintain the onboard clock with less error relative to the GPST even when the GPS signal is disrupted. In this study, we analyzed two major factors that affects the quality of the GPS measurements: the number of the visible satellites and the geometry of the satellites. Then, we proposed a weighted model for a Chip-Scale Atomic Clock (CSAC) that mitigates the clock error relative to the GPST while considering the two factors. Based on this model, a stand-alone CSAC could maintain its error less than 4 microseconds, even in a situation where no GPS signals are received for 12 hours., Comment: 6 pages

In recent years, quadruped robots have attracted significant attention due to their practical advantages in maneuverability, particularly when navigating rough terrain and climbing stairs. As these robots become more integrated into various industries, including construction and healthcare, researchers have increasingly focused on developing intuitive interaction methods such as speech and gestures that do not require separate devices such as keyboards or joysticks. This paper aims at investigating a comfortable and efficient interaction method with quadruped robots that possess a familiar form factor. To this end, we conducted two preliminary studies to observe how individuals naturally interact with a quadruped robot in natural and controlled settings, followed by a prototype experiment to examine human preferences for body-based and hand-based gesture controls using a Unitree Go1 Pro quadruped robot. We assessed the user experience of 13 participants using the User Experience Questionnaire and measured the time taken to complete specific tasks. The findings of our preliminary results indicate that humans have a natural preference for communicating with robots through hand and body gestures rather than speech. In addition, participants reported higher satisfaction and completed tasks more quickly when using body gestures to interact with the robot. This contradicts the fact that most gesture-based control technologies for quadruped robots are hand-based. The video is available at https://youtu.be/rysv1p1zvp4., Comment: 16 pages

A high-brightness entangled photon pair (HBEPP) source is essential for conducting entanglement-based quantum key distribution (QKD) between a satellite and a ground station. While an ultrabright source can overcome significant losses in satellite-based QKD (SQKD) and increase the sifted key rate, it also induces the multi-photon effect, raising the system's error rate. To accurately estimate system performance, we first present an analytical model for calculating the measurement probabilities of HBEPP distribution through an asymmetric loss channel. Based on this model, we propose the use of a passive-intensity HBEPP source for SQKD systems, assuming a polarization-independent channel and threshold detectors for measurement. We confirm that fixing the mean photon number at $\bar{\mu}=0.1$ achieves a performance of $99.7\%$ compared to the ideal one-way communication entanglement-based SQKD protocol, which is effectively optimizing the HBEPP source brightness in accordance with system losses., Comment: 9 pages, 5 figures, 3 tables

We study star formation (SF) quenching of satellite galaxies with $M_{*} > 10^7\,M_{\odot}$ within two low-mass groups ($M_{\rm vir}=10^{12.9}$ and $10^{12.7} \,M_{\odot}$) using the NewHorizon simulation. We confirm that satellite galaxies ($M_{*}\lesssim10^{10}\,M_{\odot}$) are more prone to quenching than their field counterparts. This quenched fraction decreases with increasing stellar mass, consistent with recent studies. Similar to the findings in cluster environments, we note a correlation between the orbital motions of galaxies within these groups and the phenomenon of SF quenching. Specifically, SF is suppressed at the group center, and for galaxies with $M_{*} > 10^{9.1}\,M_{\odot}$, there is often a notable rejuvenation phase following a temporary quenching period. The SF quenching at the group center is primarily driven by changes in star formation efficiency and the amount of gas available, both of which are influenced by hydrodynamic interactions between the interstellar medium and surrounding hot gas within the group. Conversely, satellite galaxies with $M_{*} < 10^{8.2}\,M_{\odot}$ experience significant gas removal within the group, leading to SF quenching. Our analysis highlights the complexity of SF quenching in satellite galaxies in group environments, which involves an intricate competition between the efficiency of star formation (which depends on the dynamical state of the gas) on the one hand, and the availability of cold dense gas on the other hand. This challenges the typical understanding of environmental effects based on gas stripping through ram pressure, suggesting a need for a new description of galaxy evolution under mild environmental effects., Comment: 20 pages, 10 figures, Published in Astrophysical Journal

Modern graphics processing units (GPUs) provide an unprecedented level of computing power. In this study, we present a high-performance, multi-GPU implementation of the analytical nuclear gradient for Kohn-Sham time-dependent density functional theory (TDDFT), employing the Tamm-Dancoff approximation (TDA) and Gaussian-type atomic orbitals as basis functions. We discuss GPU-efficient algorithms for the derivatives of electron repulsion integrals and exchange-correlation functionals within the range-separated scheme. As an illustrative example, we calculated the TDA-TDDFT gradient of the S1 state of a full-scale green fluorescent protein with explicit water solvent molecules, totaling 4353 atoms, at the wB97X/def2-SVP level of theory. Our algorithm demonstrates favorable parallel efficiencies on a high-speed distributed system equipped with 256 Nvidia A100 GPUs, achieving >70% with up to 64 GPUs and 31% with 256 GPUs, effectively leveraging the capabilities of modern high-performance computing systems., Comment: 13 pages, 9 figures

The objective of Domain Generalization (DG) is to devise algorithms and models capable of achieving high performance on previously unseen test distributions. In the pursuit of this objective, average measure has been employed as the prevalent measure for evaluating models and comparing algorithms in the existing DG studies. Despite its significance, a comprehensive exploration of the average measure has been lacking and its suitability in approximating the true domain generalization performance has been questionable. In this study, we carefully investigate the limitations inherent in the average measure and propose worst+gap measure as a robust alternative. We establish theoretical grounds of the proposed measure by deriving two theorems starting from two different assumptions. We conduct extensive experimental investigations to compare the proposed worst+gap measure with the conventional average measure. Given the indispensable need to access the true DG performance for studying measures, we modify five existing datasets to come up with SR-CMNIST, C-Cats&Dogs, L-CIFAR10, PACS-corrupted, and VLCS-corrupted datasets. The experiment results unveil an inferior performance of the average measure in approximating the true DG performance and confirm the robustness of the theoretically supported worst+gap measure.

Adiabatic quantum annealers encounter scalability challenges due to exponentially fast diminishing energy gaps between ground and excited states with qubit-count increase. This introduces errors in identifying ground states compounded by a thermal noise. We propose a novel algorithmic scheme called statistical qubit freezing (SQF) that selectively fixes the state of statistically deterministic qubit in the annealing Hamiltonian model of the given problem. Applying freezing repeatedly, SQF significantly enhances the spectral gap between of an adiabatic process, as an example, by up to 60\% compared to traditional annealing methods in the standard D-Wave's quantum Ising machine solution, effectively overcoming the fundamental limitations., Comment: 11 pages, 6 figures

The latest advancements in unsupervised learning of sentence embeddings predominantly involve employing contrastive learning-based (CL-based) fine-tuning over pre-trained language models. In this study, we analyze the latest sentence embedding methods by adopting representation rank as the primary tool of analysis. We first define Phase 1 and Phase 2 of fine-tuning based on when representation rank peaks. Utilizing these phases, we conduct a thorough analysis and obtain essential findings across key aspects, including alignment and uniformity, linguistic abilities, and correlation between performance and rank. For instance, we find that the dynamics of the key aspects can undergo significant changes as fine-tuning transitions from Phase 1 to Phase 2. Based on these findings, we experiment with a rank reduction (RR) strategy that facilitates rapid and stable fine-tuning of the latest CL-based methods. Through empirical investigations, we showcase the efficacy of RR in enhancing the performance and stability of five state-of-the-art sentence embedding methods.

We introduce the state-of-the-art semi-analytic model {\small FEGA} (Formation and Evolution of GAlaxies), which incorporates updated prescriptions for key physical processes in galaxy formation. Notably, {\small FEGA} features an unprecedented semi-analytic modeling of positive Active Galactic Nuclei (AGN) feedback. The model combines the latest prescriptions for gas infall and cooling, a revised star formation recipe that incorporates the extended Kennicutt-Schmidt relation, disk instability, updated supernovae feedback, reincorporation of ejected gas, hot gas stripping from satellite galaxies, and the formation of diffuse light. A novel description of AGN feedback is introduced, describing the positive mode as a burst of star formation from a cooling gas fraction. {\small FEGA} is rigorously calibrated using an MCMC procedure to match the evolution of the stellar mass function from high redshift to the present. Subsequently, the model is tested against several observed and predicted scaling relations, including the star formation rate-mass, black hole-bulge and stellar mass, stellar-to-halo mass, and red fraction-mass relations. Additionally, we test {\small FEGA} against other galaxy properties such as the distribution of specific star formation rates, stellar metallicity and morphology. Our results demonstrate that the inclusion of positive AGN feedback can co-exist with its negative counterpart without drastic alterations to other prescriptions. Importantly, this inclusion improves the ability of the model to describe the primary scaling relations observed in galaxies., Comment: 30 pages, 12 figures and 1 table. Minor and major changes. Accepted for publication in ApJS

The formation pathways of compact stellar systems (CSSs) are still under debate. We utilize the \NH\ simulation to investigate the origins of such objects in the field environment. We identified 55 CSS candidates in the simulation whose properties are similar to those of the observed ultra-compact dwarfs and compact ellipticals. All but two most massive objects (compact elliptical candidates) are a result of a short starburst. Sixteen are formed by tidal stripping, while the other 39 are intrinsically compact from their birth. The stripped objects originate from dwarf-like galaxies with a dark halo, but most of their dark matter is stripped through their orbital motion around a more massive neighbor galaxy. The 39 intrinsically compact systems are further divided into ``associated'' or ``isolated'' groups, depending on whether they were born near a massive dark halo or not. The isolated intrinsic compact objects (7) are born in a dark halo and their stellar properties are older and metal-poor compared to the associated counterparts (32). The stripped compact objects occupy a distinct region in the age-metallicity plane from the intrinsic compact objects. The associated intrinsic compact objects in our sample have never had a dark halo; they are the surviving star clumps of a massive galaxy., Comment: 12 pages, 7 figures, 1 table. submitted to ApJ

Starting from a general molecular Hamiltonian expressed in the basis of adiabatic electronic and nuclear position states, where a compact and complete expression for nonadiabatic derivative coupling (NDC) Hamiltonian term is obtained, we provide a general analysis of the Fermi's golden rule (FGR) rate expression for nonadiabatic transitions between adiabatic states. We then consider a quasi-adiabatic approximation that uses crude adiabatic states evaluated at the minimum potential energy configuration of the initial adiabatic state as the basis for the zeroth order adiabatic and NDC coupling terms of the Hamiltonian. Although application of this approximation is rather limited, it allows deriving a general FGR rate expression without further approximation and still accounts for non-Condon effect arising from momentum operators of NDC terms and its coupling with vibronic displacements. For a generic and widely used model where all nuclear degrees of freedom and environmental effects are represented as linearly coupled harmonic oscillators, we derive a closed form FGR rate expression that requires only Fourier transform. The resulting rate expression includes quadratic contributions of NDC terms and their couplings to Franck-Condon modes, which require evaluation of two additional bath spectral densities in addition to conventional one that appears in a typical FGR rate theory based on the Condon approximation. Model calculations for the case where nuclear vibrations consist of both a sharp high frequency mode and an Ohmic bath spectral density illustrate new features and implications of the rate expression. We then apply our theoretical expression to the nonradiative decay from the first excited singlet state of azulene, which illustrates the utility and implications of our theoretical results., Comment: 20 pages, 5 figures

The tight relationship between infrared luminosity (L$_\mathrm{TIR}$) and 1.4 GHz radio continuum luminosity (L$_\mathrm{1.4GHz}$) has proven useful for understanding star formation free from dust obscuration. Infrared emission in star-forming galaxies typically arises from recently formed, dust-enshrouded stars, whereas radio synchrotron emission is expected from subsequent supernovae. By leveraging the wealth of ancillary far-ultraviolet - far-infrared photometry from the Deep Extragalactic VIsible Legacy Survey (DEVILS) and Galaxy and Mass Assembly (GAMA) surveys, combined with 1.4 GHz observations from the MeerKAT International GHz Tiered Extragalactic Exploration (MIGHTEE) survey and Deep Investigation of Neutral Gas Origins (DINGO) projects, we investigate the impact of timescale differences between far-ultraviolet - far-infrared and radio-derived star formation rate (SFR) tracers. We examine how the SED-derived star formation histories (SFH) of galaxies can be used to explain discrepancies in these SFR tracers, which are sensitive to different timescales. Galaxies exhibiting an increasing SFH have systematically higher L$_\mathrm{TIR}$ and SED-derived SFRs than predicted from their 1.4 GHz radio luminosity. This indicates that insufficient time has passed for subsequent supernovae-driven radio emission to accumulate. We show that backtracking the SFR(t) of galaxies along their SED-derived SFHs to a time several hundred megayears prior to their observed epoch will both linearise the SFR-L$_\mathrm{1.4GHz}$ relation and reduce the overall scatter. The minimum scatter in the SFR(t)-L$_\mathrm{1.4GHz}$ is reached at 200 - 300 Myr prior, consistent with theoretical predictions for the timescales required to disperse the cosmic ray electrons responsible for the synchrotron emission., Comment: 23 pages, 13 figures, 2 tables. Accepted for publication in MNRAS

Stimulated by the sophisticated reasoning capabilities of recent Large Language Models (LLMs), a variety of strategies for bridging video modality have been devised. A prominent strategy involves Video Language Models (VideoLMs), which train a learnable interface with video data to connect advanced vision encoders with LLMs. Recently, an alternative strategy has surfaced, employing readily available foundation models, such as VideoLMs and LLMs, across multiple stages for modality bridging. In this study, we introduce a simple yet novel strategy where only a single Vision Language Model (VLM) is utilized. Our starting point is the plain insight that a video comprises a series of images, or frames, interwoven with temporal information. The essence of video comprehension lies in adeptly managing the temporal aspects along with the spatial details of each frame. Initially, we transform a video into a single composite image by arranging multiple frames in a grid layout. The resulting single image is termed as an image grid. This format, while maintaining the appearance of a solitary image, effectively retains temporal information within the grid structure. Therefore, the image grid approach enables direct application of a single high-performance VLM without necessitating any video-data training. Our extensive experimental analysis across ten zero-shot video question answering benchmarks, including five open-ended and five multiple-choice benchmarks, reveals that the proposed Image Grid Vision Language Model (IG-VLM) surpasses the existing methods in nine out of ten benchmarks., Comment: Our code is available at https://github.com/imagegridworth/IG-VLM

Class Incremental Learning (CIL) constitutes a pivotal subfield within continual learning, aimed at enabling models to progressively learn new classification tasks while retaining knowledge obtained from prior tasks. Although previous studies have predominantly focused on backward compatible approaches to mitigate catastrophic forgetting, recent investigations have introduced forward compatible methods to enhance performance on novel tasks and complement existing backward compatible methods. In this study, we introduce an effective-Rank based Feature Richness enhancement (RFR) method, designed for improving forward compatibility. Specifically, this method increases the effective rank of representations during the base session, thereby facilitating the incorporation of more informative features pertinent to unseen novel tasks. Consequently, RFR achieves dual objectives in backward and forward compatibility: minimizing feature extractor modifications and enhancing novel task performance, respectively. To validate the efficacy of our approach, we establish a theoretical connection between effective rank and the Shannon entropy of representations. Subsequently, we conduct comprehensive experiments by integrating RFR into eleven well-known CIL methods. Our results demonstrate the effectiveness of our approach in enhancing novel-task performance while mitigating catastrophic forgetting. Furthermore, our method notably improves the average incremental accuracy across all eleven cases examined.

In text-to-image personalization, a timely and crucial challenge is the tendency of generated images overfitting to the biases present in the reference images. We initiate our study with a comprehensive categorization of the biases into background, nearby-object, tied-object, substance (in style re-contextualization), and pose biases. These biases manifest in the generated images due to their entanglement into the subject embedding. This undesired embedding entanglement not only results in the reflection of biases from the reference images into the generated images but also notably diminishes the alignment of the generated images with the given generation prompt. To address this challenge, we propose SID~(Selectively Informative Description), a text description strategy that deviates from the prevalent approach of only characterizing the subject's class identification. SID is generated utilizing multimodal GPT-4 and can be seamlessly integrated into optimization-based models. We present comprehensive experimental results along with analyses of cross-attention maps, subject-alignment, non-subject-disentanglement, and text-alignment., Comment: Published at CVPR 2024

In a surge of text-to-image (T2I) models and their customization methods that generate new images of a user-provided subject, current works focus on alleviating the costs incurred by a lengthy per-subject optimization. These zero-shot customization methods encode the image of a specified subject into a visual embedding which is then utilized alongside the textual embedding for diffusion guidance. The visual embedding incorporates intrinsic information about the subject, while the textual embedding provides a new, transient context. However, the existing methods often 1) are significantly affected by the input images, eg., generating images with the same pose, and 2) exhibit deterioration in the subject's identity. We first pin down the problem and show that redundant pose information in the visual embedding interferes with the textual embedding containing the desired pose information. To address this issue, we propose orthogonal visual embedding which effectively harmonizes with the given textual embedding. We also adopt the visual-only embedding and inject the subject's clear features utilizing a self-attention swap. Our results demonstrate the effectiveness and robustness of our method, which offers highly flexible zero-shot generation while effectively maintaining the subject's identity., Comment: Project page: https://ldynx.github.io/harmony-zero-t2i/