Search

The success of visual instruction tuning has accelerated the development of large language and vision models (LLVMs). Following the scaling laws of instruction-tuned large language models (LLMs), LLVMs either have further increased their sizes, reaching 26B, 34B, and even 80B parameters. While this increase in model size has yielded significant performance gains, it demands substantially more hardware resources for both training and inference. Consequently, there naturally exists a strong need for efficient LLVMs that achieve the performance of larger models while being smaller in size. To achieve this need, we present a new efficient LLVM family with model sizes of 0.5B, 1.8B, 3.8B, and 7B parameters, Phantom, which significantly enhances learning capabilities within limited structures. By temporarily increasing the latent hidden dimension during multi-head self-attention (MHSA), we make LLVMs prepare to look and understand much more vision-language knowledge on the latent, without substantially increasing physical model sizes. To maximize its advantage, we introduce Phantom Optimization (PO) using both autoregressive supervised fine-tuning (SFT) and direct preference optimization (DPO)-like concept, which effectively follows correct answers while eliminating incorrect and ambiguous ones. Phantom outperforms numerous larger open- and closed-source LLVMs, positioning itself as a leading solution in the landscape of efficient LLVMs., Comment: Code is available in https://github.com/ByungKwanLee/Phantom

Lip reading aims to predict spoken language by analyzing lip movements. Despite advancements in lip reading technologies, performance degrades when models are applied to unseen speakers due to their sensitivity to variations in visual information such as lip appearances. To address this challenge, speaker adaptive lip reading technologies have advanced by focusing on effectively adapting a lip reading model to target speakers in the visual modality. The effectiveness of adapting language information, such as vocabulary choice, of the target speaker has not been explored in the previous works. Moreover, existing datasets for speaker adaptation have limited vocabulary size and pose variations, limiting the validation of previous speaker-adaptive methods in real-world scenarios. To address these issues, we propose a novel speaker-adaptive lip reading method that adapts a pre-trained model to target speakers at both vision and language levels. Specifically, we integrate prompt tuning and the LoRA approach, applying them to a pre-trained lip reading model to effectively adapt the model to target speakers. In addition, to validate its effectiveness in real-world scenarios, we introduce a new dataset, VoxLRS-SA, derived from VoxCeleb2 and LRS3. It contains a vocabulary of approximately 100K words, offers diverse pose variations, and enables the validation of adaptation methods in wild, sentence-level lip reading for the first time. Through various experiments, we demonstrate that the existing speaker-adaptive method also improves performance in the wild at the sentence level. Moreover, with the proposed adaptation method, we show that the proposed method achieves larger improvements when applied to the target speaker, compared to the previous works., Comment: Code available: https://github.com/JeongHun0716/Personalized-Lip-Reading

Large language and vision models (LLVMs) have been driven by the generalization power of large language models (LLMs) and the advent of visual instruction tuning. Along with scaling them up directly, these models enable LLVMs to showcase powerful vision language (VL) performances by covering diverse tasks via natural language instructions. However, existing open-source LLVMs that perform comparably to closed-source LLVMs such as GPT-4V are often considered too large (e.g., 26B, 34B, and 110B parameters), having a larger number of layers. These large models demand costly, high-end resources for both training and inference. To address this issue, we present a new efficient LLVM family with 1.8B, 3.8B, and 7B LLM model sizes, Traversal of Layers (TroL), which enables the reuse of layers in a token-wise manner. This layer traversing technique simulates the effect of looking back and retracing the answering stream while increasing the number of forward propagation layers without physically adding more layers. We demonstrate that TroL employs a simple layer traversing approach yet efficiently outperforms the open-source LLVMs with larger model sizes and rivals the performances of the closed-source LLVMs with substantial sizes., Comment: EMNLP 2024. Code is available in https://github.com/ByungKwanLee/TroL

In this paper, we introduce a novel Face-to-Face spoken dialogue model. It processes audio-visual speech from user input and generates audio-visual speech as the response, marking the initial step towards creating an avatar chatbot system without relying on intermediate text. To this end, we newly introduce MultiDialog, the first large-scale multimodal (i.e., audio and visual) spoken dialogue corpus containing 340 hours of approximately 9,000 dialogues, recorded based on the open domain dialogue dataset, TopicalChat. The MultiDialog contains parallel audio-visual recordings of conversation partners acting according to the given script with emotion annotations, which we expect to open up research opportunities in multimodal synthesis. Our Face-to-Face spoken dialogue model incorporates a textually pretrained large language model and adapts it into the audio-visual spoken dialogue domain by incorporating speech-text joint pretraining. Through extensive experiments, we validate the effectiveness of our model in facilitating a face-to-face conversation. Demo and data are available at https://multidialog.github.io and https://huggingface.co/datasets/IVLLab/MultiDialog, respectively., Comment: Accepted to ACL 2024 (Oral)

The rapid development of large language and vision models (LLVMs) has been driven by advances in visual instruction tuning. Recently, open-source LLVMs have curated high-quality visual instruction tuning datasets and utilized additional vision encoders or multiple computer vision models in order to narrow the performance gap with powerful closed-source LLVMs. These advancements are attributed to multifaceted information required for diverse capabilities, including fundamental image understanding, real-world knowledge about common-sense and non-object concepts (e.g., charts, diagrams, symbols, signs, and math problems), and step-by-step procedures for solving complex questions. Drawing from the multifaceted information, we present a new efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages multifaceted rationale to enhance understanding and answering capabilities. To embed lengthy rationales containing abundant information, we employ the Mamba architecture, capable of processing sequential data with linear time complexity. We introduce a new concept of traversal of rationale that facilitates efficient embedding of rationale. Subsequently, the backbone multimodal language model (MLM) is trained to generate answers with the aid of rationale. Through these steps, Meteor achieves significant improvements in vision language performances across multiple evaluation benchmarks requiring diverse capabilities, without scaling up the model size or employing additional vision encoders and computer vision models., Comment: Code is available in https://github.com/ByungKwanLee/Meteor

The rise of large language models (LLMs) and instruction tuning has led to the current trend of instruction-tuned large language and vision models (LLVMs). This trend involves either meticulously curating numerous instruction tuning datasets tailored to specific objectives or enlarging LLVMs to manage vast amounts of vision language (VL) data. However, current LLVMs have disregarded the detailed and comprehensive real-world scene understanding available from specialized computer vision (CV) models in visual perception tasks such as segmentation, detection, scene graph generation (SGG), and optical character recognition (OCR). Instead, the existing LLVMs rely mainly on the large capacity and emergent capabilities of their LLM backbones. Therefore, we present a new LLVM, Mixture of All Intelligence (MoAI), which leverages auxiliary visual information obtained from the outputs of external segmentation, detection, SGG, and OCR models. MoAI operates through two newly introduced modules: MoAI-Compressor and MoAI-Mixer. After verbalizing the outputs of the external CV models, the MoAI-Compressor aligns and condenses them to efficiently use relevant auxiliary visual information for VL tasks. MoAI-Mixer then blends three types of intelligence (1) visual features, (2) auxiliary features from the external CV models, and (3) language features by utilizing the concept of Mixture of Experts. Through this integration, MoAI significantly outperforms both open-source and closed-source LLVMs in numerous zero-shot VL tasks, particularly those related to real-world scene understanding such as object existence, positions, relations, and OCR without enlarging the model size or curating extra visual instruction tuning datasets., Comment: ECCV 2024. Code available: https://github.com/ByungKwanLee/MoAI

Providing emotional support through dialogue systems is becoming increasingly important in today's world, as it can support both mental health and social interactions in many conversation scenarios. Previous works have shown that using persona is effective for generating empathetic and supportive responses. They have often relied on pre-provided persona rather than inferring them during conversations. However, it is not always possible to obtain a user persona before the conversation begins. To address this challenge, we propose PESS (Persona Extraction through Semantic Similarity), a novel framework that can automatically infer informative and consistent persona from dialogues. We devise completeness loss and consistency loss based on semantic similarity scores. The completeness loss encourages the model to generate missing persona information, and the consistency loss guides the model to distinguish between consistent and inconsistent persona. Our experimental results demonstrate that high-quality persona information inferred by PESS is effective in generating emotionally supportive responses., Comment: Accepted by ICASSP2024

The remarkable success of Large Language Models (LLMs) and instruction tuning drives the evolution of Vision Language Models (VLMs) towards a versatile general-purpose model. Yet, it remains unexplored whether current VLMs genuinely possess quality object-level image understanding capabilities determined from 'what objects are in the image?' or 'which object corresponds to a specified bounding box?'. Our findings reveal that the image understanding capabilities of current VLMs are strongly correlated with their zero-shot performance on vision language (VL) tasks. This suggests that prioritizing basic image understanding is crucial for VLMs to excel at VL tasks. To enhance object-level image understanding, we propose Crayon Large Language and Vision mOdel (CoLLaVO), which incorporates instruction tuning with Crayon Prompt as a new visual prompt tuning scheme based on panoptic color maps. Furthermore, we present a learning strategy of Dual QLoRA to preserve object-level image understanding without forgetting it during visual instruction tuning, thereby achieving a significant leap in numerous VL benchmarks in a zero-shot setting., Comment: ACL 2024 Findings. Code available: https://github.com/ByungKwanLee/CoLLaVO

Elemental phosphorus exhibits fascinating structural varieties and versatile properties. The unique nature of phosphorus bonds can lead to the formation of extremely complex structures, and detailed structural information on some phosphorus polymorphs is yet to be investigated. In this study, we investigated an unidentified crystalline phase of phosphorus, type-II red phosphorus (RP), by combining state-of-the-art structural characterization techniques. Electron diffraction tomography, atomic-resolution scanning transmission electron microscopy (STEM), powder X-ray diffraction, and Raman spectroscopy were concurrently used to elucidate the hidden structural motifs and their packing in type-II RP. Electron diffraction tomography, performed using individual crystalline nanowires, was used to identify a triclinic unit cell with volume of 5330 {\AA}^3, the largest unit cell for elemental phosphorus crystals up to now, which contains approximately 250 phosphorus atoms. Atomic-resolution STEM imaging, which was performed along different crystal-zone axes, confirmed that the twisted wavy tubular motif is the basic building block of type-II RP. Our study discovered and presented a new variation of building blocks in phosphorus, and it provides insights to clarify the complexities observed in phosphorus as well as other relevant systems., Comment: 16 pages, 5 figures, 1 table

Forced alignment refers to a technology that time-aligns a given transcription with a corresponding speech. However, as the forced alignment technologies have developed using speech audio, they might fail in alignment when the input speech audio is noise-corrupted or is not accessible. We focus on that there is another component that the speech can be inferred from, the speech video (i.e., talking face video). Since the drawbacks of audio-based forced alignment can be complemented using the visual information when the audio signal is under poor condition, we try to develop a novel video-based forced alignment method. However, different from audio forced alignment, it is challenging to develop a reliable visual forced alignment technology for the following two reasons: 1) Visual Speech Recognition (VSR) has a much lower performance compared to audio-based Automatic Speech Recognition (ASR), and 2) the translation from text to video is not reliable, so the method typically used for building audio forced alignment cannot be utilized in developing visual forced alignment. In order to alleviate these challenges, in this paper, we propose a new method that is appropriate for visual forced alignment, namely Deep Visual Forced Alignment (DVFA). The proposed DVFA can align the input transcription (i.e., sentence) with the talking face video without accessing the speech audio. Moreover, by augmenting the alignment task with anomaly case detection, DVFA can detect mismatches between the input transcription and the input video while performing the alignment. Therefore, we can robustly align the text with the talking face video even if there exist error words in the text. Through extensive experiments, we show the effectiveness of the proposed DVFA not only in the alignment task but also in interpreting the outputs of VSR models., Comment: Accepted in AAAI2023

Understanding melting in metals is a hot topic of present research. This may be accomplished by pumping the system with infrared (IR) laser radiation, and probing it with hard X-rays produced by an X-ray Free-Electron Laser (XFEL). In this work we studied nanostructured polycrystalline thin films of platinum that were illuminated by IR radiation of increasing fluences. We characterized the structural response as well as the nucleation and propagation of the liquid phase as a function of time delay between the IR pump and X-ray probe. We observed partial melting of the samples for IR fluences higher than 200 mJ$\cdot$cm$^{-2}$. To fit the contribution of the liquid phase to the scattering pattern in platinum we applied a model of liquid metal. The two-temperature model simulations were performed to understand the solid-state fraction of the sample heating process as a function of time delay and fluence.

There has been a longstanding expectation that the optical resolution embodiment of photoacoustic tomography could have a substantial impact on gastrointestinal endoscopy by enabling microscopic visualization of the vasculature based on the endogenous contrast mechanism. Although multiple studies have demonstrated the in vivo imaging capability of a developed imaging device over the last decade, the implementation of such an endoscopic system that can be applied immediately when necessary via the instrument channel of a video endoscope has been a challenge. In this study, we developed a 3.38 mm diameter catheter based, integrated optical resolution photoacoustic and ultrasonic miniprobe system and successfully demonstrated its intrainstrument channel workability for the standard 3.7 mm diameter instrument channel of a clinical video endoscope based on a swine model. Through the instrument channel, we acquired the first in vivo photoacoustic and ultrasonic dual mode endoscopic images from the esophagogastric junction, which is one of the most frequently cited anatomical sites in the gastrointestinal tract in relation to Barretts esophagus. Moreover, from a rat colorectum in vivo imaging experiment, we visualized hierarchically developed mesh like capillary networks with a hole size as small as 50 microns, which suggests the potential level of image details that could be photoacoustically provided in clinical settings in the future., Comment: 28 pages

Fair division is a significant, long-standing problem and is closely related to social and economic justice. The conventional division methods such as cut-and-choose are hardly applicable to realworld problems because of their complexity and unrealistic assumptions about human behaviors. Here we propose a fair division method from a completely different perspective, using the Boltzmann distribution. The Boltzmann distribution adopted from the physical sciences gives the most probable and unbiased distribution derived from a goods-centric, rather than a player-centric, division process. The mathematical model of the Boltzmann fair division was developed for both homogeneous and heterogeneous division problems, and the players' key factors (contributions, needs, and preferences) could be successfully integrated. We show that the Boltzmann fair division is a well-balanced division method maximizing the players' total utility, and it could be easily finetuned and applicable to complex real-world problems such as income/wealth redistribution or international negotiations on fighting climate change., Comment: 36 pages, 6 figures