Search

In this paper, we present Cross Language Agent -- Simultaneous Interpretation, CLASI, a high-quality and human-like Simultaneous Speech Translation (SiST) System. Inspired by professional human interpreters, we utilize a novel data-driven read-write strategy to balance the translation quality and latency. To address the challenge of translating in-domain terminologies, CLASI employs a multi-modal retrieving module to obtain relevant information to augment the translation. Supported by LLMs, our approach can generate error-tolerated translation by considering the input audio, historical context, and retrieved information. Experimental results show that our system outperforms other systems by significant margins. Aligned with professional human interpreters, we evaluate CLASI with a better human evaluation metric, valid information proportion (VIP), which measures the amount of information that can be successfully conveyed to the listeners. In the real-world scenarios, where the speeches are often disfluent, informal, and unclear, CLASI achieves VIP of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese translation directions, respectively. In contrast, state-of-the-art commercial or open-source systems only achieve 35.4% and 41.6%. On the extremely hard dataset, where other systems achieve under 13% VIP, CLASI can still achieve 70% VIP., Comment: Authors are listed in alphabetical order by last name. Demonstrations and human-annotated test sets are available at https://byteresearchcla.github.io/clasi

This paper presents a review of the NTIRE 2024 challenge on night photography rendering. The goal of the challenge was to find solutions that process raw camera images taken in nighttime conditions, and thereby produce a photo-quality output images in the standard RGB (sRGB) space. Unlike the previous year's competition, the challenge images were collected with a mobile phone and the speed of algorithms was also measured alongside the quality of their output. To evaluate the results, a sufficient number of viewers were asked to assess the visual quality of the proposed solutions, considering the subjective nature of the task. There were 2 nominations: quality and efficiency. Top 5 solutions in terms of output quality were sorted by evaluation time (see Fig. 1). The top ranking participants' solutions effectively represent the state-of-the-art in nighttime photography rendering. More results can be found at https://nightimaging.org., Comment: 10 pages, 10 figures

The emerging video LMMs (Large Multimodal Models) have achieved significant improvements on generic video understanding in the form of VQA (Visual Question Answering), where the raw videos are captured by cameras. However, a large portion of videos in real-world applications are edited videos, \textit{e.g.}, users usually cut and add effects/modifications to the raw video before publishing it on social media platforms. The edited videos usually have high view counts but they are not covered in existing benchmarks of video LMMs, \textit{i.e.}, ActivityNet-QA, or VideoChatGPT benchmark. In this paper, we leverage the edited videos on a popular short video platform, \textit{i.e.}, TikTok, and build a video VQA benchmark (named EditVid-QA) covering four typical editing categories, i.e., effect, funny, meme, and game. Funny and meme videos benchmark nuanced understanding and high-level reasoning, while effect and game evaluate the understanding capability of artificial design. Most of the open-source video LMMs perform poorly on the EditVid-QA benchmark, indicating a huge domain gap between edited short videos on social media and regular raw videos. To improve the generalization ability of LMMs, we collect a training set for the proposed benchmark based on both Panda-70M/WebVid raw videos and small-scale TikTok/CapCut edited videos, which boosts the performance on the proposed EditVid-QA benchmark, indicating the effectiveness of high-quality training data. We also identified a serious issue in the existing evaluation protocol using the GPT-3.5 judge, namely a "sorry" attack, where a sorry-style naive answer can achieve an extremely high rating from the GPT judge, e.g., over 4.3 for correctness score on VideoChatGPT evaluation protocol. To avoid the "sorry" attacks, we evaluate results with GPT-4 judge and keyword filtering. The datasets will be released for academic purposes only.

The success of Large Vision Models (LVMs) is accompanied by vast data volumes, which are prohibitively expensive in medical diagnosis.To address this, recent efforts exploit Parameter-Efficient Fine-Tuning (PEFT), which trains a small number of weights while freezing the rest.However, they typically assign trainable weights to the same positions in LVMs in a heuristic manner, regardless of task differences, making them suboptimal for professional applications like medical diagnosis.To address this, we statistically reveal the nature of sparsity and hybridity during diagnostic-targeted fine-tuning, i.e., a small portion of key weights significantly impacts performance, and these key weights are hybrid, including both task-specific and task-agnostic parts.Based on this, we propose a novel Sparsity- and Hybridity-inspired Parameter Efficient Fine-Tuning (SH-PEFT).It selects and trains a small portion of weights based on their importance, which is innovatively estimated by hybridizing both task-specific and task-agnostic strategies.Validated on six medical datasets of different modalities, we demonstrate that SH-PEFT achieves state-of-the-art performance in transferring LVMs to medical diagnosis in terms of accuracy. By tuning around 0.01% number of weights, it outperforms full model fine-tuning.Moreover, SH-PEFT also achieves comparable performance to other models deliberately optimized for specific medical tasks.Extensive experiments demonstrate the effectiveness of each design and reveal that large model transfer holds great potential in medical diagnosis.

Recent advancements in Multimodal Large Language Models (LLMs) have focused primarily on scaling by increasing text-image pair data and enhancing LLMs to improve performance on multimodal tasks. However, these scaling approaches are computationally expensive and overlook the significance of improving model capabilities from the vision side. Inspired by the successful applications of Mixture-of-Experts (MoE) in LLMs, which improves model scalability during training while keeping inference costs similar to those of smaller models, we propose CuMo. CuMo incorporates Co-upcycled Top-K sparsely-gated Mixture-of-experts blocks into both the vision encoder and the MLP connector, thereby enhancing the multimodal LLMs with minimal additional activated parameters during inference. CuMo first pre-trains the MLP blocks and then initializes each expert in the MoE block from the pre-trained MLP block during the visual instruction tuning stage. Auxiliary losses are used to ensure a balanced loading of experts. CuMo outperforms state-of-the-art multimodal LLMs across various VQA and visual-instruction-following benchmarks using models within each model size group, all while training exclusively on open-sourced datasets. The code and model weights for CuMo are open-sourced at https://github.com/SHI-Labs/CuMo.

Traditional recommender systems heavily rely on ID features, which often encounter challenges related to cold-start and generalization. Modeling pre-extracted content features can mitigate these issues, but is still a suboptimal solution due to the discrepancies between training tasks and model parameters. End-to-end training presents a promising solution for these problems, yet most of the existing works mainly focus on retrieval models, leaving the multimodal techniques under-utilized. In this paper, we propose an industrial multimodal recommendation framework named EM3: End-to-end training of Multimodal Model and ranking Model, which sufficiently utilizes multimodal information and allows personalized ranking tasks to directly train the core modules in the multimodal model to obtain more task-oriented content features, without overburdening resource consumption. First, we propose Fusion-Q-Former, which consists of transformers and a set of trainable queries, to fuse different modalities and generate fixed-length and robust multimodal embeddings. Second, in our sequential modeling for user content interest, we utilize Low-Rank Adaptation technique to alleviate the conflict between huge resource consumption and long sequence length. Third, we propose a novel Content-ID-Contrastive learning task to complement the advantages of content and ID by aligning them with each other, obtaining more task-oriented content embeddings and more generalized ID embeddings. In experiments, we implement EM3 on different ranking models in two scenario, achieving significant improvements in both offline evaluation and online A/B test, verifying the generalizability of our method. Ablation studies and visualization are also performed. Furthermore, we also conduct experiments on two public datasets to show that our proposed method outperforms the state-of-the-art methods., Comment: 9 pages, 8 figures

Holistic understanding of urban scenes based on RGB images is a challenging yet important problem. It encompasses understanding both the geometry and appearance to enable novel view synthesis, parsing semantic labels, and tracking moving objects. Despite considerable progress, existing approaches often focus on specific aspects of this task and require additional inputs such as LiDAR scans or manually annotated 3D bounding boxes. In this paper, we introduce a novel pipeline that utilizes 3D Gaussian Splatting for holistic urban scene understanding. Our main idea involves the joint optimization of geometry, appearance, semantics, and motion using a combination of static and dynamic 3D Gaussians, where moving object poses are regularized via physical constraints. Our approach offers the ability to render new viewpoints in real-time, yielding 2D and 3D semantic information with high accuracy, and reconstruct dynamic scenes, even in scenarios where 3D bounding box detection are highly noisy. Experimental results on KITTI, KITTI-360, and Virtual KITTI 2 demonstrate the effectiveness of our approach., Comment: Our project page is at https://xdimlab.github.io/hugs_website

We consider an open interacting particle system on a finite lattice. The particles perform asymmetric simple exclusion and are randomly created or destroyed at all sites, with rates that grow rapidly near the boundaries. We study the hydrodynamic limit for the particle density at the hyperbolic space-time scale and obtain the entropy solution to a boundary-driven quasilinear conservation law with a relaxation term. Different from the usual boundary conditions introduced in [Bardos, Roux, and Nedelec, (1979), Comm. Part. Diff. Equ], discontinuity (boundary layer) does not formulate at the boundaries due to the strong relaxation scheme.

We establish the incompressible limit of weakly asymmetric simple exclusion processes coupled through particle collisions. The incompressible limit depends on various parameters in the particle system and is linked to fluid dynamics equations. Our main contributions to previous results are the extension of the parameter space and the focus on local particle jumps. Our proof uses the relative entropy method. The main novelties in the proof are a Boltzmann-Gibbs principle (a replacement lemma) and a spectral gap estimate., Comment: 42 pages

The quantum dynamics of entanglement is widely revealed in photosynthetic light-harvesting complexes. Different from the previous work, we explore the properties of exciton transport and photosynthesis assisted by the quantum entanglement in two adjacent pigment molecules, which are measured by the population dynamics behaviors, the $j$-$V$ characteristics and by the output power via a photosynthetic quantum heat engine (QHE) model. A more robust exciton transport dynamic behavior is compared with those without quantum entanglement, and the photosynthetic characteristics evaluated by the output current and power were proved to be enhanced by the quantum entanglement at different ambient temperatures. These results may point toward the possibility for artificial photosynthetic nanostructures inspired by this quantum biological systems., Comment: 11 pages, 4 figures

One of the promising approaches to revealing the photosynthetic efficiency of close to one unit is to investigate the quantum regime of excitation energy transfer (EET). The majority of studies, however, have concluded that different pigment molecules contribute equally to EET, rather than differently. We investigate the roles of different site-energies in EET by evaluating the correlated fluctuations of site-energies in two adjacent pigment molecules (namely Site 1 and Site 2), and we attempt to demonstrate different site roles in EET with the j-V characteristics and power via a photosynthetic quantum heat engine (QHE) model rather than an actual photosynthetic protein. The results show that fluctuations at Site 1 (the pigment molecule absorbing solar photons) provide ascending and then descending EET. At Site 2, the EET is reduced through the use of correlated fluctuation increments (the pigment molecule acting as the charge-transfer excited state). Furthermore, when investigating the correlated fluctuations at Site 2, the different gap differences of the output terminal play a positive role in EET, but a sharply decreasing EET process is also achieved with less correlated fluctuations at Site 2 compared to those at Site 1.The findings show that different pigment molecules contribute differently to EET. The significance of this work is that it not only clarifies the roles of different pigment molecules in EET, but it also deepens our understanding of the fundamental physics of EET as it transports through the molecular chain in photosynthetic light-harvesting complexes. Furthermore, the results are appropriate to the EET in organic semiconductors, photovoltaic devices, and quantum networks, when these systems couple to the environment of photons via the vibrational motion of sites in the molecular chain., Comment: 14 pages, 7 figures

A viable strategy for enhancing photovoltaic performance in a double quantum dot (DQD) photocell is to comprehend the underlying quantum physical regime of charge transfer. This work explores the photovoltaic performance dependent spatially correlated fluctuation in a DQD photocell. A suggested DQD photocell model was used to examine the effects of spatially correlated variation on charge transfer and output photovoltaic efficiency. The charge transfer process and the process of reaching peak solar efficiency were both significantly delayed as a result of the spatially correlated fluctuation, and the anti-spatial correlation fluctuation also resulted in lower output photovoltaic efficiency. Further results revealed that some structural parameters, such as gap difference and tunneling coefficient within two dots, could suppress the delayed response, and a natural adjustment feature was demonstrated on the delayed response in this DQD photocell model. Subsequent investigation verified that the delayed response was caused by the spatial correlation fluctuation, which slowed the generative process of noise-induced coherence, which had previously been proven to improve quantum photovoltaic performance in quantum photocells. While anti-spatial correlation fluctuation and a hotter thermal ambient environment could diminish the condition for noise-induced coherence, as demonstrated by the reduced photovoltaic capabilities in this suggested DQD photocell model. As a result, we expect that regulated noise-induced coherence, via spatially correlated fluctuation, will have a major impact on photovoltaic qualities in a DQD photocell system. The discovery of its underlying physical regime of quantum fluctuation will broaden and deepen understanding of quantum features of electron transfer, as well as provide some indications concerning quantum techniques for high efficiency DQD solar cells., Comment: 16 pages, 5 figures

Large Language Models (LLMs) have succeeded remarkably in understanding long-form contents. However, exploring their capability for generating long-form contents, such as reports and articles, has been relatively unexplored and inadequately assessed by existing benchmarks. The prevalent evaluation methods, which predominantly rely on crowdsourcing, are recognized for their labor-intensive nature and lack of efficiency, whereas automated metrics, such as the ROUGE score, demonstrate discordance with human judgment criteria. In this paper, we propose ProxyQA, an innovative framework dedicated to assessing long-text generation. ProxyQA comprises in-depth human-curated meta-questions spanning various domains, each accompanied by specific proxy-questions with pre-annotated answers. LLMs are tasked to generate extensive content in response to these meta-questions, by engaging an evaluator and incorporating the generated texts as contextual background, ProxyQA assesses the generated content's quality through the evaluator's accuracy in addressing the proxy-questions. We examine multiple LLMs, emphasizing ProxyQA's demanding nature as a high-quality assessment tool. Human evaluation demonstrates that the proxy-question method is notably self-consistent and aligns closely with human evaluative standards. The dataset and leaderboard is available at \url{https://proxy-qa.com}., Comment: Accepted to ACL 2024 main conference

Conversational recommender systems (CRS) aim to recommend relevant items to users by eliciting user preference through natural language conversation. Prior work often utilizes external knowledge graphs for items' semantic information, a language model for dialogue generation, and a recommendation module for ranking relevant items. This combination of multiple components suffers from a cumbersome training process, and leads to semantic misalignment issues between dialogue generation and item recommendation. In this paper, we represent items in natural language and formulate CRS as a natural language processing task. Accordingly, we leverage the power of pre-trained language models to encode items, understand user intent via conversation, perform item recommendation through semantic matching, and generate dialogues. As a unified model, our PECRS (Parameter-Efficient CRS), can be optimized in a single stage, without relying on non-textual metadata such as a knowledge graph. Experiments on two benchmark CRS datasets, ReDial and INSPIRED, demonstrate the effectiveness of PECRS on recommendation and conversation. Our code is available at: https://github.com/Ravoxsg/efficient_unified_crs., Comment: 9 pages, 4 figures, 8 tables, EACL 2024 conference, fixed typo

Understanding model's sensitivity to its training data is crucial but can also be challenging and costly, especially during training. To simplify such issues, we present the Memory-Perturbation Equation (MPE) which relates model's sensitivity to perturbation in its training data. Derived using Bayesian principles, the MPE unifies existing sensitivity measures, generalizes them to a wide-variety of models and algorithms, and unravels useful properties regarding sensitivities. Our empirical results show that sensitivity estimates obtained during training can be used to faithfully predict generalization on unseen test data. The proposed equation is expected to be useful for future research on robust and adaptive learning., Comment: 37th Conference on Neural Information Processing Systems (NeurIPS 2023)

We consider a scalar conservation law with source in a bounded open interval $\Omega\subseteq\mathbb R$. The equation arises from the macroscopic evolution of an interacting particle system. The source term models an external effort driving the solution to a given function $\varrho$ with an intensity function $V:\Omega\to\mathbb R_+$ that grows to infinity at $\partial\Omega$. We define the entropy solution $u \in L^\infty$ and prove the uniqueness. When $V$ is integrable, $u$ satisfies the boundary conditions introduced in [F. Otto, C. R. Acad. Sci. Paris 1996], which allows the solution to attain values at $\partial\Omega$ different from the given boundary data. When the integral of $V$ blows up, $u$ satisfies an energy estimate and presents essential continuity at $\partial\Omega$ in a weak sense.

While prompt tuning approaches have achieved competitive performance with high efficiency, we observe that they invariably employ the same initialization process, wherein the soft prompt is either randomly initialized or derived from an existing embedding vocabulary. In contrast to these conventional methods, this study aims to investigate an alternative way to derive soft prompt. Our empirical studies show that the soft prompt typically exhibits a low intrinsic rank characteristic. With such observations, we propose decomposed prompt tuning, a novel approach that utilizes low-rank matrices to initialize the soft prompt. Through the low-rank reparameterization, our method significantly reduces the number of trainable parameters while maintaining effectiveness. Experimental results on the SuperGLUE benchmark in both high-resource and low-resource scenarios demonstrate the effectiveness of the proposed method., Comment: Accepted to Findings of EMNLP 2023

Scene graph generation aims to detect visual relationship triplets, (subject, predicate, object). Due to biases in data, current models tend to predict common predicates, e.g. "on" and "at", instead of informative ones, e.g. "standing on" and "looking at". This tendency results in the loss of precise information and overall performance. If a model only uses "stone on road" rather than "stone blocking road" to describe an image, it may be a grave misunderstanding. We argue that this phenomenon is caused by two imbalances: semantic space level imbalance and training sample level imbalance. For this problem, we propose DB-SGG, an effective framework based on debiasing but not the conventional distribution fitting. It integrates two components: Semantic Debiasing (SD) and Balanced Predicate Learning (BPL), for these imbalances. SD utilizes a confusion matrix and a bipartite graph to construct predicate relationships. BPL adopts a random undersampling strategy and an ambiguity removing strategy to focus on informative predicates. Benefiting from the model-agnostic process, our method can be easily applied to SGG models and outperforms Transformer by 136.3%, 119.5%, and 122.6% on mR@20 at three SGG sub-tasks on the SGG-VG dataset. Our method is further verified on another complex SGG dataset (SGG-GQA) and two downstream tasks (sentence-to-graph retrieval and image captioning)., Comment: arXiv admin note: substantial text overlap with arXiv:2108.13129

Motivated from one-dimensional rigidity results of entire solutions to Liouville equation, we consider the semilinear equation \begin{align} \label{liouvilleequationab} \Delta u=G(u) \quad \mbox{in $\mathbb{R}^n$}, \end{align}where $G>0, G'<0$ and $GG^{''}\le A(G')^2$, with $A>0$. Let $u$ be a smooth convex solution and $\sigma_k(D^2 u)$ be the $k$-th elementary symmetric polynomial with respect to $D^2u$. We prove stronger constant rank theorems in the following sense. (1) When $A\le 2$, if $\sigma_2(D^2u)$ takes a local minimum, then $D^2 u$ has constant rank $1$. (2) When $A\le \frac{n}{n-1}$, if $\sigma_n(D^2 u)$ takes a local minimum, then $\sigma_n(D^2 u)$ is always zero in the domain.

Fueled by deep learning, computer-aided diagnosis achieves huge advances. However, out of controlled lab environments, algorithms could face multiple challenges. Open set recognition (OSR), as an important one, states that categories unseen in training could appear in testing. In medical fields, it could derive from incompletely collected training datasets and the constantly emerging new or rare diseases. OSR requires an algorithm to not only correctly classify known classes, but also recognize unknown classes and forward them to experts for further diagnosis. To tackle OSR, we assume that known classes could densely occupy small parts of the embedding space and the remaining sparse regions could be recognized as unknowns. Following it, we propose Open Margin Cosine Loss (OMCL) unifying two mechanisms. The former, called Margin Loss with Adaptive Scale (MLAS), introduces angular margin for reinforcing intra-class compactness and inter-class separability, together with an adaptive scaling factor to strengthen the generalization capacity. The latter, called Open-Space Suppression (OSS), opens the classifier by recognizing sparse embedding space as unknowns using proposed feature space descriptors. Besides, since medical OSR is still a nascent field, two publicly available benchmark datasets are proposed for comparison. Extensive ablation studies and feature visualization demonstrate the effectiveness of each design. Compared with state-of-the-art methods, MLAS achieves superior performances, measured by ACC, AUROC, and OSCR.

Relation extraction (RE) aims to extract relations from sentences and documents. Existing relation extraction models typically rely on supervised machine learning. However, recent studies showed that many RE datasets are incompletely annotated. This is known as the false negative problem in which valid relations are falsely annotated as 'no_relation'. Models trained with such data inevitably make similar mistakes during the inference stage. Self-training has been proven effective in alleviating the false negative problem. However, traditional self-training is vulnerable to confirmation bias and exhibits poor performance in minority classes. To overcome this limitation, we proposed a novel class-adaptive re-sampling self-training framework. Specifically, we re-sampled the pseudo-labels for each class by precision and recall scores. Our re-sampling strategy favored the pseudo-labels of classes with high precision and low recall, which improved the overall recall without significantly compromising precision. We conducted experiments on document-level and biomedical relation extraction datasets, and the results showed that our proposed self-training framework consistently outperforms existing competitive methods on the Re-DocRED and ChemDisgene datasets when the training data are incompletely annotated. Our code is released at https://github.com/DAMO-NLP-SG/CAST., Comment: ACL 2023 Findings

We propose a novel scheme called advanced dual-chirped optical parametric amplification (DC-OPA) that employs two kinds of nonlinear crystals (BiB$_3$O$_6$ and MgO-doped lithium niobate) to overcome the bottleneck of pulse energy scalability for single-cycle mid-infrared (MIR) laser pulses. In experiments, the advanced DC-OPA scheme achieved carrier-to-envelope phase-stable MIR laser pulses for a bandwidth of over one octave (1.4-3.1 $\mu$m) with an output pulse energy of 53 mJ. The pulse duration was compressed to 8.58 fs, which corresponds to 1.05 cycles with a central wavelength of 2.44 $\mu$m and peak power of 6 TW. To our knowledge, the obtained values for the pulse energy and peak power are the highest achieved for optical parametric amplification of single-cycle MIR laser pulses. Thanks to the energy scalability of the advanced DC-OPA scheme, it is potentially applicable to multi-TW sub-cycle laser pulses., Comment: 18 pages, 6 figures

Chemists have been pursuing the general mathematical laws to explain and predict molecular properties for a long time. However, most of the traditional quantitative structure-activity relationship (QSAR) models have limited application domains, e.g., they tend to have poor generalization performance when applied to molecules with parent structures different from those of the trained molecules. This paper attempts to develop a new QSAR method that is theoretically possible to predict various properties of molecules with diverse structures. The proposed deep electron cloud-activity relationships (DECAR) and deep field-activity relationships (DFAR) methods consist of three essentials: (1) A large number of molecule entities with activity data as training objects and responses; (2) three-dimensional electron cloud density (ECD) or related field data by the accurate density functional theory methods as input descriptors; (3) a deep learning model that is sufficiently flexible and powerful to learn the large data described above. DECAR and DFAR are used to distinguish 977 sweet and 1965 non-sweet molecules (with 6-fold data augmentation) and the classification performance is demonstrated to be significantly better than the traditional least squares support vector machine (LS-SVM) models using traditional descriptors. DECAR and DFAR would provide a possible way to establish a widely applicable, cumulative, and shareable artificial intelligence-driven QSAR system. They are likely to promote the development of an interactive platform to collect and share the accurate ECD and field data of millions of molecules with annotated activities. With enough input data, we envision the appearance of several deep networks trained for various molecular activities. Finally, we could anticipate a single DECAR or DFAR network to learn and infer various properties of interest for chemical molecules., Comment: 21 pages, 4 figures, 1 table