Search

Large Language Model-based Dense Retrieval (LLM-DR) optimizes over numerous heterogeneous fine-tuning collections from different domains. However, the discussion about its training data distribution is still minimal. Previous studies rely on empirically assigned dataset choices or sampling ratios, which inevitably leads to sub-optimal retrieval performances. In this paper, we propose a new task-level Distributionally Robust Optimization (tDRO) algorithm for LLM-DR fine-tuning, targeted at improving the universal domain generalization ability by end-to-end reweighting the data distribution of each task. The tDRO parameterizes the domain weights and updates them with scaled domain gradients. The optimized weights are then transferred to the LLM-DR fine-tuning to train more robust retrievers. Experiments show optimal improvements in large-scale retrieval benchmarks and reduce up to 30% dataset usage after applying our optimization algorithm with a series of different-sized LLM-DR models.

The coldness of the universe is a thermodynamic resource that has largely remained untapped for renewable energy generation. Recently, a growing interest in this area has led to a number of studies with the aim to realize the potential of tapping this vast resource for energy generation. While the theoretical calculation based on thermodynamic principles places an upper limit in the power density at the level of 6000 mW/m$^2$, most experimental demonstrations so far result in much lower power density at the level of tens of mW/m$^2$. Here we demonstrate, through design optimization involving the tailoring of the thermal radiation spectrum, the minimization of parasitic heat leakage, and the maximum conversion of heat to electricity, an energy generation system harvesting electricity from the thermal radiation of the ambient heat to the cold universe that achieves a sustained power density of 350 mW/m$^2$. We further demonstrate a power density at the 1000 mW/m$^2$ level using an additional heat source or heat storage that provides access to heat at a temperature above ambient. Our work here shows that the coldness of the universe can be harvested to generate renewable energy at the power density level that approaches the established bound.

Objects in aerial images are typically embedded in complex backgrounds and exhibit arbitrary orientations. When employing oriented bounding boxes (OBB) to represent arbitrary oriented objects, the periodicity of angles could lead to discontinuities in label regression values at the boundaries, inducing abrupt fluctuations in the loss function. To address this problem, an OBB representation based on the complex plane is introduced in the oriented detection framework, and a trigonometric loss function is proposed. Moreover, leveraging prior knowledge of complex background environments and significant differences in large objects in aerial images, a conformer RPN head is constructed to predict angle information. The proposed loss function and conformer RPN head jointly generate high-quality oriented proposals. A category-aware dynamic label assignment based on predicted category feedback is proposed to address the limitations of solely relying on IoU for proposal label assignment. This method makes negative sample selection more representative, ensuring consistency between classification and regression features. Experiments were conducted on four realistic oriented detection datasets, and the results demonstrate superior performance in oriented object detection with minimal parameter tuning and time costs. Specifically, mean average precision (mAP) scores of 82.02%, 71.99%, 69.87%, and 98.77% were achieved on the DOTA-v1.0, DOTA-v1.5, DIOR-R, and HRSC2016 datasets, respectively.

In recent years, Large Language Models (LLMs) have made significant strides towards Artificial General Intelligence. However, training these models from scratch requires substantial computational resources and vast amounts of text data. In this paper, we explore an alternative approach to constructing an LLM for a new language by continually pretraining (CPT) from existing pretrained LLMs, instead of using randomly initialized parameters. Based on parallel experiments on 40 model sizes ranging from 40M to 5B parameters, we find that 1) CPT converges faster and saves significant resources in a scalable manner; 2) CPT adheres to an extended scaling law derived from Hoffmann et al. (2022) with a joint data-parameter scaling term; 3) The compute-optimal data-parameter allocation for CPT markedly differs based on our estimated scaling factors; 4) The effectiveness of transfer at scale is influenced by training duration and linguistic properties, while robust to data replaying, a method that effectively mitigates catastrophic forgetting in CPT. We hope our findings provide deeper insights into the transferability of LLMs at scale for the research community., Comment: 8 pages

Malicious social bots achieve their malicious purposes by spreading misinformation and inciting social public opinion, seriously endangering social security, making their detection a critical concern. Recently, graph-based bot detection methods have achieved state-of-the-art (SOTA) performance. However, our research finds many isolated and poorly linked nodes in social networks, as shown in Fig.1, which graph-based methods cannot effectively detect. To address this problem, our research focuses on effectively utilizing node semantics and network structure to jointly detect sparsely linked nodes. Given the excellent performance of language models (LMs) in natural language understanding (NLU), we propose a novel social bot detection framework LGB, which consists of two main components: language model (LM) and graph neural network (GNN). Specifically, the social account information is first extracted into unified user textual sequences, which is then used to perform supervised fine-tuning (SFT) of the language model to improve its ability to understand social account semantics. Next, the semantically enriched node representation is fed into the pre-trained GNN to further enhance the node representation by aggregating information from neighbors. Finally, LGB fuses the information from both modalities to improve the detection performance of sparsely linked nodes. Extensive experiments on two real-world datasets demonstrate that LGB consistently outperforms state-of-the-art baseline models by up to 10.95%. LGB is already online: https://botdetection.aminer.cn/robotmain.

Multimodal Sentiment Analysis (MSA) endeavors to understand human sentiment by leveraging language, visual, and acoustic modalities. Despite the remarkable performance exhibited by previous MSA approaches, the presence of inherent multimodal heterogeneities poses a challenge, with the contribution of different modalities varying considerably. Past research predominantly focused on improving representation learning techniques and feature fusion strategies. However, many of these efforts overlooked the variation in semantic richness among different modalities, treating each modality uniformly. This approach may lead to underestimating the significance of strong modalities while overemphasizing the importance of weak ones. Motivated by these insights, we introduce a Text-oriented Cross-Attention Network (TCAN), emphasizing the predominant role of the text modality in MSA. Specifically, for each multimodal sample, by taking unaligned sequences of the three modalities as inputs, we initially allocate the extracted unimodal features into a visual-text and an acoustic-text pair. Subsequently, we implement self-attention on the text modality and apply text-queried cross-attention to the visual and acoustic modalities. To mitigate the influence of noise signals and redundant features, we incorporate a gated control mechanism into the framework. Additionally, we introduce unimodal joint learning to gain a deeper understanding of homogeneous emotional tendencies across diverse modalities through backpropagation. Experimental results demonstrate that TCAN consistently outperforms state-of-the-art MSA methods on two datasets (CMU-MOSI and CMU-MOSEI).

Building embodied agents on integrating Large Language Models (LLMs) and Reinforcement Learning (RL) have revolutionized human-AI interaction: researchers can now leverage language instructions to plan decision-making for open-ended tasks. However, existing research faces challenges in meeting the requirement of open-endedness. They typically either train LLM/RL models to adapt to a fixed counterpart, limiting exploration of novel skills and hindering the efficacy of human-AI interaction. To this end, we present OpenPAL, a co-training framework comprising two stages: (1) fine-tuning a pre-trained LLM to translate human instructions into goals for planning, and goal-conditioned training a policy for decision-making; (2) co-training to align the LLM and policy, achieving instruction open-endedness. We conducted experiments using Contra, an open-ended FPS game, demonstrating that an agent trained with OpenPAL not only comprehends arbitrary instructions but also exhibits efficient execution. These results suggest that OpenPAL holds the potential to construct open-ended embodied agents in practical scenarios.

We have established and released a new stellar index library of the Ca II Triplet, which serves as an indicator for characterizing the chromospheric activity of stars. The library is based on data from the Large Sky Area Multi-Object Fiber Spectroscopic Telescope (LAMOST) Low-Resolution Spectroscopic Survey (LRS) Data Release 9 (DR9). To better reflect the chromospheric activity of stars, we have defined new indices $R$ and $R^{+}$. The library includes measurements of $R$ and $R^{+}$ for each Ca II infrared triplet (IRT) from 699,348 spectra of 562,863 F, G and K-type solar-like stars with Signal-to-Noise Ratio (SNR) higher than 100, as well as the stellar atmospheric parameters and basic information inherited from the LAMOST LRS Catalog. We compared the differences between the 3 individual index of the Ca II Triplet and also conducted a comparative analysis of $R^{+}_{\lambda8542}$ to the Ca II H&K $S$ and $R^+_{HK}$ index database. We find the fraction of low active stars decreases with $T_{eff}$ and the fraction of high active first decrease with decreasing temperature and turn to increase with decreasing temperature at 5800K. We also find a significant fraction of stars that show high activity index in both Ca II H&K and IRT are binaries with low activity, some of them could be discriminated in Ca II H&K $S$ index and $R^{+}_{\lambda8542}$ space. This newly stellar library serves as a valuable resource for studying chromospheric activity in stars and can be used to improve our comprehension of stellar magnetic activity and other astrophysical phenomena., Comment: 15 pages, 13 figures, 5 table, submitted to ApJS

Prevalent supervised learning methods in natural language processing (NLP) are notoriously data-hungry, which demand large amounts of high-quality annotated data. In practice, acquiring such data is a costly endeavor. Recently, the superior few-shot performance of large language models (LLMs) has propelled the development of dataset generation, where the training data are solely synthesized from LLMs. However, such an approach usually suffers from low-quality issues, and requires orders of magnitude more labeled data to achieve satisfactory performance. To fully exploit the potential of LLMs and make use of massive unlabeled data, we propose LLMaAA, which takes LLMs as annotators and puts them into an active learning loop to determine what to annotate efficiently. To learn robustly with pseudo labels, we optimize both the annotation and training processes: (1) we draw k-NN examples from a small demonstration pool as in-context examples, and (2) we adopt the example reweighting technique to assign training samples with learnable weights. Compared with previous approaches, LLMaAA features both efficiency and reliability. We conduct experiments and analysis on two classic NLP tasks, named entity recognition and relation extraction. With LLMaAA, task-specific models trained from LLM-generated labels can outperform the teacher within only hundreds of annotated examples, which is much more cost-effective than other baselines., Comment: Findings of EMNLP 2023 camera ready

In the framework of nonrelativistic QCD (NRQCD) factorization, we compute both the polarized and the unpolarized decay widths for the processes $\eta_b(\chi_{bJ})\to J/\psi J/\psi$, accurate up to next-to-next-to-leading-order (NNLO) in $\alpha_s$. For the first time, we confirm that the NRQCD factorization does hold at NNLO for the process involving triple quarkonia. We find the radiative corrections are considerable. In particular for $\chi_{b2}$, both $\mathcal{O}(\alpha_s)$ and $\mathcal{O}(\alpha_s^2)$ corrections are sizable and negative, and can significantly reduce the leading order prediction. At NNLO, the branching fractions are $8.2\times 10^{-7}$, $6.2\times 10^{-6}$, $7.2\times 10^{-7}$ and $2.7\times 10^{-6}$ for $\eta_b$, $\chi_{b0}$, $\chi_{b1}$ and $\chi_{b2}$ decay, respectively. Our theoretical predictions are consistent with the upper limits measured by the {\tt Belle} Collaboration. Moreover, we investigate the dependence of the theoretical predictions on the ratio of the charm quark mass and the bottom quark mass. By fixing $m_b$ and varying $m_c$ from $1.25$ to $1.9$ GeV, we find the branching fraction can change a factor of $2$, $3$, and $6$ for $\eta_b$, $\chi_{b0}$, and $\chi_{b1}$, respectively. In the phenomenological analysis, with the integrate luminosity $\mathcal{L}=100\,{\rm fb}^{-1}$, we expect about $(5-10)\times 10^3$ $\eta_b(\chi_{bJ})\to J/\psi J/\psi\to \ell \bar{\ell}\ell \bar{\ell}$ events produced at the {\tt LHC}, thus it might be hopeful to search for these processes. On the other hand, there are less than $100$ $\eta_b(\chi_{bJ})\to J/\psi J/\psi$ signal events at the B factory, so it seems the experimental measurements on these channels are quite challenging based on current dataset., Comment: 17 pages, 3 figures, 4 tables; The affiliation and acknowledgement of the first author are updated

Incorporating multiple knowledge sources is proven to be beneficial for answering complex factoid questions. To utilize multiple knowledge bases (KB), previous works merge all KBs into a single graph via entity alignment and reduce the problem to question-answering (QA) over the fused KB. In reality, various link relations between KBs might be adopted in QA over multi-KBs. In addition to the identity between the alignable entities (i.e. full link), unalignable entities expressing the different aspects or types of an abstract concept may also be treated identical in a question (i.e. partial link). Hence, the KB fusion in prior works fails to represent all types of links, restricting their ability to comprehend multi-KBs for QA. In this work, we formulate the novel Multi-KB-QA task that leverages the full and partial links among multiple KBs to derive correct answers, a benchmark with diversified link and query types is also constructed to efficiently evaluate Multi-KB-QA performance. Finally, we propose a method for Multi-KB-QA that encodes all link relations in the KB embedding to score and rank candidate answers. Experiments show that our method markedly surpasses conventional KB-QA systems in Multi-KB-QA, justifying the necessity of devising this task.

We have calculated the three-dimensional potential energy surface (PES) for the fission of compound nucleus $^{236}$U using the covariant density functional theory with constraints on the axial quadrupole and octupole deformations $(\beta_2, \beta_3)$ as well as the nucleon number in the neck $q_N$. By considering the additonal degree of freedom $q_N$, coexistence of the elongated and compact fission modes is predicted for $0.9\lesssim \beta_3 \lesssim 1.3$. Remarkably, the PES becomes very shallow across a large range of quadrupole and octupole deformations for small $q_N$, and consequently, the scission line in $(\beta_2, \beta_3)$ plane will extend to a shallow band, which leads to a fluctuation for the estimated total kinetic energies by several to ten MeV and for the fragment masses by several to about ten nucleons.

The pervasive uncertainty and dynamic nature of real-world environments present significant challenges for the widespread implementation of machine-driven Intelligent Decision-Making (IDM) systems. Consequently, IDM should possess the ability to continuously acquire new skills and effectively generalize across a broad range of applications. The advancement of Artificial General Intelligence (AGI) that transcends task and application boundaries is critical for enhancing IDM. Recent studies have extensively investigated the Transformer neural architecture as a foundational model for various tasks, including computer vision, natural language processing, and reinforcement learning. We propose that a Foundation Decision Model (FDM) can be developed by formulating diverse decision-making tasks as sequence decoding tasks using the Transformer architecture, offering a promising solution for expanding IDM applications in complex real-world situations. In this paper, we discuss the efficiency and generalization improvements offered by a foundation decision model for IDM and explore its potential applications in multi-agent game AI, production scheduling, and robotics tasks. Lastly, we present a case study demonstrating our FDM implementation, DigitalBrain (DB1) with 1.3 billion parameters, achieving human-level performance in 870 tasks, such as text generation, image captioning, video game playing, robotic control, and traveling salesman problems. As a foundation decision model, DB1 represents an initial step toward more autonomous and efficient real-world IDM applications., Comment: 26 pages, 4 figures

Convergence rates of block iterations for solving eigenvalue problems typically measure errors of Ritz values approximating eigenvalues. The errors of the Ritz values are commonly bounded in terms of principal angles between the initial or iterative subspace and the invariant subspace associated with the target eigenvalues. Such bounds thus cannot be applied repeatedly as needed for restarted block eigensolvers, since the left- and right-hand sides of the bounds use different terms. They must be combined with additional bounds which could cause an overestimation. Alternative repeatable bounds that are angle-free and depend only on the errors of the Ritz values have been pioneered for Hermitian eigenvalue problems in doi:10.1515/rnam.1987.2.5.371 but only for a single extreme Ritz value. We extend this result to all Ritz values and achieve robustness for clustered eigenvalues by utilizing nonconsecutive eigenvalues. Our new bounds cover the restarted block Lanczos method and its modifications with shift-and-invert and deflation, and are numerically advantageous., Comment: 24 pages, 4 figures

The functions of complex networks are usually determined by a small set of vital nodes. Finding the best set of vital nodes (eigenshield nodes) is critical to the network's robustness against rumor spreading and cascading failures, which makes it one of the fundamental problems in network science. The problem is challenging as it requires to maximize the influence of nodes in the set while simultaneously minimizing the redundancies between the set's nodes. However, the redundancy mechanism is rarely investigated by previous studies. Here we introduce the matrix perturbation framework to find a small ``eigenshield" set of nodes that, when removed, lead to the largest drop in the network's spectral radius. We show that finding the ``eigenshield" nodes can be translated into the optimization of an objective function that simultaneously accounts for the individual influence of each node and redundancy between different nodes. We analytically quantify the influence redundancy that explains why an important node might play an insignificant role in the ``eigenshield" node set. Extensive experiments under diverse influence maximization problems, ranging from network dismantling to spreading maximization, demonstrate that the eigenshield detection tends to significantly outperforms state-of-the-art methods across most problems. Our findings shed light on the mechanisms that may lie at the core of the function of vital nodes in complex network., Comment: 24 pages, 16 figures

Discriminative pre-trained language models (PrLMs) can be generalized as denoising auto-encoders that work with two procedures, ennoising and denoising. First, an ennoising process corrupts texts with arbitrary noising functions to construct training instances. Then, a denoising language model is trained to restore the corrupted tokens. Existing studies have made progress by optimizing independent strategies of either ennoising or denosing. They treat training instances equally throughout the training process, with little attention on the individual contribution of those instances. To model explicit signals of instance contribution, this work proposes to estimate the complexity of restoring the original sentences from corrupted ones in language model pre-training. The estimations involve the corruption degree in the ennoising data construction process and the prediction confidence in the denoising counterpart. Experimental results on natural language understanding and reading comprehension benchmarks show that our approach improves pre-training efficiency, effectiveness, and robustness. Code is publicly available at https://github.com/cooelf/InstanceReg, Comment: Accepted to EMNLP 2022

Panorama synthesis endeavors to craft captivating 360-degree visual landscapes, immersing users in the heart of virtual worlds. Nevertheless, contemporary panoramic synthesis techniques grapple with the challenge of semantically guiding the content generation process. Although recent breakthroughs in visual synthesis have unlocked the potential for semantic control in 2D flat images, a direct application of these methods to panorama synthesis yields distorted content. In this study, we unveil an innovative framework for generating high-resolution panoramas, adeptly addressing the issues of spherical distortion and edge discontinuity through sophisticated spherical modeling. Our pioneering approach empowers users with semantic control, harnessing both image and text inputs, while concurrently streamlining the generation of high-resolution panoramas using parallel decoding. We rigorously evaluate our methodology on a diverse array of indoor and outdoor datasets, establishing its superiority over recent related work, in terms of both quantitative and qualitative performance metrics. Our research elevates the controllability, efficiency, and fidelity of panorama synthesis to new levels., Comment: AAAI 2024 main conference

In this paper, we evaluate, up to QCD next-to-next-to-next-to-leading order, the $B_c^*$ decay constant, which, within the nonrelativistic QCD (NRQCD) framework, is factorized as the product of the short-distance coefficient (SDC) and the long-distance matrix element. For the first time, the renormalization constant and the anomalous dimension for the NRQCD vector current composed of $c\bar{b}$, which are functions of the charm quark mass $m_c$, the bottom quark mass $m_b$ and the factorization scale $\mu_\Lambda$, are obtained analytically at $\mathcal{O}(\alpha_s^2)$ and $\mathcal{O}(\alpha_s^3)$. The SDC is calculated up to $\mathcal{O}(\alpha_s^3)$ in perturbative expansion. Explicitly, the $\mathcal{O}(\alpha_s^2)$ correction to the SDC is analytically calculated in terms of logarithmic and polylogarithmic functions of $r\equiv m_b/m_c$, and the $\mathcal{O}(\alpha_s^3)$ correction to the SDC is numerically calculated at a series of values of $r$, ranging from $2.1$ to $4.0$. Surprisingly, we find that the nontrivial part of the SDC at $\mathcal{O}(\alpha_s^3)$ can be well estimated by a linear function of $r$. In addition, We find that the $\mathcal{O}(\alpha_s^2)$ and $\mathcal{O}(\alpha_s^3)$ corrections to the decay constant and decay width are considerable and very significant, which indicates a very poor convergence for the perturbative expansion., Comment: 15 pages, 3 figures, 2 tables, a minor typo corrected

Transformer-based pre-trained models have gained much advance in recent years, becoming one of the most important backbones in natural language processing. Recent work shows that the attention mechanism inside Transformer may not be necessary, both convolutional neural networks and multi-layer perceptron based models have also been investigated as Transformer alternatives. In this paper, we consider a graph recurrent network for language model pre-training, which builds a graph structure for each sequence with local token-level communications, together with a sentence-level representation decoupled from other tokens. The original model performs well in domain-specific text classification under supervised training, however, its potential in learning transfer knowledge by self-supervised way has not been fully exploited. We fill this gap by optimizing the architecture and verifying its effectiveness in more general language understanding tasks, for both English and Chinese languages. As for model efficiency, instead of the quadratic complexity in Transformer-based models, our model has linear complexity and performs more efficiently during inference. Moreover, we find that our model can generate more diverse outputs with less contextualized feature redundancy than existing attention-based models., Comment: NeurIPS Efficient Natural Language and Speech Processing (ENLSP) Workshop 2022

Gradient-type iterative methods for solving Hermitian eigenvalue problems can be accelerated by using preconditioning and deflation techniques. A preconditioned steepest descent iteration with implicit deflation (PSD-id) is one of such methods. The convergence behavior of the PSD-id is recently investigated based on the pioneering work of Samokish on the preconditioned steepest descent method (PSD). The resulting non-asymptotic estimates indicate a superlinear convergence of the PSD-id under strong assumptions on the initial guess. The present paper utilizes an alternative convergence analysis of the PSD by Neymeyr under much weaker assumptions. We embed Neymeyr's approach into the analysis of the PSD-id using a restricted formulation of the PSD-id. More importantly, we extend the new convergence analysis of the PSD-id to a practically preferred block version of the PSD-id, or BPSD-id, and show the cluster robustness of the BPSD-id. Numerical examples are provided to validate the theoretical estimates., Comment: 26 pages, 10 figures

Videos contain multi-modal content, and exploring multi-level cross-modal interactions with natural language queries can provide great prominence to text-video retrieval task (TVR). However, new trending methods applying large-scale pre-trained model CLIP for TVR do not focus on multi-modal cues in videos. Furthermore, the traditional methods simply concatenating multi-modal features do not exploit fine-grained cross-modal information in videos. In this paper, we propose a multi-level multi-modal hybrid fusion (M2HF) network to explore comprehensive interactions between text queries and each modality content in videos. Specifically, M2HF first utilizes visual features extracted by CLIP to early fuse with audio and motion features extracted from videos, obtaining audio-visual fusion features and motion-visual fusion features respectively. Multi-modal alignment problem is also considered in this process. Then, visual features, audio-visual fusion features, motion-visual fusion features, and texts extracted from videos establish cross-modal relationships with caption queries in a multi-level way. Finally, the retrieval outputs from all levels are late fused to obtain final text-video retrieval results. Our framework provides two kinds of training strategies, including an ensemble manner and an end-to-end manner. Moreover, a novel multi-modal balance loss function is proposed to balance the contributions of each modality for efficient end-to-end training. M2HF allows us to obtain state-of-the-art results on various benchmarks, eg, Rank@1 of 64.9\%, 68.2\%, 33.2\%, 57.1\%, 57.8\% on MSR-VTT, MSVD, LSMDC, DiDeMo, and ActivityNet, respectively., Comment: 1 1pages, 3 figures, 5 tables

Task generalization has been a long standing challenge in Natural Language Processing (NLP). Recent research attempts to improve the task generalization ability of pre-trained language models by mapping NLP tasks into human-readable prompted forms. However, these approaches require laborious and inflexible manual collection of prompts, and different prompts on the same downstream task may receive unstable performance. We propose Unified Schema Prompt, a flexible and extensible prompting method, which automatically customizes the learnable prompts for each task according to the task input schema. It models the shared knowledge between tasks, while keeping the characteristics of different task schema, and thus enhances task generalization ability. The schema prompt takes the explicit data structure of each task to formulate prompts so that little human effort is involved. To test the task generalization ability of schema prompt at scale, we conduct schema prompt-based multitask pre-training on a wide variety of general NLP tasks. The framework achieves strong zero-shot and few-shot generalization performance on 16 unseen downstream tasks from 8 task types (e.g., QA, NLI, etc). Furthermore, comprehensive analyses demonstrate the effectiveness of each component in the schema prompt, its flexibility in task compositionality, and its ability to improve performance under a full-data fine-tuning setting.