Search

Recent studies demonstrate that query expansions generated by large language models (LLMs) can considerably enhance information retrieval systems by generating hypothetical documents that answer the queries as expansions. However, challenges arise from misalignments between the expansions and the retrieval corpus, resulting in issues like hallucinations and outdated information due to the limited intrinsic knowledge of LLMs. Inspired by Pseudo Relevance Feedback (PRF), we introduce Corpus-Steered Query Expansion (CSQE) to promote the incorporation of knowledge embedded within the corpus. CSQE utilizes the relevance assessing capability of LLMs to systematically identify pivotal sentences in the initially-retrieved documents. These corpus-originated texts are subsequently used to expand the query together with LLM-knowledge empowered expansions, improving the relevance prediction between the query and the target documents. Extensive experiments reveal that CSQE exhibits strong performance without necessitating any training, especially with queries for which LLMs lack knowledge., Comment: EACL 2024 (Short)

In this work, we introduce a new unsupervised embedding method, Meta-Task Prompting with Explicit One-Word Limitation (MetaEOL), for generating high-quality sentence embeddings from Large Language Models (LLMs) without the need for model fine-tuning or task-specific engineering. Leveraging meta-task prompting, MetaEOL guides LLMs to produce embeddings through a series of carefully designed prompts that address multiple representational aspects. Our comprehensive experiments demonstrate that embeddings averaged from various meta-tasks yield competitive performance on Semantic Textual Similarity (STS) benchmarks and excel in downstream tasks, surpassing contrastive-trained models. Our findings suggest a new scaling law for embedding generation, offering a versatile, resource-efficient approach for embedding extraction across diverse sentence-centric scenarios.

In the era of Large Language Models (LLMs), Knowledge Distillation (KD) emerges as a pivotal methodology for transferring advanced capabilities from leading proprietary LLMs, such as GPT-4, to their open-source counterparts like LLaMA and Mistral. Additionally, as open-source LLMs flourish, KD plays a crucial role in both compressing these models, and facilitating their self-improvement by employing themselves as teachers. This paper presents a comprehensive survey of KD's role within the realm of LLM, highlighting its critical function in imparting advanced knowledge to smaller models and its utility in model compression and self-improvement. Our survey is meticulously structured around three foundational pillars: \textit{algorithm}, \textit{skill}, and \textit{verticalization} -- providing a comprehensive examination of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields. Crucially, the survey navigates the intricate interplay between data augmentation (DA) and KD, illustrating how DA emerges as a powerful paradigm within the KD framework to bolster LLMs' performance. By leveraging DA to generate context-rich, skill-specific training data, KD transcends traditional boundaries, enabling open-source models to approximate the contextual adeptness, ethical alignment, and deep semantic insights characteristic of their proprietary counterparts. This work aims to provide an insightful guide for researchers and practitioners, offering a detailed overview of current methodologies in KD and proposing future research directions. Importantly, we firmly advocate for compliance with the legal terms that regulate the use of LLMs, ensuring ethical and lawful application of KD of LLMs. An associated Github repository is available at https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs., Comment: 44 pages

Catastrophic forgetting emerges as a critical challenge when fine-tuning multi-modal large language models (MLLMs), where improving performance on unseen tasks often leads to a significant performance drop on the original tasks. This paper presents a comprehensive analysis of catastrophic forgetting in MLLMs and introduces a post-training adjustment method called Model Tailor. Our method primarily preserves the pre-trained parameters while replacing a small number ($\leq$ 10\%) of fine-tuned parameters, maintaining $\sim$ 99\% effectiveness on original tasks versus pre-training, and achieving $\sim$ 97\% on new tasks compared to standard fine-tuning. Specifically, we derive a sparse mask to identify the "model patch", based on a fusion strategy that integrates salience and sensitivity analysis. Subsequently, a compensation mechanism is introduced to "decorate the patch", enhancing the model's performance on both target and original tasks. Additionally, our method is adaptable to multi-task scenarios. Through extensive experiments on InstructBLIP and LLaVA-1.5 in both image captioning and visual question answering tasks, our approach demonstrates significant task adaptability while preserving inherent pre-trained capabilities.

Large language models (LLMs) have demonstrated promising potential in various downstream tasks, including machine translation. However, prior work on LLM-based machine translation has mainly focused on better utilizing training data, demonstrations, or pre-defined and universal knowledge to improve performance, with a lack of consideration of decision-making like human translators. In this paper, we incorporate Thinker with the Drift-Diffusion Model (Thinker-DDM) to address this issue. We then redefine the Drift-Diffusion process to emulate human translators' dynamic decision-making under constrained resources. We conduct extensive experiments under the high-resource, low-resource, and commonsense translation settings using the WMT22 and CommonMT datasets, in which Thinker-DDM outperforms baselines in the first two scenarios. We also perform additional analysis and evaluation on commonsense translation to illustrate the high effectiveness and efficacy of the proposed method., Comment: Under review

In deep learning, stochastic gradient descent often yields functionally similar yet widely scattered solutions in the weight space even under the same initialization, causing barriers in the Linear Mode Connectivity (LMC) landscape. Overcoming these barriers is crucial for understanding deep learning dynamics and enhancing model-fusion algorithms. Previous studies highlight the role of permutation symmetry in reducing post-training barriers through network permutation. However, these post-hoc methods, demanding extra computations, are less effective for larger, complex models (e.g., ViT, LLM) due to numerous permutation matrices. Thus, in this paper, we study training-time neuron alignment. Our hypothesis suggests that training-time permutation subspace can reduce LMC barriers for free. We find that pruning at initialization supports this. Beyond pruning, we introduce TNA-PFN, a simple yet lossless algorithm using a partial gradient mask during training. TNA-PFN is theoretically and empirically validated for reducing LMC barriers. It excels in wide model fusion applications, especially in federated learning, two algorithms based on TNA-FPN that are proposed to show its prospects even under heterogeneous datasets. Moreover, TNA-PFN can enhance the generalization of model soup for vision transformers and ColD fusion for pretrained language models., Comment: preprint

In the rapidly evolving domain of Natural Language Generation (NLG) evaluation, introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. This survey aims to provide a thorough overview of leveraging LLMs for NLG evaluation, a burgeoning area that lacks a systematic analysis. We propose a coherent taxonomy for organizing existing LLM-based evaluation metrics, offering a structured framework to understand and compare these methods. Our detailed exploration includes critically assessing various LLM-based methodologies, as well as comparing their strengths and limitations in evaluating NLG outputs. By discussing unresolved challenges, including bias, robustness, domain-specificity, and unified evaluation, this survey seeks to offer insights to researchers and advocate for fairer and more advanced NLG evaluation techniques., Comment: 19pages

Recently, foundation models, particularly large language models (LLMs), have demonstrated an impressive ability to adapt to various tasks by fine-tuning large amounts of instruction data. Notably, federated foundation models emerge as a privacy preservation method to fine-tune models collaboratively under federated learning (FL) settings by leveraging many distributed datasets with non-IID data. To alleviate communication and computation overhead, parameter-efficient methods are introduced for efficiency, and some research adapted personalization methods to federated foundation models for better user preferences alignment. However, a critical gap in existing research is the neglect of test-time distribution shifts in real-world applications. Therefore, to bridge this gap, we propose a new setting, termed test-time personalization, which not only concentrates on the targeted local task but also extends to other tasks that exhibit test-time distribution shifts. To address challenges in this new setting, we explore a simple yet effective solution to learn a comprehensive foundation model. Specifically, a dual-personalizing adapter architecture (FedDPA) is proposed, comprising a global adapter and a local adapter for addressing test-time distribution shifts and personalization, respectively. Additionally, we introduce an instance-wise dynamic weighting mechanism to optimize the balance between the global and local adapters, enhancing overall performance. The effectiveness of the proposed method has been evaluated on benchmark datasets across different NLP tasks.

Recent breakthroughs in large models have highlighted the critical significance of data scale, labels and modals. In this paper, we introduce MS MARCO Web Search, the first large-scale information-rich web dataset, featuring millions of real clicked query-document labels. This dataset closely mimics real-world web document and query distribution, provides rich information for various kinds of downstream tasks and encourages research in various areas, such as generic end-to-end neural indexer models, generic embedding models, and next generation information access system with large language models. MS MARCO Web Search offers a retrieval benchmark with three web retrieval challenge tasks that demand innovations in both machine learning and information retrieval system research domains. As the first dataset that meets large, real and rich data requirements, MS MARCO Web Search paves the way for future advancements in AI and system research. MS MARCO Web Search dataset is available at: https://github.com/microsoft/MS-MARCO-Web-Search., Comment: 10 pages, 6 figures, for associated dataset, see http://github.com/microsoft/MS-MARCO-Web-Search

Androgen receptor (AR) signaling is crucial for driving prostate cancer (PCa), the most diagnosed and the second leading cause of death in male patients with cancer in the United States. Androgen deprivation therapy is initially effective in most instances of AR-positive advanced or metastatic PCa. However, patients inevitably develop lethal castration-resistant PCa (CRPC), which is also resistant to the next-generation AR signaling inhibitors. Most CRPCs maintain AR expression, and blocking AR signaling remains a main therapeutic approach. GATA2 is a pioneer transcription factor emerging as a key therapeutic target for PCa because it promotes AR expression and activation. While directly inhibiting GATA2 transcriptional activity remains challenging, enhancing GATA2 degradation is a plausible therapeutic strategy. How GATA2 protein stability is regulated in PCa remains unknown. Here, we show that constitutive photomorphogenesis protein 1 (COP1), an E3 ubiquitin ligase, drives GATA2 ubiquitination at K419/K424 for degradation. GATA2 lacks a conserved [D/E](x)xxVP[D/E] degron but uses alternate BR1/BR2 motifs to bind COP1. By promoting GATA2 degradation, COP1 inhibits AR expression and activation and represses PCa cell and xenograft growth and castration resistance. Accordingly, GATA2 overexpression or COP1 mutations that disrupt COP1-GATA2 binding block COP1 tumor-suppressing activities. We conclude that GATA2 is a major COP1 substrate in PCa and that COP1 promotion of GATA2 degradation is a direct mechanism for regulating AR expression and activation, PCa growth, and castration resistance.

Leaf rust, caused by Puccinia triticina Eriksson (Pt), is one of the most severe foliar diseases of wheat. Breeding for leaf rust resistance is a practical and sustainable method to control this devastating disease. Here, we report the identification of Lr47, a broadly effective leaf rust resistance gene introgressed into wheat from Aegilops speltoides. Lr47 encodes a coiled-coil nucleotide-binding leucine-rich repeat protein that is both necessary and sufficient to confer Pt resistance, as demonstrated by loss-of-function mutations and transgenic complementation. Lr47 introgression lines with no or reduced linkage drag are generated using the Pairing homoeologous1 mutation, and a diagnostic molecular marker for Lr47 is developed. The coiled-coil domain of the Lr47 protein is unable to induce cell death, nor does it have self-protein interaction. The cloning of Lr47 expands the number of leaf rust resistance genes that can be incorporated into multigene transgenic cassettes to control this devastating disease.

Purpose: From the two dimensions of aesthetic nature-relatedness and experience nature-relatedness, we systematically construct an evaluation index system of public art nature-relatedness effect composed of the attraction effect index, communication effect index, reflection effect index, and experience effect index. The construction of the evaluation index system has laid a perfect foundation for the nature-relatedness evaluation of public art landscape. Theoretical framework: 'Natural-relatedness' is a kind of relationship, which contains the individual's recognition, praise, tolerance, and understanding of the interconnection between man and nature. The content involved not only includes the individual's feeling of facing nature but also the self-concept contained in nature, as well as the individual's values of nature, self, and society. The natural-relatedness has a positive effect on people's green ecological view and healthy outlook on life. Design/methodology/approach: This study used literature analysis, field research, analytic hierarchy process, and case analysis. Through literature analysis, we sort out the current situation and problems in regional environmental planning at home and abroad. The evaluation index system of the natural coupling effect of the public art landscape is constructed by the analytic hierarchy process. The case analysis method is used to analyze the evaluation results. Findings: The evaluation index system includes four secondary evaluation indexes: 'the attraction effect, '' reflection effect, '' communication effect ' and ' the experience effect, 'and 13 tertiary indexes under the secondary indexes. The construction of the evaluation index of the nature-relatedness effect provides a theoretical basis for studying natural public connection in public art landscape design. Research, Practical & Social implications: This study combines theory with practice, takes the practical problems encountered in the sustainable developme

Weakly-Supervised Video Grounding (WSVG) aims to localize events of interest in untrimmed videos with only video-level annotations. To date, most of the state-of-the-art WSVG methods follow a two-stage pipeline, i.e., firstly generating potential temporal proposals and then grounding with these proposal candidates. Despite the recent progress, existing proposal generation methods suffer from two draw-backs: 1) lack of explicit correspondence modeling; and 2) partial coverage of complex events. To this end, we propose a novel IteRative prOposal refiNement network (dubbed as IRON) to gradually distill the prior knowledge into each proposal and encourage proposals with more complete coverage. Specifically, we set up two lightweight distillation branches to uncover the cross-modal correspondence on both the semantic and conceptual levels. Then, an iterative Label Propagation (LP) strategy is devised to prevent the network from focusing excessively on the most discriminative events instead of the whole sentence content. Precisely, during each iteration, the proposal with the minimal distillation loss and its adjacent ones are regarded as the positive samples, which refines proposal confidence scores in a cascaded manner. Extensive experiments and ablation studies on two challenging WSVG datasets have attested to the effectiveness of our IRON. The code will be available at https://github.com/mengcaopku/IRON. © 2023 IEEE.

To leverage the enormous amount of unlabeled data on distributed edge devices, we formulate a new problem in federated learning called federated unsupervised representation learning (FURL) to learn a common representation model without supervision while preserving data privacy. FURL poses two new challenges: (1) data distribution shift (non-independent and identically distributed, non-IID) among clients would make local models focus on different categories, leading to the inconsistency of representation spaces; (2) without unified information among the clients in FURL, the representations across clients would be misaligned. To address these challenges, we propose the federated contrastive averaging with dictionary and alignment (FedCA) algorithm. FedCA is composed of two key modules: a dictionary module to aggregate the representations of samples from each client which can be shared with all clients for consistency of representation space and an alignment module to align the representation of each client on a base model trained on public data. We adopt the contrastive approach for local model training. Through extensive experiments with three evaluation protocols in IID and non-IID settings, we demonstrate that FedCA outperforms all baselines with significant margins.

Image-text retrieval (ITR) is a task to retrieve the relevant images/texts, given the query from another modality. The conventional dense retrieval paradigm relies on encoding images and texts into dense representations using dual-stream encoders, however, it faces challenges with low retrieval speed in large-scale retrieval scenarios. In this work, we propose the lexicon-weighting paradigm, where sparse representations in vocabulary space are learned for images and texts to take advantage of the bag-of-words models and efficient inverted indexes, resulting in significantly reduced retrieval latency. A crucial gap arises from the continuous nature of image data, and the requirement for a sparse vocabulary space representation. To bridge this gap, we introduce a novel pre-training framework, Lexicon-Bottlenecked Language-Image Pre-Training (LexLIP), that learns importance-aware lexicon representations. This framework features lexicon-bottlenecked modules between the dual-stream encoders and weakened text decoders, allowing for constructing continuous bag-of-words bottlenecks to learn lexicon-importance distributions. Upon pre-training with same-scale data, our LexLIP achieves state-of-the-art performance on two benchmark ITR datasets, MSCOCO and Flickr30k. Furthermore, in large-scale retrieval scenarios, LexLIP outperforms CLIP with a 5.5 ~ 221.3X faster retrieval speed and 13.2 ~ 48.8X less index storage memory.

To tackle the global climate challenge, it urgently needs to develop a collaborative platform for comprehensive weather forecasting on large-scale meteorological data. Despite urgency, heterogeneous meteorological sensors across countries and regions, inevitably causing multivariate heterogeneity and data exposure, become the main barrier. This paper develops a foundation model across regions capable of understanding complex meteorological data and providing weather forecasting. To relieve the data exposure concern across regions, a novel federated learning approach has been proposed to collaboratively learn a brand-new spatio-temporal Transformer-based foundation model across participants with heterogeneous meteorological data. Moreover, a novel prompt learning mechanism has been adopted to satisfy low-resourced sensors' communication and computational constraints. The effectiveness of the proposed method has been demonstrated on classical weather forecasting tasks using three meteorological datasets with multivariate time series., Comment: Accepted by IJCAI'23 (32nd International Joint Conference on Artificial Intelligence)

Yingying Pan,1,* Xianrong Feng,2,* Wei Song,1,* Xin Zhou,1 Zhen Zhou,3 Gaoyang Chen,1 Tao Shen,1 Xiaobo Zhang1 1School of Basic Medicine, Chengdu University of Traditional Chinese Medicine, Chengdu, People’s Republic of China; 2Hospital of Chengdu University of Traditional Chinese Medicine, Chengdu, People’s Republic of China; 3Menzies Institute for Medical Research, University of Tasmania, Hobart, Tasmania, Australia*These authors contributed equally to this workCorrespondence: Xiaobo Zhang; Tao Shen, School of Basic Medicine, Chengdu University of Traditional Chinese Medicine, Chengdu, People’s Republic of China, Email zhangxiaobo@stu.cdutcm.edu.cn; shentaotcm@aliyun.comBackground: Atherosclerosis (AS) is an immunoinflammatory disease associated with dyslipidemia. Zhuyu Pill (ZYP) is a classic Chinese herbal compound that has been shown to exhibit anti-inflammatory and lipid-lowering effects on AS in our previous studies. However, the underlying mechanisms by which ZYP ameliorates atherosclerosis have not yet been fully investigated. In this study, network pharmacology and in vivo experiments were conducted to explore the underlying pharmacological mechanisms of ZYP on ameliorating AS.Methods: The active ingredients of ZYP were acquired from our previous study. The putative targets of ZYP relevant to AS were obtained from TCMSP, SwissTargetPrediction, STITCH, DisGeNET, and GeneCards databases. Protein-protein interactions (PPI) network, Gene Ontology (GO), and Kyoto Encyclopedia of Genes and Genomes (KEGG) analysis were conducted using the Cytoscape software. Furthermore, in vivo experiments were carried out for target validation in apolipoprotein E (ApoE) -/- mice.Results: Animal experiments revealed that ZYP ameliorated AS mainly through lowering blood lipids, alleviating vascular inflammation, and decreasing the levels of vascular cell adhesion molecule-1 (VCAM1), intercellular adhesion molecule-1 (ICAM1), monocyte chemota

The field of protein folding research has been greatly advanced by deep learning methods, with AlphaFold2 (AF2) demonstrating exceptional performance and atomic-level precision. As co-evolution is integral to protein structure prediction, AF2's accuracy is significantly influenced by the depth of multiple sequence alignment (MSA), which requires extensive exploration of a large protein database for similar sequences. However, not all protein sequences possess abundant homologous families, and consequently, AF2's performance can degrade on such queries, at times failing to produce meaningful results. To address this, we introduce a novel generative language model, MSA-Augmenter, which leverages protein-specific attention mechanisms and large-scale MSAs to generate useful, novel protein sequences not currently found in databases. These sequences supplement shallow MSAs, enhancing the accuracy of structural property predictions. Our experiments on CASP14 demonstrate that MSA-Augmenter can generate de novo sequences that retain co-evolutionary information from inferior MSAs, thereby improving protein structure prediction quality on top of strong AF2.

On-device intelligence for weather forecasting uses local deep learning models to analyze weather patterns without centralized cloud computing, holds significance for supporting human activates. Federated Learning is a promising solution for such forecasting by enabling collaborative model training without sharing raw data. However, it faces three main challenges that hinder its reliability: (1) data heterogeneity among devices due to geographic differences; (2) data homogeneity within individual devices and (3) communication overload from sending large model parameters for collaboration. To address these challenges, this paper propose Federated Prompt Learning for Weather Foundation Models on Devices (FedPoD), which enables devices to obtain highly customized models while maintaining communication efficiency. Concretely, our Adaptive Prompt Tuning leverages lightweight prompts guide frozen foundation model to generate more precise predictions, also conducts prompt-based multi-level communication to encourage multi-source knowledge fusion and regulate optimization. Additionally, Dynamic Graph Modeling constructs graphs from prompts, prioritizing collaborative training among devices with similar data distributions to against heterogeneity. Extensive experiments demonstrates FedPoD leads the performance among state-of-the-art baselines across various setting in real-world on-device weather forecasting datasets., Comment: Accepted by Main Track in IJCAI'24 (the 33rd International Joint Conference on Artificial Intelligence)

Deep learning-based approaches, such as AlphaFold2 (AF2), have significantly advanced protein tertiary structure prediction, achieving results comparable to real biological experimental methods. While AF2 has shown limitations in predicting the effects of mutations, its robustness against sequence mutations remains to be determined. Starting with the wild-type (WT) sequence, we investigate adversarial sequences generated via an evolutionary approach, which AF2 predicts to be substantially different from WT. Our experiments on CASP14 reveal that by modifying merely three residues in the protein sequence using a combination of replacement, deletion, and insertion strategies, the alteration in AF2's predictions, as measured by the Local Distance Difference Test (lDDT), reaches 46.61. Moreover, when applied to a specific protein, SPNS2, our proposed algorithm successfully identifies biologically meaningful residues critical to protein structure determination and potentially indicates alternative conformations, thus significantly expediting the experimental process.

Information retrieval (IR) plays a crucial role in locating relevant resources from vast amounts of data, and its applications have evolved from traditional knowledge bases to modern retrieval models (RMs). The emergence of large language models (LLMs) has further revolutionized the IR field by enabling users to interact with search systems in natural languages. In this paper, we explore the advantages and disadvantages of LLMs and RMs, highlighting their respective strengths in understanding user-issued queries and retrieving up-to-date information. To leverage the benefits of both paradigms while circumventing their limitations, we propose InteR, a novel framework that facilitates information refinement through synergy between RMs and LLMs. InteR allows RMs to expand knowledge in queries using LLM-generated knowledge collections and enables LLMs to enhance prompt formulation using retrieved documents. This iterative refinement process augments the inputs of RMs and LLMs, leading to more accurate retrieval. Experiments on large-scale retrieval benchmarks involving web search and low-resource retrieval tasks demonstrate that InteR achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods, even those using relevance judgment. Source code is available at https://github.com/Cyril-JZ/InteR, Comment: Pre-print. Work in progress

In this work, we propose a simple method that applies a large language model (LLM) to large-scale retrieval in zero-shot scenarios. Our method, the Language language model as Retriever (LameR), is built upon no other neural models but an LLM, while breaking brute-force combinations of retrievers with LLMs and lifting the performance of zero-shot retrieval to be very competitive on benchmark datasets. Essentially, we propose to augment a query with its potential answers by prompting LLMs with a composition of the query and the query's in-domain candidates. The candidates, regardless of correct or wrong, are obtained by a vanilla retrieval procedure on the target collection. As a part of the prompts, they are likely to help LLM generate more precise answers by pattern imitation or candidate summarization. Even if all the candidates are wrong, the prompts at least make LLM aware of in-collection patterns and genres. Moreover, due to the low performance of a self-supervised retriever, the LLM-based query augmentation becomes less effective as the retriever bottlenecks the whole pipeline. Therefore, we propose to leverage a non-parametric lexicon-based method (e.g., BM25) as the retrieval module to capture query-document overlap in a literal fashion. As such, LameR makes the retrieval procedure transparent to the LLM, thus circumventing the performance bottleneck., Comment: Work in progress

Aligning the recent large language models (LLMs) with computer vision models leads to large vision-language models (LVLMs), which have paved the way for zero-shot image reasoning tasks. However, LVLMs are usually trained on short high-level captions only referring to sparse focus regions in images. Such a ``tunnel vision'' limits LVLMs to exploring other relevant contexts in complex scenes. To address this challenge, we introduce Question-Driven Visual Exploration (QVix), a novel prompting strategy that enhances the exploratory capabilities of LVLMs in zero-shot reasoning tasks. QVix leverages LLMs' strong language prior to generate input-exploratory questions with more details than the original query, guiding LVLMs to explore visual content more comprehensively and uncover subtle or peripheral details. QVix enables a wider exploration of visual scenes, improving the LVLMs' reasoning accuracy and depth in tasks such as visual question answering and visual entailment. Our evaluations on various challenging zero-shot vision-language benchmarks, including ScienceQA and fine-grained visual classification, demonstrate that QVix significantly outperforms existing methods, highlighting its effectiveness in bridging the gap between complex visual data and LVLMs' exploratory abilities.

Large Language Models (LLMs) have ushered in a transformative era in the field of natural language processing, excelling in tasks related to text comprehension and generation. Nevertheless, they encounter difficulties when confronted with chaotic contexts (e.g., distractors rather than long irrelevant context), leading to the inadvertent omission of certain details within the chaotic context. In response to these challenges, we introduce the "Thread of Thought" (ThoT) strategy, which draws inspiration from human cognitive processes. ThoT systematically segments and analyzes extended contexts while adeptly selecting pertinent information. This strategy serves as a versatile "plug-and-play" module, seamlessly integrating with various LLMs and prompting techniques. In the experiments, we utilize the PopQA and EntityQ datasets, as well as a Multi-Turn Conversation Response dataset (MTCR) we collected, to illustrate that ThoT significantly improves reasoning performance compared to other prompting techniques., Comment: 11 pages, 7 figures, 5 tables

In recent research, non-orthogonal artificial noise (NORAN) has been proposed as an alternative to orthogonal artificial noise (AN). However, NORAN introduces additional noise into the channel, which reduces the capacity of the legitimate channel (LC). At the same time, selecting a NORAN design with ideal security performance from a large number of design options is also a challenging problem. To address these two issues, a novel NORAN based on a pilot information codebook is proposed in this letter. The codebook associates different suboptimal NORANs with pilot information as the key under different channel state information (CSI). The receiver interrogates the codebook using the pilot information to obtain the NORAN that the transmitter will transmit in the next moment, in order to eliminate the NORAN when receiving information. Therefore, NORAN based on pilot information codebooks can improve the secrecy capacity (SC) of the communication system by directly using suboptimal NORAN design schemes without increasing the noise in the LC. Numerical simulations and analyses show that the introduction of NORAN with a novel design using pilot information codebooks significantly enhances the security and improves the SC of the communication system.

To enhance the reasoning capabilities of off-the-shelf Large Language Models (LLMs), we introduce a simple, yet general and effective prompting method, Re2, i.e., \textbf{Re}-\textbf{Re}ading the question as input. Unlike most thought-eliciting prompting methods, such as Chain-of-Thought (CoT), which aim to elicit the reasoning process in the output, Re2 shifts the focus to the input by processing questions twice, thereby enhancing the understanding process. Consequently, Re2 demonstrates strong generality and compatibility with most thought-eliciting prompting methods, including CoT. Crucially, Re2 facilitates a "bidirectional" encoding in unidirectional decoder-only LLMs because the first pass could provide global information for the second pass. We begin with a preliminary empirical study as the foundation of Re2, illustrating its potential to enable "bidirectional" attention mechanisms. We then evaluate Re2 on extensive reasoning benchmarks across 14 datasets, spanning 112 experiments, to validate its effectiveness and generality. Our findings indicate that, with the exception of a few scenarios on vanilla ChatGPT, Re2 consistently enhances the reasoning performance of LLMs through a simple re-reading strategy. Further analyses reveal Re2's adaptability, showing how it can be effectively integrated with different LLMs, thought-eliciting prompting, and ensemble strategies. Our code is available at \url{https://github.com/Tebmer/Rereading-LLM-Reasoning/}, Comment: 25 pages

Prostate cancer (PCa) is the second leading cause of cancer death in American men. Androgen receptor (AR) signaling is essential for PCa cell growth/survival and remains a key therapeutic target for lethal castration-resistant PCa (CRPC). GATA2 is a pioneer transcription factor crucial for inducing AR expression/activation. We recently reported that MAPK4, an atypical MAPK, promotes tumor progression via noncanonical activation of AKT. Here, we demonstrated that MAPK4 activated AR by enhancing GATA2 transcriptional expression and stabilizing GATA2 protein through repression of GATA2 ubiquitination/degradation. MAPK4 expression correlated with AR activation in human CRPC. Concerted activation of both GATA2/AR and AKT by MAPK4 promoted PCa cell proliferation, anchorage-independent growth, xenograft growth, and castration resistance. Conversely, knockdown of MAPK4 decreased activation of both AR and AKT and inhibited PCa cell and xenograft growth, including castration-resistant growth. Both GATA2/AR and AKT activation were necessary for MAPK4 tumor-promoting activity. Interestingly, combined overexpression of GATA2 plus a constitutively activated AKT was sufficient to drive PCa growth and castration resistance, shedding light on an alternative, MAPK4-independent tumor-promoting pathway in human PCa. We concluded that MAPK4 promotes PCa growth and castration resistance by cooperating parallel pathways of activating GATA2/AR and AKT and that MAPK4 is a novel therapeutic target in PCa, especially CRPC.

About 15-20% of breast cancer (BCa) is triple-negative BCa (TNBC), a devastating disease with limited therapeutic options. Aberrations in the PI3K/PTEN signaling pathway are common in TNBC. However, the therapeutic impact of PI3K inhibitors in TNBC has been limited and the mechanism(s) underlying this lack of efficacy remain elusive. Here, we demonstrate that a large subset of TNBC expresses significant levels of MAPK4, and this expression is critical for driving AKT activation independent of PI3K and promoting TNBC cell and xenograft growth. The ability of MAPK4 to bypass PI3K for AKT activation potentially provides a direct mechanism regulating tumor sensitivity to PI3K inhibition. Accordingly, repressing MAPK4 greatly sensitizes TNBC cells and xenografts to PI3K blockade. Altogether, we conclude that high MAPK4 expression defines a large subset or subtype of TNBC responsive to MAPK4 blockage. Targeting MAPK4 in this subset/subtype of TNBC both represses growth and sensitizes tumors to PI3K blockade.

In order to improve the performance of the automatic apple grading and sorting system, in this paper, an ensemble model of ordinal classification based on neural network with ordered partitions and Dempster–Shafer theory is proposed. As a non-destructive grading method, apples are graded into three grades based on the Soluble Solids Content value, with features extracted from the preprocessed near-infrared spectrum of apple serving as model inputs. Considering the uncertainty in grading labels, mass generation approach and evidential encoding scheme for ordinal label are proposed, with uncertainty handled within the framework of Dempster–Shafer theory. Constructing neural network with ordered partitions as the base learner, the learning procedure of the Bagging-based ensemble model is detailed. Experiments on Yantai Red Fuji apples demonstrate the satisfactory grading performances of proposed evidential ensemble model for ordinal classification. © 2022 The Authors. CAAI Transactions on Intelligence Technology published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology and Chongqing University of Technology., open access

In order to improve the performance of the automatic apple grading and sorting system, in this paper, an ensemble model of ordinal classification based on neural network with ordered partitions and Dempster–Shafer theory is proposed. As a non-destructive grading method, apples are graded into three grades based on the Soluble Solids Content value, with features extracted from the preprocessed near-infrared spectrum of apple serving as model inputs. Considering the uncertainty in grading labels, mass generation approach and evidential encoding scheme for ordinal label are proposed, with uncertainty handled within the framework of Dempster–Shafer theory. Constructing neural network with ordered partitions as the base learner, the learning procedure of the Bagging-based ensemble model is detailed. Experiments on Yantai Red Fuji apples demonstrate the satisfactory grading performances of proposed evidential ensemble model for ordinal classification. © 2022 The Authors. CAAI Transactions on Intelligence Technology published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology and Chongqing University of Technology., open access

Objectives: The aim of this study was to analyze long-term outcomes of Chinese coronary artery disease (CAD) patients with (and without) chronic obstructive pulmonary disease (COPD) after percutaneous coronary intervention (PCI). Background: Chronic obstructive pulmonary disease is a chronic condition which often develops in conjunction with CAD. PCI is a core therapy for CAD, although we still need to understand CAD-COPD outcomes and to identify factors that influence prognoses, across ethnicities. Methods: This double-cohort study involved 12,343 Chinese CAD patients who received PCI. Baseline characteristics were collected in two independent, specialty centers. Propensity-score matching was performed to control confounding factors, using a nearest neighbor matching method within a 0.02 caliper and on a propensity score scale of 0.1 for each center. Comorbid CAD-COPD cases were compared to non-COPD patients in terms of major adverse cardiac events (MACEs). Results: Patients with COPD were generally older than those without COPD (65.4 ± 9.2 vs. 58.2 ± 10.3, p < 0.001). There were no significant differences in the end points between COPD and non-COPD groups after PCI (All p > 0.05); however, the incidence of MACEs increased after 450 days. Further subgroup analysis suggests that COPD is approximately four times more prevalent among those aged over 75 years (HR, 3.818; 95%CI, 1.10–13.29; p = 0.027) and those aged below 55 years (HR = 4.254; 95% CI, 1.55–11.72; p = 0.003). Conclusion: Having COPD does not appear to have a significant impact on CAD outcomes 2 years after PCI, and beyond. However, an increasing number of MACEs was observed after 450 days, which suggests that there may be a double-stage effect of COPD on PCI prognosis. There is a need for focused comorbidity management, specifically for those aged below 55 years and above 75 years.

About 15-20% of breast cancer (BCa) is triple-negative BCa (TNBC), a devastating disease with limited therapeutic options. Aberrations in the PI3K/PTEN signaling pathway are common in TNBC. However, the therapeutic impact of PI3K inhibitors in TNBC has been limited and the mechanism(s) underlying this lack of efficacy remain elusive. Here, we demonstrate that a large subset of TNBC expresses significant levels of MAPK4, and this expression is critical for driving AKT activation independent of PI3K and promoting TNBC cell and xenograft growth. The ability of MAPK4 to bypass PI3K for AKT activation potentially provides a direct mechanism regulating tumor sensitivity to PI3K inhibition. Accordingly, repressing MAPK4 greatly sensitizes TNBC cells and xenografts to PI3K blockade. Altogether, we conclude that high MAPK4 expression defines a large subset or subtype of TNBC responsive to MAPK4 blockage. Targeting MAPK4 in this subset/subtype of TNBC both represses growth and sensitizes tumors to PI3K blockade.

Objectives: The aim of this study was to analyze long-term outcomes of Chinese coronary artery disease (CAD) patients with (and without) chronic obstructive pulmonary disease (COPD) after percutaneous coronary intervention (PCI). Background: Chronic obstructive pulmonary disease is a chronic condition which often develops in conjunction with CAD. PCI is a core therapy for CAD, although we still need to understand CAD-COPD outcomes and to identify factors that influence prognoses, across ethnicities. Methods: This double-cohort study involved 12,343 Chinese CAD patients who received PCI. Baseline characteristics were collected in two independent, specialty centers. Propensity-score matching was performed to control confounding factors, using a nearest neighbor matching method within a 0.02 caliper and on a propensity score scale of 0.1 for each center. Comorbid CAD-COPD cases were compared to non-COPD patients in terms of major adverse cardiac events (MACEs). Results: Patients with COPD were generally older than those without COPD (65.4 ± 9.2 vs. 58.2 ± 10.3, p < 0.001). There were no significant differences in the end points between COPD and non-COPD groups after PCI (All p > 0.05); however, the incidence of MACEs increased after 450 days. Further subgroup analysis suggests that COPD is approximately four times more prevalent among those aged over 75 years (HR, 3.818; 95%CI, 1.10–13.29; p = 0.027) and those aged below 55 years (HR = 4.254; 95% CI, 1.55–11.72; p = 0.003). Conclusion: Having COPD does not appear to have a significant impact on CAD outcomes 2 years after PCI, and beyond. However, an increasing number of MACEs was observed after 450 days, which suggests that there may be a double-stage effect of COPD on PCI prognosis. There is a need for focused comorbidity management, specifically for those aged below 55 years and above 75 years.

Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. However, in contrast to passages or sentences, retrieval on long documents suffers from the scope hypothesis that a long document may cover multiple topics. This maximizes their structure heterogeneity and poses a granular-mismatch issue, leading to an inferior distillation efficacy. In this work, we propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers. While preserving the conventional dense retrieval paradigm, it first produces global-consistent representations crossing different fine granularity and then applies multi-granular aligned distillation merely during training. In experiments, we evaluate our framework on two long-document retrieval benchmarks, which show state-of-the-art performance., Comment: 13 pages, 5 figures, 5 tables

To improve the performance of the dual-encoder retriever, one effective approach is knowledge distillation from the cross-encoder ranker. Existing works construct the candidate passages following the supervised learning setting where a query is paired with a positive passage and a batch of negatives. However, through empirical observation, we find that even the hard negatives from advanced methods are still too trivial for the teacher to distinguish, preventing the teacher from transferring abundant dark knowledge to the student through its soft label. To alleviate this issue, we propose ADAM, a knowledge distillation framework that can better transfer the dark knowledge held in the teacher with Adaptive Dark exAMples. Different from previous works that only rely on one positive and hard negatives as candidate passages, we create dark examples that all have moderate relevance to the query through mixing-up and masking in discrete space. Furthermore, as the quality of knowledge held in different training instances varies as measured by the teacher's confidence score, we propose a self-paced distillation strategy that adaptively concentrates on a subset of high-quality instances to conduct our dark-example-based knowledge distillation to help the student learn better. We conduct experiments on two widely-used benchmarks and verify the effectiveness of our method., Comment: 13 pages, 3 figures

Recently, prefix-tuning was proposed to efficiently adapt pre-trained language models to a broad spectrum of natural language classification tasks. It leverages soft prefix as task-specific indicators and language verbalizers as categorical-label mentions to narrow the formulation gap from pre-training language models. However, when the label space increases considerably (i.e., many-class classification), such a tuning technique suffers from a verbalizer ambiguity problem since the many-class labels are represented by semantic-similar verbalizers in short language phrases. To overcome this, inspired by the human-decision process that the most ambiguous classes would be mulled over for each instance, we propose a brand-new prefix-tuning method, Counterfactual Contrastive Prefix-tuning (CCPrefix), for many-class classification. Basically, an instance-dependent soft prefix, derived from fact-counterfactual pairs in the label space, is leveraged to complement the language verbalizers in many-class classification. We conduct experiments on many-class benchmark datasets in both the fully supervised setting and the few-shot setting, which indicates that our model outperforms former baselines., Comment: has been accepted by EACL 2024

Device Model Generalization (DMG) is a practical yet under-investigated research topic for on-device machine learning applications. It aims to improve the generalization ability of pre-trained models when deployed on resource-constrained devices, such as improving the performance of pre-trained cloud models on smart mobiles. While quite a lot of works have investigated the data distribution shift across clouds and devices, most of them focus on model fine-tuning on personalized data for individual devices to facilitate DMG. Despite their promising, these approaches require on-device re-training, which is practically infeasible due to the overfitting problem and high time delay when performing gradient calculation on real-time data. In this paper, we argue that the computational cost brought by fine-tuning can be rather unnecessary. We consequently present a novel perspective to improving DMG without increasing computational cost, i.e., device-specific parameter generation which directly maps data distribution to parameters. Specifically, we propose an efficient Device-cloUd collaborative parametErs generaTion framework DUET. DUET is deployed on a powerful cloud server that only requires the low cost of forwarding propagation and low time delay of data transmission between the device and the cloud. By doing so, DUET can rehearse the device-specific model weight realizations conditioned on the personalized real-time data for an individual device. Importantly, our DUET elegantly connects the cloud and device as a 'duet' collaboration, frees the DMG from fine-tuning, and enables a faster and more accurate DMG paradigm. We conduct an extensive experimental study of DUET on three public datasets, and the experimental results confirm our framework's effectiveness and generalisability for different DMG tasks.

In large-scale retrieval, the lexicon-weighting paradigm, learning weighted sparse representations in vocabulary space, has shown promising results with high quality and low latency. Despite it deeply exploiting the lexicon-representing capability of pre-trained language models, a crucial gap remains between language modeling and lexicon-weighting retrieval -- the former preferring certain or low-entropy words whereas the latter favoring pivot or high-entropy words -- becoming the main barrier to lexicon-weighting performance for large-scale retrieval. To bridge this gap, we propose a brand-new pre-training framework, lexicon-bottlenecked masked autoencoder (LexMAE), to learn importance-aware lexicon representations. Essentially, we present a lexicon-bottlenecked module between a normal language modeling encoder and a weakened decoder, where a continuous bag-of-words bottleneck is constructed to learn a lexicon-importance distribution in an unsupervised fashion. The pre-trained LexMAE is readily transferred to the lexicon-weighting retrieval via fine-tuning. On the ad-hoc retrieval benchmark, MS-Marco, it achieves 42.6% MRR@10 with 45.8 QPS for the passage dataset and 44.4% MRR@100 with 134.8 QPS for the document dataset, by a CPU machine. And LexMAE shows state-of-the-art zero-shot transfer capability on BEIR benchmark with 12 datasets., Comment: Appeared at ICLR 2023

Retrieval models based on dense representations in semantic space have become an indispensable branch for first-stage retrieval. These retrievers benefit from surging advances in representation learning towards compressive global sequence-level embeddings. However, they are prone to overlook local salient phrases and entity mentions in texts, which usually play pivot roles in first-stage retrieval. To mitigate this weakness, we propose to make a dense retriever align a well-performing lexicon-aware representation model. The alignment is achieved by weakened knowledge distillations to enlighten the retriever via two aspects -- 1) a lexicon-augmented contrastive objective to challenge the dense encoder and 2) a pair-wise rank-consistent regularization to make dense model's behavior incline to the other. We evaluate our model on three public benchmarks, which shows that with a comparable lexicon-aware retriever as the teacher, our proposed dense one can bring consistent and significant improvements, and even outdo its teacher. In addition, we found our improvement on the dense retriever is complementary to the standard ranker distillation, which can further lift state-of-the-art performance., Comment: 14 pages, 6 tables, 4 figures. WWW 2023

RNA structure determination and prediction can promote RNA-targeted drug development and engineerable synthetic elements design. But due to the intrinsic structural flexibility of RNAs, all the three mainstream structure determination methods (X-ray crystallography, NMR, and Cryo-EM) encounter challenges when resolving the RNA structures, which leads to the scarcity of the resolved RNA structures. Computational prediction approaches emerge as complementary to the experimental techniques. However, none of the \textit{de novo} approaches is based on deep learning since too few structures are available. Instead, most of them apply the time-consuming sampling-based strategies, and their performance seems to hit the plateau. In this work, we develop the first end-to-end deep learning approach, E2Efold-3D, to accurately perform the \textit{de novo} RNA structure prediction. Several novel components are proposed to overcome the data scarcity, such as a fully-differentiable end-to-end pipeline, secondary structure-assisted self-distillation, and parameter-efficient backbone formulation. Such designs are validated on the independent, non-overlapping RNA puzzle testing dataset and reach an average sub-4 \AA{} root-mean-square deviation, demonstrating its superior performance compared to state-of-the-art approaches. Interestingly, it also achieves promising results when predicting RNA complex structures, a feat that none of the previous systems could accomplish. When E2Efold-3D is coupled with the experimental techniques, the RNA structure prediction field can be greatly advanced.

Anomaly detection is an important part of machine learning. Detection of outliers in the field of transportation can provide valid data for future traffic predictions or traffic flow analysis. This paper builds a model based on XGBoost to detect outliers in IoT data. The data is preprocessed first, followed by model building. Then we use the grid search to adjust the parameters and substitute the optimal parameters into the building model. To validate the model, we cross-checked it with two benchmark models, iFroset and Random Forest. The final experimental results show that the model constructed in this paper can accurately detect outliers in traffic flow and the accuracy is better than that of the baseline model. © 2022, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

IoT (internet of things) data is a topic we have discussed a lot in recent years. Traffic data is an important part of IoT data. Traffic flow prediction not only facilitates people’s travel and saves our time, but also provides effective technical support for highway traffic control and scheduling. To achieve accurate traffic flow prediction, this study aims to build a machine learning-based traffic flow prediction model. We first screen out the features that have a greater impact on traffic flow. On this basis, this work establishes a traffic flow prediction model based on CatBoost. By comparing with other machine learning models, conclusions can be drawn: CatBoost model can accurately predict traffic flow; CatBoost outperforms traditional machine learning models. © 2022, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

In order to improve the performance of the automatic apple grading and sorting system, in this paper, an ensemble model of ordinal classification based on neural network with ordered partitions and Dempster–Shafer theory is proposed. As a non-destructive grading method, apples are graded into three grades based on the Soluble Solids Content value, with features extracted from the preprocessed near-infrared spectrum of apple serving as model inputs. Considering the uncertainty in grading labels, mass generation approach and evidential encoding scheme for ordinal label are proposed, with uncertainty handled within the framework of Dempster–Shafer theory. Constructing neural network with ordered partitions as the base learner, the learning procedure of the Bagging-based ensemble model is detailed. Experiments on Yantai Red Fuji apples demonstrate the satisfactory grading performances of proposed evidential ensemble model for ordinal classification. © 2022 The Authors. CAAI Transactions on Intelligence Technology published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology and Chongqing University of Technology., open access

In order to improve the performance of the automatic apple grading and sorting system, in this paper, an ensemble model of ordinal classification based on neural network with ordered partitions and Dempster–Shafer theory is proposed. As a non-destructive grading method, apples are graded into three grades based on the Soluble Solids Content value, with features extracted from the preprocessed near-infrared spectrum of apple serving as model inputs. Considering the uncertainty in grading labels, mass generation approach and evidential encoding scheme for ordinal label are proposed, with uncertainty handled within the framework of Dempster–Shafer theory. Constructing neural network with ordered partitions as the base learner, the learning procedure of the Bagging-based ensemble model is detailed. Experiments on Yantai Red Fuji apples demonstrate the satisfactory grading performances of proposed evidential ensemble model for ordinal classification. © 2022 The Authors. CAAI Transactions on Intelligence Technology published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology and Chongqing University of Technology., open access

In order to improve the performance of the automatic apple grading and sorting system, in this paper, an ensemble model of ordinal classification based on neural network with ordered partitions and Dempster–Shafer theory is proposed. As a non-destructive grading method, apples are graded into three grades based on the Soluble Solids Content value, with features extracted from the preprocessed near-infrared spectrum of apple serving as model inputs. Considering the uncertainty in grading labels, mass generation approach and evidential encoding scheme for ordinal label are proposed, with uncertainty handled within the framework of Dempster–Shafer theory. Constructing neural network with ordered partitions as the base learner, the learning procedure of the Bagging-based ensemble model is detailed. Experiments on Yantai Red Fuji apples demonstrate the satisfactory grading performances of proposed evidential ensemble model for ordinal classification. © 2022 The Authors. CAAI Transactions on Intelligence Technology published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology and Chongqing University of Technology., open access

Large-scale retrieval is to recall relevant documents from a huge collection given a query. It relies on representation learning to embed documents and queries into a common semantic encoding space. According to the encoding space, recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. These two paradigms unveil the PLMs' representation capability in different granularities, i.e., global sequence-level compression and local word-level contexts, respectively. Inspired by their complementary global-local contextualization and distinct representing views, we propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability. Experiments on passage retrieval benchmarks verify its effectiveness in both paradigms. A uni-retrieval scheme is further presented with even better retrieval quality. We lastly evaluate the model on BEIR benchmark to verify its transferability., Comment: To appear at KDD ADS 2023

Generating new events given context with correlated ones plays a crucial role in many event-centric reasoning tasks. Existing works either limit their scope to specific scenarios or overlook event-level correlations. In this paper, we propose to pre-train a general Correlation-aware context-to-Event Transformer (ClarET) for event-centric reasoning. To achieve this, we propose three novel event-centric objectives, i.e., whole event recovering, contrastive event-correlation encoding and prompt-based event locating, which highlight event-level correlations with effective training. The proposed ClarET is applicable to a wide range of event-centric reasoning scenarios, considering its versatility of (i) event-correlation types (e.g., causal, temporal, contrast), (ii) application formulations (i.e., generation and classification), and (iii) reasoning types (e.g., abductive, counterfactual and ending reasoning). Empirical fine-tuning results, as well as zero- and few-shot learning, on 9 benchmarks (5 generation and 4 classification tasks covering 4 reasoning types with diverse event correlations), verify its effectiveness and generalization ability., Comment: ACL 2022 camera-ready version

Learning sentence embeddings in an unsupervised manner is fundamental in natural language processing. Recent common practice is to couple pre-trained language models with unsupervised contrastive learning, whose success relies on augmenting a sentence with a semantically-close positive instance to construct contrastive pairs. Nonetheless, existing approaches usually depend on a mono-augmenting strategy, which causes learning shortcuts towards the augmenting biases and thus corrupts the quality of sentence embeddings. A straightforward solution is resorting to more diverse positives from a multi-augmenting strategy, while an open question remains about how to unsupervisedly learn from the diverse positives but with uneven augmenting qualities in the text field. As one answer, we propose a novel Peer-Contrastive Learning (PCL) with diverse augmentations. PCL constructs diverse contrastive positives and negatives at the group level for unsupervised sentence embeddings. PCL performs peer-positive contrast as well as peer-network cooperation, which offers an inherent anti-bias ability and an effective way to learn from diverse augmentations. Experiments on STS benchmarks verify the effectiveness of PCL against its competitors in unsupervised sentence embeddings., Comment: To appear at EMNLP 2022

Applying knowledge distillation to personalized cross-silo federated learning can well alleviate the problem of user heterogeneity. This approach, however, requires a proxy dataset, which is difficult to obtain in the real world. Moreover, the global model based on parameter averaging will lead to the leakage of user privacy. We introduce a distributed three-player GAN to implement datafree co-distillation between clients. This technique mitigates the user heterogeneity problem and better protects user privacy. We confirmed that thefake samples generated by GAN can make federated distillation more efficient and robust, and the co-distillation can achieve good performance for individual clients on the basis of obtaining global knowledge. Our extensive experiments on benchmark datasets demonstrate the superior generalization performance of the proposed methods, compared with the state-of-the-art.

This paper focuses on the data augmentation for low-resource NLP tasks where the training set is limited. The existing solutions either leverage task-independent heuristic rules (e.g., Synonym Replacement) or fine-tune general-purpose pre-trained language models (e.g., GPT2) using the limited training instances to produce new synthetic data. Consequently, they have trivial task-specific knowledge and are limited to yielding low-quality synthetic data. To combat this issue, we propose Knowledge Mixture Data Augmentation Model (KnowDA) which is an Seq2Seq language model pre-trained on a mixture of diverse NLP tasks under a novel framework of Knowledge Mixture Training (KoMT). The goal of KoMT is to condense diverse NLP task-specific knowledge into the single KnowDA model (i.e., all-in-one) such that KnowDA could utilize these knowledge to quickly grasp the inherent synthesis law of the target task through limited training instances. Specifically, KoMT reformulates input examples from various heterogeneous NLP tasks into a unified text-to-text format, and employs denoising training objectives in different granularity to learn to reconstruct partial or complete samples. To the best of our knowledge, we are the first attempt to apply 100+ NLP multi-task training for data augmentation. Extensive experiments show that i) the synthetic data produced by KnowDA successfully improves performance of the strong pre-trained language models (i.e., Bert, ALBert and Deberta) by a large margin on the low-resource NLP benchmark FewGLUE, CoNLL'03 and WikiAnn; ii) KnowDA successfully transfers the task knowledge to NLP tasks whose types are seen and unseen in KoMT., Comment: Accepted by ICLR 2023 main track at https://openreview.net/forum?id=2nocgE1m0A