Search

Mixture-of-Experts (MoE) has emerged as a prominent architecture for scaling model size while maintaining computational efficiency. In MoE, each token in the input sequence activates a different subset of experts determined by a routing mechanism. However, the unchosen experts in MoE models do not contribute to the output, potentially leading to underutilization of the model's capacity. In this work, we first conduct exploratory studies to demonstrate that increasing the number of activated experts does not necessarily improve and can even degrade the output quality. Then, we show that output distributions from an MoE model using different routing strategies substantially differ, indicating that different experts do not always act synergistically. Motivated by these findings, we propose Self-Contrast Mixture-of-Experts (SCMoE), a training-free strategy that utilizes unchosen experts in a self-contrast manner during inference. In SCMoE, the next-token probabilities are determined by contrasting the outputs from strong and weak activation using the same MoE model. Our method is conceptually simple and computationally lightweight, as it incurs minimal latency compared to greedy decoding. Experiments on several benchmarks (GSM8K, StrategyQA, MBPP and HumanEval) demonstrate that SCMoE can consistently enhance Mixtral 8x7B's reasoning capability across various domains. For example, it improves the accuracy on GSM8K from 61.79 to 66.94. Moreover, combining SCMoE with self-consistency yields additional gains, increasing major@20 accuracy from 75.59 to 78.31.

Question answering plays a pivotal role in human daily life because it involves our acquisition of knowledge about the world. However, due to the dynamic and ever-changing nature of real-world facts, the answer can be completely different when the time constraint in the question changes. Recently, Large Language Models (LLMs) have shown remarkable intelligence in question answering, while our experiments reveal that the aforementioned problems still pose a significant challenge to existing LLMs. This can be attributed to the LLMs' inability to perform rigorous reasoning based on surface-level text semantics. To overcome this limitation, rather than requiring LLMs to directly answer the question, we propose a novel approach where we reframe the $\textbf{Q}$uestion $\textbf{A}$nswering task $\textbf{a}$s $\textbf{P}$rogramming ($\textbf{QAaP}$). Concretely, by leveraging modern LLMs' superior capability in understanding both natural language and programming language, we endeavor to harness LLMs to represent diversely expressed text as well-structured code and select the best matching answer from multiple candidates through programming. We evaluate our QAaP framework on several time-sensitive question answering datasets and achieve decent improvement, up to $14.5$% over strong baselines. Our codes and data are available at https://github.com/TianHongZXY/qaap, Comment: Accepted to EMNLP 2023 Main Conference

Large language models with instruction-following abilities have revolutionized the field of artificial intelligence. These models show exceptional generalizability to tackle various real-world tasks through their natural language interfaces. However, their performance heavily relies on high-quality exemplar data, which is often difficult to obtain. This challenge is further exacerbated when it comes to multimodal instruction following. We introduce TextBind, an almost annotation-free framework for empowering larger language models with the multi-turn interleaved multimodal instruction-following capabilities. Our approach requires only image-caption pairs and generates multi-turn multimodal instruction-response conversations from a language model. To accommodate interleaved image-text inputs and outputs, we devise MIM, a language model-centric architecture that seamlessly integrates image encoder and decoder models. We release our dataset, model, and demo to foster future research in the area of multimodal instruction following., Comment: Findings of ACL 2024

Hot news is one of the most popular topics in daily conversations. However, news grounded conversation has long been stymied by the lack of well-designed task definition and scarce data. In this paper, we propose a novel task, Proactive News Grounded Conversation, in which a dialogue system can proactively lead the conversation based on some key topics of the news. In addition, both information-seeking and chit-chat scenarios are included realistically, where the user may ask a series of questions about the news details or express their opinions and be eager to chat. To further develop this novel task, we collect a human-to-human Chinese dialogue dataset \ts{NewsDialogues}, which includes 1K conversations with a total of 14.6K utterances and detailed annotations for target topics and knowledge spans. Furthermore, we propose a method named Predict-Generate-Rank, consisting of a generator for grounded knowledge prediction and response generation, and a ranker for the ranking of multiple responses to alleviate the exposure bias. We conduct comprehensive experiments to demonstrate the effectiveness of the proposed method and further present several key findings and challenges to prompt future research., Comment: Accepted to ACL 2023 Conference (Long Paper; Findings)

Information-seeking conversation, which aims to help users gather information through conversation, has achieved great progress in recent years. However, the research is still stymied by the scarcity of training data. To alleviate this problem, we propose AutoConv for synthetic conversation generation, which takes advantage of the few-shot learning ability and generation capacity of large language models (LLM). Specifically, we formulate the conversation generation problem as a language modeling task, then finetune an LLM with a few human conversations to capture the characteristics of the information-seeking process and use it for generating synthetic conversations with high quality. Experimental results on two frequently-used datasets verify that AutoConv has substantial improvements over strong baselines and alleviates the dependence on human annotation. In addition, we also provide several analysis studies to promote future research., Comment: Accepted to ACL 2023 Main Conference (Short)

The top-down and bottom-up methods are two mainstreams of referring segmentation, while both methods have their own intrinsic weaknesses. Top-down methods are chiefly disturbed by Polar Negative (PN) errors owing to the lack of fine-grained cross-modal alignment. Bottom-up methods are mainly perturbed by Inferior Positive (IP) errors due to the lack of prior object information. Nevertheless, we discover that two types of methods are highly complementary for restraining respective weaknesses but the direct average combination leads to harmful interference. In this context, we build Win-win Cooperation (WiCo) to exploit complementary nature of two types of methods on both interaction and integration aspects for achieving a win-win improvement. For the interaction aspect, Complementary Feature Interaction (CFI) provides fine-grained information to top-down branch and introduces prior object information to bottom-up branch for complementary feature enhancement. For the integration aspect, Gaussian Scoring Integration (GSI) models the gaussian performance distributions of two branches and weightedly integrates results by sampling confident scores from the distributions. With our WiCo, several prominent top-down and bottom-up combinations achieve remarkable improvements on three common datasets with reasonable extra costs, which justifies effectiveness and generality of our method., Comment: Accepted to IJCAI2023

Weakly supervised semantic segmentation is typically inspired by class activation maps, which serve as pseudo masks with class-discriminative regions highlighted. Although tremendous efforts have been made to recall precise and complete locations for each class, existing methods still commonly suffer from the unsolicited Out-of-Candidate (OC) error predictions that not belongs to the label candidates, which could be avoidable since the contradiction with image-level class tags is easy to be detected. In this paper, we develop a group ranking-based Out-of-Candidate Rectification (OCR) mechanism in a plug-and-play fashion. Firstly, we adaptively split the semantic categories into In-Candidate (IC) and OC groups for each OC pixel according to their prior annotation correlation and posterior prediction correlation. Then, we derive a differentiable rectification loss to force OC pixels to shift to the IC group. Incorporating our OCR with seminal baselines (e.g., AffinityNet, SEAM, MCTformer), we can achieve remarkable performance gains on both Pascal VOC (+3.2%, +3.3%, +0.8% mIoU) and MS COCO (+1.0%, +1.3%, +0.5% mIoU) datasets with negligible extra training overhead, which justifies the effectiveness and generality of our OCR., Comment: Accepted to CVPR2023

Zero-shot entity retrieval, aiming to link mentions to candidate entities under the zero-shot setting, is vital for many tasks in Natural Language Processing. Most existing methods represent mentions/entities via the sentence embeddings of corresponding context from the Pre-trained Language Model. However, we argue that such coarse-grained sentence embeddings can not fully model the mentions/entities, especially when the attention scores towards mentions/entities are relatively low. In this work, we propose GER, a \textbf{G}raph enhanced \textbf{E}ntity \textbf{R}etrieval framework, to capture more fine-grained information as complementary to sentence embeddings. We extract the knowledge units from the corresponding context and then construct a mention/entity centralized graph. Hence, we can learn the fine-grained information about mention/entity by aggregating information from these knowledge units. To avoid the graph information bottleneck for the central mention/entity node, we construct a hierarchical graph and design a novel Hierarchical Graph Attention Network~(HGAN). Experimental results on popular benchmarks demonstrate that our proposed GER framework performs better than previous state-of-the-art models. The code has been available at https://github.com/wutaiqiang/GER-WSDM2023., Comment: 9 pages, 5 figures

Chinese Spelling Correction (CSC) is gaining increasing attention due to its promise of automatically detecting and correcting spelling errors in Chinese texts. Despite its extensive use in many applications, like search engines and optical character recognition systems, little has been explored in medical scenarios in which complex and uncommon medical entities are easily misspelled. Correcting the misspellings of medical entities is arguably more difficult than those in the open domain due to its requirements of specificdomain knowledge. In this work, we define the task of Medical-domain Chinese Spelling Correction and propose MCSCSet, a large scale specialist-annotated dataset that contains about 200k samples. In contrast to the existing open-domain CSC datasets, MCSCSet involves: i) extensive real-world medical queries collected from Tencent Yidian, ii) corresponding misspelled sentences manually annotated by medical specialists. To ensure automated dataset curation, MCSCSet further offers a medical confusion set consisting of the commonly misspelled characters of given Chinese medical terms. This enables one to create the medical misspelling dataset automatically. Extensive empirical studies have shown significant performance gaps between the open-domain and medical-domain spelling correction, highlighting the need to develop high-quality datasets that allow for Chinese spelling correction in specific domains. Moreover, our work benchmarks several representative Chinese spelling correction models, establishing baselines for future work., Comment: The full version of CIKM 2022 accepted resource paper "MCSCSet: A Specialist-annotated Dataset for Medical-domain Chinese Spelling Correction". (https://dl.acm.org/doi/10.1145/3511808.3557636)

Training multiple agents to perform safe and cooperative control in the complex scenarios of autonomous driving has been a challenge. For a small fleet of cars moving together, this paper proposes Lepus, a new approach to training multiple agents. Lepus adopts a pure cooperative manner for training multiple agents, featured with the shared parameters of policy networks and the shared reward function of multiple agents. In particular, Lepus pre-trains the policy networks via an adversarial process, improving its collaborative decision-making capability and further the stability of car driving. Moreover, for alleviating the problem of sparse rewards, Lepus learns an approximate reward function from expert trajectories by combining a random network and a distillation network. We conduct extensive experiments on the MADRaS simulation platform. The experimental results show that multiple agents trained by Lepus can avoid collisions as many as possible while driving simultaneously and outperform the other four methods, that is, DDPG-FDE, PSDDPG, MADDPG, and MAGAIL(DDPG) in terms of stability., Comment: 8pages, IJCNN2022, Accepted

In empathetic conversations, humans express their empathy to others with empathetic intents. However, most existing empathetic conversational methods suffer from a lack of empathetic intents, which leads to monotonous empathy. To address the bias of the empathetic intents distribution between empathetic dialogue models and humans, we propose a novel model to generate empathetic responses with human-consistent empathetic intents, EmpHi for short. Precisely, EmpHi learns the distribution of potential empathetic intents with a discrete latent variable, then combines both implicit and explicit intent representation to generate responses with various empathetic intents. Experiments show that EmpHi outperforms state-of-the-art models in terms of empathy, relevance, and diversity on both automatic and human evaluation. Moreover, the case studies demonstrate the high interpretability and outstanding performance of our model., Comment: Accepted to NAACL 2022

Lei Yang,1,2 Weimin Dang,2,3 Xin Yu,1,2 Yi He,4,5 Chuan Shi,1,2 Jialong Yi,1,2 Lu Gao,6 Siheng Li,7 Tianhang Zhou,1,2 Yantao Ma1,2 1Department of Clinical Research, Peking University Sixth Hospital (Institute of Mental Health), Beijing, People’s Republic of China; 2NHC Key Laboratory of Mental Health (Peking University), National Clinical Research Center for Mental Disorders (Peking University Sixth Hospital), Beijing, People’s Republic of China; 3The Division of Medical Affairs, Peking University Sixth Hospital (Institute of Mental Health), Beijing, People’s Republic of China; 4Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Beijing, People’s Republic of China; 5Department of Psycho-Oncology, Peking University Cancer Hospital & Institute, Beijing, People’s Republic of China; 6Department of Stomatology, Beijing Children’s Hospital, Capital Medical University, Beijing, People’s Republic of China; 7Department of Neurology, Peking University First Hospital, Beijing, People’s Republic of ChinaCorrespondence: Xin Yu, Peking University Sixth Hospital, No. 51 Hua Yuan Bei Road, Haidian District, Beijing, 100191, People’s Republic of China, Tel +86 13718747842, Fax +86-10-62026310, Email yuxin@bjmu.edu.cnIntroduction: Processed betel quid product chewing is a public health problem in areca non-plant areas in China. However, there is no valid instrument to screen for betel quid use disorder (BQUD) in mainland China. We developed a self-administered screening test for betel quid use disorders (SST-BQUD) and tested its reliability and validity in a sample of betel quid chewers (BQCers) in Hunan, China.Methods: Items of SST-BQUD were selected from the test results of an item pool, which includes 52 questions related to the psycho-social and behavioral presentations of BQUD. All participants, in a self-administered manner, completed the item pool. A subsample completed the re-test one week later. Two psychiatrists interv

Quan Chen,1 Siheng Li,2 Zixiong Feng,1 Meng Wang,3 Chengzhi Cai,2 Jufang Wang,3 Lijuan Zhang1 1School of Chemistry and Chemical Engineering, South China University of Technology, Guangzhou, People’s Republic of China; 2Department of Chemistry, University of Houston, Houston, TX, USA; 3School of Bioscience & Bioengineering, South China University of Technology, Guangzhou, People’s Republic of China Abstract: We have demonstrated a novel drug delivery system to improve the selectivity of the current chemotherapy by pH-responsive, polymeric micelle carriers. The micelle carriers were prepared by the self-assembly of copolymers containing the polybasic poly(2-(diethylamino)ethyl methacrylate) (PDEAEMA) block. The mixed copolymers exhibited a comparatively low critical micelle concentration (CMC; 1.95–5.25 mg/L). The resultant mixed micelles were found to be <100 nm and were used to encapsulate the anticancer drug doxorubicin (DOX) with pretty good drug-loading content (24%) and entrapment efficiency (55%). Most importantly, the micelle carrier exhibited a pH-dependent conformational conversion and promoted the DOX release at the tumorous pH. Our in vitro studies demonstrated the comparable level of DOX-loaded mixed micelle delivery into tumor cells with the free DOX (80% of the tumor cells were killed after 48 h incubation). The DOX-loaded mixed micelles were effective to inhibit the proliferation of tumor cells after prolonged incubation. Overall, the pH-responsive mixed micelle system provided desirable potential in the controlled release of anticancer therapeutics. Keywords: PDEAEMA, copolymers, pH-responsive, mixed micelle, DOX, targeting delivery

Quan Chen,1 Siheng Li,2 Zixiong Feng,1 Meng Wang,3 Chengzhi Cai,2 Jufang Wang,3 Lijuan Zhang1 1School of Chemistry and Chemical Engineering, South China University of Technology, Guangzhou, People’s Republic of China; 2Department of Chemistry, University of Houston, Houston, TX, USA; 3School of Bioscience & Bioengineering, South China University of Technology, Guangzhou, People’s Republic of China Abstract: We have demonstrated a novel drug delivery system to improve the selectivity of the current chemotherapy by pH-responsive, polymeric micelle carriers. The micelle carriers were prepared by the self-assembly of copolymers containing the polybasic poly(2-(diethylamino)ethyl methacrylate) (PDEAEMA) block. The mixed copolymers exhibited a comparatively low critical micelle concentration (CMC; 1.95–5.25 mg/L). The resultant mixed micelles were found to be <100 nm and were used to encapsulate the anticancer drug doxorubicin (DOX) with pretty good drug-loading content (24%) and entrapment efficiency (55%). Most importantly, the micelle carrier exhibited a pH-dependent conformational conversion and promoted the DOX release at the tumorous pH. Our in vitro studies demonstrated the comparable level of DOX-loaded mixed micelle delivery into tumor cells with the free DOX (80% of the tumor cells were killed after 48 h incubation). The DOX-loaded mixed micelles were effective to inhibit the proliferation of tumor cells after prolonged incubation. Overall, the pH-responsive mixed micelle system provided desirable potential in the controlled release of anticancer therapeutics. Keywords: PDEAEMA, copolymers, pH-responsive, mixed micelle, DOX, targeting delivery