Search

Your search keyword '"Liu, Qun"' showing total 218 results
Start Over Author "Liu, Qun" Publisher association for computational linguistics

Refine your results

more »

more »

218 results on '"Liu, Qun"'

10. TRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Models

Author

Xiong, Jing, primary, Shen, Jianhao, additional, Yuan, Ye, additional, Wang, Haiming, additional, Yin, Yichun, additional, Liu, Zhengying, additional, Li, Lin, additional, Guo, Zhijiang, additional, Cao, Qingxing, additional, Huang, Yinya, additional, Zheng, Chuanyang, additional, Liang, Xiaodan, additional, Zhang, Ming, additional, and Liu, Qun, additional

Published
2023
Full Text
View/download PDF

15. Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

Author

Dai, Wenliang, Hou, Lu, Shang, Lifeng, Jiang, Xin, Liu, Qun, Fung, Pascale Ngan, Dai, Wenliang, Hou, Lu, Shang, Lifeng, Jiang, Xin, Liu, Qun, and Fung, Pascale Ngan

Abstract

The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficient compared to the pre-training from scratch. Experimental results show that the resulting model has strong zero-shot performance on multimodal generation tasks, such as open-ended visual question answering and image captioning. For example, it achieves 44.5% zero-shot accuracy on the VQAv2 dataset, surpassing the previous state-of-the-art zero-shot model with 7× fewer parameters. Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.

Published
2022

16. Controlled text generation using dictionary prior in variational autoencoders

Author

Fang, Xianghong, Li, Jian, Shang, Lifeng, Jiang, Xin, Liu, Qun, Yeung, Dit Yan, Fang, Xianghong, Li, Jian, Shang, Lifeng, Jiang, Xin, Liu, Qun, and Yeung, Dit Yan

Abstract

While variational autoencoders (VAEs) have been widely applied in text generation tasks, they are troubled by two challenges: insufficient representation capacity and poor controllability. The former results from the posterior collapse and restrictive assumption, which impede better representation learning. The latter arises as continuous latent variables in traditional formulations hinder VAEs from interpretability and controllability. In this paper, we propose Dictionary Prior (DPrior), a new data-driven prior that enjoys the merits of expressivity and controllability. To facilitate controlled text generation with DPrior, we propose to employ contrastive learning to separate the latent space into several parts. Extensive experiments on both language modeling and controlled text generation demonstrate the effectiveness of the proposed approach.

Published
2022

17. Read before Generate! Faithful Long Form Question Answering with Machine Reading

Author

Su, Dan, Li, Xiaoguang, Zhang, Jindi, Shang, Lifeng, Jiang, Xin, Liu, Qun, Fung, Pascale Ngan, Su, Dan, Li, Xiaoguang, Zhang, Jindi, Shang, Lifeng, Jiang, Xin, Liu, Qun, and Fung, Pascale Ngan

Abstract

Long-form question answering (LFQA) aims to generate a paragraph-length answer for a given question. While current work on LFQA using large pre-trained model for generation are effective at producing fluent and somewhat relevant content, one primary challenge lies in how to generate a faithful answer that has less hallucinated content. We propose a new end-to-end framework that jointly models answer generation and machine reading. The key idea is to augment the generation model with fine-grained, answer-related salient information which can be viewed as an emphasis on faithful facts. State-of-the-art results on two LFQA datasets, ELI5 and MS MARCO, demonstrate the effectiveness of our method, in comparison with strong baselines on automatic and human evaluation metrics. A detailed analysis further proves the competency of our methods in generating fluent, relevant, and more faithful answers.

Published
2022

18. Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answering

Author

Zhou, Jiawei, Li, Xiaoguang, Shang, Lifeng, Luo, Lan, Zhan, Ke, Hu, Enrui, Zhang, Xinyu, Jiang, Hao, Cao, Zhao, Yu, Fan, Jiang, Xin, Liu, Qun, and Chen, Lei

Subjects
FOS: Computer and information sciences, Computer Science - Computation and Language, Computation and Language (cs.CL), Information Retrieval (cs.IR), Computer Science - Information Retrieval
Abstract

To alleviate the data scarcity problem in training question answering systems, recent works propose additional intermediate pre-training for dense passage retrieval (DPR). However, there still remains a large discrepancy between the provided upstream signals and the downstream question-passage relevance, which leads to less improvement. To bridge this gap, we propose the HyperLink-induced Pre-training (HLP), a method to pre-train the dense retriever with the text relevance induced by hyperlink-based topology within Web documents. We demonstrate that the hyperlink-based structures of dual-link and co-mention can provide effective relevance signals for large-scale pre-training that better facilitate downstream passage retrieval. We investigate the effectiveness of our approach across a wide range of open-domain QA datasets under zero-shot, few-shot, multi-hop, and out-of-domain scenarios. The experiments show our HLP outperforms the BM25 by up to 7 points as well as other pre-training methods by more than 10 points in terms of top-20 retrieval accuracy under the zero-shot scenario. Furthermore, HLP significantly outperforms other pre-training methods under the other scenarios., Accepted by ACL 2022 main conference; The dataset and code are available at https://github.com/jzhoubu/HLP

Published
2022

19. Compression of Generative Pre-trained Language Models via Quantization

Author

Tao, Chaofan, Hou, Lu, Zhang, Wei, Shang, Lifeng, Jiang, Xin, Liu, Qun, Luo, Ping, and Wong, Ngai

Subjects
FOS: Computer and information sciences, Computer Science - Computation and Language, Computer Vision and Pattern Recognition (cs.CV), Computer Science - Computer Vision and Pattern Recognition, Data_CODINGANDINFORMATIONTHEORY, Computation and Language (cs.CL)
Abstract

The increasing size of generative Pre-trained Language Models (PLMs) has greatly increased the demand for model compression. Despite various methods to compress BERT or its variants, there are few attempts to compress generative PLMs, and the underlying difficulty remains unclear. In this paper, we compress generative PLMs by quantization. We find that previous quantization methods fail on generative tasks due to the \textit{homogeneous word embeddings} caused by reduced capacity, and \textit{varied distribution of weights}. Correspondingly, we propose a token-level contrastive distillation to learn distinguishable word embeddings, and a module-wise dynamic scaling to make quantizers adaptive to different modules. Empirical results on various tasks show that our proposed method outperforms the state-of-the-art compression methods on generative PLMs by a clear margin. With comparable performance with the full-precision models, we achieve 14.4x and 13.4x compression rates on GPT-2 and BART, respectively., ACL 2022

Published
2022

31. bert2BERT: Towards Reusable Pretrained Language Models

Author

Chen, Cheng, primary, Yin, Yichun, additional, Shang, Lifeng, additional, Jiang, Xin, additional, Qin, Yujia, additional, Wang, Fengyu, additional, Wang, Zhi, additional, Chen, Xiao, additional, Liu, Zhiyuan, additional, and Liu, Qun, additional

Published
2022
Full Text
View/download PDF

40. Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Processing

Author

Ghaddar, Abbas, primary, Wu, Yimeng, additional, Bagga, Sunyam, additional, Rashid, Ahmad, additional, Bibi, Khalil, additional, Rezagholizadeh, Mehdi, additional, Xing, Chao, additional, Wang, Yasheng, additional, Duan, Xinyu, additional, Wang, Zhefeng, additional, Huai, Baoxing, additional, Jiang, Xin, additional, Liu, Qun, additional, and Langlais, Phillippe, additional

Published
2022
Full Text
View/download PDF

45. Improving unsupervised question answering via summarization-informed question generation

Author

Lyu, Chenyang, Shang, Lifeng, Graham, Yvette, Foster, Jennifer, Jiang, Xin, Liu, Qun, Lyu, Chenyang, Shang, Lifeng, Graham, Yvette, Foster, Jennifer, Jiang, Xin, and Liu, Qun

Abstract

Question Generation (QG) is the task of generating a plausible question for a given \textlesspassage, answer\textgreater pair. Template-based QG uses linguistically-informed heuristics to transform declarative sentences into interrogatives, whereas supervised QG uses existing Question Answering (QA) datasets to train a system to generate a question given a passage and an answer. A disadvantage of the heuristic approach is that the generated questions are heavily tied to their declarative counterparts. A disadvantage of the supervised approach is that they are heavily tied to the domain/language of the QA dataset used as training data. In order to overcome these shortcomings, we propose a distantly-supervised QG method which uses questions generated heuristically from summaries as a source of training data for a QG system. We make use of freely available news summary data, transforming declarative summary sentences into appropriate questions using heuristics informed by dependency parsing, named entity recognition and semantic role labeling. The resulting questions are then combined with the original news articles to train an end-to-end neural QG model. We extrinsically evaluate our approach using unsupervised QA: our QG model is used to generate synthetic QA pairs for training a QA model. Experimental results show that, trained with only 20k English Wikipedia-based synthetic QA pairs, the QA model substantially outperforms previous unsupervised models on three in-domain datasets (SQuAD1.1, Natural Questions, TriviaQA) and three out-of-domain datasets (NewsQA, BioASQ, DuoRC), demonstrating the transferability of the approach.

Published
2021
Full Text
View/download PDF

46. DyLex: Incorporating Dynamic Lexicons into BERT for Sequence Labeling

Author

Wang, Baojun, Zhang, Zhao, Xu, Kun, Hao, Guangyuan, Zhang, Yuyang, Shang, Lifeng, Li, Linlin, Chen, Xiao, Jiang, Xiao, Liu, Qun, Wang, Baojun, Zhang, Zhao, Xu, Kun, Hao, Guangyuan, Zhang, Yuyang, Shang, Lifeng, Li, Linlin, Chen, Xiao, Jiang, Xiao, and Liu, Qun

Abstract

Incorporating lexical knowledge into deep learning models has been proved to be very effective for sequence labeling tasks. However, previous works commonly have difficulty dealing with large-scale dynamic lexicons which often cause excessive matching noise and problems of frequent updates. In this paper, we propose DyLex, a plug-in lexicon incorporation approach for BERT based sequence labeling tasks. Instead of leveraging embeddings of words in the lexicon as in conventional methods, we adopt word-agnostic tag embeddings to avoid re-training the representation while updating the lexicon. Moreover, we employ an effective supervised lexical knowledge denoising method to smooth out matching noise. Finally, we introduce a col-wise attention based knowledge fusion mechanism to guarantee the pluggability of the proposed framework. Experiments on ten datasets of three tasks show that the proposed framework achieves new SOTA, even with very large scale lexicons. © 2021 Association for Computational Linguistics

Published
2021

Catalog

Books, media, physical & digital resources
See catalog results