Search

Your search keyword '"Collier, Nigel"' showing total 71 results
Start Over Author "Collier, Nigel" Publisher association for computational linguistics
71 results on '"Collier, Nigel"'

6. Prix-LM: Pretraining for Multilingual Knowledge Base Construction

Author

Zhou, Wenxuan, Liu, Fangyu, Vulić, Ivan, Collier, Nigel, and Chen, Muhao

Subjects
FOS: Computer and information sciences, Computer Science - Computation and Language, Computation and Language (cs.CL)
Abstract

Knowledge bases (KBs) contain plenty of structured world and commonsense knowledge. As such, they often complement distributional text-based information and facilitate various downstream tasks. Since their manual construction is resource- and time-intensive, recent efforts have tried leveraging large pretrained language models (PLMs) to generate additional monolingual knowledge facts for KBs. However, such methods have not been attempted for building and enriching multilingual KBs. Besides wider application, such multilingual KBs can provide richer combined knowledge than monolingual (e.g., English) KBs. Knowledge expressed in different languages may be complementary and unequally distributed: this implies that the knowledge available in high-resource languages can be transferred to low-resource ones. To achieve this, it is crucial to represent multilingual knowledge in a shared/unified space. To this end, we propose a unified representation model, Prix-LM, for multilingual KB construction and completion. We leverage two types of knowledge, monolingual triples and cross-lingual links, extracted from existing multilingual KBs, and tune a multilingual language encoder XLM-R via a causal language modeling objective. Prix-LM integrates useful multilingual and KB-based factual knowledge into a single model. Experiments on standard entity-related tasks, such as link prediction in multiple languages, cross-lingual entity linking and bilingual lexicon induction, demonstrate its effectiveness, with gains reported over strong task-specialised baselines., Comment: ACL 2022

Published
2023
Full Text
View/download PDF

7. Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT

Author

Meng, Zaiqiao, Liu, Fangyu, Clark, Thomas, Shareghi, Ehsan, and Collier, Nigel

Subjects
FOS: Computer and information sciences, Computer Science - Computation and Language, cs.CL, Computation and Language (cs.CL)
Abstract

Infusing factual knowledge into pre-trained models is fundamental for many knowledge-intensive tasks. In this paper, we proposed Mixture-of-Partitions (MoP), an infusion approach that can handle a very large knowledge graph (KG) by partitioning it into smaller sub-graphs and infusing their specific knowledge into various BERT models using lightweight adapters. To leverage the overall factual knowledge for a target task, these sub-graph adapters are further fine-tuned along with the underlying BERT through a mixture layer. We evaluate our MoP with three biomedical BERTs (SciBERT, BioBERT, PubmedBERT) on six downstream tasks (inc. NLI, QA, Classification), and the results show that our MoP consistently enhances the underlying BERTs in task performance, and achieves new SOTA performances on five evaluated datasets., Comment: EMNLP 2021 camera-ready version

Published
2022
Full Text
View/download PDF

8. Plan-then-Generate: Controlled Data-to-Text Generation via Planning

Author

Su, Yixuan, Vandyke, David, Wang, Sihui, Fang, Yimai, and Collier, Nigel

Subjects
FOS: Computer and information sciences, Computer Science - Computation and Language, Computation and Language (cs.CL)
Abstract

Recent developments in neural networks have led to the advance in data-to-text generation. However, the lack of ability of neural models to control the structure of generated output can be limiting in certain real-world applications. In this study, we propose a novel Plan-then-Generate (PlanGen) framework to improve the controllability of neural data-to-text models. Extensive experiments and analyses are conducted on two benchmark datasets, ToTTo and WebNLG. The results show that our model is able to control both the intra-sentence and inter-sentence structure of the generated output. Furthermore, empirical comparisons against previous state-of-the-art methods show that our model improves the generation quality as well as the output diversity as judged by human and automatic evaluations., Comment: Accepted to Findings of EMNLP 2021

Published
2022
Full Text
View/download PDF

9. Improving Word Translation via Two-Stage Contrastive Learning

Author

Li, Yaoyiran, Liu, Fangyu, Collier, Nigel, Korhonen, Anna, and Vulić, Ivan

Subjects
FOS: Computer and information sciences, Computer Science - Machine Learning, Artificial Intelligence (cs.AI), Computer Science - Computation and Language, Computer Science - Artificial Intelligence, Computation and Language (cs.CL), Information Retrieval (cs.IR), Machine Learning (cs.LG), Computer Science - Information Retrieval
Abstract

Word translation or bilingual lexicon induction (BLI) is a key cross-lingual task, aiming to bridge the lexical gap between different languages. In this work, we propose a robust and effective two-stage contrastive learning framework for the BLI task. At Stage C1, we propose to refine standard cross-lingual linear maps between static word embeddings (WEs) via a contrastive learning objective; we also show how to integrate it into the self-learning procedure for even more refined cross-lingual maps. In Stage C2, we conduct BLI-oriented contrastive fine-tuning of mBERT, unlocking its word translation capability. We also show that static WEs induced from the `C2-tuned' mBERT complement static WEs from Stage C1. Comprehensive experiments on standard BLI datasets for diverse languages and different experimental setups demonstrate substantial gains achieved by our framework. While the BLI method from Stage C1 already yields substantial gains over all state-of-the-art BLI methods in our comparison, even stronger improvements are met with the full two-stage framework: e.g., we report gains for 112/112 BLI setups, spanning 28 language pairs., ACL 2022 Main

Published
2022

15. Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders

Author

Liu, Fangyu, Vulic, Ivan, Korhonen, Anna, and Collier, Nigel

Subjects
FOS: Computer and information sciences, Computer Science - Machine Learning, Computer Science - Computation and Language, Computer Science - Artificial Intelligence, cs.LG, 4704 Linguistics, cs.CL, cs.AI, Machine Learning (cs.LG), Artificial Intelligence (cs.AI), 52 Psychology, 5204 Cognitive and Computational Psychology, 47 Language, Communication and Culture, Computation and Language (cs.CL)
Abstract

Pretrained Masked Language Models (MLMs) have revolutionised NLP in recent years. However, previous work has indicated that off-the-shelf MLMs are not effective as universal lexical or sentence encoders without further task-specific fine-tuning on NLI, sentence similarity, or paraphrasing tasks using annotated task data. In this work, we demonstrate that it is possible to turn MLMs into effective universal lexical and sentence encoders even without any additional data and without any supervision. We propose an extremely simple, fast and effective contrastive learning technique, termed Mirror-BERT, which converts MLMs (e.g., BERT and RoBERTa) into such encoders in 20-30 seconds without any additional external knowledge. Mirror-BERT relies on fully identical or slightly modified string pairs as positive (i.e., synonymous) fine-tuning examples, and aims to maximise their similarity during identity fine-tuning. We report huge gains over off-the-shelf MLMs with Mirror-BERT in both lexical-level and sentence-level tasks, across different domains and different languages. Notably, in the standard sentence semantic similarity (STS) tasks, our self-supervised Mirror-BERT model even matches the performance of the task-tuned Sentence-BERT models from prior work. Finally, we delve deeper into the inner workings of MLMs, and suggest some evidence on why this simple approach can yield effective universal lexical and sentence encoders., EMNLP 2021 camera-ready version

Published
2021

Catalog

Books, media, physical & digital resources
See catalog results