Search

Pre-trained language models (e.g., BERT (Devlin et al., 2018) and its variants) have achieved remarkable success in varieties of NLP tasks. However, these models usually consist of hundreds of millions of parameters which brings challenges for fine-tuning and online serving in real-life applications due to latency and capacity constraints. In this work, we present a simple and effective approach to compress large Transformer (Vaswani et al., 2017) based pre-trained models, termed as deep self-attention distillation. The small model (student) is trained by deeply mimicking the self-attention module, which plays a vital role in Transformer networks, of the large model (teacher). Specifically, we propose distilling the self-attention module of the last Transformer layer of the teacher, which is effective and flexible for the student. Furthermore, we introduce the scaled dot-product between values in the self-attention module as the new deep self-attention knowledge, in addition to the attention distributions (i.e., the scaled dot-product of queries and keys) that have been used in existing works. Moreover, we show that introducing a teacher assistant (Mirzadeh et al., 2019) also helps the distillation of large pre-trained Transformer models. Experimental results demonstrate that our monolingual model outperforms state-of-the-art baselines in different parameter size of student models. In particular, it retains more than 99% accuracy on SQuAD 2.0 and several GLUE benchmark tasks using 50% of the Transformer parameters and computations of the teacher model. We also obtain competitive results in applying deep self-attention distillation to multilingual pre-trained models., Comment: Code and models: https://github.com/microsoft/unilm/tree/master/minilm

In this work we focus on transferring supervision signals of natural language generation (NLG) tasks between multiple languages. We propose to pretrain the encoder and the decoder of a sequence-to-sequence model under both monolingual and cross-lingual settings. The pre-training objective encourages the model to represent different languages in the shared space, so that we can conduct zero-shot cross-lingual transfer. After the pre-training procedure, we use monolingual data to fine-tune the pre-trained model on downstream NLG tasks. Then the sequence-to-sequence model trained in a single language can be directly evaluated beyond that language (i.e., accepting multi-lingual input and producing multi-lingual output). Experimental results on question generation and abstractive summarization show that our model outperforms the machine-translation-based pipeline methods for zero-shot cross-lingual generation. Moreover, cross-lingual transfer improves NLG performance of low-resource languages by leveraging rich-resource language data. Our implementation and data are available at https://github.com/CZWin32768/xnlg., Comment: Accepted by AAAI-20

A fast silicon-graphene hybrid plasmonic waveguide photodetectors beyond 1.55 {\mu}m is proposed and realized by introducing an ultra-thin wide silicon-on-insulator ridge core region with a narrow metal cap. With this novel design, the light absorption in graphene is enhanced while the metal absorption loss is reduced simultaneously, which helps greatly improve the responsivity as well as shorten the absorption region for achieving fast responses. Furthermore, metal-graphene-metal sandwiched electrodes are introduced to reduce the metal-graphene contact resistance, which is also helpful for improving the response speed. When the photodetector operates at 2 {\mu}m, the measured 3dB-bandwidth is >20 GHz (which is limited by the experimental setup) while the 3dB-bandwith calculated from the equivalent circuit with the parameters extracted from the measured S11 is as high as ~100 GHz. To the best of our knowledge, it is the first time to report the waveguide photodetector at 2 {\mu}m with a 3dB-bandwidth over 20 GHz. Besides, the present photodetectors also work very well at 1.55 {\mu}m. The measured responsivity is about 0.4 A/W under a bias voltage of -0.3 V for an optical power of 0.16 mW, while the measured 3dB-bandwidth is over 40 GHz (limited by the test setup) and the 3 dB-bandwidth estimated from the equivalent circuit is also as high as ~100 GHz, which is one of the best results reported for silicon-graphene photodetectors at 1.55 {\mu}m., Comment: 22 pages, 7 figures, 1 table. Supplementary Materials are available

Atomically thin hexagonal boron nitride (h-BN) is often regarded as an elastic film that is impermeable to gases. The high stabilities in thermal and chemical properties allow h-BN to serve as a gas barrier under extreme conditions.In this work, we demonstrate the isolation of hydrogen in bubbles of h-BN via plasma treatment.Detailed characterizations reveal that the substrates do not show chemical change after treatment. The bubbles are found to withstand thermal treatment in air,even at 800 degree celsius. Scanning transmission electron microscopy investigation shows that the h-BN multilayer has a unique aligned porous stacking nature, which is essential for the character of being transparent to atomic hydrogen but impermeable to hydrogen molecules. We successfully demonstrated the extraction of hydrogen gases from gaseous compounds or mixtures containing hydrogen element. The successful production of hydrogen bubbles on h-BN flakes has potential for further application in nano/micro-electromechanical systems and hydrogen storage., Comment: 55 pages, 33figures

Machine reading comprehension with unanswerable questions is a challenging task. In this work, we propose a data augmentation technique by automatically generating relevant unanswerable questions according to an answerable question paired with its corresponding paragraph that contains the answer. We introduce a pair-to-sequence model for unanswerable question generation, which effectively captures the interactions between the question and the paragraph. We also present a way to construct training data for our question generation models by leveraging the existing reading comprehension dataset. Experimental results show that the pair-to-sequence model performs consistently better compared with the sequence-to-sequence baseline. We further use the automatically generated unanswerable questions as a means of data augmentation on the SQuAD 2.0 dataset, yielding 1.9 absolute F1 improvement with BERT-base model and 1.7 absolute F1 improvement with BERT-large model., Comment: ACL 2019 (long paper)

This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. UniLM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, UniLM achieves new state-of-the-art results on five natural language generation datasets, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.51 (2.04 absolute improvement), the Gigaword abstractive summarization ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7 document-grounded dialog response generation NIST-4 to 2.67 (human performance is 2.65). The code and pre-trained models are available at https://github.com/microsoft/unilm., Comment: Accepted by NeurIPS-19. Code and pre-trained models: https://github.com/microsoft/unilm

Defect induced trap states are essential in determining the performance of semiconductor photodetectors. The de-trap time of carriers from a deep trap could be prolonged by several orders of magnitude as compared to shallow trap, resulting in additional decay/response time of the device. Here, we demonstrate that the trap states in two-dimensional ReS2 could be efficiently modulated by defect engineering through molecule decoration. The deep traps that greatly prolong the response time could be mostly filled by Protoporphyrin (H2PP) molecules. At the same time, carrier recombination and shallow traps would in-turn play dominant roles in determining the decay time of the device, which can be several orders of magnitude faster than the as-prepared device. Moreover, the specific detectivity of the device is enhanced (as high as ~1.89 x 10^13 Jones) due to the significant reduction of dark current through charge transfer between ReS2 and molecules. Defect engineering of trap states therefore provides a solution to achieve photodetectors with both high responsivity and fast response., Comment: 34 pages, 17 figures, to appear in Advanced Materials