Search

In recent years,pretrained language models have developed rapidly,pushing natural language processing into a whole new stage of development.To help researchers understand where and how the powerful pretrained language models can be applied in natural language processing,this paper surveys the state-of-the-art of its application.Specifically,we first briefly review typical pretrained language models,including monolingual,multilingual and Chinese pretrained models.Then,we discuss these pretrained language models' contributions to five different natural language processing tasks:information extraction,sentiment analysis,question answering,text summarization,and machine translation.Finally,we discuss some challenges faced by the applications of pretrained language models.

Since the introduction of neural networks,text summarization techniques continue to attract the attention of resear-chers.Similarly,generative adversarial networks(GANs)can be used for text summarization because they can generate text features or learn the distribution of the entire sample and produce correlated sample points.In this paper,we exploit the features of generative adversarial networks(GANs)and use them for abstractive text summarization tasks.The proposed generative adversa-rial model has three components:a generator,which encodes the input sentences into shorter representations;a readability discriminator,which forces the generator to create comprehensible summaries;and a similarity discriminator,which acts on the generator to curb the discorrelation between the outputted text summarization and the inputted text summarization.In addition,Chinese WordNet is used as an external knowledge base in the similarity discriminator to enhance the discriminator.The generator is optimized using policy gradient algorithm,converting the problem into reinforcement learning.Experimental results show that the proposed model gets high ROUGE evaluation scores.