Search

This paper describes neural models developed for the First Workshop on Scope Detection of the Peer Review Articles shared task collocated with PAKDD 2021. The aim of the task is to identify topics or category of scientific abstracts. We investigate the use of several fine-tuned language representation models pretrained on different large-scale corpora. In addition, we conduct experiments on combining BERT-based models and document topic vectors for scientific text classification. The topic vectors are obtained using LDA topic modeling. The topic-informed soft voting ensemble of neural networks achieved F1-score of 93.82%.

We study the BERT language representation model and the sequence generation model with BERT encoder for the multi-label text classification task. We show that the Sequence Generating BERT model achieves decent results in significantly fewer training epochs compared to the standard BERT. We also introduce and experimentally examine a mixed model, an ensemble of BERT and Sequence Generating BERT models. Our experiments demonstrate that the proposed model outperforms current baselines in several metrics on three well-studied multi-label classification datasets with English texts and two private Yandex Taxi datasets with Russian texts.

Статья посвящена анализу языковой репрезентации ментальных представлений о подобии человека и технических устройств. В данной работе исследуются образные переосмысления, выявляющие антропоцентрическую сущность механизмов в исторической динамике. Материал исследования составили существительные русского языка, обозначающие различные приспособления, от самых примитивных, изобретенных в древности, до сложных агрегатов наших дней. Проанализировано множество примеров их употребления в словарях, произведениях русских писателей, в материалах публицистического характера, а также в Национальном корпусе русского языка., The article analyzes the linguistic forms of mental representations of the similarity between a person and technical devices. The study examines image reinterpretations that reveal the anthropocentric essence of mechanisms in historical dynamics. The research material consists of Russian language nouns that denote various devices, from the most primitive, invented in ancient times, to complex aggregates of our days. The authors also analyze various examples of their use in dictionaries, Russian literary works, and journalistic materials, as well as in the National Corpus of the Russian Language., Russian Linguistic Bulletin, Выпуск 3 (27) 2021, Pages 26-29

Background and objectives: We provide a case analysis for a 28-year-old, native Dutch-speaking lady who developed Foreign Accent Syndrome (FAS), a few weeks after falling down the staircase. In addition to FAS, which gave the impression she spoke with a German accent, German(-like) words and structures occurred. Speech symptoms were aggravated by increased stress, fatigue or emotional pressure, and this triggered jargon speech. It was hypothesized her FAS and jargon developed on a functional basis.Methods: In-depth analyses of the patient's medical background, neuropsychological and neurolinguistic tests and psychodiagnostic exams were done. The patient participated in an fMRI experiment. In a syllable repetition paradigm, motor speech activations were compared to those of healthy individuals, to see whether they were altered, which would be expected in case of a neurological etiology.Results: Medical history disclosed prior traumatic experiences for which she sought help, but no neurological incidents. Repeated neuropsychological and neurolinguistic tests showed deficits in recent memory and executive functioning. The patient demonstrated great difficulties with picture naming. Clinically, language switching and mixing as well as recurring jargon speech was found. Formal psychodiagnostic tests did not identify a clear disorder, but psychodiagnostic interviews were consistent with a DSM-5 conversion disorder. The fMRI study demonstrated that speech network activations corresponded to those found in healthy participants.Conclusion: The clinical neurolinguistic characteristics, outcome of the fMRI experiment, together with the clinical psychodiagnostic findings were strongly indicative for an underlying functional etiology for the FAS and jargon speech, presenting as symptoms of conversion disorder. (C) 2020 Asociacion Universitaria de Zaragoza para el Progreso de la Psiquiatria y la Salud Mental. Published by Elsevier Espana, S.L.U. All rights reserved.

This paper describes the training process of the first Czech monolingual language representation models based on BERT and ALBERT architectures. We pre-train our models on more than 340K of sentences, which is 50 times more than multilingual models that include Czech data. We outperform the multilingual models on 9 out of 11 datasets. In addition, we establish the new state-of-the-art results on nine datasets. At the end, we discuss properties of monolingual and multilingual models based upon our results. We publish all the pre-trained and fine-tuned models freely for the research community., Comment: 13 pages

BERT has been shown to be extremely effective on a wide variety of natural language processing tasks, including sentiment analysis and emotion detection. However, the proposed pretraining objectives of BERT do not induce any sentiment or emotion-specific biases into the model. In this paper, we present Emotion Masked Language Modelling, a variation of Masked Language Modelling aimed at improving the BERT language representation model for emotion detection and sentiment analysis tasks. Using the same pre-training corpora as the original model, Wikipedia and BookCorpus, our BERT variation manages to improve the downstream performance on 4 tasks from emotion detection and sentiment analysis by an average of 1.2% F-1. Moreover, our approach shows an increased performance in our task-specific robustness tests.

Understanding of nerve-organ interactions is crucial to facilitate the development of effective bioelectronic treatments. Towards the end of developing a systematized and computable wiring diagram of the autonomic nervous system (ANS), we introduce a curated ANS connectivity corpus together with several neural language representation model based connectivity relation extraction systems. We also show that active learning guided curation for labeled corpus expansion significantly outperforms randomly selecting connectivity relation candidates minimizing curation effort. Our final relation extraction system achieves F1 = 72.8% on anatomical connectivity and F1 = 74.6% on functional connectivity relation extraction.

Velké zvukové archivy s mluveným obsahem jsou přirozenými kandidáty pro systémy odpovídající na otázky. Archivy orální historie obecně obsahují mnoho faktů a příběhů, které by bylo jinak těžké získat bez poslechu mnoha hodin nahrávek. Snažíme se učinit archiv přístupnějším tím, že umožňujeme hledat odpovědi na otázky položené v přirozeném jazyce. V tomto článku popisujeme výzvy, které naše datová sada představuje. Navrhujeme náš počáteční přístup, který využívá otázky a odpovědi získané ze samotného archivu a hodnotíme výkon v experimentech s modely s předem natrénovanou jazykovou reprezentací a s předtrénovanými modely odpovědí na otázky. Large audio archives with spoken content are natural candidates for question answering systems. Oral history archives generally contain many facts and stories that would be otherwise hard to obtain without listening to hours of recordings. We strive for making the archive more accessible by allowing natural language question answering. In this paper, we present challenges our dataset poses. We propose our initial approach that uses questions and answers mined from the archive itself and evaluate the performance in experiments with pretrained language representation and question answering models.

AI and BERT (Bidirectional Encoder Representations from Transformers) have been increasingly adopted in the human resources (HR) industry for recruitment. The increased efficiency (e.g., fairness) will help remove biases in machine learning, help organizations find a qualified candidate, and remove bias in the labor market. BERT has further improved the performance of language representation models by using an auto-encoding model which incorporates larger bidirectional contexts. However, BERT’s underlying mechanisms that enhance its effectiveness, such as tokenization, masking, and leveraging the attention mechanism to compute vector score, are not well understood.

Unsupervised cross-lingual language representation initialization methods, together with mechanisms such as denoising and back-translation, have advanced unsupervised neural machine translation (UNMT), which has achieved impressive results. Meanwhile, there are still several challenges for UNMT. This tutorial first introduces the background and the latest progress of UNMT. We then examine a number of challenges to UNMT and give empirical results on how well the technology currently holds up.

Learning effective language representations from crowdsourced labels is crucial for many real-world machine learning tasks. A challenging aspect of this problem is that the quality of crowdsourced labels suffer high intra- and inter-observer variability. Since the high-capacity deep neural networks can easily memorize all disagreements among crowdsourced labels, directly applying existing supervised language representation learning algorithms may yield suboptimal solutions. In this paper, we propose \emph{TACMA}, a \underline{t}emporal-\underline{a}ware language representation learning heuristic for \underline{c}rowdsourced labels with \underline{m}ultiple \underline{a}nnotators. The proposed approach (1) explicitly models the intra-observer variability with attention mechanism; (2) computes and aggregates per-sample confidence scores from multiple workers to address the inter-observer disagreements. The proposed heuristic is extremely easy to implement in around 5 lines of code. The proposed heuristic is evaluated on four synthetic and four real-world data sets. The results show that our approach outperforms a wide range of state-of-the-art baselines in terms of prediction accuracy and AUC. To encourage the reproducible results, we make our code publicly available at \url{https://github.com/CrowdsourcingMining/TACMA}., Comment: The 59th Annual Meeting of the Association for Computational Linguistics Workshop on Representation Learning for NLP (ACL RepL4NLP 2021)

We present RobeCzech, a monolingual RoBERTa language representation model trained on Czech data. RoBERTa is a robustly optimized Transformer-based pretraining approach. We show that RobeCzech considerably outperforms equally-sized multilingual and Czech-trained contextualized language representation models, surpasses current state of the art in all five evaluated NLP tasks and reaches state-of-the-art results in four of them. The RobeCzech model is released publicly at https://hdl.handle.net/11234/1-3691 and https://huggingface.co/ufal/robeczech-base.

The following system description presents our approach to the detection of persuasion techniques in texts and images. The given task has been framed as a multi-label classification problem with the different techniques serving as class labels. The multi-label classification problem is one in which a list of target variables such as our class labels is associated with every input chunk and assumes that a document can simultaneously and independently be assigned to multiple labels or classes. In order to assign class labels to the given memes, we opted for RoBERTa (A Robustly Optimized BERT Pretraining Approach) as a neural network architecture for token and sequence classification. Starting off with a pre-trained model for language representation we fine-tuned this model on the given classification task with the provided annotated data in supervised training steps. To incorporate image features in the multi-modal setting, we rely on the pre-trained VGG-16 model architecture.

China, a multinational country with 56 nationalities, has a complex linguistic configuration with more than two hundred languages. The prestige and hegemony of contemporary standardised Chinese relegates the languages and cultures of the 55 ethnic minorities to the shadows, in contrast to a state policy proclaiming the "flourishing of the 56 nationalities in the great family of the Chinese Nation". In the current circumstances where China is participating in accelerated urbanisation and globalisation, this thesis aims to shed light on the language practices, glottopolitical interventions, language representations, and sociolinguistic identity of a group of young Tibetan students in their aspirations for academic and social success in mainland China, specifically in Shandong province, the homeland of Confucius and a place strongly marked by the emphasis on the pursuit of academic success., La Chine, un pays multinational qui possède 56 nationalités, présente une configuration linguistique complexe avec plus de deux cents langues. Le prestige et la place hégémonique du chinois contemporain standardisé relèguent les langues et les cultures des 55 minorités ethniques dans l’ombre, en contraste avec une politique d’État proclamant l’“épanouissement des 56 nationalités dans la grande famille de la Nation Chinoise”. Dans les circonstances actuelles où la Chine participe d’une urbanisation et d’une mondialisation accélérées, cette thèse a pour but de mettre en lumière les pratiques langagières, les interventions glottopolitiques, les représentations des langues, l’identité sociolinguistique d’un groupe de jeunes étudiants tibétains dans leurs aspirations à la réussite scolaire et sociale en Chine continentale, plus précisément dans la province du Shandong, pays originaire de Confucius et lieu très marqué par la place prépondérante accordée à la poursuite de la réussite scolaire et universitaire.

Data scarcity is a common issue in the development of Dialogue Systems from scratch, where it is difficult to find dialogue data. This scenario is more likely to happen when the system’s language differs from English. This paper proposes a first text augmentation approach that selects samples similar to annotated user utterances from existing corpora, even if they differ in style, domain or content, in order to improve the detection of Out-of-Domain (OOD) user inputs. Three different sampling methods based on word-vectors extracted from BERT language representation model are compared. The evaluation is carried out using a Spanish chatbot corpus for OOD utterances detection, which has been artificially reduced to simulate various scenarios with different amounts of data. The presented approach is shown to be capable of enhancing the detection of OOD user utterances, achieving greater improvements when less annotated data is available.

Large pre-trained language representation models (LMs) have recently collected a huge number of successes in many NLP tasks. In 2018 BERT, and later its successors (e.g. RoBERTa), obtained state-of-the-art results in classical benchmark tasks, such as GLUE benchmark. After that, works about adversarial attacks have been published to test their generalization proprieties and robustness. In this work, we design Evolutionary Fooling Sentences Generator (EFSG), a model- and task-agnostic adversarial attack algorithm built using an evolutionary approach to generate false-positive sentences for binary classification tasks. We successfully apply EFSG to CoLA and MRPC tasks, on BERT and RoBERTa, comparing performances. Results prove the presence of weak spots in state-of-the-art LMs. We finally test adversarial training as a data augmentation defence approach against EFSG, obtaining stronger improved models with no loss of accuracy when tested on the original datasets., 13 pages, 19 figures

OBJECTIVE: To demonstrate the potential of machine learning and capability of natural language processing (NLP) to predict disposition of patients based on triage notes in the ED. METHODS: A retrospective cohort of ED triage notes from St Vincent's Hospital (Melbourne) was used to develop a deep-learning algorithm that predicts patient disposition. Bidirectional Encoder Representations from Transformers, a recent language representation model developed by Google, was utilised for NLP. Eighty percent of the dataset was used for training the model and 20% was used to test the algorithm performance. Ktrain library, a wrapper for TensorFlow Keras, was employed to develop the model. RESULTS: The accuracy of the algorithm was 83% and the area under the curve was 0.88. Sensitivity, specificity, precision and F1-score of the algorithm were 72%, 86%, 56% and 63%, respectively. CONCLUSION: Machine learning and NLP can be together applied to the ED triage note to predict patient disposition with a high level of accuracy. The algorithm can potentially assist ED clinicians in early identification of patients requiring admission by mitigating the cognitive load, thus optimises resource allocation in EDs.

Recently, Natural Language Processing has shown significant development, especially in text mining and analysis. An important task in this area is learning vector-space representations of text. Since various machine learning algorithms require representing their inputs in a vector format. In this paper, we highlight the most important language representation learning models used in the literature, ranging from the free contextual approaches like word2vec and Glove until the appearance of recent modern contextualized approaches such as ELMo, BERT, and XLNet. We show and discuss their main architectures and their main strengths and limits.

Researchers have shown that sentiment analysis of software artifacts can potentially improve various software engineering tools, including API and library recommendation systems, code suggestion tools, and tools for improving communication among software developers. However, sentiment analysis techniques applied to software artifacts still have not yet yielded very high accuracy. Recent adaptations of sentiment analysis tools to the software domain have reported some improvements, but the f-measures for the positive and negative sentences still remain in the 0.4-0.64 range, which deters their practical usefulness for software engineering tools.In this paper, we explore the potential effectiveness of customizing BERT, a language representation model, which has recently achieved very good results on various Natural Language Processing tasks on English texts, for the task of sentiment analysis of software artifacts. We describe our application of BERT to analyzing sentiments of sentences in Stack Overflow posts and compare the impact of a BERT sentiment classifier to state-of-the-art sentiment analysis techniques when used on a domain-specific data set created from Stack Overflow posts. We also investigate how the performance of sentiment analysis changes when using a much (3 times) larger data set than previous studies. Our results show that the BERT classifier achieves reliable performance for sentiment analysis of software engineering texts. BERT combined with the larger data set achieves an overall f-measure of 0.87, with the f-measures for the negative and positive sentences reaching 0.91 and 0.78 respectively, a significant improvement over the state-of-the-art.

This paper describes BUT-FIT’s submission at SemEval-2020 Task 5: Modelling Causal Reasoning in Language: Detecting Counterfactuals. The challenge focused on detecting whether a given statement contains a counterfactual (Subtask 1) and extracting both antecedent and consequent parts of the counterfactual from the text (Subtask 2). We experimented with various state-of-the-art language representation models (LRMs). We found RoBERTa LRM to perform the best in both subtasks. We achieved the first place in both exact match and F1 for Subtask 2 and ranked second for Subtask 1.

This paper aims to extract summarization and keywords from scientific articles simultaneously, while abstract extraction (AE) and key extraction (KE) are considered as auxiliary tasks to each other. For the data scarcity in scientific AE and KE tasks, we propose a multi-task learning framework which uses huge unlabeled data to learn scientific language representation (pre-training) and uses smaller annotated data to transfer the learned representation to AE and KE (fine-tuning). Although the pre-trained language model performs well in universal natural language tasks, its capacity still has a margin of improvement for specific tasks. Inspired by this intuition, we use another two tasks keyword masking and key sentence prediction before the fine-tuning phase to enhance the language representation for AE and KE. This language representation enhancing stage uses the same labeled data but different optimization objectives with the fine-tuning phase. In order to evaluate our model, we develop and release a high-quality annotated corpus for scientific papers with keywords and abstract. We conduct comparative experiments on this dataset, and experimental results show that our multi-task learning framework achieves the state-of-the-art performance, proving the effectiveness of the language model enhancing mechanism.

In this paper, we have identified and described verbal and nonverbal means of representing the negative emotional state of anger. In the text of artistic works, the authors most vividly describe the emotional states experienced by the characters. Currently, a large number of scientists are turning to the problem of studying emotions and emotional states that a person experiences. Anger refers to negative emotional states. The paper presents the results of the analysis of the implementation of negative emotions in the framework of the artistic work of Kerstin Gier "Ein unmoralisches Sonderangebot". The purpose of this study is to examine and identify the main verbal and nonverbal language means of representing the emotion of anger displayed by a character in the text of fiction. We have described verbal and nonverbal means of implementing a negative emotional state of anger. It is concluded that nonverbal means play a major role in the manifestation of the emotional state. Identifying the main language means of representing the emotional state of anger in verbal and non-verbal communication based on a literary text. It is with the help of verbal and nonverbal means used by the author that we see the representation of the emotional state of the character., В представленной работе нами были выявлены и описаны вербальные и невербальные средства репрезентации отрицательного эмоционального состояния гнева. В тексте художественных произведений авторами наиболее ярко описаны эмоциональные состояния, испытываемые персонажами. В настоящее время большое количество ученых обращаются к проблеме изучения эмоций и эмоциональных состояний, которые испытывает человек. Гнев относится к отрицательным эмоциональным состояниям. В работе представлены результаты анализа реализации отрицательной эмоции в рамках художественного произведения Kerstin Gier «Ein unmoralisches Sonderangebot». Целью данного исследования является рассмотрение и выявление основных вербальных и невербальных языковых средств представления эмоции гнева, проявляемой персонажем в тексте художественной прозы. Нами были описаны вербальные и невербальные средства реализации отрицательного эмоционального состояния гнева. Сделан вывод о том, что именно невербальные средства играют главную роль при манифестации эмоционального состояния. Выявлении основных языковых средств репрезентации эмоционального состояния гнева в вербальной и невербальной коммуникации на основе художественного текста. Именно с помощью вербальных и невербальных средств, которые использует автор, мы видим репрезентацию эмоционального состояния персонажа.

Neural language representation models such as BERT have recently shown state of the art performance in downstream NLP tasks and bio-medical domain adaptation of BERT (Bio-BERT) has shown same behavior on biomedical text mining tasks. However, due to their large model size and resulting increased computational need, practical application of models such as BERT is challenging making smaller models with comparable performance desirable for real word applications. Recently, a new language transformers based language representation model named ELECTRA is introduced, that makes efficient usage of training data in a generative-discriminative neural model setting that shows performance gains over BERT. These gains are especially impressive for smaller models. Here, we introduce two small ELECTRA based model named Bio-ELECTRA and Bio-ELECTRA++ that are eight times smaller than BERT Base and Bio-BERT and achieves comparable or better performance on biomedical question answering, yes/no question answer classification, question answer candidate ranking and relation extraction tasks. Bio-ELECTRA is pre-trained from scratch on PubMed abstracts using a consumer grade GPU with only 8GB memory. Bio-ELECTRA++ is the further pre-trained version of Bio-ELECTRA trained on a corpus of open access full papers from PubMed Central. While, for biomedical named entity recognition, large BERT Base model outperforms Bio-ELECTRA++, Bio-ELECTRA and ELECTRA-Small++, with hyperparameter tuning Bio-ELECTRA++ achieves results comparable to BERT.

Recommender systems are paramount in providing personalized content and intelligent content filtering on any social media platform, web portal, and online application. In line with the current trends in the field directed towards mapping problem and data encoding representations from other fields, this research investigates the feasibility of augmenting a graph-based recommender system for Amazon products with two state-of-the-art representation models. In particular, the potential benefits of using the language representation model BERT and GraphSage based representations of nodes and edges for improving the quality of the recommendations were investigated. Link prediction and link attribute inference were used to identify the products that the users will buy and predict the rating they will give to a product, respectively. The initial results of our exploratory study are encouraging and point to potential directions for future research.

This chapter discusses the research field on bilingualism and its purported effects on cognition, both for children and across the lifespan. It explores some of the factors behind the conflicting and often contradictory results and interpretations. The chapter also discusses the contributions of neuroscience to the debate thus far and identify future opportunities in the field of research. The implications of bilingualism in education also extend beyond English as an additional language. There is the question of when best to introduce the acquisition of additional languages. Numerous studies, both behavioural and neuroimaging, have yielded at best inconsistent and at worst contradictory results. The reasons for some of these inconsistent results have been explained already. A critical but unresolved issue, therefore, is how multi-linguistic experience impacts on crucial cognitive processes across the lifespan, from infancy to old age. Neuroscience has helped to identify the regions of the brain associated with both the language representation and other non-linguistic cognitive activity in bilinguals.

Solving arithmetic word problem automatically has been a challenge both in terms of attaining robustness to unseen problems and achieving high problem-solving accuracy. In this paper, we propose a Template based – Multi-Task Deep Neural Network (T-MTDNN) framework, which utilizes two types of techniques. First, by generating normalized equation templates, we achieve robustness by enabling a more general language representation of a given linguistic task. Second, by applying MTDNN [1], which uses BERT with number and operator classification as multi-tasks, we gain higher problem solving accuracy compared to T-RNN [2], which is the state-of-the-art model. Specifically, with MAWPS dataset, the accuracy of T-MTDNN is 78.88% compared to the accuracy of T-RNN at 66.8%. With Math23K dataset, the accuracy of T-MTDNN is 72.6% compared to the accuracy of T-RNN at 66.9%.

We introduce Sentence-level Language Modeling, a new pre-training objective for learning a discourse language representation in a fully self-supervised manner. Recent pre-training methods in NLP focus on learning either bottom or top-level language representations: contextualized word representations derived from language model objectives at one extreme and a whole sequence representation learned by order classification of two given textual segments at the other. However, these models are not directly encouraged to capture representations of intermediate-size structures that exist in natural languages such as sentences and the relationships among them. To that end, we propose a new approach to encourage learning of a contextualized sentence-level representation by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. Through experiments on downstream tasks such as GLUE, SQuAD, and DiscoEval, we show that this feature of our model improves the performance of the original BERT by large margins., EMNLP 2020

Мельгунова Анастасия Геннадьевна, кандидат филологических наук, доцент, Байкальский государственный университет (Иркутск), an_gen2002@yahoo.com Китова Елена Борисовна, кандидат филологических наук, Байкальский государственный университет (Иркутск), lkitova@yahoo.com Anastasia G. Melgunova, Candidate of Philological Science, Associate Professor, Baikal State University (Irkutsk), an_gen2002@yahoo.com Elena B. Kitova, Candidate of Philological Science, Associate Professor, Baikal State University (Irkutsk), lkitova@yahoo.com В данной статье рассматривается метафоризация значений некоторых английских глаголов со значением сопротивления и их использование для описания внутренних эмоционально-психологических состояний человека. Рассмотрено явление и виды внутреннего сопротивления человека и языковые единицы английского языка, использующиеся для описания внутреннего сопротивления. Проанализированы зафиксированные в словарях значения и примеры употребления глаголов сопротивления в прямом номинативном значении, а также примеры метафоризации их значений. Выделены языковые единицы, представляющие интерес как средства концептуализации эмоциональных состояний и ментальных процессов в виде физических сил. Сделан вывод о том, что некоторые эмоционально-психические процессы, такие как внутреннее сопротивление, четко осмысляются в терминах борьбы с материальными физическими силами, что и объясняет использование в английском языке глаголов физической борьбы fight и struggle для описания высокого градуса эмоционального напряжения. Отмечается также, что соответствующие значения данных двух глаголов не зафиксированы в их словарных дефинициях, что свидетельствует о недостаточной изученности их значений и сферы употребления. The paper studies metaphorical meaning of some English verbs of resistance, and their usage as means of conceptualization of human’s inner emotional and psychological states. The article provides a general overview of the phenomenon and types of human’s inner resistance, and units of the English language that are used to describe this phenomenon. Based on a case study of English language units, the authors analyze the direct nominative meaning of the verbs, as well as their metaphorical meaning, and outline the language units that help to conceptualize emotional states and mental processes as physical forces. The inference implies that some emotional and mental processes, namely, inner resistance, are conceptualized in human’s consciousness in terms of physical fight against material forces. The usage of the verbs fight and struggle for this purpose is explained by the necessity to express a high intensity of mental strain that cannot be represented by the direct nominative meaning of other verbs. The described metaphorical meaning of these two verbs cannot be found in their definitions in dictionary entries, which indicates insufficient study of their meaning and usage.

Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy with four perspectives. Next, we describe how to adapt the knowledge of PTMs to the downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks., Comment: Invited Review of Science China Technological Sciences

Multimodal sentiment analysis aims to learn a joint representation of multiple features. As demonstrated by previous studies, it is shown that the language modality may contain more semantic information than that of other modalities. Based on this observation, we propose a Multi-perspective Fusion Network(MPFN) focusing on Sense Attentive Language for multimodal sentiment analysis. Different from previous studies, we use the language modality as the main part of the final joint representation, and propose a multi-stage and uni-stage fusion strategy to get the fusion representation of the multiple modalities to assist the final language-dominated multimodal representation. In our model, a Sense-Level Attention Network is proposed to dynamically learn the word representation which is guided by the fusion of the multiple modalities. As in turn, the learned language representation can also help the multi-stage and uni-stage fusion of the different modalities. In this way, the model can jointly learn a well integrated final representation focusing on the language and the interactions between the multiple modalities both on multi-stage and uni-stage. Several experiments are carried on the CMU-MOSI, the CMU-MOSEI and the YouTube public datasets. The experiments show that our model performs better or competitive results compared with the baseline models.

Medical reports are an essential medium in recording a patient’s condition throughout a clinical trial. They contain valuable information that can be extracted to generate a large labeled dataset needed for the development of clinical tools. However, the majority of medical reports are stored in an unregularized format, and a trained human annotator (typically a doctor) must manually assess and label each case, resulting in an expensive and time consuming procedure. In this work, we present a framework for developing a multilingual breast MRI report classifier using a custom-built language representation called LAMBR. Our proposed method overcomes practical challenges faced in clinical settings, and we demonstrate improved performance in extracting labels from medical reports when compared with conventional approaches.

Research on cognitive obstacles related to fraction sense in preservice mathematics teachers is significant, because their success depends on their skills. The acquisition of fraction sense is a complicated problem requiring a strategy to solve it. This study presents cognitive obstacles with fraction sense tests in preservice who will teach in secondary schools. It focuses on the following categories of cognitive obstacles: epistemological (language representation, tendency to generalise and rely on intuition) and didactic (less meaningful learning, and strategy). This paper takes a qualitative descriptive approach to examine 20 preservice mathematics teachers. The preservice teachers who encountered cognitive obstacles related to fraction sense testing were then grouped based on the similarity of their answers, and seven of them were selected to be interviewed. The research findings showed that five preservice teachers had overlapping obstacles: language representation and tendency to generalise; tendency to generalise and less meaningful learning; language representation, tendency to rely on intuition and trial and error strategy in; language representation and trial and error; and language representation and tendency to rely on intuition.

Online product reviews help consumers make purchase decisions when shopping online. As such, many computational models have been constructed to automatically evaluate the helpfulness of customer product reviews. However, many existing models are based on simple explanatory variables, including those extracted from low quality reviews that can be misleading and lead to confusion. Quality feature selection is essential for predicting the helpfulness of online customer reviews. The Bidirectional Encoder Representations from Transformers (BERT) is a very recently developed language representation model which can attain state-of-the-art results on many natural language processing tasks. In this study, a predictive model for determining helpfulness scores of customer reviews based on incorporation of BERT features with deep learning techniques is proposed. The application analyzes the Amazon product reviews dataset, and uses a BERT features based algorithm expected to be useful in help consumers to make a better purchase decisions.

This paper describes our system that participated in the SemEval-2020 task 4: Commonsense Validation and Explanation. For this task, it is obvious that external knowledge, such as Knowledge graph, can help the model understand commonsense in natural language statements. But how to select the right triples for statements remains unsolved, so how to reduce the interference of irrelevant triples on model performance is a research focus. This paper adopt a modified K-BERT as the language encoder, to enhance language representation through triples from knowledge graphs. Experiments show that our method is better than models without external knowledge, and is slightly better than the original K-BERT. We got an accuracy score of 0.97 in subtaskA, ranking 1/45, and got an accuracy score of 0.948, ranking 2/35.

Illustration is the use of the graphic images, in line with the principle of the unity of aesthetics and practice, trying to make lines and shapes clear and lucid, easy to produce. Illustration is a universal language in the world. Its design is usually divided into characters, animals and commodity images in the commercial applications. In recent years, with the popularization of computers and the network, the modern illustration creation in China has been greatly developed. Illustration has gradually been widely used in various fields of our life and work. The diversified characteristics of the modern illustration development can no longer cover the application scope of the traditional illustration. With the development of the media technology, the application of the streaming media technology in the comprehensive language representation system of illustrations has developed into the technical connotation of the illustration art display.

Chinese named entity recognition (NER) has been an important problem in natural language processing (NLP) field. Most existing methods mainly use traditional deep learning models which cannot fully leverage contextual dependencies that are very important for capturing the relations between words or characters for modeling. To address this problem, various language representation methods such as BERT have been proposed to learn the global context information. Although these methods can achieve good results, the large number of parameters limited the efficiency and application in real-world scenarios. To improve both of the performance and efficiency, this paper proposes an ALBERT-based Chinese NER method which uses ALBERT, a Lite version of BERT, as the pre-trained model to reduce model parameters and to improve the performance through sharing cross-layer parameters. Besides, it uses conditional random field (CRF) to capture the sentence-level correlation information between words or characters to alleviate the tagging inconsistency problems. Experimental results demonstrate that our method outperforms the comparison methods over 4.23–11.17% in terms of relative F1-measure with only 4% of BERT’s parameters.

Rutuli are the seventh most numerous indigenous people of Dagestan. Rutul call themselves “makhabir”. According to the results of the 2010 census, there are 35,240 Rutuls, but this is a very low figure, since the Rutulis living in Azerbaijan (more than 20,000) are indicated by Azerbaijanis by nationality, Rutuls of the older generation and hnovsky Rutuls - as Lezgins. In fact there are about 80 thousand Rutuls. The material of the study of this work was the idea of the concept of Abyr "Ethics" in the language picture of Rutul, largely determining the mentality, ethical and spiritual basis of the mountain people of Dagestan.

The paper presents the most common patterns of representation of subject and object relations in the definitions of terms, and identifies the ways of implementing subject relations between elements of scientific knowledge in active and passive structures that are part of the structure of applied models of organization of scientific knowledge. In addition, the article raises the question of the potential of various grammatical structures in the context of the transfer of subject and object relations. The results of the study show that system relation S can be represented by lexical and grammatical means. The lexical verbalizes of this relation are the key words of blocks of subject relations, and among the grammatical language means it is possible to distinguish the category of case.

The article attempts to consider the concept of “concept” as an object of interdisciplinary research, defines the areas of functioning and research fixation of concepts, identifies differences between the usual and artistic concepts that serve as a productive way of penetrating into the secrets of poetic skill and creative laboratory of artists of the word. The work is devoted to the study of the conceptual foundations of L. N. Andreeva and identification of the specificity of his linguistic personality through the study of linguistic representation of the concept of “Life” associated with the General socio-philosophical and religious-philosophical quest of the author. The paper describes the parameters of the artistic concept and the ways of its linguistic expression, the main of which is the associative-semantic field; the structure of the associative-semantic field “Life” is analyzed on the material of the text of the play by L.N. Andreeva “human Life”; its nuclear and peripheral components are indicated, the ratio of which allows to approach the General understanding of the phenomenon of human life by the writer, coupled with the concepts of Fate, rock and artistic images of candles, stairs based on associative connections. Additional means of linguistic expression of the concept “Life” are listed, which emphasize, on the one hand, the value essence of life, and, on the other hand, its fragility, unpredictability, ruthlessness towards a Person. Particular attention is paid to the artistic techniques involved in the representation of the concept of “Life” in the author’s artistic discourse. The value of the scientific approach outlined in the work is not in the contrast comparison of the most interesting and significant points of view of domestic and foreign scientists, expressed regarding the nature and content of the concept as a phenomenon of language, thinking, consciousness and speech, and in their organic mutual addition, which allows us to present the most complete and comprehensive understanding of this phenomenon in the light of the various paradigms of modern scientific knowledge.

Рассматриваются когнитивные основания ментального феномена воображения и языковая репрезентация способности создания и формирования образа. Анализ языковой концептуализации воображения опирается на метод когнитивного моделирования ситуации. Эмпирический материал свидетельствует о том, что ситуация воображения может быть категоризована глаголами, которые по-разному ее характеризуют. Обобщение компонентов семантической структуры языковых выражений и контекстов их репрезентации позволило построить когнитивную модель ситуации воображения, участниками которой будут являться субъект воображения, процесс воображения как создание мысленных образов в определенной пространственно-временной среде и объект воображения (воображаемое).

Background Effectively utilizing disease-relevant text information from unstructured clinical notes for medical research presents many challenges. BERT (Bidirectional Encoder Representation from Transformers) related models such as BioBERT and ClinicalBERT, pre-trained on biomedical corpora and general clinical information, have shown promising performance in various biomedical language processing tasks. Objectives This study aims to explore whether a BERT-based model pre-trained on disease-related clinical information can be more effective for cerebrovascular disease-relevant research. Methods This study proposed the StrokeBERT which was initialized from BioBERT and pre-trained on large-scale cerebrovascular disease related clinical text information. The pre-trained corpora contained 113,590 discharge notes, 105,743 radiology reports, and 38,199 neurological reports. Two real-world empirical clinical tasks were conducted to validate StrokeBERT's performance. The first task identified extracranial and intracranial artery stenosis from two independent sets of radiology angiography reports. The second task predicted the risk of recurrent ischemic stroke based on patients’ first discharge information. Results In stenosis detection, StrokeBERT showed improved performance on targeted carotid arteries, with an average AUC compared to that of ClinicalBERT of 0.968 ± 0.021 and 0.956 ± 0.018, respectively. In recurrent ischemic stroke prediction, after 10-fold cross-validation on 1,700 discharge information, StrokeBERT presented better prediction ability (AUC±SD = 0.838 ± 0.017) than ClinicalBERT (AUC±SD = 0.808 ± 0.045). The attention scores of StrokeBERT showed better ability to detect and associate cerebrovascular disease related terms than current BERT based models. Conclusions This study shows that a disease-specific BERT model improved the performance and accuracy of various disease-specific language processing tasks and can readily be fine-tuned to advance cerebrovascular disease research and further developed for clinical applications.

Place descriptions record qualitative information related to places and their spatial relationships, thus, the geospatial semantics of a place can be extracted from place descriptions. In this study, geotagged microblog short texts recorded in 2017 from the Tibetan Autonomous Region and Qinghai Province were used to extract the place semantics of the Qinghai–Tibetan Plateau (QTP). ERNIE, a language representation model enhanced by knowledge, was employed to extract thematic topics from the microblog short texts, which were then geolocated and used to analyze the place semantics of the QTP. Considering the large number of microblogs published by tourists in both Qinghai and Tibet, we separated the texts into four datasets according to the user, i.e., local users in Tibet, tourists in Tibet, local users in Qinghai, and tourists in Qinghai, to explore the place semantics of the QTP from different perspectives. The results revealed clear spatial variability in the thematic topics. Tibet is characterized by travel- and scenery-related language, whereas Qinghai is characterized by emotion, work, and beauty salon-related language. The human cognition of place semantics differs between local residents and tourists, and with a greater difference between the two in Tibet than in Qinghai. Weibo texts also indicate that local residents and tourists are concerned with different aspects of the same thematic topics. The cities on the QTP can be classified into three groups according to their geospatial semantic components, i.e., tourism-focused, life-focused, and religion-focused cities.