Search

The standard BERT adopts subword-based tokenization, which may break a word into two or more wordpieces (e.g., converting "lossless" to "loss" and "less"). This will bring inconvenience in following situations: (1) what is the best way to obtain the contextual vector of a word that is divided into multiple wordpieces? (2) how to predict a word via cloze test without knowing the number of wordpieces in advance? In this work, we explore to develop a BERT-style pretrained model over a vocabulary of words instead of wordpieces. We call such word-level BERT model as WordBERT. We train models with different vocabulary sizes, initialization configurations and languages. Results show that, compared to standard wordpiece-based BERT, WordBERT makes significant improvements on cloze test and machine reading comprehension. On many other natural language understanding tasks, including POS tagging, chunking and NER, WordBERT consistently performs better than BERT. Model analysis indicates that the major advantage of WordBERT over BERT lies in the understanding for low-frequency words and rare words. Furthermore, since the pipeline is language-independent, we train WordBERT for Chinese language and obtain significant gains on five natural language understanding datasets. Lastly, the analyse on inference speed illustrates WordBERT has comparable time cost to BERT in natural language understanding tasks. [ABSTRACT FROM AUTHOR]

Most of the bilingual lexicon induction (BLI) models learn a mapping function that can transfer word embedding (WE) spaces from one language to another. This usually relies on the isomorphism hypothesis, which posits that words in different languages share the same structures and relationships (i.e. similar in geometric structure). However, WE's isomorphism weakens substantially in distant language pairs, resulting in low accuracy of BLI. To address this problem, we propose a novel BLI method incorporating synonymous knowledge. The main idea is to stabilize the distance between words to optimize the monolingual WE space, yielding higher isomorphism. Specifically, we first induce monolingual synonym pairs from Wordnet and construct monolingual synonym lexicons. We then generate pseudo-sentences by substituting words in the training corpus with synonyms. Finally, the original sentences and pseudo-sentences are jointly used to generate monolingual WEs, enabling the word vectors of synonyms to be closer naturally. Comprehensive experiments on standard BLI datasets in diverse distant languages demonstrate that our method significantly outperforms the strong BLI systems in word translation. [ABSTRACT FROM AUTHOR]

During the past years, we have seen a steady increase in the number of social networks worldwide. Among them, Twitter has consolidated its position as one of the most influential social platforms, with Brazilian Portuguese speakers holding the fifth position in the number of users. Due to the informal linguistic style of tweets, the discovery of information in such an environment poses a challenge to Natural Language Processing (NLP) tasks such as sentiment analysis. In this work, we state sentiment analysis as a binary (positive and negative) and multiclass (positive, negative, and neutral) classification task at the Portuguese-written tweet level. Following a feature extraction approach, embeddings are initially gathered for a tweet and then given as input to learning a classifier. This study was designed to evaluate the effectiveness of different word representations, from the original pre-trained language model to continued pre-training strategies, to improve the predictive performance of sentiment classification, using three different classifier algorithms and eight Portuguese tweets datasets. Because of the lack of a language model specific to Brazilian Portuguese tweets, we have expanded our evaluation to consider six different embeddings: fastText, GloVe, Word2Vec, BERT-multilingual (mBERT), BERTweet, and BERTimbau. The experiments showed that embeddings trained from scratch solely using the target Portuguese language, BERTimbau, outperform the static representations, fastText, GloVe, and Word2Vec, and the Transformer-based models BERT multilingual and BERTweet. In addition, we show that extracting the contextualized embedding without any adjustment to the pre-trained language model is the best approach for most datasets. [ABSTRACT FROM AUTHOR]

Affective tasks, including sentiment analysis, emotion classification, and sarcasm detection have drawn a lot of attention in recent years due to a broad range of useful applications in various domains. The main goal of affect detection tasks is to recognize states such as mood, sentiment, and emotions from textual data (e.g., news articles or product reviews). Despite the importance of utilizing preprocessing steps in different stages (i.e., word representation learning and building a classification model) of affect detection tasks, this topic has not been studied well. To that end, we explore whether applying various preprocessing methods (stemming, lemmatization, stopword removal, punctuation removal and so on) and their combinations in different stages of the affect detection pipeline can improve the model performance. The are many preprocessing approaches that can be utilized in affect detection tasks. However, their influence on the final performance depends on the type of preprocessing and the stages that they are applied. Moreover, the preprocessing impacts vary across different affective tasks. Our analysis provides thorough insights into how preprocessing steps can be applied in building an effect detection pipeline and their respective influence on performance. [ABSTRACT FROM AUTHOR]

User-generated content on social media platforms has reached big data levels. Sentiment analysis of this data provides opportunities to gain valuable insights into any domain. However, analyzing real-world data may confront the challenge of class imbalance, which can adversely affect the generalization ability of models due to majority class overfitting. Therefore, having an efficient model that manages any scenario of imbalanced data is practically needed. In this light, this work proposes different models based on studying the impact of data quality and transfer learning through pre-trained embeddings on boosting minority class detection. The proposed models are tested on imbalanced datasets related to social media and education. The experimental results highlight the effectiveness of Wor2vec, Glove, and Fasttext embeddings with preprocessed data. In contrast, BERT embeddings present better results with no-preprocessed data. Furthermore, in comparison with other methods, the best-performing model resulting from this study shows outperformance with notable improvements. [ABSTRACT FROM AUTHOR]

With the development of technology, e-marketing and business competition, reviews significantly affect people's lives. Consumers of e-commerce sites share their experiences, referencing other user reviews when making consumption decisions and evaluating product quality. People's dependence on review information and business competition has triggered the emergence of false reviews. False reviews refer to the promotion or demotion of the quality of the product. False reviews can easily confuse new users' opinions or judgments, due to which both the consumer and the company are affected. Addressing this issue, spam review is a new research problem today, and even humans do not quickly classify spam reviews with high accuracy. Machine learning methods and natural language processing techniques effectively recognise false reviews and help users get benign review information. In this research, we have categorized and classified spam review detection methods in the direction of review features, reviewer features, and the spammer's group features. In this article, we focus on three approaches for detecting false reviews: spam review detection, spammer detection, and spammer-group detection. This article provides a motivational analysis of the three types of research and, in particular, compares them to aspects of feature design, model methods, datasets, and rating indicators. [ABSTRACT FROM AUTHOR]

IntroductionThe notion of a single localized store of word representations has become increasingly less plausible as evidence has accumulated for the widely distributed neural representation of wordform grounded in motor, perceptual, and conceptual processes. Here, we attempt to combine machine learning methods and neurobiological frameworks to propose a computational model of brain systems potentially responsible for wordform representation. We tested the hypothesis that the functional specialization of word representation in the brain is driven partly by computational optimization. This hypothesis directly addresses the unique problem of mapping sound and articulation vs. mapping sound and meaning.ResultsWe found that artificial neural networks trained on the mapping between sound and articulation performed poorly in recognizing the mapping between sound and meaning and vice versa. Moreover, a network trained on both tasks simultaneously could not discover the features required for efficient mapping between sound and higher-level cognitive states compared to the other two models. Furthermore, these networks developed internal representations reflecting specialized task-optimized functions without explicit training.DiscussionTogether, these findings demonstrate that different task-directed representations lead to more focused responses and better performance of a machine or algorithm and, hypothetically, the brain. Thus, we imply that the functional specialization of word representation mirrors a computational optimization strategy given the nature of the tasks that the human brain faces.

Information retrieval is a difficult process due to the overabundance of information on the web. Nowadays, search result responds to user queries with too many results although only a few are relevant. Therefore, the existing clustering methods that fail in clustering snippets (short texts) of web documents due to the low frequencies of document terms should be deeply investigated. One of the approaches that can be used to solve this problem is the expansion of document terms with semantically similar terms. Hence, a list of terms with their closest and accurate semantically similar words (word representation) must be built. This study aims to design and develop a new framework to enhance the performance of web search result clustering (WSRC). The research also presents a new unsupervised distributed word representation scheme where each word is represented by a vector of its semantically related words; such as scheme expands snippets and user queries. The proposed framework consists of several activities, such as (1) various standard datasets (Open Directory Project [ODP]-239 and MORESQUE) that are used for evaluating search result clustering algorithms for most cited dataset works, (2) text pre-processing, (3) document representation based on a new wiki-based k-nearest neighbors (KNN) representation method, (4) effect of the proposed model on the performance of traditional clustering methods (k-means, k-medoids, single-linkage, and complete-linkage) for WSRC, and (5) evaluation stage of the proposed method. Results indicate that enhanced clustering methods, according to the new wiki-KNN based representation method in comparison with the baseline methods, show a significant improvement in WSRC. Furthermore, the new data representation scheme has enhanced the overall performance of clustering methods.

Words, unlike images, are symbolic representations. The associative details inherent within a word's meaning and the visual imagery it generates, are inextricably connected to the way words are processed and represented. It is well recognised that the hippocampus associatively binds components of a memory to form a lasting representation, and here we show that the hippocampus is especially sensitive to abstract word processing. Using fMRI during recognition, we found that the increased abstractness of words produced increased hippocampal activation regardless of memory outcome. Interestingly, word recollection produced hippocampal activation regardless of word content, while the parahippocampal cortex was sensitive to concreteness of word representations, regardless of memory outcome. We reason that the hippocampus has assumed a critical role in the representation of uncontextualized abstract word meaning, as its information-binding ability allows the retrieval of the semantic and visual associates that, when bound together, generate the abstract concept represented by word symbols. These insights have implications for research on word representation, memory, and hippocampal function, perhaps shedding light on how the human brain has adapted to encode and represent abstract concepts. • Explored how words are represented and recognised while varying their abstractness. • Hippocampal activity tracked the degree of abstractness independent of memory. • Recollection of words engaged the hippocampus, regardless of word content. • Abstractness, rather than familiarity, is predicted by hippocampal activity. • Hippocampus helps form associative representations of abstract words. [ABSTRACT FROM AUTHOR]

In this paper, inspired from the idea that the contextual distances and positions may have a substantial impact on distinguishing the relationships between words, a novel word association method with two weighting schemes for refining contexts, named as Weighted Point-wise Mutual Information with Contextual Distances and Positions (PMIDP), is proposed to eliminate the noisy and redundant information hidden in an imbalanced corpus. One weighting scheme, called PMIdist, revises the Point-wise Mutual Information (PMI) method by scaling the co-occurrence counts according to the distance between a word and the context. The second weighting scheme is a ratio that can measure the contextual position variation within the window of a given word. Then, the refined word association in PMIDP is defined as the multiplication of the two proposed weighting schemes, which essentially aims to flexibly adjust the word association when solving target-oriented similarity tasks. The proposed word association method has been applied on two widely known models, i.e., the positive PMI matrix with truncated Singular Vector Decomposition (PPMI-SVD) model and the Global Vectors (GloVe) model. Experimental results demonstrate that the PMIDP method can significantly improve the performances of the two models on both semantic and relational similarity tasks and show advantages when compared with other state-of-the-art models. [ABSTRACT FROM AUTHOR]

Due to the rapid growth of textual information on the web, analyzing users' opinions about particular products, events or services is now considered a crucial and challenging task that has changed sentiment analysis from an academic endeavor to an essential analytic tool in cognitive science and natural language understanding. Despite the remarkable success of deep learning models for textual sentiment classification, they are still confronted with some limitations. Convolutional neural network is one of the deep learning models that has been excelled at sentiment classification but tends to need a large amount of training data while it considers that all words in a sentence have equal contribution in the polarity of a sentence and its performance is highly dependent on its accompanying hyper-parameters. To overcome these issues, an Attention-Based Convolutional Neural Network with Transfer Learning (ACNN-TL) is proposed in this paper that not only tries to take advantage of both attention mechanism and transfer learning to boost the performance of sentiment classification but also language models, namely Word2Vec and BERT, are used as its the backbone to better express sentence semantics as word vector. We conducted our experiment on widely-studied sentiment classification datasets and according to the empirical results, not only the proposed ACNN-TL achieved comparable or even better classification results but also employing contextual representation and transfer learning yielded remarkable improvement in the classification accuracy. [ABSTRACT FROM AUTHOR]

This paper proposes a lyrics retrieval method based on word-embedding technology to perform music recommendations that matched a tourist attraction. In the proposed method, word importance is calculated by Term Frequency ‒ Inverse Document Frequency (TFIDF) and Smooth Inverse Frequency (SIF). We built a vectorization model from the lyrics corpus using fastText with Continuous Bag of Words (CBOW) and applied this model to both the lyrics corpus and the tourist attraction reviews corpus to create wordembedding vectors for lyrics and tourist attraction reviews. And, the review vectors are integrated for each tourist attraction to generate tourist attraction vectors. Based on the similarity calculation between the tourist attraction vectors and the lyric vectors, the song with the most suitable lyrics for the tourist attraction comes to be the recommended result. Subjective evaluation experiments on the recommendation results of the proposed method was conducted. The results of the experiments showed that the subjects reacted positively to the lyrics of the recommended songs. However, their evaluations of the music of the recommended songs were not as positive as those accorded to the lyrics were. A joint study of acoustic information and text information is an approach to the issue for the future. [ABSTRACT FROM AUTHOR]

The detection of mentioned aspects in product reviews is one of the significant and complex tasks in opinion mining. Recently, contextual-based approaches have significantly improved the accuracy of aspect extraction over non-contextual embeddings. However, these approaches are often computationally expensive and time-consuming; thus, applying such heavy models with insufficient resources and within runtime systems is impractical in many realistic scenarios. The present investigation sought an efficient, practical deep-learning-based model that relies on the complementary power of various existing non-contextual embeddings. In this regard, two morphology-based (character and FastText) and two syntax-based (POS and extended dependency skip-gram) embeddings were used alongside a base word embedding (GloVe) to form an enriched word representation layer. The presented model was integrated into the proposed network architecture (extended BiGRU). Finally, two novel post-processing rules were applied to refine the errors in the model's predictions. The proposed model achieved F-scores of 0.86, 0.91, 0.79, and 0.80 for the SemEval 2014 laptop domain and the SemEval 2015–2016 restaurant domain, respectively. Furthermore, the results were validated by comparing the computational and temporal efficiency of the proposed model with seven BERT-family transformers through statistical tests. [ABSTRACT FROM AUTHOR]

Information retrieval is a difficult process due to the overabundance of information on the web. Nowadays, search result responds to user queries with too many results although only a few are relevant. Therefore, the existing clustering methods that fail in clustering snippets (short texts) of web documents due to the low frequencies of document terms should be deeply investigated. One of the approaches that can be used to solve this problem is the expansion of document terms with semantically similar terms. Hence, a list of terms with their closest and accurate semantically similar words (word representation) must be built. This study aims to design and develop a new framework to enhance the performance of web search result clustering (WSRC). The research also presents a new unsupervised distributed word representation scheme where each word is represented by a vector of its semantically related words; such as scheme expands snippets and user queries. The proposed framework consists of several activities, such as (1) various standard datasets (Open Directory Project [ODP]-239 and MORESQUE) that are used for evaluating search result clustering algorithms for most cited dataset works, (2) text pre-processing, (3) document representation based on a new wiki-based k-nearest neighbors (KNN) representation method, (4) effect of the proposed model on the performance of traditional clustering methods (k-means, k-medoids, single-linkage, and complete-linkage) for WSRC, and (5) evaluation stage of the proposed method. Results indicate that enhanced clustering methods, according to the new wiki-KNN based representation method in comparison with the baseline methods, show a significant improvement in WSRC. Furthermore, the new data representation scheme has enhanced the overall performance of clustering methods. [ABSTRACT FROM AUTHOR]

Feature-listing tasks are an invaluable resource for exploring how words, categories, and metaphors are represented. However, manually coding the generated features is time-consuming and expensive, and involves subjective judgments from the experimenter. The purpose of this paper is to introduce the "RK processor", a program that was developed in our lab to analyse metaphor feature data but which can also be applied to other feature-listing data. After detailing the steps of processing, we demonstrate that the processed feature data align with previous findings in which metaphor features were processed manually and that the processed features predict dimensions of metaphor judgments pertaining to comprehensibility and metaphor goodness. Lastly, we present several other applications for research on word similarity, compound words, categories and concepts, semantic ambiguity, incongruity resolution and computational modelling. The RK processor offers researchers a valuable tool to save time and resources and to maintain consistency in processing. [ABSTRACT FROM AUTHOR]

Variability is pervasive in spoken language, in particular if one is exposed to two varieties of the same language (e.g., the standard variety and a dialect). Unlike in bilingual settings, standard and dialectal forms are often phonologically related, increasing the variability in word forms (e.g., German Fuß "foot" is produced as [fu s] in Standard German and as [f s] in the Alemannic dialect). We investigate whether dialectal variability in children's input affects their ability to recognize words in Standard German, testing non-dialectal vs. dialectal children. Non-dialectal children, who typically grow up in urban areas, mostly hear Standard German forms, and hence encounter little segmental variability in their input. Dialectal children in turn, who typically grow up in rural areas, hear both Standard German and dialectal forms, and are hence exposed to a large amount of variability in their input. We employ the familiar word paradigm for German children aged 12–18 months. Since dialectal children from rural areas are hard to recruit for laboratory studies, we programmed an App that allows all parents to test their children at home. Looking times to familiar vs. non-familiar words were analyzed using a semi-automatic procedure based on neural networks. Our results replicate the familiarity preference for non-dialectal German 12–18-month-old children (longer looking times to familiar words than vs. non-familiar words). Non-dialectal children in the same age range, on the other hand, showed a novelty preference. One explanation for the novelty preference in dialectal children may be more mature linguistic processing, caused by more variability of word forms in the input. This linguistic maturation hypothesis is addressed in Experiment 2, in which we tested older children (18–24-month-olds). These children, who are not exposed to dialectal forms, also showed a novelty preference. Taken together, our findings show that both dialectal and non-dialectal German children recognized the familiar Standard German word forms, but their looking pattern differed as a function of the variability in the input. Frequent exposure to both dialectal and Standard German word forms may hence have affected the nature of (prelexical and/or) lexical representations, leading to more mature processing capacities. [ABSTRACT FROM AUTHOR]

Cross-domain word representation aims to learn high-quality semantic representations in an under-resourced domain by leveraging information in a resourceful domain. However, most existing methods mainly transfer the semantics of common words across domains, ignoring the semantic relations among domain-specific words. In this paper, we propose a domain structure-based transfer learning method to learn cross-domain representations by leveraging the relations among domain-specific words. To accomplish this, we first construct a semantic graph to capture the latent domain structure using domain-specific co-occurrence information. Then, in the domain adaptation process, beyond domain alignment, we employ Laplacian Eigenmaps to ensure the domain structure is consistently distributed in the learned embedding space. As such, the learned cross-domain word representations not only capture shared semantics across domains, but also maintain the latent domain structure. We performed extensive experiments on two tasks, namely sentiment analysis and query expansion. The experiment results show the effectiveness of our method for tasks in under-resourced domains. • A cross-domain word representation method for tasks with limited resources. • Fusing the latent semantics of the source and target domains. • Propagating common semantic knowledge from source domain to target domain. • Preserving domain structure in the domain adaptation process. • Effective and efficient for text-related and low-resource tasks. [ABSTRACT FROM AUTHOR]

Interpretability is a significant aspect of the distributed word representation learning model. Although the most advanced pretrained models have achieved the best results till date, the interpretability of a pretrained model is difficult to explain clearly. For this reason, based on the interpretability of distributed word embeddings, this paper presents a method of learning word representation using joint context. At present, the existing distributed word representation models for learning word representations usually focus on either neighbor or syntactic context. We argue that it is necessary to simultaneously model both contexts. In particular, the point mutual information obtained by combining the two types of contexts can efficiently express the correlation between the words. We propose two alternative distribution models for learning word representations by employing the neighbor and syntactic contexts via a simple and effective joint learning framework. Furthermore, the proposed models are trained on a public corpus, and the learned representations are evaluated in word analogy, word similarity, and sentence classification tasks. The experimental results demonstrate the potential of the proposed method. [ABSTRACT FROM AUTHOR]

In the field of natural language processing, combining multiple pre-trained word embeddings has become a viable approach to improve word representations. However, there is still a lack of understanding of why such improvements can be achieved. In this paper, we investigate this issue by firstly proposing a novel word meta-embedding method. The proposed method tends to disentangle common and individual information from different word embeddings and learns representations for both. Based on the proposed method, we then carry out a systematic evaluation to provide a perspective on how common and individual information contributes to different tasks. Our intrinsic evaluation results suggest that common information is critical for word-level representations in terms of word similarity and relatedness. While, based on natural language inference, our extrinsic evaluation results show that common and individual information plays different roles and can complement each other. Further, both intrinsic and extrinsic evaluations reveal that explicitly separating common and individual information is beneficial for learning word meta-embeddings.

Bug triage processes are intended to assign bug reports to appropriate developers effectively, but they typically become bottlenecks in the development process-especially for large-scale software projects. Recently, several machine learning approaches, including deep learning-based approaches, have been proposed to recommend an appropriate developer automatically by learning past assignment patterns. In this paper, we propose a deep learning-based bug triage technique using a convolutional neural network (CNN) with three different word representation techniques: Word to Vector (Word2Vec), Global Vector (GloVe), and Embeddings from Language Models (ELMo). Experiments were performed on datasets from well-known large-scale open-source projects, such as Eclipse and Mozilla, and top-k accuracy was measured as an evaluation metric. The experimental results suggest that the ELMo-based CNN approach performs best for the bug triage problem. GloVe-based CNN slightly outperforms Word2Vec-based CNN in many cases. Word2Vec-based CNN outperforms GloVe-based CNN when the number of samples per class in the dataset is high enough.

Word embedding (i.e., word representation) transforms words into computable mathematical expressions (usually vectors) according to semantics. Compared with human semantic representation, these purely text-based models are severely deficient because they lack perceptual information attached to the physical world. This observation promotes the development of multimodal word representation models. Multimodal models have been proven to outperform text-based models on learning semantic word representations, and almost all previous multimodal models only focus on introducing perceptual information. However, it is obvious that syntactic information can effectively improve the performance of multimodal models on downstream tasks. Therefore, this article proposes an effective multimodal word representation model that uses two gate mechanisms to explicitly embed syntactic and phonetic information into multimodal representations and uses supervised learning to train the model. We select Chinese and English as examples and evaluate the model using several downstream tasks. The results show that our approach outperforms the existing models. We have made the source code of the model available to encourage reproducible research.

Variability is pervasive in spoken language, in particular if one is exposed to two varieties of the same language (e.g., the standard variety and a dialect). Unlike in bilingual settings, standard and dialectal forms are often phonologically related, increasing the variability in word forms (e.g., German Fuß “foot” is produced as [fus] in Standard German and as [fs] in the Alemannic dialect). We investigate whether dialectal variability in children’s input affects their ability to recognize words in Standard German, testing non-dialectal vs. dialectal children. Non-dialectal children, who typically grow up in urban areas, mostly hear Standard German forms, and hence encounter little segmental variability in their input. Dialectal children in turn, who typically grow up in rural areas, hear both Standard German and dialectal forms, and are hence exposed to a large amount of variability in their input. We employ the familiar word paradigm for German children aged 12–18 months. Since dialectal children from rural areas are hard to recruit for laboratory studies, we programmed an App that allows all parents to test their children at home. Looking times to familiar vs. non-familiar words were analyzed using a semi-automatic procedure based on neural networks. Our results replicate the familiarity preference for non-dialectal German 12–18-month-old children (longer looking times to familiar words than vs. non-familiar words). Non-dialectal children in the same age range, on the other hand, showed a novelty preference. One explanation for the novelty preference in dialectal children may be more mature linguistic processing, caused by more variability of word forms in the input. This linguistic maturation hypothesis is addressed in Experiment 2, in which we tested older children (18–24-month-olds). These children, who are not exposed to dialectal forms, also showed a novelty preference. Taken together, our findings show that both dialectal and non-dialectal German children recognized the familiar Standard German word forms, but their looking pattern differed as a function of the variability in the input. Frequent exposure to both dialectal and Standard German word forms may hence have affected the nature of (prelexical and/or) lexical representations, leading to more mature processing capacities.

Making inference on clinical texts is a task which has not been fully studied. With the newly released, expert annotated MedNLI dataset, this task is being boosted. Compared with open domain data, clinical texts present unique linguistic phenomena, e.g., a large number of medical terms and abbreviations, different written forms for the same medical concept, which make inference much harder. Incorporating domain-specific knowledge is a way to eliminate this problem, in this paper, we assemble a new incorporating medical concept definitions module on the classic enhanced sequential inference model (ESIM), which first extracts the most relevant medical concept for each word, if it exists, then encodes the definition of this medical concept with a bidirectional long short-term network (BiLSTM) to obtain domain-specific definition representations, and attends these definition representations over vanilla word embeddings. The empirical evaluations are conducted to demonstrate that our model improves the prediction performance and achieves a high level of accuracy on the MedNLI dataset. Specifically, the knowledge enhanced word representations contribute significantly to entailment class.

Inspired from the idea that the contexts in which a word occurs are of different significance, this paper proposes a novel method, called word representation with Salient Features (SaFe), to represent words using salient features selected from the context words. The SaFe method employs the point-wise mutual information (PMI) method with scaled context window to measure word association between a target word and its context. Then, contexts having word associations will be selected as salient features, where the number of salient features for a given word is decided by the ratio between the number of unique contexts and the total counts of occurrences in the whole corpus. The SaFe method can be used with the positive PMI matrix (PPMI), with each row representing a word, hence the name SaFe-PPMI. Moreover, the SaFe-PPMI model can be further decomposed by using the truncated singular vector decomposition technique to obtain dense vectors. In addition to efficient computation, the new models can achieve remarkable improvements in seven semantic relatedness tasks, and they show superior performance when compared with the state-of-the-art models.

With the rapid development of the Internet, the number of Internet users has grown rapidly, and the Internet has become more and more influential on people's lives. As a result, the amount of network text is increasing rapidly, and it is difficult to extract interested event information from it only by manual reading. Therefore, event extraction technique automatically extracting useful information from a large amount of unstructured texts becomes increasingly important. Event detection is the first step of event extraction task and plays a vital role in it. However, current event detection research lacks comprehensive consideration of the context of the trigger words. A Chinese event detection method based on multi-feature fusion and BiLSTM is proposed in this paper. The contextual information of word is divided into sentence-level and document-level in the method. The contextual information is captured based on BiLSTM model. At the same time, a word representation method suitable for trigger word classification tasks is proposed in this paper. The word representation incorporates semantic information, grammar information, and document-level context information of word. The word vectors in the sentence are sequentially inputted into BiLSTM model to obtain output vectors containing sentence-level contextual information. Finally, output vectors of BiLSTM are inputted into the Softmax classifier to realize the identification of the trigger words. The experimental results show that Chinese Event Detection Based on Multi-feature Fusion and BiLSTM method proposed in this paper has high accuracy.

• Syntactically-informed representations based on static and/or contextual representations. • Pre-encode syntactic information with automatically annotated data. • Improve performance over base representations on three information extraction tasks. • Syntactic dependencies can be beneficial for both static and contextual embeddings. • Easily adapt in different linguistic tasks than fine-tuning large models. Most deep language understanding models depend only on word representations, which are mainly based on language modelling derived from a large amount of raw text. These models encode distributional knowledge without considering syntactic structural information, although several studies have shown benefits of including such information. Therefore, we propose new syntactically-informed word representations (SIWRs), which allow us to enrich the pre-trained word representations with syntactic information without training language models from scratch. To obtain SIWRs, a graph-based neural model is built on top of either static or contextualised word representations such as GloVe, ELMo and BERT. The model is first pre-trained with only a relatively modest amount of task-independent data that are automatically annotated using existing syntactic tools. SIWRs are then obtained by applying the model to downstream task data and extracting the intermediate word representations. We finally replace word representations in downstream models with SIWRs for applications. We evaluate SIWRs on three information extraction tasks, namely nested named entity recognition (NER), binary and n -ary relation extractions (REs). The results demonstrate that our SIWRs yield performance gains over the base representations in these NLP tasks with 3–9% relative error reduction. Our SIWRs also perform better than fine-tuning BERT in binary RE. We also conduct extensive experiments to analyse the proposed method. [ABSTRACT FROM AUTHOR]

Abstract Background Biomedical event extraction is a crucial task in biomedical text mining. As the primary forum for international evaluation of different biomedical event extraction technologies, BioNLP Shared Task represents a trend in biomedical text mining toward fine-grained information extraction (IE). The fourth series of BioNLP Shared Task in 2016 (BioNLP-ST’16) proposed three tasks, in which the Bacteria Biotope event extraction (BB) task has been put forward in the earlier BioNLP-ST. Deep learning methods provide an effective way to automatically extract more complex features and achieve notable results in various natural language processing tasks. Results The experimental results show that the presented approach can achieve an F-score of 57.42% in the test set, which outperforms previous state-of-the-art official submissions to BioNLP-ST 2016. Conclusions In this paper, we propose a novel Gated Recurrent Unit Networks framework integrating attention mechanism for extracting biomedical events between biotope and bacteria from biomedical literature, utilizing the corpus from the BioNLP’16 Shared Task on Bacteria Biotope task. The experimental results demonstrate the potential and effectiveness of the proposed framework.

Convolutional neural networks (CNNs) are widely used in many natural language processing tasks, which employ some convolutional filters to capture useful semantic features of a text. However, a small window size convolutional filter is short of the ability to capture contextual information, simply increasing the window size may bring the problems of data sparsity and enormous parameters. To capture the contextual information, we propose to use the weighted sum operation to obtain contextual word representation. We present one implicit weighting method and two explicit category-aware weighting methods to assign the weights of the contextual information. Experimental results on five text classification datasets show the effectiveness of our proposed methods. [ABSTRACT FROM AUTHOR]

The meaning of a word includes a conceptual meaning and a distributive meaning. Word embedding based on distribution suffers from insufficient conceptual semantic representation caused by data sparsity, especially for low-frequency words. In knowledge bases, manually annotated semantic knowledge is stable and the essential attributes of words are accurately denoted. In this paper, we propose a Conceptual Semantics Enhanced Word Representation (CEWR) model, computing the synset embedding and hypernym embedding of Chinese words based on the Tongyici Cilin thesaurus, and aggregating it with distributed word representation to have both distributed information and the conceptual meaning encoded in the representation of words. We evaluate the CEWR model on two tasks: word similarity computation and short text classification. The Spearman correlation between model results and human judgement are improved to 64.71%, 81.84%, and 85.16% on Wordsim297, MC30, and RG65, respectively. Moreover, CEWR improves the F1 score by 3% in the short text classification task. The experimental results show that CEWR can represent words in a more informative approach than distributed word embedding. This proves that conceptual semantics, especially hypernymous information, is a good complement to distributed word representation. [ABSTRACT FROM AUTHOR]

Word similarity (WS) is a fundamental and critical task in natural language processing. Existing approaches to WS are mainly to calculate the similarity or relatedness of word pairs based on word embedding obtained by massive and high-quality corpus. However, it may suffer from poor performance for insufficient corpus in some specific fields, and cannot capture rich semantic and sentimental information. To address these above problems, we propose an enhancing embedding-based word similarity evaluation with character-word concepts and synonyms knowledge, namely EWS-CS model, which can provide extra semantic information to enhance word similarity evaluation. The core of our approach contains knowledge encoder and word encoder. In knowledge encoder, we incorporate the semantic knowledge extracted from knowledge resources, including character-word concepts, synonyms and sentiment lexicons, to obtain knowledge representation. Word encoder is to learn enhancing embedding-based word representation from pre-trained model and knowledge representation based on similarity task. Finally, compared with baseline models, the experiments on four similarity evaluation datasets validate the effectiveness of our EWS-CS model in WS task. [ABSTRACT FROM AUTHOR]

In the history of natural language processing (NLP) development, the representation of words has always been a significant research topic. In this survey, we provide a comprehensive typology of word representation models from a novel perspective that the development from static to dynamic embeddings can effectively address the polysemy problem, which has been a great challenge in this field. Then the survey covers the main evaluation metrics and applications of these word embeddings. And, we further discuss the development of word embeddings from static to dynamic in cross-lingual scenario. Finally, we point out some open issues and future works. [ABSTRACT FROM AUTHOR]