Search

Word Embedding plays a crucial role in natural language processing, and other related domains. The vast variety of language modelling and feature learning techniques often concludes in a quandary. The motivation behind this work was to produce comparative analysis among these methods and finally use them to flag hate-speech on social media. The progress in these word embedding techniques has led to remarkable results by incorporating various natural language applications. Understanding the different context of polysemous words is one of the features that evolved over time with these word embedding models. A systematic review on varying word embedding methodologies has been performed in this paper. Various experimental metrics have been used and detailed analysis has been done on each word embedding model. It is shown that analysis involves various aspects of the model like dealing with multi-sense words, and rarely occurring words, etc., and finally a coherent analysis report is presented. The various models under analysis are—Word2Vec (Skip-Gram, CBOW), GloVe, Fast-Text and ELMo. These models are then put to a real-life application in the form of Hate Speech detection of twitter data, and their individual capacities and accuracies are compared. Through this paper we show how ELMo uses different word embeddings for polysemous words to capture the context. We show how Hate speech can be better detected by ELMo because such speech requires better understanding of context of words for segregation from normal speech/text.

Word embedding or universal embeddings are the representation of textual data into vectors of real numbers. It acts as a link between the human understanding of a text to that of the machine. It maps the high-dimensional textual data into a vector space of low dimension, which represents the complex relationships existing in the data. Thus, it boosts the performance of tasks involving natural language processing. To know about the popularity gain and the impact of this field, research literature between 2000 and 2019 is analyzed with the help of a scientometric mapping for research done in word embeddings. The paper visualizes year-wise analysis, demographic analysis, category-wise distribution, and document type-wise distribution of the publications indexed in Web of Science (WoS). The paper also comprises a comparative study of Word2Vec, FastText, global vector representation of words, and bidirectional encoder representations for transformers. Pre-trained models are used for experimental comparison. Performance is measured by calculating the deviation of the similarity score of two words given by the models from manually assigned similarity scores by experts, repeated over a list of words on various datasets. The least deviation is shown by FastText due to the usage of morphological information in the skip-gram model and n-gram architecture.

The exponentially evolving size of information today has made it difficult to find relevant information quickly and efficiently. A good extractive text summarizer not only provides the most significant information from the document but also helps the user to decide the relevance of the information. The proposed method is a knowledge-based, generic, extractive text summarization technique. Our approach is based on the centrality of a sentences in the graphical representation of the documents. The graph is constructed using the pair-wise softcosine similarity measures between the sentences derived using the S semantic relations presented in WordNet lexical database. Eigenvector centrality measure outperforms the weighted degree, betweenness and closeness centrality measures. The resultant summary is compared against the gold-standard summaries of BBC news articles from year 2004 to 2005 and DUC 2007 datasets. The ROUGE-I, -2 and -Lmetrices are used to evaluate the results and found that our approach performs better than LexRank, TextRank, Luhn and LSA baseline text summarizers.

In this article, we propose a system called “UTTAM,” for correcting spelling errors in Hindi language text using supervised learning. Unlike other languages, Hindi contains a large set of characters, words with inflections and complex characters, phonetically similar sets of characters, and so on. The complexity increases the possibility of confusion and occasionally leads to entering a wrong character in a word. The existence of spelling errors in text significantly decreases the accuracy of the available resources, like search engine, text editor, and so on. The proposed work is the first approach to correct non-word (Out of Vocabulary) errors as well as real-word errors simultaneously in a sentence of Hindi language. The proposed method investigates the human behavior, i.e., the type and frequency of spelling errors done by humans in Hindi text. Based on the type and frequency of spelling errors, the heterogeneous data is collected in matrices. This data in matrices is used to generate the suitable candidate words for an input word. After generating candidate words, the Viterbi algorithm is applied to perform the word correction. The Viterbi algorithm finds the best sequence of candidate words to correct the input sentence. For Hindi, this work is the first attempt for real-word error correction. For non-word errors, the experiments show that “UTTAM” performs better than the existing systems SpellGuru and Saksham.

Warning: This manuscript may contain upsetting language. Social media has become a bedrock for people to voice their opinions worldwide. Due to the greater sense of freedom with the anonymity feature, it is possible to disregard social etiquette online and attack others without facing severe consequences, inevitably propagating hate speech. The current measures to sift the online content and offset the hatred spread do not go far enough. One factor contributing to this is the prevalence of regional languages in social media and the paucity of language flexible hate speech detectors. The proposed work focuses on analyzing hate speech in Hindi–English code-switched language. Our method explores transformation techniques to capture precise text representation. To contain the structure of data and yet use it with existing algorithms, we developed ‘MoH’ or (Map Only Hindi), which means ‘Love’ in Hindi. ‘MoH’ pipeline which consists of language identification, Roman to Devanagari Hindi transliteration using a knowledge base of Roman Hindi words, and finally employs the fine-tuned Multilingual Bert, and MuRIL language models. We conducted several quantitative experiment studies on three datasets, and evaluated performance using Precision, Recall and F1 metrics. The first experiment studies ‘MoH’ mapped text’s performance with classical machine learning models and shows an average increase of 13% in F1 scores. The second compares the proposed work’s scores with those of the baseline models and shows a rise in performance by 6%. Finally, the third compares the proposed ‘MoH’ technique with various data simulations using the existing transliteration library. Here, ‘MoH’ outperforms the rest by 15%. Our results demonstrate a significant improvement in the state-of-the-art scores on all three datasets.

Here we propose an open source algorithm, L,M&A(Lyrics, Mine and Analyse) to create a dataset of lyrics of the works of various artists. The aim of this approach is to facilitate the generation of a large data set that can be used for improving accuracy of song recommendation algorithms. The limited availability of such datasets has excluded the sentiment analysis of lyrics from music recommendation systems. By using the L,M&A algorithm, it is possible to generate a large dataset which can function as training dataset for future classifier systems. We have used iterative API requests from musixmatch and Genius servers to text mine lyrics data of songs by multiple artists. The data is processed and then analysed for sentiment using lexicons provided in the Tidytext package (BING, AFINN, NRC) and the overall sentiment of artist was determined through modal counts. The occurrence of each sentiments was evaluated and visualized using ggplot2. This representation exhibits the merit of our approach and the applicability of our data. The key feature of our approach is the open source platforms utilized and simplicity of input required from user.

This paper presents keyword extraction using lexical chains and graph centrality measures, derived from the semantic similarity of the words by analysis of the graphical network created using WordNet. The hypothesis is presented using a small-world approach where every paragraph in a document is constrained to a local point, while the document in all is centered on a global concept. Creating lexical chains for each paragraph and combining the best via scoring methods and graph based algorithms, we present parallels to baseline system to extract the keywords from the document.

This research article deals with translating a sentence from one language toanother. Our motive is to provide a technique that converts web-pages from one language to another automatically using Resource Description Framework (RDF) graphs. We particularly focus on making translation independent of human interference. Therefore, this research paper presents a framework and provides its implementationfor converting English sentences to Hindi using RDF and Pythonprogramming language. As a result of our research work, we successfullytranslatedour sample sentences present in English to Hindi.

Hindi, the majority language of India, is still in its infancy stage concerning to natural language processing applications and research. Spelling detection and correction for Hindi language is an important process which has received inadequate attention. Today, there is lots of work available for English Spelling detection and correction but for Hindi, not much work has been done. Unlike English which has only two types of letters i.e. 21 consonants and 5 vowels, Hindi language in addition to 40 consonants, 10 vowels, consists of various types of symbols (ten vowel sign (matras), half letters & halant etc.). So the methods available for English cannot be directly applied for Hindi. Various documents, newspaper, novels, books, magazines, government notice etc. are typed in Hindi so there is need for development of spelling detection and correction tools for Hindi. In this paper author focused on detection and correction of spelling errors in Hindi language for non-word errors.