Search

The proliferation of Internet of Things (IoT) devices has introduced significant security challenges, including weak authentication, insufficient data protection, and firmware vulnerabilities. To address these issues, we propose a linguistic secret sharing scheme tailored for IoT applications. This scheme leverages neural networks to embed private data within texts transmitted by IoT devices, using an ambiguous token selection algorithm that maintains the textual integrity of the cover messages. Our approach eliminates the need to share additional information for accurate data extraction while also enhancing security through a secret sharing mechanism. Experimental results demonstrate that the proposed scheme achieves approximately 50% accuracy in detecting steganographic text across two steganalysis networks. Additionally, the generated steganographic text preserves the semantic information of the cover text, evidenced by a BERT score of 0.948. This indicates that the proposed scheme performs well in terms of security. [ABSTRACT FROM AUTHOR]

Word embeddings are a fundamental tool in natural language processing. Currently, word embedding methods are evaluated on the basis of empirical performance on benchmark data sets, and there is a lack of rigorous understanding of their theoretical properties. This paper studies word embeddings from a statistical theoretical perspective, which is essential for formal inference and uncertainty quantification. We propose a copula-based statistical model for text data and show that under this model, the now-classical Word2Vec method can be interpreted as a statistical estimation method for estimating the theoretical pointwise mutual information (PMI). We further illustrate the utility of this statistical model by using it to develop a missing value-based estimator as a statistically tractable and interpretable alternative to the Word2Vec approach. The estimation error of this estimator is comparable to Word2Vec and improves upon the truncation-based method proposed by Levy and Goldberg (Adv. Neural Inf. Process. Syst., 27, 2177–2185 2014). The resulting estimator also is comparable to Word2Vec in a benchmark sentiment analysis task on the IMDb Movie Reviews data set and a part-of-speech tagging task on the OntoNotes data set. [ABSTRACT FROM AUTHOR]

• The accurate identification of phosphorylation sites in SARS-CoV-2-infected host cells would contribute to the investigation of SARS-CoV-2 pathogenic mechanism and the mining of candidate therapy targets. • This work constructed a phosphorylation site computational approach (named PhosBERT) based on a pre-trained protein language model through self-supervised learning from the BERT architecture. • Prediction accuracy and AUC for SARS-CoV-2-infected serine and threonine phosphorylation sites are 81.9% and 0.896. • Prediction accuracy and AUC for SARS-CoV-2-infected tyrosine phosphorylation sites are 87.1% and 0.902. Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is a single-stranded RNA virus, which mainly causes respiratory and enteric diseases and is responsible for the outbreak of coronavirus disease 19 (COVID-19). Numerous studies have demonstrated that SARS-CoV-2 infection will lead to a significant dysregulation of protein post-translational modification profile in human cells. The accurate recognition of phosphorylation sites in host cells will contribute to a deep understanding of the pathogenic mechanisms of SARS-CoV-2 and also help to screen drugs and compounds with antiviral potential. Therefore, there is a need to develop cost-effective and high-precision computational strategies for specifically identifying SARS-CoV-2-infected phosphorylation sites. In this work, we first implemented a custom neural network model (named PhosBERT) on the basis of a pre-trained protein language model of ProtBert, which was a self-supervised learning approach developed on the Bidirectional Encoder Representation from Transformers (BERT) architecture. PhosBERT was then trained and validated on serine (S) and threonine (T) phosphorylation dataset and tyrosine (Y) phosphorylation dataset with 5-fold cross-validation, respectively. Independent validation results showed that PhosBERT could identify S/T phosphorylation sites with high accuracy and AUC (area under the receiver operating characteristic) value of 81.9% and 0.896. The prediction accuracy and AUC value of Y phosphorylation sites reached up to 87.1% and 0.902. It indicated that the proposed model was of good prediction ability and stability and would provide a new approach for studying SARS-CoV-2 phosphorylation sites. [ABSTRACT FROM AUTHOR]

The flight arrival scheduling problem is one of the critical tasks in air traffic operations, aiming to ensure that the flight arrive in the correct sequence safely. Existing methods primarily focus on the terminal area and often overlook the presence of training flight at the airport. Due to the limited generalization of traditional methods and varying control practices at different airports, training flight at airports still rely on manual control for arrival sorting. To effectively address these issues, we propose a novel method for slot allocation that leverages the strong reasoning capabilities and generalization potential of large language models (LLMs). Our method conceptualizes the dynamic scheduling problem for training flight as a language modeling problem, a perspective not previously explored. Specifically, we represent the allocator's inputs and outputs as language tokens, utilizing LLMs to generate conflict-free results based on a language description of requested landing information and assigned training flight information. Additionally, we employ a reset strategy to create a small dataset for scenario-specific samples, enabling LLMs to quickly learn allocation schemes from the dataset. We demonstrated the capability of LLMs in addressing time conflicts by evaluating metrics such as answer accuracy, conflict rate, and total delay time (without the wrong answer). These findings underscore the feasibility of employing LLMs in the field of air traffic control. [ABSTRACT FROM AUTHOR]

Cephalo is presented as a series of multimodal vision large language models (V‐LLMs) designed for materials science applications, integrating visual and linguistic data for enhanced understanding. A key innovation of Cephalo is its advanced dataset generation method. Cephalo is trained on integrated image and text data from thousands of scientific papers and science‐focused Wikipedia data demonstrates it can interpret complex visual scenes, generate precise language descriptions, and answer queries about images effectively. The combination of a vision encoder with an autoregressive transformer supports multimodal natural language understanding, which can be coupled with other generative methods to create an image‐to‐text‐to‐3D pipeline. To develop more capable models from smaller ones, both mixture‐of‐expert methods and model merging are reported. The models are examined in diverse use cases that incorporate biological materials, fracture and engineering analysis, protein biophysics, and bio‐inspired design based on insect behavior. Generative applications include bio‐inspired designs, including pollen‐inspired architected materials, as well as the synthesis of bio‐inspired material microstructures from a photograph of a solar eclipse. Additional model fine‐tuning with a series of molecular dynamics results demonstrate Cephalo's enhanced capabilities to accurately predict statistical features of stress and atomic energy distributions, as well as crack dynamics and damage in materials. [ABSTRACT FROM AUTHOR]

Quantized large language models are large language models (LLMs) optimized for model size while preserving their efficacy. They can be executed on consumer-grade computers without the powerful features of dedicated servers needed to execute regular (non-quantized) LLMs. Because of their ability to summarize, answer questions, and provide insights, LLMs are being used to analyze large texts/documents. One of these types of large texts/documents are Internet of Things (IoT) privacy policies, which are documents specifying how smart home gadgets, health-monitoring wearables, and personal voice assistants (among others) collect and manage consumer/user data on behalf of Internet companies providing services. Even though privacy policies are important, they are difficult to comprehend due to their length and how they are written, which makes them attractive for analysis using LLMs. This study evaluates how quantized LLMs are modeling the language of privacy policies to be potentially used to transform IoT privacy policies into simpler, more usable formats, thus aiding comprehension. While the long-term goal is to achieve this usable transformation, our work focuses on evaluating quantized LLM models used for IoT privacy policy language. Particularly, we study 4-bit, 5-bit, and 8-bit quantized versions of the large language model Meta AI version 2 (Llama 2) and the base Llama 2 model (zero-shot, without fine-tuning) under different metrics and prompts to determine how well these quantized versions model the language of IoT privacy policy documents by completing and generating privacy policy text. [ABSTRACT FROM AUTHOR]

This essay demonstrates the value of using artificial intelligence (AI) technologies to address specific kinds of research questions in rhetoric. The essay builds on a study of a novel rhetorical object first observed by Yang on the Reddit subreddit r/wallstreetbets. We demonstrate how the rhetorical structure of "pathologics" (1) generated a kind of rhetorical authority that can be measured by higher-than-average user engagement on Reddit and (2) circulated from Reddit into more traditional legacy media. Through our research on the rhetorical circulation of pathologics, we argue that researching rhetoric with AI can center new ways of knowing about concepts relevant in rhetoric, like circulation and rhetorical ecosystems. Further, we argue that researching rhetoric with AI always also entails considering a "rhetoric of AI," requiring critical attention to the platforms, infrastructures, and data sources connected to AI systems. [ABSTRACT FROM AUTHOR]

The flight arrival scheduling problem is one of the critical tasks in air traffic operations, aiming to ensure that the flight arrive in the correct sequence safely. Existing methods primarily focus on the terminal area and often overlook the presence of training flight at the airport. Due to the limited generalization of traditional methods and varying control practices at different airports, training flight at airports still rely on manual control for arrival sorting. To effectively address these issues, we propose a novel method for slot allocation that leverages the strong reasoning capabilities and generalization potential of large language models (LLMs). Our method conceptualizes the dynamic scheduling problem for training flight as a language modeling problem, a perspective not previously explored. Specifically, we represent the allocator’s inputs and outputs as language tokens, utilizing LLMs to generate conflict-free results based on a language description of requested landing information and assigned training flight information. Additionally, we employ a reset strategy to create a small dataset for scenario-specific samples, enabling LLMs to quickly learn allocation schemes from the dataset. We demonstrated the capability of LLMs in addressing time conflicts by evaluating metrics such as answer accuracy, conflict rate, and total delay time (without the wrong answer). These findings underscore the feasibility of employing LLMs in the field of air traffic control.

Spoken keyword detection is a technique used to identify specific keywords or phrases in spoken language. It is often used in the field of speech recognition to trigger specific actions or responses. For example, a spoken keyword detection system might be used to activate a voicecontrolled assistant or to initiate a search query. Keyword detection systems can be trained to recognize a wide range of keywords, depending on the specific needs of the application. They may use machine learning techniques to analyze the audio signal and identify patterns that correspond to specific keywords. [ABSTRACT FROM AUTHOR]

Abstract This article presents the research work on improving speech recognition systems for the morphologically complex Malayalam language using subword tokens for language modeling. The speech recognition system is built using a deep neural network–hidden Markov model (DNN-HMM)-based automatic speech recognition (ASR). We propose a novel method, syllable-byte pair encoding (S-BPE), that combines linguistically informed syllable tokenization with the data-driven tokenization method of byte pair encoding (BPE). The proposed method ensures words are always segmented at valid pronunciation boundaries. On a text corpus that has been divided into tokens using the proposed method, we construct statistical n-gram language models and assess the modeling effectiveness in terms of both information-theoretic and corpus linguistic metrics. A comparative study of the proposed method with other data-driven (BPE, Morfessor, and Unigram), linguistic (Syllable), and baseline (Word) tokenization algorithms is also presented. Pronunciation lexicons of subword tokenized units are built with pronunciation described as graphemes. We develop ASR systems employing the subword tokenized language models and pronunciation lexicons. The resulting ASR models are comprehensively evaluated to answer the research questions regarding the impact of subword tokenization algorithms on language modeling complexity and on ASR performance. Our study highlights the strong performance of the hybrid S-BPE tokens, achieving a notable 10.6% word error rate (WER), which represents a substantial 16.8% improvement over the baseline word-level ASR system. The ablation study has revealed that the performance of S-BPE segmentation, which initially underperformed compared to syllable tokens with lower amounts of textual data for language modeling, exhibited steady improvement with the increase in LM training data. The extensive ablation study indicates that there is a limited advantage in raising the n-gram order of the language model beyond $$n=3$$ n = 3 . Such an increase results in considerable model size growth without significant improvements in WER. The implementation of the algorithm and all associated experiments are available under an open license, allowing for reproduction, adaptation, and reuse.

Under the background of the development of network technology, this paper aims to promote rural English teaching and constructs an English teaching model that combines English recognition technology and rural teaching. The main process of speech recognition is examined by analyzing different speech recognition technologies. Using a deep learning network, an English speech recognition model has been established. Combined with the English acoustic features in the network data, fluency of English speech is evaluated. Data embedding is performed on the English sequences in the network, combined with the sequence probability in the English data, so as to determine whether the English speech is correct or not. The Eval value for the English recognition model based on deep learning is 5.49%, while the test value is 5.89%, as per the results. As the English dataset increases, so does the English recognition technique proposed in this paper, and the accuracy remains above 0.6, and when the dataset is 500, the speech recognition accuracy is 0.8. The teaching model that combines speech recognition techniques with English teaching improves students’ English to a certain extent.

This article provides a comprehensive survey of contemporary language modeling approaches within the realm of natural language processing (NLP) tasks. This paper conducts an analytical exploration of diverse methodologies employed in the creation of language models. This exploration encompasses the architecture, training processes, and optimization strategies inherent in these models. The detailed discussion covers various models ranging from traditional n-gram and hidden Markov models to state-of-the-art neural network approaches such as BERT, GPT, LLAMA, and Bard. This article delves into different modifications and enhancements applied to both standard and neural network architectures for constructing language models. Special attention is given to addressing challenges specific to agglutinative languages within the context of developing language models for various NLP tasks, particularly for Arabic and Turkish. The research highlights that contemporary transformer-based methods demonstrate results comparable to those achieved by traditional methods employing Hidden Markov Models. These transformer-based approaches boast simpler configurations and exhibit faster performance during both training and analysis. An integral component of the article is the examination of popular and actively evolving libraries and tools essential for constructing language models. Notable tools such as NLTK, TensorFlow, PyTorch, and Gensim are reviewed, with a comparative analysis considering their simplicity and accessibility for implementing diverse language models. The aim is to provide readers with insights into the landscape of contemporary language modeling methodologies and the tools available for their implementation. [ABSTRACT FROM AUTHOR]

Parallel natural language processing systems were previously successfully tested on the tasks of part-of-speech tagging and authorship attribution through mini-language modeling, for which they achieved significantly better results than independent methods in the cases of seven European languages. The aim of this paper is to present the advantages of using composite language models in the processing and evaluation of texts written in arbitrary highly inflective and morphology-rich natural language, particularly Serbian. A perplexity-based dataset, the main asset for the methodology assessment, was created using a series of generative pre-trained transformers trained on different representations of the Serbian language corpus and a set of sentences classified into three groups (expert translations, corrupted translations, and machine translations). The paper describes a comparative analysis of calculated perplexities in order to measure the classification capability of different models on two binary classification tasks. In the course of the experiment, we tested three standalone language models (baseline) and two composite language models (which are based on perplexities outputted by all three standalone models). The presented results single out a complex stacked classifier using a multitude of features extracted from perplexity vectors as the optimal architecture of composite language models for both tasks. [ABSTRACT FROM AUTHOR]

This article presents the research work on improving speech recognition systems for the morphologically complex Malayalam language using subword tokens for language modeling. The speech recognition system is built using a deep neural network–hidden Markov model (DNN-HMM)-based automatic speech recognition (ASR). We propose a novel method, syllable-byte pair encoding (S-BPE), that combines linguistically informed syllable tokenization with the data-driven tokenization method of byte pair encoding (BPE). The proposed method ensures words are always segmented at valid pronunciation boundaries. On a text corpus that has been divided into tokens using the proposed method, we construct statistical n-gram language models and assess the modeling effectiveness in terms of both information-theoretic and corpus linguistic metrics. A comparative study of the proposed method with other data-driven (BPE, Morfessor, and Unigram), linguistic (Syllable), and baseline (Word) tokenization algorithms is also presented. Pronunciation lexicons of subword tokenized units are built with pronunciation described as graphemes. We develop ASR systems employing the subword tokenized language models and pronunciation lexicons. The resulting ASR models are comprehensively evaluated to answer the research questions regarding the impact of subword tokenization algorithms on language modeling complexity and on ASR performance. Our study highlights the strong performance of the hybrid S-BPE tokens, achieving a notable 10.6% word error rate (WER), which represents a substantial 16.8% improvement over the baseline word-level ASR system. The ablation study has revealed that the performance of S-BPE segmentation, which initially underperformed compared to syllable tokens with lower amounts of textual data for language modeling, exhibited steady improvement with the increase in LM training data. The extensive ablation study indicates that there is a limited advantage in raising the n-gram order of the language model beyond n = 3 . Such an increase results in considerable model size growth without significant improvements in WER. The implementation of the algorithm and all associated experiments are available under an open license, allowing for reproduction, adaptation, and reuse. [ABSTRACT FROM AUTHOR]

Quantized large language models are large language models (LLMs) optimized for model size while preserving their efficacy. They can be executed on consumer-grade computers without the powerful features of dedicated servers needed to execute regular (non-quantized) LLMs. Because of their ability to summarize, answer questions, and provide insights, LLMs are being used to analyze large texts/documents. One of these types of large texts/documents are Internet of Things (IoT) privacy policies, which are documents specifying how smart home gadgets, health-monitoring wearables, and personal voice assistants (among others) collect and manage consumer/user data on behalf of Internet companies providing services. Even though privacy policies are important, they are difficult to comprehend due to their length and how they are written, which makes them attractive for analysis using LLMs. This study evaluates how quantized LLMs are modeling the language of privacy policies to be potentially used to transform IoT privacy policies into simpler, more usable formats, thus aiding comprehension. While the long-term goal is to achieve this usable transformation, our work focuses on evaluating quantized LLM models used for IoT privacy policy language. Particularly, we study 4-bit, 5-bit, and 8-bit quantized versions of the large language model Meta AI version 2 (Llama 2) and the base Llama 2 model (zero-shot, without fine-tuning) under different metrics and prompts to determine how well these quantized versions model the language of IoT privacy policy documents by completing and generating privacy policy text.

One-dimensional convolutional models are used for various natural language processing tasks. This study revisits Dynamic Convolutional Neural Network (DCNN) architecture. The study investigates the effect of language modeling pre-training on Wikicorpus on published experiment results for DCNN. Therefore, the reference study integrates a top layer for the language-modeling training into DCNN. Also, benchmarks were reported for the original DCNN compared to the pre-trained language model version. The revisited model was then benchmarked for sentiment classification and question classification tasks. Benchmarks include transfer learning from pre-trained DCNN for language modeling and ground-up trained versions of DCNN on Stanford Sentiment Tree Bank and TREC Question Classification datasets. [ABSTRACT FROM AUTHOR]

To model behavioral and neural correlates of language comprehension in naturalistic environments, researchers have turned to broad‐coverage tools from natural‐language processing and machine learning. Where syntactic structure is explicitly modeled, prior work has relied predominantly on context‐free grammars (CFGs), yet such formalisms are not sufficiently expressive for human languages. Combinatory categorial grammars (CCGs) are sufficiently expressive directly compositional models of grammar with flexible constituency that affords incremental interpretation. In this work, we evaluate whether a more expressive CCG provides a better model than a CFG for human neural signals collected with functional magnetic resonance imaging (fMRI) while participants listen to an audiobook story. We further test between variants of CCG that differ in how they handle optional adjuncts. These evaluations are carried out against a baseline that includes estimates of next‐word predictability from a transformer neural network language model. Such a comparison reveals unique contributions of CCG structure‐building predominantly in the left posterior temporal lobe: CCG‐derived measures offer a superior fit to neural signals compared to those derived from a CFG. These effects are spatially distinct from bilateral superior temporal effects that are unique to predictability. Neural effects for structure‐building are thus separable from predictability during naturalistic listening, and those effects are best characterized by a grammar whose expressive power is motivated on independent linguistic grounds. [ABSTRACT FROM AUTHOR]

This paper describes the process of building the first large-scale machinetranslated question answering dataset SQuAD-sk for the Slovak language. The dataset was automatically translated from the original English SQuAD v2.0 using the Marian neural machine translation together with the Helsinki-NLP Opus English-Slovak model. Moreover, we proposed an effective approach for the approximate search of the translated answer in the translated paragraph based on measuring their similarity using their word vectors. In this way, we obtained more than 92% of the translated questions and answers from the original English dataset. We then used this machine-translated dataset to train the Slovak question answering system by fine-tuning monolingual and multilingual BERT-based language models. The scores achieved by EM = 69.48% and F1 = 78.87% for the fine-tuned mBERT model show comparable results of question answering with recently published machinetranslated SQuAD datasets for other European languages. [ABSTRACT FROM AUTHOR]

Long text generation is a challenging yet unsolved task. To generate long, coherent, and consistent text, existing approaches need to increase the language model length accordingly. However, the cost of the computational and memory resources grows as the square of the length. Even trained with thousands of GPUs, the length of language models is still limited to a few thousand, which may cause the generation of longer texts to be inconsistent with the topics and ideas in their preceding texts. To address this, we propose a novel Transformer architecture called Transformer with Local and Global Memory (Transformer LGM). It is inspired by the way people write long articles, which generate a key idea first and then guide the writing of the entire article with the idea in mind. Such a "key idea" can be put into the fixed global memory of the Transformer LGM to guide the whole generation process. On the contrary, the local memory, which is responsible for local coherence, could shift and drop with the increasing length of the generated text. We implement the global memory by introducing a negative positional embedding, while the traditional positive positional embedding is still used for the local memory. Experiments show that by utilizing the global memory, our model could generate long, coherent, and consistent text without enlarging the length of the language model. [ABSTRACT FROM AUTHOR]

Summary: Text recognition has attracted increased attention recently as a result of the complexity of natural settings and the variety of text instances. Various text or character recognition methods are introduced to distinguish the text from the natural scene, but existing methods struggle with the distorted and highly curved text instances. Consequently, an effective method for occluded text or character detection from object‐background images is developed using the suggested elephant herding exponential sailfish optimizer‐based generative adversarial network (EHESFO‐based GAN). In order to build the proposed EHESFO, elephant herding optimization and Exponential SailFish Optimizer (ESFO) are merged. ESFO was created by fusing exponentially weighted moving average and SailFish Optimizer. With GAN, features extracted from the background and foreground of an image are efficiently used for image annotation and text recognition. The best features from the background and foreground images are extracted to create the optimal solution, which increases the efficacy and efficiency of text recognition. While taking the occlusion as 0.4, the proposed EHESFO‐based GAN achieved higher accuracy of 98.1090% and lower error of 1.4%. [ABSTRACT FROM AUTHOR]

Leveraging generative Artificial Intelligence (AI), we have transformed a dataset comprising 1000 scientific papers focused on biological materials into a comprehensive ontological knowledge graph. Through an in-depth structural analysis of this graph, we have calculated node degrees, identified communities along with their connectivities, and evaluated clustering coefficients and betweenness centrality of pivotal nodes, uncovering fascinating knowledge architectures. We find that the graph has an inherently scale-free nature, shows a high level of connectedness, and can be used as a rich source for downstream graph reasoning by taking advantage of transitive and isomorphic properties to reveal insights into unprecedented interdisciplinary relationships that can be used to answer queries, identify gaps in knowledge, propose never-before-seen material designs, and predict material behaviors. Using a large language embedding model we compute deep node representations and use combinatorial node similarity ranking to develop a path sampling strategy that allows us to link dissimilar concepts that have previously not been related. One comparison revealed detailed structural parallels between biological materials and Beethoven’s 9th Symphony, highlighting shared patterns of complexity through isomorphic mapping. In another example, the algorithm proposed an innovative hierarchical mycelium-based composite based on integrating path sampling with principles extracted from Kandinsky’s ‘Composition VII’ painting. The resulting material integrates an innovative set of concepts that include a balance of chaos and order, adjustable porosity, mechanical strength, and complex patterned chemical functionalization. We uncover other isomorphisms across science, technology and art, revealing a nuanced ontology of immanence that reveal a context-dependent heterarchical interplay of constituents. Because our method transcends established disciplinary boundaries through diverse data modalities (graphs, images, text, numerical data, etc), graph-based generative AI achieves a far higher degree of novelty, explorative capacity, and technical detail, than conventional approaches and establishes a widely useful framework for innovation by revealing hidden connections.

Data imbalance in datasets is a common issue where the number of instances in one or more categories far exceeds the others, so is the case with the educational domain. Collecting feedback on a course on a large scale and the lack of publicly available datasets in this domain limits models’ performance, especially for deep neural network based models which are data hungry. A model trained on such an imbalanced dataset would naturally favor the majority class. However, the minority class could be critical for decision-making in prediction systems, and therefore it is usually desirable to train a model with equally high class-level accuracy. This paper addresses the data imbalance issue for the sentiment analysis of users’ opinions task on two educational feedback datasets utilizing synthetic text generation deep learning models. Two state-of-the-art text generation GAN models namely CatGAN and SentiGAN, are employed for synthesizing text used to balance the highly imbalanced datasets in this study. Particular emphasis is given to the diversity of synthetically generated samples for populating minority classes. Experimental results on highly imbalanced datasets show significant improvement in models’ performance on CR23K and CR100K after balancing with synthetic data for the sentiment classification task.

Although language modeling has been trending upwards steadily, models available for low-resourced languages are limited to large multilingual models such as mBERT and XLM-RoBERTa, which come with significant overheads for deployment vis-à-vis their model size, inference speeds, etc. We attempt to tackle this problem by proposing a novel methodology to apply knowledge distillation techniques to filter language-specific information from a large multilingual model into a small, fast monolingual model that can often outperform the teacher model. We demonstrate the viability of this methodology on two downstream tasks each for six languages. We further dive into the possible modifications to the basic setup for low-resourced languages by exploring ideas to tune the final vocabulary of the distilled models. Lastly, we perform a detailed ablation study to understand the different components of the setup better and find out what works best for the two under-resourced languages, Swahili and Slovene. [ABSTRACT FROM AUTHOR]

The community question-answering (CQA) websites such as Quora1, and Reddit2, and YouTube provides a significant resource to the students. However, there is a redundancy issue which results in inadequate search results for a given question. On the other hand, e-book is primary source of knowledge for students. The Latent Dirichlet Allocation (LDA) topic model helps us to find the key topic of the e-book. The major flaw of this LDA is that it can't capture the semantic knowledge in the documents. As a result, it fails to find the semantically cohesive, and meaningful topics. To address this issue, we propose a novel sBERT-LDA model, which augments the e-books with recommended question-answers and videos. We construct SiameseBERT (Bidirectional Encoder Representations from Transformers) network which provides the semantically relevant phrase embeddings. The model identifies the key topics of the e-book, after which sBERT is used to assess the similarity between the question-answers. This effort also provides advanced video indexing methods for each recommended video, allowing videos with "Table of Contents" and "Phrase Cloud" features to make videos more consumable. Experiments were carried out on question-answers datasets (Quora (QQP), TREC QA, and Yahoo Answers) as well as e-books on various subjects and across different grades.The model outperforms previous research by a large margin. [ABSTRACT FROM AUTHOR]

Progress in machine learning (ML) comes with a cost to the environment, given that training ML models requires computational resources, energy and materials. In the present article, we aim to quantify the carbon footprint of BLOOM, a 176-billion parameter language model, across its life cycle. We estimate that BLOOM's final training emitted approximately 24.7 tonnes of CO2eq if we consider only the dynamic power consumption, and 50.5 tonnes if we account for all processes ranging from equipment manufacturing to energy-based operational consumption. We also carry out an empirical study to measure the energy requirements and carbon emissions of its deployment for inference via an API endpoint receiving user queries in real-time. We conclude with a discussion regarding the difficulty of precisely estimating the carbon footprint of ML models and future research directions that can contribute towards improving carbon emissions reporting. [ABSTRACT FROM AUTHOR]

This article provides a comprehensive survey of contemporary language modeling approaches within the realm of natural language processing (NLP) tasks. This paper conducts an analytical exploration of diverse methodologies employed in the creation of language models. This exploration encompasses the architecture, training processes, and optimization strategies inherent in these models. The detailed discussion covers various models ranging from traditional n-gram and hidden Markov models to state-of-the-art neural network approaches such as BERT, GPT, LLAMA, and Bard. This article delves into different modifications and enhancements applied to both standard and neural network architectures for constructing language models. Special attention is given to addressing challenges specific to agglutinative languages within the context of developing language models for various NLP tasks, particularly for Arabic and Turkish. The research highlights that contemporary transformer-based methods demonstrate results comparable to those achieved by traditional methods employing Hidden Markov Models. These transformer-based approaches boast simpler configurations and exhibit faster performance during both training and analysis. An integral component of the article is the examination of popular and actively evolving libraries and tools essential for constructing language models. Notable tools such as NLTK, TensorFlow, PyTorch, and Gensim are reviewed, with a comparative analysis considering their simplicity and accessibility for implementing diverse language models. The aim is to provide readers with insights into the landscape of contemporary language modeling methodologies and the tools available for their implementation.

Parallel natural language processing systems were previously successfully tested on the tasks of part-of-speech tagging and authorship attribution through mini-language modeling, for which they achieved significantly better results than independent methods in the cases of seven European languages. The aim of this paper is to present the advantages of using composite language models in the processing and evaluation of texts written in arbitrary highly inflective and morphology-rich natural language, particularly Serbian. A perplexity-based dataset, the main asset for the methodology assessment, was created using a series of generative pre-trained transformers trained on different representations of the Serbian language corpus and a set of sentences classified into three groups (expert translations, corrupted translations, and machine translations). The paper describes a comparative analysis of calculated perplexities in order to measure the classification capability of different models on two binary classification tasks. In the course of the experiment, we tested three standalone language models (baseline) and two composite language models (which are based on perplexities outputted by all three standalone models). The presented results single out a complex stacked classifier using a multitude of features extracted from perplexity vectors as the optimal architecture of composite language models for both tasks.

Prompt-based learning is a method used for language models to interpret natural language by remembering the prior knowledge acquired and the training objective. Recent prompt-based few-shot learners have achieved superior performance by alleviating the catastrophic forgetting that occurs in pretrained language models. Few-shot learning contributes towards solving the data scarcity problem, an enormous challenge in AI systems and a significant consideration in natural language processing research. In spite of the significance of few-shot learning, research on Korean language-based few-shot learning is insufficient, and whether the prompt-based approach is appropriate for the Korean language has not been thoroughly verified. As a step toward realizing a Korean-prompt-based few-shot learner, we attempt to apply prompt engineering to the Korean language understanding benchmark dataset and introduce plain template insertion to overcome data scarcity in a more practical few-shot setting. The contributions of this study are as follows: (1) presumably, this is the first study to apply prompt-based few-shot learning to Korean benchmark datasets. With 32 few-shot settings, it improves performance by +14.88, +29.04, and +1.81 in the natural language inference, semantic textual similarity, and topic classification tasks. (2) We present prompt engineering, which merely inserts a plain template and increases data efficiency without training example selection, augmentation, reformulation, and retrieval. (3) Our approach is robust to the Korean prompt’s contextual information and sentence structure and is applicable to both hard- and soft-prompt.

This paper reviews the novel concept of a controllable variational autoencoder (ControlVAE), discusses its parameter tuning to meet application needs, derives its key analytic properties, and offers useful extensions and applications. ControlVAE is a new variational autoencoder (VAE) framework that combines automatic control theory with the basic VAE to stabilize the KL-divergence of VAE models to a specified value. It leverages a non-linear PI controller, a variant of the proportional-integral-derivative (PID) controller, to dynamically tune the weight of the KL-divergence term in the evidence lower bound (ELBO) using the output KL-divergence as feedback. This allows us to precisely control the KL-divergence to a desired value (set point) that is effective in avoiding posterior collapse and learning disentangled representations. While prior work developed alternative techniques for controlling the KL divergence, we show that our PI controller has better stability properties and thus better convergence, thereby producing better disentangled representations from finite training data. In order to improve the ELBO of ControlVAE over that of the regular VAE, we provide a simplified theoretical analysis to inform the choice of set point for the KL-divergence of ControlVAE. We evaluate the proposed method on three tasks: image generation, language modeling, and disentangled representation learning. The results show that ControlVAE can achieve much better reconstruction quality than the other methods for comparable disentanglement. On the language modeling task, our method can avoid posterior collapse (KL vanishing) and improve the diversity of generated text. Moreover, it can change the optimization trajectory, improving the ELBO and the reconstruction quality for image generation. [ABSTRACT FROM AUTHOR]

Text data augmentation is essential in the field of medicine for the tasks of natural language processing (NLP). However, most of the traditional text data augmentation focuses on the English datasets, and there is little research on the Chinese datasets to augment Chinese sentences. Nevertheless, the traditional text data augmentation ignores the semantics between words in sentences, besides, it has limitations in alleviating the problem of the diversity of augmented sentences. In this paper, a novel medical data augmentation (MDA) is proposed for NLP tasks, which combines the medical knowledge graph with text data augmentation to generate augmented data. Experiments on the named entity recognition task and relational classification task demonstrate that the MDA can significantly enhance the efficiency of the deep learning models compared to cases without augmentation. [ABSTRACT FROM AUTHOR]

Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure (3-state accuracy Q3=81%-87%); (2) per-protein (pooling) predictions of protein sub-cellular location (ten-state accuracy: Q10=81%) and membrane versus water-soluble (2-state accuracy Q2=91%). For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that pLMs learned some of the grammar of the language of life. All our models are available through https://github.com/agemagician/ProtTrans. [ABSTRACT FROM AUTHOR]

The article deals with the study of the peculiarities of linguistic expression of the category of assessment, which as a universal language category operating at different language levels, plays an important role in the choice of native speakers and, consequently, culture, language means of information and is equally important for mental processes of perception, comprehension and acceptance or non-acceptance of this information. The publications in the scientific literature on various aspects of the study of the category of evaluation in terms of language have been reviewed. The analysis of the peculiarities of verbalization of the category of evaluation is carried out on the material of fictional and dramatic discourse, which is a productive language environment for studying the category of evaluation, because through verbalized national-cultural ideas and stereotypes are reflected the conceptual and linguistic pictures of the world. First of all, this category is manifested in the language and characteristics of the characters. Special attention is paid to the characters of a (chynovnyk, pan, a pani, a panna and a cossack, verbalized in the fictional and dramatic discourse of the XIX century. Evaluative characteristics, which are the elements of language modeling of fictional images, served as a basis for distinguishing the following types of evaluation: positive and negative evaluation in the structure of the text narrative, neutral evaluation, hidden positive evaluation, author's evaluation, evaluation by other characters. A positive evaluation in the language of fiction is shown by the linguistic means of creating the character of a cossack. Negative assessment was found in the linguistic means of creating the image of the pan (landlord), the images of the pani and panna are ambivalent. Evaluation is present both in the language of the characters and in the author's characteristics or comments. The object of evaluation in the fictional and dramatic genres can be both internal and external characteristics of the characters, and external characteristics are often marked by positive evaluation, while internal features have either negative or ambivalent evaluation characteristics. It was found that the ways of realization of positive or negative evaluation in the language of fiction are quite diverse. They depend on the literary and fictional methods, the creative manner of the writer and reflect his worldview.

Reading comprehension is one of the essential skills for students as they make a transition from learning to read to reading to learn. Over the last decade, the increased use of digital learning materials for promoting literacy skills (e.g., oral fluency and reading comprehension) in K-12 classrooms has been a boon for teachers. However, instant access to reading materials, as well as relevant assessment tools for evaluating students' comprehension skills, remains to be a problem. Teachers must spend many hours looking for suitable materials for their students because high-quality reading materials and assessments are primarily available through commercial literacy programs and websites. This study proposes a promising solution to this problem by employing an artificial intelligence (AI) approach. We demonstrate how to use advanced language models (e.g., OpenAI's GPT-2 and Google's T5) to automatically generate reading passages and items. Our preliminary findings suggest that with additional training and fine-tuning, opensource language models could be used to support the instruction and assessment of reading comprehension skills in the classroom. For both automatic story and item generation, the language models performed reasonably; however, the outcomes of these language models still require a human evaluation and further adjustments before sharing them with students. Practical implications of the findings and future research directions are discussed. [ABSTRACT FROM AUTHOR]

Data imbalance in datasets is a common issue where the number of instances in one or more categories far exceeds the others, so is the case with the educational domain. Collecting feedback on a course on a large scale and the lack of publicly available datasets in this domain limits models' performance, especially for deep neural network based models which are data hungry. A model trained on such an imbalanced dataset would naturally favor the majority class. However, the minority class could be critical for decision-making in prediction systems, and therefore it is usually desirable to train a model with equally high class-level accuracy. This paper addresses the data imbalance issue for the sentiment analysis of users' opinions task on two educational feedback datasets utilizing synthetic text generation deep learning models. Two state-of-the-art text generation GAN models namely CatGAN and SentiGAN, are employed for synthesizing text used to balance the highly imbalanced datasets in this study. Particular emphasis is given to the diversity of synthetically generated samples for populating minority classes. Experimental results on highly imbalanced datasets show significant improvement in models' performance on CR23K and CR100K after balancing with synthetic data for the sentiment classification task. [ABSTRACT FROM AUTHOR]

Classroom environments were analyzed to better understand adult language modeling rates and whether teacher certification practices contributed to differences with learners with deafblindness (DB). Student characteristics were also examined in relation to communication rates. When there is a dual sensory loss, access to tactile and visual communication forms (i.e., multimodal) in addition to verbal communication is needed. Data were collected from 15 teacher–student dyads from four states through behavioral coding of videotaped language samples, teacher surveys, and the Communication Matrix assessment. Overall, teachers used verbal communication significantly more than additional classroom staff. Teachers in a state that required a severe/profound certification had significantly higher rates of overall communication and visual communication, and they had students with higher communication levels. [ABSTRACT FROM AUTHOR]

The lack of Estonian NLU datasets severely affects advancing Estonian-specific NLP research. With this paper we aim to relieve the issue by publishing a new Estonian NLU dataset EstCOPA. We benchmark the task on several Estonian and multilingual transformer based language models, including a novel Estonian-centric GPT (GPT4Est). Moreover, we evaluate different low-cost alternatives for creating training and test datasets and outline strategies for future Estonian language understanding research. [ABSTRACT FROM AUTHOR]

Generating coherent and consistent long text is an important but challenging task. Despite the recent success of planning-based methods in maintaining consistency and modeling long-distance coherence for text, the existing generative model still suffers from the inconsistency problem among prompt, plan, and target text. In this paper, we propose a novel generative model MDFUT, which leverages an autoregressive model to do content planning and surface realization simultaneously. To alleviate error accumulation and performance compromising for the pre-trained language model, we introduce a novel paralleled dual decoder architecture to improve generation form. Moreover, we propose a bridging objective to minimize the bidirectional KL divergence between the distributions of the dual decoder to enhance the consistency between the plan and text. We use BART as the backbone model and extend the typical transformer to dual decoder architecture. Extensive experiments are conducted on four datasets: Wikiplots and ROCStories for long- and short-form story generation task, CMV for argument generation task, and CNNNews for news generation task. The results show that our model achieves a significant improvement in both automatic and human evaluations and can generate more consistent texts than the baselines. • MDFUT: concurrent plan-text generation with the fusion of both distributions. • Novel bridging loss for closer decoder alignment, boosting plan-text consistency. • Extensive tests show MDFUT excels in open-ended story tasks with coherent outputs. [ABSTRACT FROM AUTHOR]

Automatic item generation (AIG) has the potential to greatly expand the number of items for educational assessments, while simultaneously allowing for a more construct-driven approach to item development. However, the traditional item modeling approach in AIG is limited in scope to content areas that are relatively easy to model (such as math problems), and depends on highly skilled content experts to create each model. In this paper we describe the interactive reading task, a transformer-based deep language modeling approach for creating reading comprehension assessments. This approach allows a fully automated process for the creation of source passages together with a wide range of comprehension questions about the passages. The format of the questions allows automatic scoring of responses with high fidelity (e.g., selected response questions). We present the results of a large-scale pilot of the interactive reading task, with hundreds of passages and thousands of questions. These passages were administered as part of the practice test of the Duolingo English Test. Human review of the materials and psychometric analyses of test taker results demonstrate the feasibility of this approach for automatic creation of complex educational assessments.

Although gestalt language development incorporates some neurodiversity-affirmative strategies (e.g. embracing echolalia), it is also associated with numerous ideas, assertions, and clinical strategies that lack theoretical and empirical support. For this reason, we believe it is premature to embrace gestalt language development as a neurodiversity-affirmative practice. Given the potential for miscommunication, we must strive for clarity and precision in our terminology (e.g. making a clear distinction between delayed echolalia and the broader set of ideas that comprise gestalt language development). The field of autism will benefit from in-depth discussions about the relationship between neurodiversity-affirmative practices and evidence-based practices.It was recently suggested that a set of ideas known as gestalt language development be embraced as a neurodiversity-affirmative practice. Neurodiversity refers to the idea that people interact with the world in many different ways and that there is not a single right way to do so. Some aspects of gestalt language development, such as embracing autistic communication, are consistent with neurodiversity. However, gestalt language development is also associated with numerous ideas, assertions, and clinical strategies that lack theoretical and empirical support. For this reason, we believe it is premature to embrace gestalt language development as a neurodiversity-affirmative practice. We propose that it is important to make sure we use language that differentiates between delayed echolalia and the broader set of ideas that comprise gestalt language development. We also suggest that it is important to discuss the relationship between neurodiversity-affirmative practices and practices supported by research evidence. [ABSTRACT FROM AUTHOR]

Algorithmic automatic item generation can be used to obtain large quantities of cognitive items in the domains of knowledge and aptitude testing. However, conventional item models used by template-based automatic item generation techniques are not ideal for the creation of items for non-cognitive constructs. Progress in this area has been made recently by employing long short-term memory recurrent neural networks to produce word sequences that syntactically resemble items typically found in personality questionnaires. To date, such items have been produced unconditionally, without the possibility of selectively targeting personality domains. In this article, we offer a brief synopsis on past developments in natural language processing and explain why the automatic generation of construct-specific items has become attainable only due to recent technological progress. We propose that pre-trained causal transformer models can be fine-tuned to achieve this task using implicit parameterization in conjunction with conditional generation. We demonstrate this method in a tutorial-like fashion and finally compare aspects of validity in human- and machine-authored items using empirical data. Our study finds that approximately two-thirds of the automatically generated items show good psychometric properties (factor loadings above.40) and that one-third even have properties equivalent to established and highly curated human-authored items. Our work thus demonstrates the practical use of deep neural networks for non-cognitive automatic item generation. [ABSTRACT FROM AUTHOR]

Language skills are vital to children's learning and well-being, and the first 5 years of life are an especially critical time for language acquisition. Research suggests that when early childhood teachers create language-rich environments, children develop stronger receptive and expressive language abilities, especially children from low-income households who are more at risk for language delays. This study investigated the effects of a 10-week professional learning intervention focused on language-based interactions in early care and education settings that serve infants and toddlers on state childcare subsidy. The intervention with childcare providers was titled Filming Interactions to Nurture Development (FIND), which utilized video-based reflective practices to promote positive interactions between providers and children. Using a treatment-control design (control: n = 44; treatment: n = 42), we compared childcare providers' language practices. Provider language use was measured using the Language Environment Analysis (LENA), a wearable digital language processer. Results indicate that, as compared with the control group, FIND participation was associated with increased adult word count and conversational turns in facilities that serve the most vulnerable children. These results suggest the utility of video-based professional learning and automated data collection for supporting providers' positive interactions and improving the quality of infant/toddler programs. [ABSTRACT FROM AUTHOR]

Data augmentation (DA) is a universal technique to reduce overfitting and improve the robustness of machine learning models by increasing the quantity and variety of the training dataset. Although data augmentation is essential in vision tasks, it is rarely applied to text datasets since it is less straightforward. Some studies have concerned text data augmentation, but most of them are for the majority languages, such as English or French. There have been only a few studies on data augmentation for minority languages, e.g., Korean. This study fills the gap by demonstrating several common data augmentation methods and Korean corpora with pre-trained language models. In short, we evaluate the performance of two text data augmentation approaches, known as text transformation and back translation. We compare these augmentations among Korean corpora on four downstream tasks: semantic textual similarity (STS), natural language inference (NLI), question duplication verification (QDV), and sentiment classification (STC). Compared to cases without augmentation, the performance gains when applying text data augmentation are 2.24%, 2.19%, 0.66%, and 0.08% on the STS, NLI, QDV, and STC tasks, respectively. [ABSTRACT FROM AUTHOR]

Recently Neural Architecture Search has drawn interest from researchers because of its ability to learn neural network architectures from data automatically. The differentiable methods are widely used because they can obtain better architectures with less computational resources. However, the method is with a mismatch between training and inference. In training process, we minimize the loss on the expected output of an ensemble of models, whereas we infer a single model at test time. In this paper we present a topology-sensitive approach to neural architecture search. Unlike previous work, we do not resort to the search strategy that works in a different scope of architecture topologies with inference. Instead, the topology of a neural network is explicitly modeled in the training/search process. We experiment with our method on the PTB and WikiText language modeling tasks. On the PTB dataset, it outperforms several strong baselines. Also, the newly-found architecture shows good transferability to named entity recognition and machine translation tasks. In particular, experimental results show our method outperforms the strong baseline by a BLEU score of 0.6 on IWSLT15 English-Vietnamese task and a BLEU score of 0.9 on WMT14 English-German task.

These days, neural networks constantly prove their high capacity for nearly every application case and are considered as key technology for learning systems. However, neural networks need to continuously evolve for managing new arising challenges like increasing task complexity, explainability of decision making processes, expanded problem domains, providing resilient and robust systems etc. One possible enhancement of traditional neural networks constitutes the innovative Capsule Network (CapsNet) technology, which combines the expressiveness of distributed entity representations with an intelligent and interpretable signal propagation, named as routing-by-agreement. Since CapsNets represent a relatively young acquirement, further research is essential for gaining profound knowledge about CapsNet theory and best practices for diverse application areas. This paper wants to contribute to the progress of CapsNets for the task of text classification. For this purpose, various research questions about this technology get formulated and experimentally answered with the aid of six selected datasets. In addition, this paper serves as a possible starting point for researchers as well as for practitioners to deal with CapsNets in the text domain, by supplying a survey about its theory, text classification basics and the combination of both areas. The analysis results empirically prove the robustness of CapsNets with routing-by-agreement for a wide spectrum of net architectures, datasets and text classification tasks. Hence, CapsNets can be viewed as a next-generation neural network technology, which offers high potential as text classification method and should be topic of future research.

In 2018, Mikolov et al. [Mik+18] introduced the positional language model, which has characteristics of attention-based neural machine translation models and which achieved state-of-the-art performance on the intrinsic word analogy task. However, the positional model is not practically fast and it has never been evaluated on qualitative criteria or extrinsic tasks. We propose a constrained positional model, which adapts the sparse attention mechanism from neural machine translation to improve the speed of the positional model. We evaluate the positional and constrained positional models on three novel qualitative criteria and on language modeling. We show that the positional and constrained positional models contain interpretable information about the grammatical properties of words and outperform other shallow models on language modeling. We also show that our constrained model outperforms the positional model on language modeling and trains twice as fast. [ABSTRACT FROM AUTHOR]