Search

Aided augmentative and alternative communication (AAC) modeling is an important approach to facilitating language learning for individuals with complex communication repertoires. This study examines the implementation of this approach during COVID-19 virtual schooling. Key implementation challenges are identified, along with implications for educational practice to facilitate language learning. We conducted a qualitative case study of a 5-year-old child with Rett syndrome, a rare neurological disorder, who used an eye-tracking AAC device. We collected and triangulated data using participant observation of three one-on-one story reading sessions involving the child, her caregiver, and her teacher during virtual schooling as well as a semi-structured interview with the teacher. The study identified key steps for successful language modeling during virtual sessions. These included: ensuring access to the device, promoting attention through gestural and verbal prompts, providing modeling of AAC symbols, confirming responses, providing feedback, using visual supports, a collaborative approach, and positive reinforcement. Our findings suggest these strategies can help facilitate effective language development for children using AAC devices in online schooling contexts. The virtual context presented significant barriers to the practice of aided AAC modeling. Active participation of caregivers and teacher-caregiver collaboration is essential for effective implementation of aided AAC modeling during such schooling.

In this study, a steganography method based on BERT transformer model is proposed for hiding text data in cover text. The aim is to hide information by replacing specific words within the text using BERT’s masked language modeling (MLM) feature. In this study, two models, fine-tuned for English and Turkish, are utilized to perform steganography on texts belonging to these languages. Furthermore, the proposed method can work with any transformer model that supports masked language modeling. While traditionally the hidden information in text is often limited, the proposed method allows for a significant amount of data to be hidden in the text without distorting its meaning. In this study, the proposed method is tested by hiding stego texts of varying lengths in cover text of different lengths in two different language scenarios. The test results are analyzed in terms of perplexity, KL divergence and semantic similarity. Upon examining the results, the proposed method has achieved the best results compared to other methods found in the literature, with KL divergence of 7.93 and semantic similarity of 0.99. It can be observed that the proposed method has low detectability and demonstrates success in the data hiding process.

Abstract The generation of plausible crystal structures is often the first step in predicting the structure and properties of a material from its chemical composition. However, most current methods for crystal structure prediction are computationally expensive, slowing the pace of innovation. Seeding structure prediction algorithms with quality generated candidates can overcome a major bottleneck. Here, we introduce CrystaLLM, a methodology for the versatile generation of crystal structures, based on the autoregressive large language modeling (LLM) of the Crystallographic Information File (CIF) format. Trained on millions of CIF files, CrystaLLM focuses on modeling crystal structures through text. CrystaLLM can produce plausible crystal structures for a wide range of inorganic compounds unseen in training, as demonstrated by ab initio simulations. Our approach challenges conventional representations of crystals, and demonstrates the potential of LLMs for learning effective models of crystal chemistry, which will lead to accelerated discovery and innovation in materials science.

Abstract Life sciences research and experimentation are resource-intensive, requiring extensive trials and considerable time. Often, experiments do not achieve their intended objectives, but progress is made through trial and error, eventually leading to breakthroughs. Machine learning is transforming this traditional approach, providing methods to expedite processes and accelerate discoveries. Deep Learning is becoming increasingly prominent in chemistry, with Convolutional Graph Networks (CGN) being a key focus, though other approaches also show significant potential. This research explores the application of Natural Language Processing (NLP) to evaluate the effectiveness of chemical language representations, specifically SMILES and SELFIES, using tokenization methods such as Byte Pair Encoding (BPE) and a novel approach developed in this study, Atom Pair Encoding (APE), in BERT-based models. The primary objective is to assess how these tokenization techniques influence the performance of chemical language models in biophysics and physiology classification tasks. The findings reveal that APE, particularly when used with SMILES representations, significantly outperforms BPE by preserving the integrity and contextual relationships among chemical elements, thereby enhancing classification accuracy. Performance was evaluated in downstream classification tasks using three distinct datasets for HIV, toxicology, and blood–brain barrier penetration, with ROC-AUC serving as the evaluation metric. This study highlights the critical role of tokenization in processing chemical language and suggests that refining these techniques could lead to significant advancements in drug discovery and material science.

The textual data unlike images, videos, and audio will not get much distorted while transmitting across the network. Steganography is the process of hiding secret data inside innocent data for secured communication. Software-defined radios are promising as they transmit over air. The approach proposed is performing linguistic steganography and transmitting the stego text from source to destination through software-defined radio. The linguistic steganography is achieved through the Generative Pre-trained Transformer 3 language model. Initially, English alphabets are coded using binary uniquely, followed by generation of scrambling sequence by a pre-shared key. The secret bits and scrambling bits are XORed, and the resultant bit stream will be reversed to ensure additional security. Both sender and receiver should agree upon binary coding of English alphabets. Mapping of these alphabets to binary scrambled sequence is done after grouping it into 4 bits. In the next step secret bit stream is converted as an abbreviation. stego text is obtained by generating the full form for the obtained abbreviation in the previous step using the Generative Pre-trained Transformer 3 language model. The perplexity and human opinion scores are considered to quantify the quality of the generated text. The method demonstrates promising performance metrics, including an average perplexity as low as 2.75, an average human opinion score of 4.052, and an average embedding rate of approximately 0.72 seconds along with 4 bits/words embedding capacity. The generated text exhibits low recognition as steganography, with an accuracy value of 0.5245, while achieving 100% decoding accuracy.

Abstract Generative deep learning is reshaping drug design. Chemical language models (CLMs) – which generate molecules in the form of molecular strings – bear particular promise for this endeavor. Here, we introduce a recent deep learning architecture, termed Structured State Space Sequence (S4) model, into de novo drug design. In addition to its unprecedented performance in various fields, S4 has shown remarkable capabilities to learn the global properties of sequences. This aspect is intriguing in chemical language modeling, where complex molecular properties like bioactivity can ‘emerge’ from separated portions in the molecular string. This observation gives rise to the following question: Can S4 advance chemical language modeling for de novo design? To provide an answer, we systematically benchmark S4 with state-of-the-art CLMs on an array of drug discovery tasks, such as the identification of bioactive compounds, and the design of drug-like molecules and natural products. S4 shows a superior capacity to learn complex molecular properties, while at the same time exploring diverse scaffolds. Finally, when applied prospectively to kinase inhibition, S4 designs eight of out ten molecules that are predicted as highly active by molecular dynamics simulations. Taken together, these findings advocate for the introduction of S4 into chemical language modeling – uncovering its untapped potential in the molecular sciences.

Abstract This article presents the research work on improving speech recognition systems for the morphologically complex Malayalam language using subword tokens for language modeling. The speech recognition system is built using a deep neural network–hidden Markov model (DNN-HMM)-based automatic speech recognition (ASR). We propose a novel method, syllable-byte pair encoding (S-BPE), that combines linguistically informed syllable tokenization with the data-driven tokenization method of byte pair encoding (BPE). The proposed method ensures words are always segmented at valid pronunciation boundaries. On a text corpus that has been divided into tokens using the proposed method, we construct statistical n-gram language models and assess the modeling effectiveness in terms of both information-theoretic and corpus linguistic metrics. A comparative study of the proposed method with other data-driven (BPE, Morfessor, and Unigram), linguistic (Syllable), and baseline (Word) tokenization algorithms is also presented. Pronunciation lexicons of subword tokenized units are built with pronunciation described as graphemes. We develop ASR systems employing the subword tokenized language models and pronunciation lexicons. The resulting ASR models are comprehensively evaluated to answer the research questions regarding the impact of subword tokenization algorithms on language modeling complexity and on ASR performance. Our study highlights the strong performance of the hybrid S-BPE tokens, achieving a notable 10.6% word error rate (WER), which represents a substantial 16.8% improvement over the baseline word-level ASR system. The ablation study has revealed that the performance of S-BPE segmentation, which initially underperformed compared to syllable tokens with lower amounts of textual data for language modeling, exhibited steady improvement with the increase in LM training data. The extensive ablation study indicates that there is a limited advantage in raising the n-gram order of the language model beyond $$n=3$$ n = 3 . Such an increase results in considerable model size growth without significant improvements in WER. The implementation of the algorithm and all associated experiments are available under an open license, allowing for reproduction, adaptation, and reuse.

Over the past two decades, language modeling (LM) has emerged as a primary methodology for language understanding and generation. This technology has become a cornerstone within the field of natural language processing (NLP). At its core, LM is designed to train models to predict the probability of the next word or token, thereby generating natural and fluent language. The advent of large language models (LLMs), such as Bidirectional Encoder Representations from Transformers and GPT-3, marks a significant milestone in the evolution of LM. These LLMs have left a profound impact on the field of artificial intelligence (AI) while also paving the way for advancements in other domains. This progression underscores the power and efficacy of AI, illustrating how the landscape of AI research has been reshaped by the rapid advancement of LLMs. This paper provides a comprehensive review of the evolution of LLMs, focusing on the technical architecture, model scale, training methods, optimization techniques, and evaluation metrics. Language models have evolved significantly over time, starting from initial statistical language models, moving onto neural network-based models, and now embracing the era of advanced pre-trained language models. As the scale of these models has expanded, so has their performance in language understanding and generation. This has led to notable results across various sectors, including education, healthcare, finance, and industry. However, the application of LLMs also presents certain challenges, such as data quality, model generalization capabilities, and computational resources. This paper delves into these issues, providing an analysis of the strengths and limitations of LLMs. Furthermore, the rise of LLMs has sparked a series of ethical, privacy, and security concerns. For instance, LLMs may generate discriminatory, false, or misleading information, infringe on personal privacy, or even be exploited for malicious activities such as cyber-attacks. To tackle these issues, this paper explores relevant technical measures, such as model interpretability, privacy protection, and security assessments. Ultimately, the paper outlines potential future research trends of LLMs. With ongoing enhancements to model scale and efficiency, LLMs are expected to play an even greater role in multimodal processing and societal impact. For example, by integrating information from different modalities, such as images and sound, LLMs can better understand and generate language. Additionally, they can be employed for societal impact assessment, providing support for policy formulation and decision-making. By thoroughly analyzing the current state of research and potential future directions, this paper aims to offer researchers valuable insights and inspiration regarding LLMs, thereby fostering further advancement in the field.

Paraeducators are the most frequent communication partners during the school day for students who use augmentative and alternative communication (AAC), yet they often lack training in AAC best practices. This intervention study examined the effect of an in-class coaching intervention on the aided language modeling (ALM) skills of paraeducators who work with students who use AAC. An intervention protocol using evidence-based coaching strategies was used to support paraeducator implementation of ALM in typical classroom activities. The multiple-baseline single-subject design measured the use of ALM by four paraeducators. Data were analyzed visually and by calculating Tau-U and gain scores. Results suggest a strong effect from the coaching intervention on ALM skills for each of the paraeducators. Challenges and benefits of paraeducator-focused interventions in classroom settings are presented.

Abstract Background Protein solubility is a critically important physicochemical property closely related to protein expression. For example, it is one of the main factors to be considered in the design and production of antibody drugs and a prerequisite for realizing various protein functions. Although several solubility prediction models have emerged in recent years, many of these models are limited to capturing information embedded in one-dimensional amino acid sequences, resulting in unsatisfactory predictive performance. Results In this study, we introduce a novel Graph Attention network-based protein Solubility model, GATSol, which represents the 3D structure of proteins as a protein graph. In addition to the node features of amino acids extracted by the state-of-the-art protein large language model, GATSol utilizes amino acid distance maps generated using the latest AlphaFold technology. Rigorous testing on independent eSOL and the Saccharomyces cerevisiae test datasets has shown that GATSol outperforms most recently introduced models, especially with respect to the coefficient of determination R2, which reaches 0.517 and 0.424, respectively. It outperforms the current state-of-the-art GraphSol by 18.4% on the S. cerevisiae_test set. Conclusions GATSol captures 3D dimensional features of proteins by building protein graphs, which significantly improves the accuracy of protein solubility prediction. Recent advances in protein structure modeling allow our method to incorporate spatial structure features extracted from predicted structures into the model by relying only on the input of protein sequences, which simplifies the entire graph neural network prediction process, making it more user-friendly and efficient. As a result, GATSol may help prioritize highly soluble proteins, ultimately reducing the cost and effort of experimental work. The source code and data of the GATSol model are freely available at https://github.com/binbinbinv/GATSol .

Detecting plagiarism poses a significant challenge for academic institutions, research centers, and content-centric organizations, especially in cases involving subtle paraphrasing and content manipulation where conventional methods often prove inadequate. Our paper proposes FTLM (Fuzzy TOPSIS Language Modeling), a novel method for detecting plagiarism within decision science. FTLM integrates language models with fuzzy sorting techniques to assess plagiarism severity by evaluating the similarity of potential solutions to a reference. The method involves two stages: leveraging language modeling to define criteria and alternatives and implementing enhanced fuzzy TOPSIS. Word usage patterns, grammatical structures, and semantic coherence represent fuzzy membership functions. Moreover, pre-trained language models enhance semantic similarity analysis. This approach highlights the benefits of combining fuzzy logic’s tolerance for imprecision with the semantic evaluation capabilities of advanced language models, thereby offering a comprehensive and contextually aware method for analyzing plagiarism severity. The experimental results on the benchmark dataset demonstrate effective features that enhance performance on the user-defined severity ranking order.

The Languages of the Indian subcontinent are less represented in current NLP literature. To mitigate this gap, we present the IndicDialogue dataset, which contains subtitles and dialogues in 10 major Indic languages: Hindi, Bengali, Marathi, Telugu, Tamil, Urdu, Odia, Sindhi, Nepali, and Assamese. This dataset is sourced from OpenSubtitles.org, with subtitles pre-processed to remove irrelevant tags, timestamps, square brackets, and links, ensuring the retention of relevant dialogues in JSONL files. The IndicDialogue dataset comprises 7750 raw subtitle files (SRT), 11 JSONL files, 6,853,518 dialogues, and 42,188,569 words. It is designed to serve as a foundation for language model pre-training for low-resource languages, enabling a wide range of downstream tasks including word embeddings, topic modeling, conversation synthesis, neural machine translation, and text summarization.

[Purpose/Significance] Deep learning for language modeling is one of the major methods and advanced technologies to enhance language intelligence of machines at present, which has become an indispensable important technical means for automatic processing and analysis of data resources, and intelligent mining of information and knowledge. However, there are still some difficulties in using deep learning for language modeling for technology development and application service in the library and information science (LIS) field. Therefore, this study systematically reviews and reveals the research progress, technical principles, and development methods of deep learning for language modeling, with the aim at providing reliable theoretical basis and feasible methodological paths for the deep understanding and application of deep learning for language modeling for librarians and fellow practitioners. [Method/Process] The data used in this study were collected from the WOS core database, CNKI literature database, arXiv preprint repository, GitHub open-source software hosting platform and the open resources on the Internet. Based on these data, this paper first systematically investigates the background, basic feature representation algorithms, and representative application development tools of deep learning for language modeling, reveals their dynamic evolution and technical principles, and analyzes the advantages and disadvantages and applicability of each algorithm model and development tool. Second, an in-depth analysis of the possible challenging problems faced by the development and application of deep learning for language modeling was performed, and two strategic approaches to expand their application capabilities were put forward. [Results/Conclusions] The important challenges faced by the application and development of deep learning for language modeling include numerous parameters and difficulties to adjust accuracy, relying on a large amount of accurate training data, difficulties in making changes, and the intellectual property and information security issues. In the future, we will start from two aspects of specific domains and feature engineering to expand and improve the application capabilities of deep learning for language modeling. Specifically, we focus on consideration of the collection and preparation of domain data, selection of model architecture, participation of domain experts, and optimization for specific tasks, in order to ensure that the data source of the model is more reliable and secure, and the application effect is more accurate and practical. Moreover, the strategic methods for feature engineering to expand the application capabilities of deep learning for language modeling include selecting appropriate features, feature pre-processing, feature selection, and feature dimensionality reduction. These strategies can help improve the performance and efficiency of deep learning for language models, making them more suitable for specific tasks or domains. To sum up, LIS institutions should leverage the deep learning for language modeling related technologies, guided by the needs of scientific research and social development, and based on advantages of existing literature data resources and knowledge services; they should carry out innovative professional or vertical domain intelligent knowledge management and application service, and develop technology and systems with independent intellectual property rights, which is their long-term sustainable development path.

This paper discusses the multi-scenario application of ChatGPT-based language modeling in English language teaching, and empirical experiments are conducted to support the research findings. The study includes constructing and analyzing English composition scoring and similarity detection models. The BERT-BiLSTM algorithm was utilized and compared to the Word2Vec-BiLSTM model. The BERT-BiLSTM-based English composition scoring model has a high correlation and consistency with the original scores, with an average correlation of 0.72 and a consistency of 82%. Conversely, the Word2Vec-BiLSTM model has a lesser correlation and consistency. We created a model and used different K values for the experiment to detect English composition similarity. The correctness, recall, and F1 measures were higher at a K value 200, with F1 values fluctuating between 89.35% and 95.14%. These support the high accuracy and efficiency of ChatGPT-based language modeling in English language teaching.

Abstract Background Longitudinal data on key cancer outcomes for clinical research, such as response to treatment and disease progression, are not captured in standard cancer registry reporting. Manual extraction of such outcomes from unstructured electronic health records is a slow, resource-intensive process. Natural language processing (NLP) methods can accelerate outcome annotation, but they require substantial labeled data. Transfer learning based on language modeling, particularly using the Transformer architecture, has achieved improvements in NLP performance. However, there has been no systematic evaluation of NLP model training strategies on the extraction of cancer outcomes from unstructured text. Results We evaluated the performance of nine NLP models at the two tasks of identifying cancer response and cancer progression within imaging reports at a single academic center among patients with non-small cell lung cancer. We trained the classification models under different conditions, including training sample size, classification architecture, and language model pre-training. The training involved a labeled dataset of 14,218 imaging reports for 1112 patients with lung cancer. A subset of models was based on a pre-trained language model, DFCI-ImagingBERT, created by further pre-training a BERT-based model using an unlabeled dataset of 662,579 reports from 27,483 patients with cancer from our center. A classifier based on our DFCI-ImagingBERT, trained on more than 200 patients, achieved the best results in most experiments; however, these results were marginally better than simpler “bag of words” or convolutional neural network models. Conclusion When developing AI models to extract outcomes from imaging reports for clinical cancer research, if computational resources are plentiful but labeled training data are limited, large language models can be used for zero- or few-shot learning to achieve reasonable performance. When computational resources are more limited but labeled training data are readily available, even simple machine learning architectures can achieve good performance for such tasks.

Based on the framework of autoregressive language modeling, this paper analyzes the word frequency characteristics and introduces a quartile inverse probability weighted sampling algorithm in probability distribution prediction to regulate the quality and diversity of the generated music. Through the effective division of the subset of high-frequency words by this algorithm, a polyphonic piano transcription model is established, which enhances the rationality of the predicted probability distribution of piano composition. Meanwhile, objective evaluation metrics are designed for the pentatonic tonal form of Gong tuning to quantitatively assess the results of automatic composition for Chinese-style piano. It is proved that the proposed model performs well in music generation, with an average generation time of only 6.9s and a model parameter count of 2.7M, which can provide strong support and validation for the automatic composition of the Chinese wind piano.

With the development of ESL education in recent years, real-time language translation has gradually become a research hotspot in the field of natural language communication. In this paper, the data processing technology of real-time translation is introduced, and then a Transducer (RNN-T) model based on real-time language translation is constructed based on the BPE subword segmentation method under the condition of probability gradient calculation algorithm and transcription network. Secondly, this model determines and tests the content involved in the process of real-time language translation and finally obtains results to improve the quality of real-time language translation. The experimental results show that the accuracy of character translation among the six language models is 85.8%, 86.2%, 88%, 93.4%, 94.03%, and 94.36%, respectively. In addition, the error rate of inserting, deleting, and replacing characters is < 20%. Compared with the other four models, the word error rate of the RNN-Transducer algorithm is reduced by 1.59%, 3.94%, 3.86%, and 9.01%, and the RTF is reduced by 3.91%, 10.96%, 10.09%, and 7.94%, respectively, which reflects the superiority of the RNN-Transducer method proposed in this paper.

Narratives posted on the internet by patients contain a vast amount of information about various concerns. This study aimed to extract multiple concerns from interviews with breast cancer patients using the natural language processing (NLP) model bidirectional encoder representations from transformers (BERT). A total of 508 interview transcriptions of breast cancer patients written in Japanese were labeled with five types of concern labels: "treatment," "physical," "psychological," "work/financial," and "family/friends." The labeled texts were used to create a multi-label classifier by fine-tuning a pre-trained BERT model. Prior to fine-tuning, we also created several classifiers with domain adaptation using (1) breast cancer patients' blog articles and (2) breast cancer patients' interview transcriptions. The performance of the classifiers was evaluated in terms of precision through 5-fold cross-validation. The multi-label classifiers with only fine-tuning had precision values of over 0.80 for "physical" and "work/financial" out of the five concerns. On the other hand, precision for "treatment" was low at approximately 0.25. However, for the classifiers using domain adaptation, the precision of this label took a range of 0.40-0.51, with some cases improving by more than 0.2. This study showed combining domain adaptation with a multi-label classifier on target data made it possible to efficiently extract multiple concerns from interviews.

Recently Neural Architecture Search has drawn interest from researchers because of its ability to learn neural network architectures from data automatically. The differentiable methods are widely used because they can obtain better architectures with less computational resources. However, the method is with a mismatch between training and inference. In training process, we minimize the loss on the expected output of an ensemble of models, whereas we infer a single model at test time. In this paper we present a topology-sensitive approach to neural architecture search. Unlike previous work, we do not resort to the search strategy that works in a different scope of architecture topologies with inference. Instead, the topology of a neural network is explicitly modeled in the training/search process. We experiment with our method on the PTB and WikiText language modeling tasks. On the PTB dataset, it outperforms several strong baselines. Also, the newly-found architecture shows good transferability to named entity recognition and machine translation tasks. In particular, experimental results show our method outperforms the strong baseline by a BLEU score of 0.6 on IWSLT15 English-Vietnamese task and a BLEU score of 0.9 on WMT14 English-German task.