Search

(1) Background: Artificial intelligence (AI) has flourished in recent years. More specifically, generative AI has had broad applications in many disciplines. While mental illness is on the rise, AI has proven valuable in aiding the diagnosis and treatment of mental disorders. However, there is little to no research about precisely how much interest there is in AI technology. (2) Methods: We performed a Google Trends search for "AI and mental health" and compared relative search volume (RSV) indices of "AI", "AI and Depression", and "AI and anxiety". This time series study employed Box-Jenkins time series modeling to forecast long-term interest through the end of 2024. (3) Results: Within the United States, AI interest steadily increased throughout 2023, with some anomalies due to media reporting. Through predictive models, we found that this trend is predicted to increase 114% through the end of the year 2024, with public interest in AI applications being on the rise. (4) Conclusions: According to our study, we found that the awareness of AI has drastically increased throughout 2023, especially in mental health. This demonstrates increasing public awareness of mental health and AI, making advocacy and education about AI technology of paramount importance.

Protein and nucleic acid binding site prediction is a critical computational task that benefits a wide range of biological processes. Previous studies have shown that feature selection holds particular significance for this prediction task, making the generation of more discriminative features a key area of interest for many researchers. Recent progress has shown the power of protein language models in handling protein sequences, in leveraging the strengths of attention networks, and in successful applications to tasks such as protein structure prediction. This naturally raises the question of the applicability of protein language models in predicting protein and nucleic acid binding sites. Various approaches have explored this potential. This paper first describes the development of protein language models. Then, a systematic review of the latest methods for predicting protein and nucleic acid binding sites is conducted by covering benchmark sets, feature generation methods, performance comparisons, and feature ablation studies. These comparisons demonstrate the importance of protein language models for the prediction task. Finally, the paper discusses the challenges of protein and nucleic acid binding site prediction and proposes possible research directions and future trends. The purpose of this survey is to furnish researchers with actionable suggestions for comprehending the methodologies used in predicting protein-nucleic acid binding sites, fostering the creation of protein-centric language models, and tackling real-world obstacles encountered in this field.

The widespread reuse of code in the open-source community has led to the proliferation of homologous vulnerabilities, which are security flaws propagated across diverse software systems through the reuse of vulnerable code. Such vulnerabilities pose serious cybersecurity risks, as attackers can exploit the same weaknesses across multiple platforms. Deep learning has emerged as a promising approach for detecting homologous vulnerabilities in binary code due to their automated feature extraction and high efficiency. However, existing deep learning methods often struggle to capture deep semantic features in binary code, limiting their effectiveness. To address this limitation, this paper presents GMN+, which is a novel graph matching neural network with enhanced attention for detecting homologous vulnerabilities. This method comprehensively considers the information contained in instructions and incorporates types of input instruction. Masked Language Modeling and Instruction Type Prediction are developed as pre-training tasks to enhance the ability of GMN+ in extracting semantic information from basic blocks. GMN+ utilizes an attention mechanism to focus concurrently on the critical semantic information within functions and differences between them, generating robust function embeddings. Experimental results indicate that GMN+ outperforms state-of-the-art methods in various tasks and achieves notable performance in real-world vulnerability detection scenarios. [ABSTRACT FROM AUTHOR]

Current language models have achieved remarkable success in NLP tasks. Nonetheless, individual decoding methods face difficulties in realizing the immense potential of these models. The challenge is primarily due to the lack of a decoding framework that can integrate language models and decoding methods. We introduce DecoStrat, which bridges the gap between language modeling and the decoding process in D2T generation. By leveraging language models, DecoStrat facilitates the exploration of alternative decoding methods tailored to specific tasks. We fine-tuned the model on the MultiWOZ dataset to meet task-specific requirements and employed it to generate output(s) through multiple interactive modules of the framework. The Director module orchestrates the decoding processes, engaging the Generator to produce output(s) text based on the selected decoding method and input data. The Manager module enforces a selection strategy, integrating Ranker and Selector to identify the optimal result. Evaluations on the stated dataset show that DecoStrat effectively produces a diverse and accurate output, with MBR variants consistently outperforming other methods. DecoStrat with the T5-small model surpasses some baseline frameworks. Generally, the findings highlight DecoStrat's potential for optimizing decoding methods in diverse real-world applications. [ABSTRACT FROM AUTHOR]

In recent years, as cybersecurity threats have become increasingly severe and cyberattacks have occurred frequently, higher requirements have been put forward for cybersecurity protection. Therefore, the Named Entity Recognition (NER) technique, which is the cornerstone of Cyber Threat Intelligence (CTI) analysis, is particularly important. However, most existing NER studies are limited to recognizing single-layer flat entities, ignoring the possible nested entities in CTI. On the other hand, most of the existing studies focus on English CTIs, and the existing models performed poorly in a limited number of Chinese CTI studies. Given the above challenges, we propose in this paper a novel unified model, RBTG, which aims to identify flat and nested entities in Chinese CTI effectively. To overcome the difficult boundary recognition problem and the direction-dependent and distance-dependent properties in Chinese CTI NER, we use Global Pointer as the decoder and TENER as the encoder layer, respectively. Specifically, the Global Pointer layer solves the problem of the insensitivity of general NER methods to entity boundaries by utilizing the relative position information and the multiplicative attention mechanism. The TENER layer adapts to the Chinese CTI NER task by introducing an attention mechanism with direction awareness and distance awareness. Meanwhile, to cope with the complex feature capture of hierarchical structure and dependencies among Chinese CTI nested entities, the TENER layer solves the problem by following the structure of multiple self-attention layers and feed-forward network layers superimposed on each other in the Transformer. In addition, to fill the gap in the Chinese CTI nested entity dataset, we further apply the Large Language Modeling (LLM) technique and domain knowledge to construct a high-quality Chinese CTI nested entity dataset, CDTinee, which consists of six entity types selected from STIX, including nearly 4000 entity types extracted from more than 3000 threatening sentences. In the experimental session, we conduct extensive experiments on multiple datasets, and the results show that the proposed model RBTG outperforms the baseline model in both flat NER and nested NER. [ABSTRACT FROM AUTHOR]

Multimodal sentiment analysis enables machines to interact with people more naturally. The integration of multimodalities can enhance the machines' ability to accurately predict emotions. The main obstacle to multimodal sentiment analysis is integrating information from different modalities. Previous research has used a variety of techniques, including long short-term memory networks (LSTM) and transformers. However, traditional fusion methods cannot better utilize the information from each modality, and some intra- and inter-modal features may be overlooked due to possible differences in feature representations. Therefore, to address this problem, we use a combined transformer that can connect different modal inputs and introduce SS-Trans (Single-Stream Transformer for Multimodal Sentiment Analysis and Emotion Recognition), a single-stream transformer that fuses textual, visual, and speech modalities. The model was pre-trained on the CMU-MOSI and CMU-MOSEI datasets: multi-modal masked image language modeling (MLM) and text–image matching (TIA). Compared to other existing models, SS-Trans improves ACC-2 on these two datasets by 1.06% and 1.33%, and improves F1 values by 1.50% and 1.62%, respectively. The experimental results show that our method achieves the state-of-the-art level. In addition, ablation experiments validate the model and the pre-training task, proving the effectiveness of the proposed model. [ABSTRACT FROM AUTHOR]

Addressing the privacy protection and data sharing issues in Chinese medical texts, this paper introduces a federated learning approach named FLCMC for Chinese medical text classification. The paper first discusses the data heterogeneity issue in federated language modeling. Then, it proposes two perturbed federated learning algorithms, FedPA and FedPAP, based on the self-attention mechanism. In these algorithms, the self-attention mechanism is incorporated within the model aggregation module, while a perturbation term, which measures the differences between the client and the server, is added to the local update module along with a customized PAdam optimizer. Secondly, to enable a fair comparison of algorithms' performance, existing federated algorithms are improved by integrating a customized Adam optimizer. Through experiments, this paper first conducts experimental analyses on hyperparameters, data heterogeneity, and validity on synthetic datasets, which proves that the proposed federated learning algorithm has significant advantages in classification performance and convergence stability when dealing with heterogeneous data. Then, the algorithm is applied to Chinese medical text datasets to verify its effectiveness on real datasets. The comparative analysis of algorithm performance and communication efficiency shows that the algorithm exhibits strong generalization ability on deep learning models for Chinese medical texts. As for the synthetic dataset, upon comparing with comparison algorithms FedAvg, FedProx, FedAtt, and their improved versions, the experimental results show that for data with general heterogeneity, both FedPA and FedPAP show significantly more accurate and stable convergence behavior. On the real Chinese medical dataset of doctor–patient conversations, IMCS-V2, with logistic regression and long short-term memory network as training models, the experiment results show that in comparison to the above three comparison algorithms and their improved versions, FedPA and FedPAP both possess the best accuracy performance and display significantly more stable and accurate convergence behavior, proving that the method in this paper has better classification effects for Chinese medical texts. [ABSTRACT FROM AUTHOR]

The flight arrival scheduling problem is one of the critical tasks in air traffic operations, aiming to ensure that the flight arrive in the correct sequence safely. Existing methods primarily focus on the terminal area and often overlook the presence of training flight at the airport. Due to the limited generalization of traditional methods and varying control practices at different airports, training flight at airports still rely on manual control for arrival sorting. To effectively address these issues, we propose a novel method for slot allocation that leverages the strong reasoning capabilities and generalization potential of large language models (LLMs). Our method conceptualizes the dynamic scheduling problem for training flight as a language modeling problem, a perspective not previously explored. Specifically, we represent the allocator's inputs and outputs as language tokens, utilizing LLMs to generate conflict-free results based on a language description of requested landing information and assigned training flight information. Additionally, we employ a reset strategy to create a small dataset for scenario-specific samples, enabling LLMs to quickly learn allocation schemes from the dataset. We demonstrated the capability of LLMs in addressing time conflicts by evaluating metrics such as answer accuracy, conflict rate, and total delay time (without the wrong answer). These findings underscore the feasibility of employing LLMs in the field of air traffic control. [ABSTRACT FROM AUTHOR]

Language is a unique ability of human beings. Although relatively simple for humans, the ability to understand human language is a highly complex task for machines. For a machine to learn a particular language, it must understand not only the words and rules used in a particular language, but also the context of sentences and the meaning that words take on in a particular context. In the experimental development we present in this paper, the goal was the development of the language model SRBerta—a language model designed to understand the formal language of Serbian legal documents. SRBerta is the first of its kind since it has been trained using Cyrillic legal texts contained within a dataset created specifically for this purpose. The main goal of SRBerta network development was to understand the formal language of Serbian legislation. The training process was carried out using minimal resources (single NVIDIA Quadro RTX 5000 GPU) and performed in two phases—base model training and fine-tuning. We will present the structure of the model, the structure of the training datasets, the training process, and the evaluation results. Further, we will explain the accuracy metric used in our case and demonstrate that SRBerta achieves a high level of accuracy for the task of masked language modeling in Serbian Cyrillic legal texts. Finally, SRBerta model and training datasets are publicly available for scientific and commercial purposes. [ABSTRACT FROM AUTHOR]

Pretrained language models have achieved great success in various natural language understanding (NLU) tasks due to their capacity to capture deep contextualized information in text using pretraining on large-scale corpora. Tokenization plays a significant role in the process of lexical analysis. Tokens become the input for other natural language processing (NLP) tasks, like semantic parsing and language modeling. However, there is a lack of research on the evaluation of the impact of tokenization on the Arabic language model. Therefore, this study aims to address this gap in the literature by evaluating the performance of various tokenizers on Arabic large language models (LLMs). In this paper, we analyze the differences between WordPiece, SentencePiece, and BBPE tokenizers by pretraining three BERT models using each tokenizer while measuring the performance of each model on seven different NLP tasks using 29 different datasets. Overall, the model pretrained with text tokenized using the SentencePiece tokenizer significantly outperforms the other two models that utilize WordPiece and BBPE tokenizers. The results of this paper will assist researchers in developing better models, making better decisions in selecting the best tokenizers, improving feature engineering, and making models more efficient, thus ultimately leading to advancements in various NLP applications. [ABSTRACT FROM AUTHOR]

Acquiring meaningful representations of gene expression is essential for the accurate prediction of downstream regulatory tasks, such as identifying promoters and transcription factor binding sites. However, the current dependency on supervised learning, constrained by the limited availability of labeled genomic data, impedes the ability to develop robust predictive models with broad generalization capabilities. In response, recent advancements have pivoted towards the application of self-supervised training for DNA sequence modeling, enabling the adaptation of pre-trained genomic representations to a variety of downstream tasks. Departing from the straightforward application of masked language learning techniques to DNA sequences, approaches such as MoDNA enrich genome language modeling with prior biological knowledge. In this study, we advance DNA language models by utilizing the Motif-oriented DNA (MoDNA) pre-training framework, which is established for self-supervised learning at the pre-training stage and is flexible enough for application across different downstream tasks. MoDNA distinguishes itself by efficiently learning semantic-level genomic representations from an extensive corpus of unlabeled genome data, offering a significant improvement in computational efficiency over previous approaches. The framework is pre-trained on a comprehensive human genome dataset and fine-tuned for targeted downstream tasks. Our enhanced analysis and evaluation in promoter prediction and transcription factor binding site prediction have further validated MoDNA's exceptional capabilities, emphasizing its contribution to advancements in genomic predictive modeling. [ABSTRACT FROM AUTHOR]

Investigating causality to establish novel criteria for training robust natural language processing (NLP) models is an active research area. However, current methods face various challenges such as the difficulties in identifying keyword lexicons and obtaining data from multiple labeled environments. In this paper, we study the problem of robust NLP from a complementary but different angle: we treat the behavior of an attack model as a complex causal mechanism and quantify its algorithmic information using the minimum description length (MDL) framework. Specifically, we use masked language modeling (MLM) to measure the "amount of effort" needed to transform from the original text to the altered text. Based on that, we develop techniques for judging whether a specified set of tokens has been altered by the attack, even in the absence of the original text data. [ABSTRACT FROM AUTHOR]

Time series forecasting has been a challenging area in the field of Artificial Intelligence. Various approaches such as linear neural networks, recurrent linear neural networks, Convolutional Neural Networks, and recently transformers have been attempted for the time series forecasting domain. Although transformer-based architectures have been outstanding in the Natural Language Processing domain, especially in autoregressive language modeling, the initial attempts to use transformers in the time series arena have met mixed success. A recent important work indicating simple linear networks outperform transformer-based designs. We investigate this paradox in detail comparing the linear neural network- and transformer-based designs, providing insights into why a certain approach may be better for a particular type of problem. We also improve upon the recently proposed simple linear neural network-based architecture by using dual pipelines with batch normalization and reversible instance normalization. Our enhanced architecture outperforms all existing architectures for time series forecasting on a majority of the popular benchmarks. [ABSTRACT FROM AUTHOR]

A typhoon disaster is a common meteorological disaster that seriously impacts natural ecology, social economy, and even human sustainable development. It is crucial to access the typhoon disaster information, and the corresponding disaster prevention and reduction strategies. However, traditional question and answering (Q&A) methods exhibit shortcomings like low information retrieval efficiency and poor interactivity. This makes it difficult to satisfy users' demands for obtaining accurate information. Consequently, this work proposes a typhoon disaster knowledge Q&A approach based on LLM (T5). This method integrates two technical paradigms of domain fine-tuning and retrieval-augmented generation (RAG) to optimize user interaction experience and improve the precision of disaster information retrieval. The process specifically includes the following steps. First, this study selects information about typhoon disasters from open-source databases, such as Baidu Encyclopedia and Wikipedia. Utilizing techniques such as slicing and masked language modeling, we generate a training set and 2204 Q&A pairs specifically focused on typhoon disaster knowledge. Second, we continuously pretrain the T5 model using the training set. This process involves encoding typhoon knowledge as parameters in the neural network's weights and fine-tuning the pretrained model with Q&A pairs to adapt the T5 model for downstream Q&A tasks. Third, when responding to user queries, we retrieve passages from external knowledge bases semantically similar to the queries to enhance the prompts. This action further improves the response quality of the fine-tuned model. Finally, we evaluate the constructed typhoon agent (Typhoon-T5) using different similarity-matching approaches. Furthermore, the method proposed in this work lays the foundation for the cross-integration of large language models with disaster information. It is expected to promote the further development of GeoAI. [ABSTRACT FROM AUTHOR]

Classification methods based on fine-tuning pre-trained language models often require a large number of labeled samples; therefore, few-shot text classification has attracted considerable attention. Prompt learning is an effective method for addressing few-shot text classification tasks in low-resource settings. The essence of prompt tuning is to insert tokens into the input, thereby converting a text classification task into a masked language modeling problem. However, constructing appropriate prompt templates and verbalizers remains challenging, as manual prompts often require expert knowledge, while auto-constructing prompts is time-consuming. In addition, the extensive knowledge contained in entities and relations should not be ignored. To address these issues, we propose a structured knowledge prompt tuning (SKPT) method, which is a knowledge-enhanced prompt tuning approach. Specifically, SKPT includes three components: prompt template, prompt verbalizer, and training strategies. First, we insert virtual tokens into the prompt template based on open triples to introduce external knowledge. Second, we use an improved knowledgeable verbalizer to expand and filter the label words. Finally, we use structured knowledge constraints during the training phase to optimize the model. Through extensive experiments on few-shot text classification tasks with different settings, the effectiveness of our model has been demonstrated. [ABSTRACT FROM AUTHOR]

Natural language processing (NLP) has significantly transformed in the last decade, especially in the field of language modeling. Large language models (LLMs) have achieved SOTA performances on natural language understanding (NLU) and natural language generation (NLG) tasks by learning language representation in self-supervised ways. This paper provides a comprehensive survey to capture the progression of advances in language models. In this paper, we examine the different aspects of language models, which started with a few million parameters but have reached the size of a trillion in a very short time. We also look at how these LLMs transitioned from task-specific to task-independent to task-and-language-independent architectures. This paper extensively discusses different pretraining objectives, benchmarks, and transfer learning methods used in LLMs. It also examines different finetuning and in-context learning techniques used in downstream tasks. Moreover, it explores how LLMs can perform well across many domains and datasets if sufficiently trained on a large and diverse dataset. Next, it discusses how, over time, the availability of cheap computational power and large datasets have improved LLM's capabilities and raised new challenges. As part of our study, we also inspect LLMs from the perspective of scalability to see how their performance is affected by the model's depth, width, and data size. Lastly, we provide an empirical comparison of existing trends and techniques and a comprehensive analysis of where the field of LLM currently stands. [ABSTRACT FROM AUTHOR]

With the advancement of the large language model (LLM), the demand for data labeling services has increased dramatically. Big models are inseparable from high-quality, specialized scene data, from training to deploying application iterations to landing generation. However, how to achieve intelligent labeling consistency and accuracy and improve labeling efficiency in distributed data middleware scenarios is the main difficulty in enhancing the quality of labeled data at present. In this paper, we proposed an asynchronous federated learning optimization method based on the combination of LLM and digital twin technology. By analysising and comparing and with other existing asynchronous federated learning algorithms, the experimental results show that our proposed method outperforms other algorithms in terms of performance, such as model accuracy and running time. The experimental validation results show that our proposed method has good performance compared with other algorithms in the process of intelligent labeling both in terms of accuracy and running solves the consistency and accuracy problems of intelligent labeling in a distributed data center. [ABSTRACT FROM AUTHOR]

Recent advances in large language models (LLMs) have shown promise in specialized fields, yet their effectiveness is often constrained by limited domain expertise. We present a renewable and hydrogen energy-focused LLM developed by fine-tuning LLaMA 3.1 8B on a curated renewable energy corpus (RE-LLaMA). Through continued pretraining on domain-specific data, we enhanced the model's capabilities in renewable energy contexts. Extensive evaluation using zero-shot and few-shot prompting demonstrated that our fine-tuned model significantly outperformed the base model across renewable and hydrogen energy tasks. This work establishes the viability of specialized, smaller-scale LLMs and provides a framework for developing domain-specific models that can support advanced research and decision-making in the renewable energy sector. Our approach represents a significant step forward in applying LLMs to the renewable and hydrogen energy sector, offering potential applications in advanced research and decision-making processes. [ABSTRACT FROM AUTHOR]

In the context of global warming, the accurate prediction of Arctic Sea Ice Concentration (SIC) is crucial for the development of Arctic shipping routes. We have therefore constructed a lightweight, non-recursive spatio-temporal prediction model, the Spatio-Temporal Decomposition Network (STDNet), to predict the daily SIC in the Arctic. The model is based on the Seasonal and Trend decomposition using Loess (STL) decomposition idea to decompose the model into trend and seasonal components. In addition, we have designed the Global Sparse Attention Module (GSAM) to help the model extract global information. STDNet not only extracts seasonal signals and trend information with periodical correspondence from the data but also obtains the spatio-temporal dependence features in the data. The experimental methodology involves predicting the next 10 days based on the first 10 days of data. The prediction results provided the following metrics for the 10-day forecast of STDNet: Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), Root Mean Square Error (RMSE), and coefficient of determination of 1.988%, 3.541%, 5.843%, and 0.979, respectively. The average Binary Accuracy (BACC) at the beginning of September for the period 2018–2022 reached 93.85%. The proposed STDNet model outperforms and is lighter than existing deep-learning-based SIC prediction models. [ABSTRACT FROM AUTHOR]

The rise in abusive language on social media is a significant threat to mental health and social cohesion. For Bengali speakers, the need for effective detection is critical. However, current methods fall short in addressing the massive volume of content. Improved techniques are urgently needed to combat online hate speech in Bengali. Traditional machine learning techniques, while useful, often require large, linguistically diverse datasets to train models effectively. This paper addresses the urgent need for improved hate speech detection methods in Bengali, aiming to fill the existing research gap. Contextual understanding is crucial in differentiating between harmful speech and benign expressions. Large language models (LLMs) have shown state-of-the-art performance in various natural language tasks due to their extensive training on vast amounts of data. We explore the application of LLMs, specifically GPT-3.5 Turbo and Gemini 1.5 Pro, for Bengali hate speech detection using Zero-Shot and Few-Shot Learning approaches. Unlike conventional methods, Zero-Shot Learning identifies hate speech without task-specific training data, making it highly adaptable to new datasets and languages. Few-Shot Learning, on the other hand, requires minimal labeled examples, allowing for efficient model training with limited resources. Our experimental results show that LLMs outperform traditional approaches. In this study, we evaluate GPT-3.5 Turbo and Gemini 1.5 Pro on multiple datasets. To further enhance our study, we consider the distribution of comments in different datasets and the challenge of class imbalance, which can affect model performance. The BD-SHS dataset consists of 35,197 comments in the training set, 7542 in the validation set, and 7542 in the test set. The Bengali Hate Speech Dataset v1.0 and v2.0 include comments distributed across various hate categories: personal hate (629), political hate (1771), religious hate (502), geopolitical hate (1179), and gender abusive hate (316). The Bengali Hate Dataset comprises 7500 non-hate and 7500 hate comments. GPT-3.5 Turbo achieved impressive results with 97.33%, 98.42%, and 98.53% accuracy. In contrast, Gemini 1.5 Pro showed lower performance across all datasets. Specifically, GPT-3.5 Turbo excelled with significantly higher accuracy compared to Gemini 1.5 Pro. These outcomes highlight a 6.28% increase in accuracy compared to traditional methods, which achieved 92.25%. Our research contributes to the growing body of literature on LLM applications in natural language processing, particularly in the context of low-resource languages. [ABSTRACT FROM AUTHOR]

The rapid proliferation of textual data across multiple sectors demands more sophisticated and efficient techniques for summarizing extensive texts. Focusing on extractive text summarization, this approach zeroes in on choosing key sentences from a document, providing an essential method for handling extensive information. While conventional methods often miss capturing deep semantic links within texts, resulting in summaries that might lack cohesion and depth, this paper introduces a novel framework called Knowledge-Enhanced Transformer Graph Summary (KETGS). Leveraging the strengths of both transformer models and Graph Neural Networks, KETGS develops a detailed graph representation of documents, embedding linguistic units from words to key entities. This structured graph is then navigated via a Transformer-Guided Graph Neural Network (TG-GNN), dynamically enhancing node features with structural connections and transformer-driven attention mechanisms. The framework adopts a Maximum Marginal Relevance (MMR) strategy for selecting sentences. Our evaluations show that KETGS outshines other leading extractive summarization models, delivering summaries that are more relevant, cohesive, and concise, thus better preserving the essence and structure of the original texts. [ABSTRACT FROM AUTHOR]

Despite the remarkable progress of large language models (LLMs) in understanding and generating unstructured text, their application in structured data domains and their multi-role capabilities remain underexplored. In particular, utilizing LLMs to perform complex reasoning tasks on knowledge graphs (KGs) is still an emerging area with limited research. To address this gap, we propose KG-EGV, a versatile framework leveraging LLMs to perform KG-based tasks. KG-EGV consists of four core steps: sentence segmentation, graph retrieval, EGV, and backward updating, each designed to segment sentences, retrieve relevant KG components, and derive logical conclusions. EGV, a novel integrated framework for LLM inference, enables comprehensive reasoning beyond retrieval by synthesizing diverse evidence, which is often unattainable via retrieval alone due to noise or hallucinations. The framework incorporates six key stages: generation expansion, expansion evaluation, document re-ranking, re-ranking evaluation, answer generation, and answer verification. Within this framework, LLMs take on various roles, such as generator, re-ranker, evaluator, and verifier, collaboratively enhancing answer precision and logical coherence. By combining the strengths of retrieval-based and generation-based evidence, KG-EGV achieves greater flexibility and accuracy in evidence gathering and answer formulation. Extensive experiments on widely used benchmarks, including FactKG, MetaQA, NQ, WebQ, and TriviaQA, demonstrate that KG-EGV achieves state-of-the-art performance in answer accuracy and evidence quality, showcasing its potential to advance QA research and applications. [ABSTRACT FROM AUTHOR]

Most deep learning models suffer from the problems of large computational complexity and insufficient feature extraction. To achieve a dynamic balance and tradeoff between computational complexity and performance, an enhanced SVTR-based scene text recognition model (SVTR-SRNet) was designed in this paper. In the SVTR-SRNet, we first created a bottom-up jump connection network that increases the number of information transfer pathways between the top and bottom features and improves the accuracy of information extraction. Second, we modified the attention mechanism by adding a new intermediate parameter called SR(Q) (Spatial Reduction (Q)), which finds a suitable compromise between the representational power and computing efficiency. In contrast to the conventional attention mechanism, the novel technique maintains the ability to model the global context while also enhancing efficiency. Ultimately, we developed a novel adaptive hybrid loss function to mitigate the shortcomings of a singular loss function's inadequate generalization capacity and enhance the model's resilience in handling a variety of challenging scenarios. Our technique outperforms existing standard models in terms of recognition performance on both the English and Chinese datasets, which deal with a high number of similar characters. As the model possesses great efficiency and outstanding cross-linguistic adaptability, it has a wide range of practical applications. [ABSTRACT FROM AUTHOR]

Prompt tuning is a mainstream technique for fine-tuning large language models (LLMs), offering minimal parameter adjustments by learning task-specific prompt vectors. However, it suffers from training costs due to network-wide backpropagation and weaker performance compared to methods like adapters and LoRA, likely due to the limited capacity of soft prompts to encode task-specific information. This study introduces Late Inception Prompt Tuning (LIPT), a novel approach to soft prompt learning that enhances performance and efficiency by shortening backpropagation paths and employing a multidimensional bottleneck network with greater capacity. LIPT surpasses existing prompt tuning techniques on various benchmark tasks, delivering a 1.3% gain over LPT and a 5% improvement compared to standard prompt tuning when applied to RoBERTa-large, while converging more rapidly. It achieves an average accuracy of 90% across ten benchmark datasets. Notably, in certain scenarios, LIPT's performance approaches that of full-parameter fine-tuning methods. To evaluate parameter-efficient fine-tuning (PEFT) comprehensively, we propose an Efficiency Indicator (EI) that balances accuracy and cost. LIPT is well suited for natural language understanding tasks, like sentiment analysis and text classification, with potential extensions to larger-scale models and tasks like text generation. This framework advances the scalability and practicality of fine-tuning methods for diverse applications. [ABSTRACT FROM AUTHOR]

Self-admitted technical debts (SATDs) refer to a solution in software development that selects suboptimal solutions to meet the current requirements and are intentionally introduced and documented by developers. SATDs in issue-tracking systems are a complement to those within source code comments. The effective identification of SATDs is crucial for software quality assurance and maintenance. Current studies focus on whether issue sections contain debt, but overlook specific SATD types. Meanwhile, they lack solutions for the challenge that SATD features are hard to learn due to the scarcity of instances containing SATDs. To address these problems, we propose a novel method, which is a weighted prompt tuning to identify SATDs, called WPTD. Specifically, WPTD employs a weighted prompt tuning to adapt the model with few-shot samples for insufficient training data. Moreover, to improve the performance of the model, WPTD constructs an SATD verbalizer by extracting keywords through mutual information and refining it with prior contextual information. Furthermore, it also improves SATD representation by extracting weights using the chi-square method and integrating them into the text. Finally, to reduce bias, WPTD computes the average score of results as final predicted distributions. We conduct comprehensive experiments on seven projects and the results show that our method significantly outperforms baseline approaches. In addition, we summarize the project-specific keywords, which can help developers better understand SATDs. [ABSTRACT FROM AUTHOR]

In the recent past, the level of cyber threats has changed drastically, leading to the current transformation of the cybersecurity landscape. For example, emerging threats like Zero-day and polymorphic malware cannot be detected by conventional detection methods like heuristic and signature-based methods, which have proven useful in the identification of malware. In view of this shift in the cybersecurity paradigm, this study proposes to discuss the utilization of transformer models to improve malware detection effectiveness and the accuracy and efficiency in detecting malicious software. In this regard, this study adopts the application of transformers in identifying different forms of malicious software: ransomware, spyware, and trojans. Transformers are endowed with the ability to handle sequential data and capture intricate patterns. By employing deep learning techniques and conducting thorough contextual analysis, these models enhance the detection process by identifying subtle indications of compromise, which traditional methods may overlook. This research also explains the challenges and limitations related to the application of transformer-based models in real-world cybersecurity settings, which include computing requirements and large-scale labeled datasets' requirements. By the end, the article suggests potential future research avenues in order to improve and integrate these models into cybersecurity systems. [ABSTRACT FROM AUTHOR]

The rapid increase in data volume and variety within the field of machine learning necessitates ethical data utilization and adherence to strict privacy protection standards. Fair federated learning (FFL) has emerged as a pivotal solution for ensuring fairness and privacy protection within distributed learning environments. FFL not only enhances privacy safeguards but also addresses inherent limitations of existing federated learning (FL) systems by fostering equitable model training across diverse participant groups, mitigating the exclusion of individual users or minorities, and improving overall model fairness. This study examines the causes of bias and fairness within existing FL systems and categorizes solutions according to data partitioning strategies, privacy mechanisms, applicable machine learning models, communication architectures, and technologies designed to manage heterogeneity. To mitigate bias, enhance fairness, and strengthen privacy protections in FL, this study also explores fairness evaluation metrics, relevant applications, and associated challenges of FFL. Addressing bias, fairness, and privacy concerns across all mechanisms serves as a valuable resource for practitioners aiming to develop efficient FL solutions. [ABSTRACT FROM AUTHOR]

Existing risk measurement methods often fail to fully consider the impact of climatic conditions on stock market risk, making it difficult to capture dynamic patterns and long-term dependencies. To address these issues, we propose the TS-GRU method: this approach utilizes a temporal convolutional network (TCN) to extract underlying features from historical data, capturing key characteristics of time series data. Subsequently, a gated recurrent unit (GRU) model is employed to capture dynamic patterns and long-term dependencies within the stock market. Finally, the TS-GRU model is optimized using the Sparrow algorithm based on collective behavior, iteratively evaluating and refining model parameters to obtain improved solutions. Experimental results demonstrate the effectiveness of the TS-GRU method in providing accurate risk assessment and forecasting. This comprehensive approach takes into account carbon finance, climate change, and environmental factors, offering valuable insights to investors to help them to understand and manage investment risks in the ever-changing stock market. [ABSTRACT FROM AUTHOR]

To tackle the issue of limited sample data in small sample fault diagnosis for rolling bearings using deep learning, we propose a fault diagnosis method that integrates a KANs-CNN network. Initially, the raw vibration signals are converted into two-dimensional time-frequency images via a continuous wavelet transform. Next, Using CNN combined with KANs for feature extraction, the nonlinear activation of KANs helps extract deep and complex features from the data. After the output of CNN-KANs, an FAN network module is added. The FAN module can employ various feature aggregation strategies, such as weighted averaging, max pooling, addition aggregation, etc., to combine information from multiple feature levels. To further tackle the small sample issue, data generation is performed on the original data through diffusion networks under conditions of fewer samples for bearings and tools, thereby increasing the sample size of the dataset and enhancing fault diagnosis accuracy. Experimental results demonstrate that, under small sample conditions, this method achieves higher accuracy compared to other approaches. [ABSTRACT FROM AUTHOR]

Dataset distillation has become an important technique for enhancing the efficiency of data when training machine learning models. It finds extensive applications across various fields, including computer vision (CV) and natural language processing (NLP). However, it essentially consists of a deep neural network (DNN) model, which remain susceptible to security and privacy vulnerabilities (e.g., backdoor attacks). Existing studies have primarily focused on optimizing the balance between computational efficiency and model performance, overlooking the accompanying security and privacy risks. This study presents the first backdoor attack targeting NLP models trained on distilled datasets. We introduce malicious triggers into synthetic data during the distillation phase to execute a backdoor attack on downstream models trained with these data. We employ several widely used datasets to assess how different architectures and dataset distillation techniques withstand our attack. The experimental findings reveal that the attack achieves strong performance with a high (above 0.9 and up to 1.0) attack success rate (ASR) in most cases. For backdoor attacks, high attack performance often comes at the cost of reduced model utility. Our attack maintains high ASR while maximizing the preservation of downstream model utility, as evidenced by results showing that the clean test accuracy (CTA) of the backdoored model is very close to that of the clean model. Additionally, we performed comprehensive ablation studies to identify key factors affecting attack performance. We tested our attack method against five defense strategies, including NAD, Neural Cleanse, ONION, SCPD, and RAP. The experimental results show that these defense methods are unable to reduce the attack success rate without compromising the model's performance on normal tasks. Therefore, these methods cannot effectively defend against our attack. [ABSTRACT FROM AUTHOR]

Performing an operational safety situation evaluation of the terminal area control system is crucial for enhancing safety management and ensuring operational safety in the terminal area. We use a combination active–passive risk source identification method to thoroughly identify the safety features of the terminal area control system, and then establish a related indicator system. After that, the system's secondary and primary indicators are used as input and output vectors for an Extreme Learning Machine (ELM). This creates an ISSA-ELM-based risk probability prediction model for the primary indicators, which gives us the predicted values of the primary indicator risks. Bayesian theory merges the output results to determine the overall safety status of the terminal area control system. The evaluation results indicate a distinct correlation between the secondary and primary indicators in a terminal area of North China. The ELM, which has been improved by the Sparrow Search Algorithm (ISSA), correctly displays this mapping and predicts the operational risks. The improved model is able to make predictions that are more than 80% accurate. A comparison of the model's evaluation results with the actual operational conditions during the same period demonstrates consistency, suggesting that this method may consistently evaluate the operational safety status of the terminal area control system. [ABSTRACT FROM AUTHOR]

Offensive content is a complex and multifaceted form of harmful material that targets individuals or groups. In recent years, offensive language (OL) has become increasingly harmful, as it incites violence and intolerance. The automatic identification of OL on social networks is essential to curtail the spread of harmful content. We address this problem by developing an architecture to effectively respond to and mitigate the impact of offensive content on society. In this paper, we use the Davidson dataset containing 24,783 samples of tweets and proposed three different architectures for detecting OL on social media platforms. Our proposed approach involves concatenation of features (TF-IDF, Word2Vec, sentiments, and FKRA/FRE) and a baseline machine learning model for the classification. We explore the effectiveness of different dimensions of GloVe embeddings in conjunction with deep learning models for classifying OL. We also propose an architecture that utilizes advanced transformer models such as BERT, ALBERT, and ELECTRA for pre-processing and encoding, with 1D CNN and neural network layers serving as the classification components. We achieve the highest precision, recall, and F1 score, i.e., 0.89, 0.90, and 0.90, respectively, for both the "bert encased preprocess/1 + small bert/L4H512A8/1 + neural network layers" model and the "bert encased preprocess/1 + electra small/2 + cnn" architecture. [ABSTRACT FROM AUTHOR]

This paper presents the Open-Source Intelligence Disaster Event Tracker (ODET), a modular platform that provides customizable endpoints and agents for each processing step. ODET enables the implementation of AI-enhanced algorithms to respond to various complex disaster scenarios. To evaluate ODET, we conducted two case studies using unmodified AI models to demonstrate its base performance and potential applications. Through our case studies on Hurricane Harvey and the 2023 Turkey earthquake, we show how complex tasks can be quickly broken down with ODET while achieving a score of up to 89% using the AlignScore metric. ODET enables compliance with Berkeley protocol requirements by ensuring data privacy and using privacy-preserving processing methods. Our results demonstrate that ODET is a robust platform for the long-term monitoring and analysis of dynamic environments and can improve the efficiency and accuracy of situational awareness reports in disaster management. [ABSTRACT FROM AUTHOR]

This paper analyzes the experience of first-person shooter (FPS) players when game difficulty is adjusted by personalizing their audio cue settings, considering the balance between player performance, modeled using Gaussian process regression (GPR), and prior data serving as designer preference. In addition, we investigate why player experience changes according to in-game audio cues. Previous studies have proposed various dynamic difficulty adjustment (DDA) methods for FPS games. However, few studies have considered the role of audio cues in the player experience. This paper compares the player experience of personalized enemy audio cue volume settings (GPR-DDA) with that of predetermined settings in an FPS game. Two comprehensive experimental phases, involving 80 participants, are conducted to assess the efficacy of GPR-DDA. The experience of our players is measured using questions taken from the Game User Experience Satisfaction Scale (GUESS) questionnaire and a final survey asking for their open-ended feedback. A large language model (LLM) is used to analyze the natural language expressions of the players according to their native languages. To ensure the LLM effectively assists a limited number of qualified human evaluators in classifying player responses, we have developed an original procedure for this task. The GUESS results show that GPR-DDA can improve the player experience. In addition, the high consistency in the classification results over multiple runs of the selected LLM, as well as the similarity between its results and those of our human evaluators, reflects the reliability of the proposed LLM-assisted procedure. [ABSTRACT FROM AUTHOR]

Featured Application: Enhancing Intelligent Transport Systems by accurately classifying traffic-related social media messages. Classifying social media (SM) messages into relevant or irrelevant categories is challenging due to data sparsity, imbalance, and ambiguity. This study aims to improve Intelligent Transport Systems (ITS) by enhancing short text classification of traffic-related SM data. Deep learning methods such as RNNs, CNNs, and BERT are effective at capturing context, but they can be computationally expensive, struggle with very short texts, and perform poorly with rare words. On the other hand, transfer learning leverages pre-trained knowledge but may be biased towards the pre-training domain. To address these challenges, we propose DLCTC, a novel system combining character-level, word-level, and context features with BiLSTM and TextCNN-based attention. By utilizing external knowledge, DLCTC ensures an accurate understanding of concepts and abbreviations in traffic-related short texts. BiLSTM captures context and term correlations; TextCNN captures local patterns. Multi-level attention focuses on important features across character, word, and concept levels. Experimental studies demonstrate DLCTC's effectiveness over well-known short-text classification approaches based on CNN, RNN, and BERT. [ABSTRACT FROM AUTHOR]

Named-Entity Recognition (NER) as a core task in Natural Language Processing (NLP) aims to automatically identify and classify specific types of entities from unstructured text. In recent years, the introduction of Transformer architecture and its derivative BERT model has pushed the performance of NER to unprecedented heights. However, these models often have high requirements for computational power and memory resources, making it difficult to train and deploy them on small computing platforms. Although ALBERT as a lightweight model uses parameter sharing and matrix decomposition strategies to reduce memory consumption to some extent consumption, it does not effectively reduce the computational load of the model. Additionally, due to its internal sharing mechanism, the model's understanding ability of text is reduced leading to poor performance in named-entity recognition tasks. To address these challenges, this manuscript proposes an efficient lightweight model called GoalBERT. The model adopts multiple fusion technologies by integrating a lightweight and efficient BiGRU that excels at handling context into part of the Transformer's self-attention layers. This reduces the high computational demand caused by stacking multiple self-attention layers while enhancing the model's ability to process context information. To solve the problem of gradient disappearance and explosion during training, residual connections are added between core layers for more stable training and steady performance improvement. Experimental results show that GoalBERT demonstrates recognition accuracy comparable to standard models with accuracy surpassing ALBERT by 10% in multi-entity type scenarios. Furthermore, compared to standard models, GoalBERT reduces memory requirements by 200% and improves training speed by nearly 230%. Experimental results indicate that GoalBERT is a high-quality lightweight model. [ABSTRACT FROM AUTHOR]

Relation extraction, a crucial task in natural language processing (NLP) for constructing knowledge graphs, entails extracting relational semantics between pairs of entities within a sentence. Given the intricacy of language, a single sentence often encompasses multiple entities that mutually influence one another. Recently, various iterations of recurrent neural networks (RNNs) have been introduced into relation extraction tasks, where the efficacy of neural network structures directly influences task performance. However, many neural networks necessitate manual determination of optimal parameters and network architectures, resulting in limited generalization capabilities for specific tasks. In this paper, we formally define the context-based relation extraction problem and propose a solution utilizing neural architecture search (NAS) to optimize RNN. Specifically, NAS employs an RNN controller to delineate an RNN cell, yielding an optimal structure to represent all relationships, thereby aiding in extracting relationships between target entities. Additionally, to enhance relation extraction performance, we leverage the XLNet pretrained model to comprehensively capture the semantic features of the sentence. Extensive experiments conducted on a real-world dataset containing words with multiple relationships demonstrate that our proposed method significantly enhances micro-F1 scores compared to state-of-the-art baselines. [ABSTRACT FROM AUTHOR]

Employees are the most valuable resources in any company, and their well-being directly influences work productivity. This research investigates integrating health parameters and sentiment analysis expressed in sent messages to enhance employee well-being within organizations in the context of Industry 5.0. Our primary aim is to develop a Well-Being Index (WBI) that quantifies employee health through various physiological and psychological parameters. A new methodology combining data collection from wearable devices from 1 January 2023 to 18 October 2024 and advanced text analytics was employed to achieve the WBI. This study uses the LbfgsMaximumEntropy ML classification algorithm to construct the Well-Being Model (WBM) and Azure Text Analytics for sentiment evaluation to assess negative messages among employees. The findings reveal a correlation between physiological metrics and self-reported well-being, highlighting the utility of the WBI in identifying areas of concern within employee behavior. We propose that the employee global indicator (EGI) is calculated based on the WBI and the dissatisfaction score component (DSC) to measure the overall state of mind of employees. The WBM exhibited a MacroAccuracy of 91.81% and a MicroAccuracy of 95.95% after 384 configurations were analyzed. Azure Text Analytics evaluated 2000 text messages, resulting in a Precision of 99.59% and an Accuracy of 99.7%. In this case, the Recall was 99.89% and F1-score was 99.73%. In the Industry 5.0 environment, which focuses on the employee, a new protocol, the Employee KPI Algorithm (EKA), is integrated to prevent and identify employee stress. This study underscores the synergy between quantitative health metrics and qualitative sentiment analysis, offering organizations a framework to address employee needs proactively. [ABSTRACT FROM AUTHOR]

Efficient and accurate fault diagnosis of rotating machinery is extremely important. Fault diagnosis methods using vibration signals based on convolutional neural networks (CNNs) have become increasingly mature. They often struggle with capturing the temporal dynamics of vibration signals. To overcome this, the application of Transformer-based Vision Transformer (ViT) methods to fault diagnosis is gaining attraction. Nonetheless, these methods typically require extensive preprocessing, which increases computational complexity, potentially reducing the efficiency of the diagnosis process. Addressing this gap, this paper presents the Time Series Vision Transformer (TSViT), tailored for effective fault diagnosis. The TSViT incorporates a convolutional layer to extract local features from vibration signals alongside a transformer encoder to discern long-term temporal patterns. A thorough experimental comparison of three diverse datasets demonstrates the TSViT's effectiveness and adaptability. Moreover, the paper delves into the influence of hyperparameter tuning on the model's performance, computational demand, and parameter count. Remarkably, the TSViT achieves an unprecedented 100% average accuracy on two of the test sets and 99.99% on the other, showcasing its exceptional fault diagnosis capabilities for rotating machinery. The implementation of this model will bring significant economic benefits. [ABSTRACT FROM AUTHOR]

Featured Application: This work can be applied to an online education platform to predict students' current learning states. In response to the challenge of low predictive accuracy in scenarios with limited data, we propose a few adjustable parameters prediction model based on lightweight prefix-tuning (FAP-Prefix). Prefix-tuning is an efficient fine-tuning method that only adjusts prefix vectors while keeping the model's original parameters frozen. In each transformer layer, the prefix vectors are connected with the internal key-value pair of the transformer structure. By training on the synthesized sequence of the prefix and original input with masked learning, the transformer model learns the features of individual learning behaviors. In addition, it can also discover hidden connections of continuous learning behaviors. During fine-tuning, all parameters of the pre-trained model are frozen, and downstream task learning is accomplished by adjusting the prefix parameters. Continuous trainable prefix vectors can influence subsequent vector representations, leading to the generation of session dropout prediction results. The experiments show that FAP-Prefix significantly outperforms traditional methods in data-limited settings, with AUC improvements of +4.58%, +3.53%, and +8.49% under 30%, 10%, and 1% data conditions, respectively. It also surpasses state-of-the-art models in prediction performance (AUC +5.42%, ACC +5.3%, F1 score +5.68%). [ABSTRACT FROM AUTHOR]

Natural disasters cause extensive infrastructure and significant economic losses, hindering sustainable development and impeding social and economic progress. More importantly, they jeopardize community well-being by causing injuries, damaging human health, and resulting in loss of life. Furthermore, communities often experience delayed disaster response. Aggravating the situation, the frequency and impact of disasters have been continuously increasing. Therefore, fast and effective disaster response management is paramount. To achieve this, disaster managers must proactively safeguard communities by developing quick and effective disaster management strategies. Disruptive technologies such as artificial intelligence (AI), machine learning (ML), and robotics and their applications in geospatial analysis, social media, and smartphone applications can significantly contribute to expediting disaster response, improving efficiency, and enhancing safety. However, despite their significant potential, limited research has examined how these technologies can be utilized for disaster response in low-income communities. The goal of this research is to explore which technologies can be effectively leveraged to improve disaster response, with a focus on low-income communities. To this end, this research conducted a comprehensive review of existing literature on disruptive technologies, using Covidence to simplify the systematic review process and NVivo 14 to synthesize findings. [ABSTRACT FROM AUTHOR]

The development and testing of a virtual assistant (VA) designed to enhance information retrieval and support in an academic environment are presented in this paper, with the Retrieval-Augmented Generation (RAG) approach being utilized alongside Google's VERTEX AI Palm-2 model. A novel integration of RAG with contextual learning is introduced in this study, specifically for applications in university contact centers, where accuracy and relevance are considered paramount. The effectiveness of the VA was evaluated through user testing, focusing on two primary hypotheses: first, that the VA can achieve accurate interpretation and response to queries with context-based information, and second, that the VA minimizes potential harm from erroneous responses. In total, 187 participants were involved in the testing, and a diverse set of inquiries was utilized, resulting in 561 query–response interactions that were analyzed. It was shown that contextual data significantly reduced hallucinations and increased response accuracy, thereby underscoring the value of the RAG method in applications requiring high levels of specificity. Furthermore, the study provides empirical insights into the impact of AI-generated hallucinations and response inconsistencies, particularly about structured or procedural data. A framework for mitigating these challenges in future implementations is also offered. The scalability and adaptability of the RAG method in specialized academic contexts are demonstrated in this work, with broader implications for integrating AI-driven VAs across educational and professional domains being highlighted. [ABSTRACT FROM AUTHOR]

Visual object tracking is a fundamental task in computer vision, with applications ranging from video surveillance to autonomous driving. Despite recent advances in transformer-based one-stream trackers, unrestricted feature interactions between the template and the search region often introduce background noise into the template, degrading the tracking performance. To address this issue, we propose FETrack, a feature-enhanced transformer-based network for visual object tracking. Specifically, we incorporate an independent template stream in the encoder of the one-stream tracker to acquire the high-quality template features while suppressing the harmful background noise effectively. Then, we employ a sequence-learning-based causal transformer in the decoder to generate the bounding box autoregressively, simplifying the prediction head network. Further, we present a dynamic threshold-based online template-updating strategy and a template-filtering approach to boost tracking robustness and reduce redundant computations. Extensive experiments demonstrate that our FETrack achieves a superior performance over state-of-the-art trackers. Specifically, the proposed FETrack achieves a 75.1% AO on GOT-10k, 81.2% AUC on LaSOT, and 89.3% P n o r m on TrackingNet. [ABSTRACT FROM AUTHOR]

Arabic raw audio datasets were initially gathered to produce a corresponding signal spectrum, which was further used to extract the Mel-Frequency Cepstral Coefficients (MFCCs). The pronunciation dictionary, language model, and acoustic model were further derived from the MFCCs' features. These output data were processed into Baidu's Deep Speech model (ASR system) to attain the text corpus. Baidu's Deep Speech model was implemented to precisely identify the global optimal value rapidly while preserving a low word and character discrepancy rate by attaining an excellent performance in isolated and end-to-end speech recognition. The desired outcome in this work is to forecast the next word and character in a sequential and systematic order that applies under natural language processing (NLP). This work combines the trained Arabic language model ARABERT with the potential of Long Short-Term Memory (LSTM) networks to predict the next word and character in an Arabic text. We used the pre-trained ARABERT embedding to improve the model's capacity and, to capture semantic relationships within the language, we educated LSTM + CNN and Markov models on Arabic text data to assess the efficacy of this model. Python libraries such as TensorFlow, Pickle, Keras, and NumPy were used to effectively design our development model. We extensively assessed the model's performance using new Arabic text, focusing on evaluation metrics like accuracy, word error rate, character error rate, BLEU score, and perplexity. The results show how well the combined LSTM + ARABERT and Markov models have outperformed the baseline models in envisaging the next word or character in the Arabic text. The accuracy rates of 64.9% for LSTM, 74.6% for ARABERT + LSTM, and 78% for Markov chain models were achieved in predicting the next word, and the accuracy rates of 72% for LSTM, 72.22% for LSTM + CNN, and 73% for ARABERET + LSTM models were achieved for the next-character prediction. This work unveils a novelty in Arabic natural language processing tasks, estimating a potential future expansion in deriving a precise next-word and next-character forecasting, which can be an efficient utility for text generation and machine translation applications. [ABSTRACT FROM AUTHOR]

Accessibility in online platforms is a critical concern in our increasingly digital world, where information and services are predominantly accessed through the Internet. The purpose of this systematic review is to provide a comprehensive overview of the current state of the art in online accessibility technologies, and it is focused on key tools such as sign language recognition, speech-to-text, text-to-speech, and voice recognition. Despite advancements in digital inclusivity, numerous technical limitations persist, which limit the accessibility of online content for individuals with disabilities. Our findings indicate that while speech and voice technologies have achieved good accuracies and low word error rates, further research is needed to improve the accuracy and usability of sign language recognition systems, especially for continuous sign language recognition, as they have low accuracy. In this review, we analyzed research articles and publications from well-known databases, including Google Scholar, Elsevier, IEEE Xplore, and Springer. In order to ensure a high standard of quality, we applied the PRISMA 2020 and PEDro methodologies to quantitatively and qualitatively filter the thousands of articles provided by these databases, and we selected only studies that were related to our study. Key areas of investigation included the performance and accuracy of sign language interfaces, speech-to-text, text-to-speech, and speech recognition applications and the compatibility of these technologies with different platforms and devices. This review also explores the role of emerging technologies such as artificial intelligence (AI) and machine learning (ML) in enhancing accessibility and personalizing user experiences. Through a critical analysis of current solutions and a discussion of existing gaps, this paper offers insights into potential improvements and future directions for creating more accessible online environments. The findings might be valuable to researchers and developers dedicated to promoting digital inclusivity and equality. [ABSTRACT FROM AUTHOR]

Accurate classification of import and export goods' Harmonized System (HS) codes is essential for ensuring tax security. Applying text classification technologies for HS code classification can significantly enhance the prevention and control of customs tax risks. However, the goods text is a semi-structured one that involves multi-domain Chinese professional vocabulary, which poses challenges for current classification models. These models often suffer from inadequate text representation and imprecise feature extraction. To address these challenges, we propose a novel classification model ERNIE-BiLSTM-Channel attention–Spatial attention (EBLCS). This model integrates ERNIE (Enhanced Representation through Knowledge Integration) with a Bidirectional Long Short-Term Memory Network (BiLSTM) and employs multi-scale attention mechanisms. The ERNIE-BiLSTM model provides a more comprehensive and accurate representation of the goods text, effectively capturing the global features of the text. By introducing channel attention and spatial attention mechanisms, greater weights are assigned to important words and word embedding dimensions, significantly enhancing the model's ability to perceive key information. The experimental results on a customs dataset demonstrate that the EBLCS model consistently outperforms various baseline models across all evaluation metrics, effectively enhancing the performance of HS code classification. [ABSTRACT FROM AUTHOR]

Kolmogorov–Arnold Networks (KANs) are a novel class of neural network architectures based on the Kolmogorov–Arnold representation theorem, which has demonstrated potential advantages in accuracy and interpretability over Multilayer Perceptron (MLP) models. This paper comprehensively evaluates the robustness of various KAN architectures—including KAN, KAN-Mixer, KANConv_KAN, and KANConv_MLP—against adversarial attacks, which constitute a critical aspect that has been underexplored in current research. We compare these models with MLP-based architectures such as MLP, MLP-Mixer, and ConvNet_MLP across three traffic sign classification datasets: GTSRB, BTSD, and CTSD. The models were subjected to various adversarial attacks (FGSM, PGD, CW, and BIM) with varying perturbation levels and were trained under different strategies, including standard training, adversarial training, and Randomized Smoothing. Our experimental results demonstrate that KAN-based models, particularly the KAN-Mixer, exhibit superior robustness to adversarial attacks compared to their MLP counterparts. Specifically, the KAN-Mixer consistently achieved lower Success Attack Rates (SARs) and Degrees of Change (DoCs) across most attack types and datasets while maintaining high accuracy on clean data. For instance, under FGSM attacks with ϵ = 0.01 , the KAN-Mixer outperformed the MLP-Mixer by maintaining higher accuracy and lower SARs. Adversarial training and Randomized Smoothing further enhanced the robustness of KAN-based models, with t-SNE visualizations revealing more stable latent space representations under adversarial perturbations. These findings underscore the potential of KAN architectures to improve neural network security and reliability in adversarial settings. [ABSTRACT FROM AUTHOR]

This study investigates the impact of rising temperatures on residential water use (RWU) in Seoul from 2015 to 2021, addressing the challenges of urban water sustainability under climate change. Using advanced models—convolutional neural networks (CNNs), long short-term memory (LSTM) Networks, eXtreme Gradient Boosting (XGBoost), and Bayesian Neural Networks (BNNs)—we examined RWU prediction accuracy and incorporated a method to quantify prediction uncertainties. As a result, the BNN model emerged as a robust alternative, demonstrating competitive predictive accuracy and the capability to account for uncertainties in predictions. Recent studies highlight a strong correlation between rising temperatures and increased RWU, especially during summer, with tropical nights (with temperatures above 25 °C) becoming more common; Seoul experienced a record 26 consecutive tropical nights in July 2024, underscoring a trend toward higher RWU. To capture these dynamics, we employed Shared Socioeconomic Pathway (SSP) scenarios and downscaled the KACE-1-0-G Global Climate Model (GCM) for Seoul, projecting a progressive increase in RWU: 0.49% in the F1 period (2011–2040), 1.53% in F2 (2041–2070), and 2.95% in F3 (2071–2100), with significant rises in maximum RWU across these intervals. Our findings highlight an urgent need for proactive measures to secure water resources in response to the anticipated increase in urban water demand driven by climate change. [ABSTRACT FROM AUTHOR]

In actual production processes, analysis and prediction tasks commonly rely on large amounts of time-series data. However, real-world scenarios often face issues such as insufficient or imbalanced data, severely impacting the accuracy of analysis and predictions. To address this challenge, this paper proposes a dual-layer transfer model based on Generative Adversarial Networks (GANs) aiming to enhance the training speed and generation quality of time-series data augmentation under small-sample conditions while reducing the reliance on large training datasets. This method introduces a module transfer strategy based on the traditional GAN framework which balances the training between the discriminator and the generator, thereby improving the model's performance and convergence speed. By employing a dual-layer network structure to transfer the features of time-series signals, the model effectively reduces the generation of noise and other irrelevant features, improving the similarity of the generated signals' characteristics. This paper uses speech signals as a case study, addressing scenarios where speech data are difficult to collect and the limited number of speech samples available for effective feature extraction and analysis. Simulated speech timbre generation is conducted, and the experimental results on the CMU-ARCTIC database show that, compared to traditional methods, this approach achieves significant improvements in enhancing the consistency of generated signal features and the model's convergence speed. [ABSTRACT FROM AUTHOR]

As user requirements become increasingly complex, the demand for product personalization is growing, but traditional hardware-centric production relies on fixed procedures that lack the flexibility to support diverse requirements. Although bespoke manufacturing has been introduced, it provides users with only a few standardized options, limiting its ability to meet a wide range of needs. To address this issue, a new manufacturing concept called the software-defined factory has emerged. It is an autonomous manufacturing system that provides reconfigurable manufacturing services to produce tailored products. Reinforcement learning has been suggested for flexible scheduling to satisfy user requirements. However, fixed rule-based methods struggle to accommodate conflicting needs. This study proposes a novel federated digital twin scheduling that combines large language models and deep reinforcement learning algorithms to meet diverse user requirements in the software-defined factory. The large language model-based literacy module analyzes requirements in natural language and assigns weights to digital twin attributes to achieve highly relevant KPIs, which are used to guide scheduling decisions. The deep reinforcement learning-based scheduling module optimizes scheduling by selecting the job and machine with the maximum reward. Different types of user requirements, such as reducing manufacturing costs and improving productivity, are input and evaluated by comparing the flow-shop scheduling with job-shop scheduling based on reinforcement learning. Experimental results indicate that in requirement case 1 (the manufacturing cost), the proposed method outperforms flow-shop scheduling by up to 14.9% and job-shop scheduling by 5.6%. For requirement case 2 (productivity), it exceeds the flow-shop method by up to 13.4% and the job-shop baseline by 7.2%. The results confirm that the literacy DRL scheduling proposed in this paper can handle the individual characteristics of requirements. [ABSTRACT FROM AUTHOR]