Search

This study employs an unsupervised learning technique to identify features of misinformation on Weibo, a popular social media platform in China. By utilizing BERT (Bidirectional Encoder Representations from Transformers) for sequence classification, we were able to detect a dataset with a mix of true and false information, achieving a perfect recall rate for false information detection. However, the precision for true information was low, with a 3% precision, 6% recall, and an F1 score of 6%, indicating a high rate of misclassification. To address this, we conducted a Latent Dirichlet Allocation (LDA) analysis on the misclassified true information, identifying specific features that led to incorrect classification. Additionally, social network analysis revealed the presence of structural holes within the information network. This study contributes to the understanding of misinformation detection mechanisms and provides insights into the social dynamics of information spread.

Transformer-based models, such as Bidirectional Encoder Representations from Transformers (BERT), cannot process long sequences because their self-attention operation scales quadratically with the sequence length. To remedy this, we introduce the Look Near and Look Far BERT (LNLF-BERT) with a two-level self-attention mechanism at the sentence and document levels, which can handle document classifications with thousands of tokens. The self-attention mechanism of LNLF-BERT retains some of the benefits of full self-attention at each level while reducing the complexity of not using full self-attention on the whole document. Our theoretical analysis shows that the LNLF-BERT mechanism is an approximator of the full self-attention model. We pretrain the LNLF-BERT from scratch and fine-tune it on downstream tasks. The experiments were also conducted to demonstrate the feasibility of LNLF-BERT in long text processing. Moreover, LNLF-BERT effectively balances local and global attention, allowing for efficient document-level understanding. Compared to other long-sequence models like Longformer and BigBird, LNLF-BERT shows competitive performance in both accuracy and computational efficiency. The architecture is scalable to various downstream tasks, making it adaptable for different applications in natural language processing.

Insider threats cause greater losses than external attacks, prompting organizations to invest in detection systems. However, there exist challenges: 1) Security and privacy concerns prevent data sharing, making it difficult to train robust models and identify new attacks. 2) The diversity and uniqueness of organizations require localized models, as a universal solution could be more effective. 3) High resource costs, delays, and data security concerns complicate building effective detection systems. This paper introduces FedITD, a flexible, hierarchy, and federated framework with local real-time detection systems, combining Large Language Models (LLM), Federated Learning (FL), Parameter Efficient Tuning (PETuning), and Transfer Learning (TF) for insider threat detection. FedITD uses FL to protect privacy while indirect integrating client information and employs PETuning methods (Adapter, BitFit, LoRA) with LLMs (BERT, RoBERTa, XLNet, DistilBERT) to reduce resource use and time delay. FedITD customizes client models and optimizes performance via transfer learning without central data transfer, further enhancing the detection of new attacks. FedITD outperforms other federated learning methods and its performance is very close to the best centrally trained method. Extensive experiment results show FedITD’s superior performance, adaptability to varied data, and reduction of resource costs, achieving an optimal balance in detection capabilities across source data, unlabeled local data, and global data. Alternative PETuning implementations are also explored in this paper.

Text classification tasks aim to comprehend and classify text content into specific classifications. This task is crucial for interpreting unstructured text, making it a foundational task in the field of Natural Language Processing(NLP). Despite advancements in large language models, lightweight text classification via these models still demands substantial computational resources. Therefore, this paper presents a multiscale interactive attention short text classification model based on BERT, which is designed to address the short text classification problem with limited resources. A corpus containing news articles, Chinese comments, and English sentiment classifications is employed for text classification. The model uses BERT pre-trained word vectors as embedding layers, connects to a multilevel feature extraction network, and further extracts contextual features after feature fusion. The experimental results on the THUCNews, Today’s headline news corpus, the SST-2 dataset, and the Touhou 38 W dataset demonstrate that our method outperforms all existing algorithms in the literature.

Depression is a prevalent mental health condition that significantly impacts individuals’ lives. Early detection of depression is crucial for timely intervention and improved outcomes. However, traditional machine learning approaches are constrained by the limited amount of annotated data and lack of model transparency. This study aims to address these challenges by leveraging social media data and advanced natural language processing techniques to develop effective and explainable models for depression detection. The study focuses on two main objectives. The first objective is to develop and evaluate fine-tuned Bidirectional Encoder Representations from Transformers (BERT), BERT with Bidirectional Long Short-Term Memory (BERT-BiLSTM), and BERT with Convolutional Neural Network (BERT-CNN) models, and compare their performance with MentalBERT, a state-of-the-art model for mental health detection. The second objective is to observe the key features used by the BERT models to make the decision-making using Transformer Interpretability Beyond Attention Visualization and Average Attention Weight methods. The study utilizes three publicly available datasets: the Depression Reddit Dataset, the Sentiment Analysis for Tweets Dataset, and the Mental Health Corpus. The results demonstrate that the proposed models, especially BERT-BiLSTM and BERT-CNN, achieve superior performance compared to MentalBERT, particularly regarding accuracy and F1-score. Notably, BERT-CNN achieved exceptional accuracy scores of 0.982, 0.961, and 1.0 on the Depression Reddit Dataset, the Mental Health Corpus, and the Sentiment Analysis for Tweets Dataset, respectively, demonstrating its robust performance across different social media contexts. The attention map visualizations provide valuable insights into the language patterns and key features associated with depression in social media posts. This study contributes to the mental health field by presenting novel and explainable models for depression detection using social media data. The proposed approaches have the potential to assist mental health professionals in early identification and intervention, ultimately improving the lives of individuals affected by depression.

It is essential for MOOCs to understand the learner’s emotions. This study aims to develop a novel sentiment analysis model to automatically classify the emotions of MOOC learners, which is beneficial to educators and instructors in improving the quality of courses and platform design in MOOCs. Firstly, a robust and interpretable model was proposed, named BB-OAM, which incorporated BERT, attention method, and BiLSTM to extract features of forum reviews as accurately as possible. To capture and differentiate among various affective tendencies more effectively in sentiment analysis, we have taken a breakthrough by introducing an orthogonal attention mechanism to enhance the model’s performance for emotion-ambiguous sentences. Compared with models in previous studies including SVM, Tree-CRF, BiRNN, LSTM, and BiLSTM, our method improved the accuracy value of sentiment analysis by 24%, 24%, 19%, 17%, and 9% respectively. Ablation experiments were conducted to systematically evaluate the impact of the orthogonal attention mechanism on sentiment analysis. Through visualization analysis, the model showed higher sensitivity in capturing sentiment-related contents, which further validated the reliability and effectiveness of the proposed method in sentiment classification tasks of MOOC reviews. In essence, this study has great methodological and theoretical insights to help educators and instructors gain a deeper understanding of the actual needs of learners, so as to optimize the efficiency of utilizing MOOC platforms and courses, and to promote effective interactions and collaboration between learners and educators.

In the era of digital transformation, customer feedback has become crucial for improving service quality. This study aims to enhance customer satisfaction through sentiment analysis utilizing machine learning techniques, with additional case studies conducted to ensure comprehensive method validation. Traditional sentiment analysis methods frequently fail to manage the complexity and volume of feedback data, yielding to less accurate insights. To address this challenge, we analyzed six machine learning models: Naïve Bayes, Support Vector Machine (SVM), Long Short-Term Memory (LSTM), Random Forest, AdaBoost, and BERT, with a particular focus on BERT. Our results demonstrate that BERT outperformed the other models in terms of both accuracy and processing speed, achieving an accuracy of up to 95%. The excellence of BERT in managing large and complex datasets provides a more precise sentiment analysis, which can significantly improve service quality and customer loyalty, while increasing company revenue by up to 15%. This research advances to the field of sentiment analysis by validating the effectiveness of BERT over traditional models through extensive comparative analysis and practical case studies.

Recent studies have shown that clustering latent variables, derived from autoencoders (AEs) through reconstruction tasks, is effective for identifying spatial domains of spatial transcriptomics (ST). Despite their utility, AEs exhibit inherent limitations due to discontinuities in latent spaces where similar inputs might not closely map. Variational Autoencoders address this by enhancing smoothness and continuity through alignment with priors, yet their interpolation can obscure necessary group distinctions. To overcome these limitations, we introduce stBERT, a BERT-based pre-training framework that transforms the traditional reconstruction task in ST into a masked language modeling task. stBERT leverages BERT’s hidden states to achieve refined and rich latent space mappings, preserves essential group distinctions in complex and high-dimensional biological data, to facilitate more effective clustering and interpretation. Our experiments demonstrate that stBERT significantly outperforms current state-of-the-art models in tasks related to spatial domain identification, including clustering validation using ground truth labels, intrinsic evaluation of clustering performance, and biological validation of clustering outcomes. The source code of the stBERT model is available at https://github.com/azusakou/stBERT.

In the era of digital communication, emojis have become the vibrant palette of people’s textual expressions, adding depth and emotion to messages. However, deciphering the subtle nuances of emojis poses a unique challenge due to their inherent ambiguity. The quest for predicting the next emoji in technological devices has emerged at the forefront of predictive analytics, demanding the analysis of extensive and diverse datasets while respecting user privacy. Enter Federated Learning (FL), a groundbreaking approach that thrives on learning from decentralized data sources without compromising confidentiality. This study delves into the unexplored realm of Federated Learning-based emoji prediction. Utilizing a tailored adaptation of BERT with a rich corpus of text (drawn from social media), encompassing both words and emojis, the author’s innovative architecture aims to predict the most fitting emoji for a given text while prioritizing user privacy. Welcome to EmojiSculpt, where the future of personalized emoji prediction takes center stage.

The learning status of students directly influences their learning outcomes. Course evaluations by students offer valuable insights into the effectiveness of teaching methodologies and students’ own perceptions of their learning environments. These evaluations encapsulate a spectrum of emotions, ranging from satisfaction to frustration, and provide a holistic view of the educational experience. Analyzing course evaluations plays a crucial role in deciphering students’ learning outcomes. By extracting sentiments from these evaluations, educators can gain a deeper understanding of students’ experiences, identifying areas of improvement and tailoring teaching strategies to better meet students’ needs. Sentiment analysis of course evaluations falls under text sentiment analysis. In this study, we leverage advanced deep learning techniques to extract sentiment from course evaluations. Specifically, we capitalize on the capabilities of BERT (Bidirectional Encoder Representations from Transformers), a state-of-the-art language model known for its proficiency in understanding context and semantic meaning. By harnessing BERT’s bidirectional encoding and feature extraction capabilities, coupled with the bidirectional influence of BiLSTM (Bidirectional Long Short-Term Memory) models, we aim to capture nuanced relationships within the text data. Moreover, we incorporate the attention mechanism into our model architecture to assign varying weights to different parts of the input sequence, enabling the model to focus on relevant information effectively. The proposed BERT-BiLSTM-ATTENTION (BBA) model represents an innovative approach that capitalizes on the synergy between these advanced techniques. Our experimental results, based on real-world data extracted from online comments on Massive Open Online Courses (MOOCs), demonstrate the superior performance of the BBA model compared to existing approaches. By thoroughly exploring bidirectional semantics and accounting for both preceding and subsequent influences, our model offers enhanced predictive capabilities and deeper insights into students’ learning experiences. This has significant implications for enhancing students’ learning outcomes and improving teaching methods.

In machine learning class imbalance is a pressing issue, where the model is biased towards the majority classes and underperforms in the minority classes. In textual data, the natural language processing (NLP) model bias significantly reduces overall accuracy, along with poor performance in minority classes. This paper investigates and compares the performance of transformer-based models, such as Multi-head Attention with the data levels and algorithmic levels approaches and BERT (Bidirectional Encoder Representations from Transformers) with LLM-based data augmentation. The research utilized the approaches, such as Random Over Sampler, Synthetic Minority Over-sampling Technique (SMOTE), SMOTEENN, data augmentation at word level, class weights, L2 regularization and leveraging GPT-3.5-Turbo’s for data augmentation to create additional data samples in imbalance dataset. The results from the experiment demonstrate that the LLM-based data augmentation with Multi-head Attention and BERT in the Myers-Briggs Type Indicator (MBTI) dataset (a highly skewed dataset) achieves the highest precision, recall and F1 score of 0.76 across terms. It indicates that the LLM-based data augmentation has significant improvements in dealing with class imbalance and improves the model’s accuracy in minority class types in the MBTI dataset.

Understanding customer sentiment through product reviews is important for businesses aiming to enhance their products and services. Traditional sentiment analysis often struggles to detect the wide range of emotional nuances, particularly those expressed through emojis, in online reviews. To address this challenge, our paper introduces a novel technique called Emoji-Enhanced BERT (EEBERT) based, on the BERT architecture. The Sentiment Adjustment Factor (SAF), created for emoji tokenization with the embeddings layer, is applied to analyze sentiment and emotion within review text content. Sentiments classification is evaluated using three different methods: (1) evaluating the full review text; (2) analyzing the text with emojis; and (3) integrating star ratings and total votes in EEBERT. These methods were fine-tuned using a labeled Amazon product reviews dataset. The approach utilizes a well-vetted dataset comprising approximately 10 million customer reviews from four different categories. This dataset is condensed to 26,818 reviews to only consider emoji contained reviews, covering 3,275 distinct emojis. A systematic approach to classifying reviews and an innovative technique for assessing the emotional semantics of emojis are developed. The proposed model achieves an accuracy of 97.00%, further applies continuous testing via 5-fold cross-validation, achieving an average accuracy of 99.21%, which supports the reliability of the EEBERT model.

Accumulating a substantial amount of textual data on power equipment defects during maintenance and inspection stages presents a valuable problem of assessing and grading these text-based information. This paper proposes a dual-channel text feature extraction model based on the pre-trained BERT model, applied to the evaluation of power equipment defect levels in textual data. Firstly, a dataset of power equipment defect levels is established, followed by data augmentation and preprocessing. Then, a neural network model is constructed, utilizing the pre-trained BERT model for initial semantic information extraction from the text, further extracting features through two modules, Bi-LSTM and CNN, on top of BERT’s output. Finally, the obtained feature vectors are concatenated to generate the output. Comparative experiments with other algorithms demonstrate that the proposed method out-performs others in multiple metrics, achieving an F1 score of 96%. The research findings can serve as a reference for achieving intelligent processing of power textual information.

In order to solve the problems of redundant information processing and high quality summary generation in existing methods, this paper proposes a two-stage text summary model which is composed of abstracted and generated models. First of all, the important information is abstracted by using an abstracted model which incorporates dilated convolution and gated convolution. Then, a replication mechanism is incorporated into the generated model to ensure that both primary and secondary information are taken into consideration, while also optimizing the cluster search algorithm. Finally, the network structure is reconfigured in the generated model to effectively integrate the coding capabilities of the two-way language model and the text generation abilities of the one-way language model. We conducted experiments on the CNewSum dataset, and achieved Rouge-1, Rouge-2, and Rouge-L values of 44.21, 27.52, and 39.03 for the two-paragraph text summary model respectively. The results indicate a significant enhancement in the performance of the two-paragraph text summary model compared to the benchmark model.

In recent years, the explosive growth of social media platforms has led to the rapid spread of vast amounts of false news and rumors on the internet. This disrupts various news sources such as online news outlets, radio and television stations, and newspapers, especially in Arab countries. Therefore, the fake news detection problem has been raised worldwide. Arabic research in this field is very little compared to English research. Previous researchers had used machine learning and deep learning techniques on a large scale, but recently they used pre-trained models in their studies. Our proposed model works by using the Arabic pre-trained Bidirectional Encoder Representations from Transformers (Arabic BERT) to extract features from the text, then uses a Convolutional Neural Network (1D-CNN or 2D-CNN) to reduce the size of the features and extract the important ones, then passes it to an artificial neural network to perform the classification process. In our experiment we introduce a novel hybrid system consists of two main parts. In the first part we try three Arabic pre-trained Bidirectional Encoder Representations from Transformers model (APBTM) which are AraBERT, GigaBERT or MARBERT, while in the second part, we use 1D-CNN or 2D-CNN. this leads to six architectures from this system. we make our experiment by train and evaluating every architecture using three datasets which are (Arabic News Stance (ANS), AraNews, and Covid19Fakes). A comparison is made between the proposed model and other modern models which used the same dataset. We made three sets of experiments depending on the used datasets. Each set includes a group of experiments, and then we present the results in tables. Our proposed model which is the hybrid model between AraBERT and 2D-CNN has achieved the best F1-scores of 0.6188,0.7837 and 0.8009 when using the ANS dataset, the Ara-News dataset, and the Covid19Fakes dataset respectively. Furthermore, the model reduces the training time by achieving better results with less number of epochs. The results indicate that the proposed model offers the best performance, with 71% accuracy in the Arabic News Stance (ANS) dataset outperforming the model made by Sorour and Abdelkader (2022) and the model made by Shishah (2022) that achieved accuracy of 67% and 66% respectively.

With the continuous development of the Internet, Massive Open Online Courses (MOOCs) have become the choice of more and more learners. MOOCs platforms provide a forum for students and teachers to interact in a timely manner and solve problems in the student learning process. Due to the large number of posts in the forum and the limited energy of teachers, many urgent posts that require priority responses are not handled promptly. In this paper, we propose a MOOC post classification model based on BERT to help teachers quickly and accurately distinguish urgent posts from non-urgent posts. The working process of the model is as follows: First, we use a pre-trained BERT to encode the post text; then, we use a downstream network based on a double pointwise convolution to perform global semantic refinement for the BERT encoding, and use a downstream network based on CNNs to perform local semantic extraction on BERT encoding, and finally merges two semantic refinement sub-vectors to classify the MOOC post. Experiments show that the urgent post’s F1-scores of our model on the three sub-datasets (Groups A, B, and C) of the Stanford MOOC post dataset achieve 83.54%, 81.79%, and 79.58%, respectively, which improves other methods on group A by 0.24%, group B by 0.89%, and group C by 0.88%. Our code is available at https://github.com/3269233844/MOOC-classification.

In recent years, the proliferation of online communication platforms and social media has given rise to a new wave of challenges, including the rapid spread of malicious bots. These bots, often programmed to impersonate human users, can infiltrate online communities, disseminate misinformation, and engage in various activities detrimental to the integrity of digital discourse. It is becoming more and more difficult to discern a text produced by deep neural networks from that created by humans. Transformer-based Pre-trained Language Models (PLMs) have recently shown excellent results in challenges involving natural language understanding (NLU). The suggested method is to employ an approach to detect bots at the tweet level by utilizing content and fine-tuning PLMs, to reduce the current threat. Building on the recent developments of the BERT (Bidirectional Encoder Representations from Transformers) and GPT-3, the suggested model employs a text embedding approach. This method offers a high-quality representation that can enhance the efficacy of detection. In addition, a Feedforward Neural Network (FNN) was used on top of the PLMs for final classification. The model was experimentally evaluated using the Twitter bot dataset. The strategy was tested using test data that came from the same distribution as their training set. The methodology in this paper involves preprocessing Twitter data, generating contextual embeddings using PLMs, and designing a classification model that learns to differentiate between human users and bots. Experiments were carried out adopting advanced Language Models to construct an encoding of the tweet to create a potential input vector on top of BERT and their variants. By employing Transformer-based models, we achieve significant improvements in bot detection F1-score (93%) compared to traditional methods such as Word2Vec and Global Vectors for Word Representation (Glove). Accuracy improvements ranging from 3% to 24% compared to baselines were achieved. The capability of GPT-4, an advanced Large Language Model (LLM), in interpreting bot-generated content is examined in this research. Additionally, explainable artificial intelligence (XAI) was utilized alongside transformer-based models for detecting bots on social media, enhancing the transparency and reliability of these models.

The development of abstractive summarization methods is a crucial task in Natural Language Processing (NLP) that presents challenges, which require the creation of intelligent systems that are capable of extracting the main idea from text effectively and generate coherent summary. Numerous existing abstractive approaches do not take into account the importance of the broader context or fail to capture the global semantics in identifying salient content for summary. Moreover, there is lack of studies that extensively evaluated abstractive summarization models for specific domains, such as medical scientific document summarization. With this motivation behind, this paper developed an integrated framework for abstractive summarization of medical scientific documents that integrates topic-aware Heterogeneous Graph Neural Network with a Transformer model. The suggested framework uses Latent Dirichlet Allocation (LDA) for topic modeling to uncover latent topics and global information, thus preserving document-level attributes important for creation of effective summaries. In addition to topic modeling, the framework utilized a Heterogeneous Graph Neural Network (HGNN), capable of capturing the relationship between sentences through graph-based document representation, and allows for the concurrent updating of both local and global information. Finally, the framework is integrated with a Transformer decoder, which greatly enhances the ability of model to produce accurate and informative abstractive summaries. The performance of proposed framework is evaluated on publicly available PubMed dataset related to medical scientific papers. Experimental results illustrate that the suggested framework for abstractive summarization showed superior performance as compared to the state-of-the-art models, achieving high F1-Scores: 46.03 for Rouge-1, 21.42 for Rouge-2, and 39.71 for Rouge-L. Our research makes a significant contribution to the field of natural language processing, particularly in the area of medical scientific document summarization. It demonstrates superior performance and provides a deeper understanding of document structure, and has the potential to impact various applications by offering efficient access to information.

This study aims to explore the application of deep learning models in multi-track music generation to enhance the efficiency and quality of music production. Considering the limited capability of traditional methods in extracting and representing audio features, a multi-track music generation model based on the Bidirectional Encoder Representations from Transformers (BERT) Transformer network is proposed. This model first utilizes the BERT model to encode and represent music data, capturing semantic and emotional information within the music data. Subsequently, the encoded music features are inputted into the Transformer network to learn the temporal relationships and structural patterns among music sequences, thereby generating new multi-track music compositions. The performance of this model is evaluated, revealing that compared to other algorithms, the proposed model achieves an accuracy of 95.98% in music generation prediction, with an improvement in precision by 4.77%. Particularly, the model demonstrates significant advantages in predicting pitch of music tracks. Hence, the multi-track music generation model proposed in this study exhibits excellent performance in accuracy and pitch prediction, offering valuable experimental reference for research and practice in the field of multi-track music generation.

Accurate detection of cyberbullying on Social Networking Sites (SNSs) is crucial for online safety, especially for individuals impacted by it. Cyberbullying language is often implicit, necessitating a comprehensive analysis of conversational context to determine intention and severity. Current studies on cyberbully detection predominantly focus on the main post, overlooking the valuable insights provided by bystander reactions. Moreover, confusion over the differences between cyber-aggression and cyberbullying can undermine the reliability of some of these studies. To ameliorate these issues, this paper addresses the gap in existing research by emphasizing the significance of bystander in fine-grained cyberbullying detection. We specifically investigate the influence of bystander for more precise identification of cyberbullying attacks. Our approach involves fine-tuning a pre-trained language model (BERT) to identify features associated with bystander, categorizing them into roles such as defender, instigator, neutral, and other. To enhance fine-grained cyberbullying detection, we propose a classifier chain that combines BERT’s output with Long Short-Term Memory (LSTM) networks. Our experiments demonstrate that involving bystander roles increases the model’s performance by 25.35%, achieving a final classification F1-score of 89%. Moreover, our approach accurately determines the level of aggression in cyberbullying incidents through considering the interdependency of bystander roles and cyberbullying classes labels, providing an innovative solution to the challenge of cyberbullying misclassification. In summary, our study highlights the significance of incorporating bystanders as a feature in cyberbullying detection and introduces a chained fine-grained model that surpasses conventional approaches, demonstrating promising outcomes in precisely classifying cyberbullying incidents.

The study presents a detailed framework designed to develop a Question-Answering System (QA System) for the Kazakh language, highlighting its importance in the field of Low Resource Languages (LRL) Text Processing. This effort aims to fill the gap in resources for languages that lack substantial digital tools. Specifically, the project focuses on geographical questions about Kazakhstan, aiming to enhance accessibility and understanding of the nation’s geography. The challenges associated with LRL text processing are addressed through the creation of a question-answer corpus, training a Bidirectional Encoder Representations from Transformers (BERT)-based model, and evaluating the system using Bilingual Evaluation Understudy (BLEU) metrics. The endeavor begins with the careful compilation of a corpus containing 50,000 questions, which supports the subsequent development phases and ensures the creation of a robust QA System. In the second phase, a BERT model equipped with 91,821,056 parameters is trained, enhancing the model’s ability to understand the complex linguistic nuances of the Kazakh language. The final phase involves a rigorous evaluation using BLEU metrics, where the system achieves an impressive average score of 0.9576. This score indicates a high level of agreement between the system-generated answers and the reference answers, demonstrating the system’s effectiveness at interpreting and responding to queries about Kazakh geography. This study significantly contributes to the field by providing a systematic and nuanced approach to QA System development and underscores the model’s effectiveness through thorough evaluation and comparative analysis.

With the development of intelligent technology, the application of detection models in various fields becomes more and more important. In this study, a novel detection model (BCOLD) is developed, which is not only suitable for language detection, but also can be widely used in fields such as medical text and image identification.The BCOLD model first utilizes BERT-generated word vectors to capture contextual details, and then combines them with CTM-generated topic vectors to understand the core themes of the text. This fusion strategy enhances the model’s detection capability and understanding of the deeper meaning of the text. The fused vectors are fed into DPCNN and TextCNN models in parallel to capture complex semantic structures and local features, and the feature representation is further optimized by the Multi-Head Attention mechanism. The experimental results show that the BCOLD model performs well in language detection, provides an efficient and accurate solution for automatic detection and classification, and exhibits a wide range of application prospects.

With the widespread use of distance learning, technological developments have also been applied in the field of education. The need for accurate and efficient assessment methods for online exams has become even more apparent, especially with remote learning taking place during the pandemic. For a more efficient evaluation process, we propose a hybrid model of the Automatic Short Answer Grading (ASAG) system based on Bidirectional Encoder Representation of Transformers (BERT). The usage of novel state-of-the-art natural language processing (NLP) techniques in our model enhances the comprehension of text. Specifically, we employ a customized multi-head attention mechanism adapted with BERT, which enables reliable identification of semantic dependencies among words within a sentence and therefore contributes to the effectiveness and trustworthiness of the scoring system. We use a parallel connection of CNN layers in our proposed BERT based ASAG system instead of their serial connection and this usage improves the performance of the system. The proposed model is assessed using common datasets frequently used for ASAG related research projects. In this evaluation process, our model produces much better results compared to other systems available in the literature.

Selecting appropriate reading materials for L2 (second language/foreign language) learners is crucial for improving their proficiency in the target language. However, the limitation of effective Chinese text readability classifiers poses a significant hurdle for students and educators in accurately gauging the precise difficulty level of texts in international Chinese education. This research conducted the readability grading of Chinese as a Second Language (CSL) texts by developing a BERT-Based CSL Readability Classifier (BCRC), which utilizes the BERT architecture specifically trained on CSL texts and incorporates multidimensional linguistic features including lexical richness, syntactic complexity and syntax patterns. The model was evaluated using a dataset of CSL texts, and the results indicate that the BCRC model performs effectively in predicting the readability levels of CSL texts. It achieves high mean accuracy of 92.9% across different readability levels, which outperforms baseline classifiers in terms of classification performance, highlighting the enhancement capabilities of multidimensional linguistic features in CSL readability classification models. This study contributes to the field of CSL education by providing a robust readability classifier as a valuable tool for educators, curriculum designers, and developers of CSL learning materials to ensure appropriate text selection based on learners’ proficiency levels.

Issues are evolving requirements in software engineering. They are the main factors that increase the cost of software evolution. To help developers manage issues, GitHub provides issue labeling mechanisms in issue management systems. However, manually labeling issue reports still requires considerable developer workload. To ease developers’ burden, researchers have proposed automatically classifying issue reports. To improve the classification accuracy, researchers adopted deep learning techniques and pretrained models. However, pretrained models in the general domain such as RoBERTa have limitations in understanding the contexts of software engineering tasks. In this paper, we create a pretrained model, IssueBERT, with issue data to understand whether a domain-specific pretrained model could improve the accuracy of issue report classification. We also adopt and explore several pretrained models in the software engineering domain, namely, CodeBERT, BERTOverflow, and seBERT. We conduct a comparative experiment on these pretrained models to evaluate their performance in classifying issue reports. Our comparison results show that IssueBERT outperforms the other pretrained models. Noticeably, IssueBERT yields an average F1 score that is 1.74% higher than that of seBERT and 3.61% higher than that of RoBERTa, even though IssueBERT was pretrained with much less data than seBERT and RoBERTa.

This study addresses the challenge of improving the downstream performance of pretrained language models for morphologically rich languages, with a focus on Turkish. Traditional BERT models use one-dimensional absolute positional embeddings, which, while effective, have limitations when dealing with complex languages. We propose BERT2D, which is a novel BERT-based model that contributes to positional embedding systems. This approach introduces a dual embedding system that targets all the words and their subwords. Remarkably, this modification, coupled with whole word masking, resulted in a significant increase in performance despite a negligible increase in the parameters. Our experiments showed that BERT2D consistently outperformed the leading Turkish-focused BERT model, BERTurk, in terms of various performance metrics in text classification, token classification, and question-answering downstream tasks. For a fair comparison, we pretrained our BERT2D language model on the same dataset as that of BERTurk. The results demonstrate that two-dimensional positional embeddings can significantly improve the performance of encoder-only models in Turkish and other morphologically rich languages, suggesting a promising direction for future research in natural language processing.

The current renaissance in the development of Arabic Pre-trained Language models (APLMs) has yielded significant advancement across many fields. Nevertheless, no study has explored the dimensions of gender bias in these models. It is argued that the bias is influenced by the resources used during the models’ pre-training process. Thus, in this study, we conducted a comprehensive analysis to qualitatively assess the representation of different genders by tracing the bias signals from the training corpus. Through applying several Natural Language Processing (NLP) techniques, including Named Entity Recognition (NER), Part of Speech Tagging (POS), and Dependency Parsing (DP), the results indicated an imbalanced corpus in terms of gender nouns and reveal verbs’ patterns associated with each gender. The second phase of this study aimed to examine the impact of the results that emerged from the corpus analysis on the recent APLMs. Leveraging Bidirectional Encoder Representations (BERT)’s ability to predict the missing tokens in quantifying gender bias, we introduce the first template-based Arabic benchmark designed to measure gender bias across various disciplines. Utilizing this benchmark, along with the list of gender-specific nouns and personal names extracted from the corpus, we evaluated the gender skew in the context of scientific and liberal arts disciplines across six APLMs. These models included: AraBERT, CAMeLBERT-CA, CAMeLBERT-MSA, GigaBERT, MAR-BERT, and ARBERT. The outcomes revealed a higher bias skew toward personal names, indicating that the presence of gender associations in the training corpus reinforced gender bias in APLMs.

Radiology reports are a valuable source of textual information used to improve clinical care and support research. In recent years, deep learning techniques have been shown to be effective in classifying radiology reports. This article investigates the use of deep learning techniques with attention mechanisms to achieve better performance in the classification of radiology reports. We focus on various Natural Language Processing approaches, such as LSTM with Attention, BERT, and GPT-4, evaluated on a chest tomography report dataset regarding neoplastic diseases collected from an Italian hospital. In particular, we compare the results with a previous machine learning system, showing that models based on attention mechanisms can achieve higher performance. The Attention Mechanism allows us to identify the most relevant bits of text used by the model to make its predictions. We show that our model achieves state-of-the-art results on the hierarchical classification of radiology reports. Moreover, we evaluate the performance of GPT-4 on the classification of these reports in a zero-shot setup through prompt engineering, showing interesting results even with a small context and a non-English language. Our findings suggest that deep learning techniques with attention mechanisms may be successful in the classification of radiology reports even in non-English languages for which it is not possible to leverage on large text corpus.

The digitization of the healthcare industry has led to a growing number of applications that use machine learning and image processing techniques to improve the diagnostic process. These applications utilize a variety of medical data, including laboratory results, clinical findings, MRI scans, tomographic images, and radiological images. In addition, free-text healthcare documentation, such as well-structured discharge summaries, contains valuable information. Natural Language Processing encompasses the development of automated systems for generating health reports. This process involves using domain-specific knowledge and prior knowledge to extract relevant information from medical records. This article investigates the use of natural language processing techniques for chest X-ray classification. A total of 14 distinct impressions derived from chest radiography findings from the MIMIC-CXR dataset were used in a multi-label classification procedure. Six distinct language models derived from the BERT language model, along with three distinct classification algorithms, were employed to evaluate the effectiveness of the models and the dataset for multi-label categorization. The experimental results showed a successful prediction rate of 80.47% for 14 distinct impressions within the dataset.

Software vulnerabilities are among the significant causes of security breaches. Vulnerabilities can severely compromise software security if exploited by malicious attacks and may result in catastrophic losses. Hence, Automatic vulnerability detection methods promise to mitigate attack risks and safeguard software security. This paper introduces a novel model for automatic vulnerability detection of source code vulnerabilities dubbed DB-CBIL using a hybrid deep learning model based on Distilled Bidirectional Encoder Representations from Transformers (DistilBERT). The proposed model considers contextualized word embeddings using the language model for the syntax and semantics of source code functions based on the Abstract Syntax Tree (AST) representation. The model includes two main phases. First, using a vulnerable code dataset, the pre-trained DistilBert transformer is fine-tuned for word embedding. Second, a hybrid deep learning model detects which code functions are vulnerable. The hybrid model is built on two Deep Neural Networks (DNN). The first model is the Convolutional Neural Network (CNN), which is used for extracting features. The second model is Bidirectional-LSTM (BiLSTM), which has been used to maintain the sequential order of the data as it can handle lengthy token sequences. The utilized source code dataset is derived from the Software Assurance Reference Database (SARD) benchmark dataset. Final experimental findings show that the proposed model outperforms the state-of-the-art approaches’ performance by improving precision, recall, F1-score, and False Negative Rate (FNR) by 2.41%-8.95%, 4.0%-16.28%, 1.85%-12.74%, and 18% respectively. The proposed model reports the lowest FNR in the literature, a significant achievement given the cost-based nature of vulnerability detectors.

Extended use of social media, leads to personality detection from online content shared by users. While it has numerous applications in different areas such as recommendation systems, most of existing studies focus on superficial, statistical, and explicit user contents, ignoring the knowledge hidden in semantic features. In this study, we proposed a method to explore psycholinguistic knowledge hidden in deep levels of user data for the task of personality prediction. We proposed a method utilizing a fine-tuned domain-specific BERT model to extract features from a sentence and enrich the outputs by leveraging emotional information to highlight the important words. Furthermore, by conducting a double-way-attention mechanism we reflected the information from highlighted words into the whole knowledge extracted from inputs. Then, created a graph by considering extracted embeddings from the last step as node features and developing a dynamic and task-related learning approach to specify edges to connect different pairs of nodes based on a neural network, and leveraged graph attention network to predict personality traits. Finally, experimental results confirmed the effectiveness of our proposed model with 80.63% accuracy, compared to other state-of-the-art studies for the essays dataset. Also, several ablations are conducted to illustrate and verify the impact of different sections and parameters of the proposed architecture.

Sentiment analysis is essential for comprehending public opinion, particularly when considering e-commerce and the expansion of online businesses. Early approaches treated sentiment analysis as a document or sentence-level classification problem, lacking the ability to capture nuanced opinions about specific aspects. This limitation was addressed by the development of aspect-based sentiment analysis (ABSA), which links sentiment to specific aspects that are mentioned explicitly or implicitly in the review. ABSA is relatively a recent field of sentiment analysis and the existing models for ABSA face three main challenges, including domain-specificity, reliance on labeled data, and a lack of exploration into the potential of newer large language models (LLMs) such as GPT, PaLM, and T5. Leveraging a diverse set of datasets, including DOTSA, MAMS, and SemEval16, we evaluate the performance of prominent models such as ATAE-LSTM, flan-t5-large-absa, DeBERTa, PaLM, and GPT-3.5-Turbo. Our findings reveal nuanced strengths and weaknesses of these models across different domains, with DeBERTa emerging as consistently high-performing and PaLM demonstrating remarkable competitiveness for aspect term sentiment analysis (ATSA) tasks. In addition, the PaLM demonstrates competitive performance for all the domains that were used in the experiments including the restaurant, hotel, books, clothing, and laptop reviews. Notably, the analysis underscores the models’ domain sensitivity, shedding light on their varying efficacy for both ATSA and ACSA tasks. These insights contribute to a deeper understanding of model applicability and highlight potential areas for improvement in ABSA research and development.

Nowadays global market products are readily accessible worldwide, and a vast array of reviews across numerous platforms are posted daily in several categories, making it challenging for customers to stay informed about their product interests. To make informed decisions regarding product quality, users require access to reviews and ratings. Owners and managers must analyze customer ratings and the underlying emotional content of reviews to enhance the product’s quality, cost, customer service, and environmental impact. The primary aim of our proposed research is to accurately predict product helpfulness through customer reviews using the Large Language Model (LLM), thereby assisting customers in saving time and money. We employed a benchmark dataset, the Amazon Fine Food Reviews, to develop numerous advanced machine-learning techniques. We introduced a novel transformer approach BERF (BERT Random Forest) for feature engineering to enhance the value of user evaluations for Amazon’s gourmet food products. The BERF method utilizes BERT embeddings and class probability features derived from product helpfulness online reviews textual data. We have balanced the dataset using the Synthetic Minority Over-sampling TEchnique (SMOTE) approach. Our comprehensive study results demonstrated that the Light Gradient Boosting Machine (LGBM) strategy outperformed existing state-of-the-art approaches, achieving an accuracy of 98%. The performance of each method is confirmed using a k-fold method and further improved through hyperparameter optimization. Our innovative study employing a transformer model has significantly enhanced the utility of customer reviews, substantially reducing online product scams and preventing wasted time and money.

Depression constitutes a significant mental health condition, impacting an individual’s emotional state, thought processes, and ability to carry out everyday tasks. Depression is defined by ongoing feelings of sadness, diminished interest in previously enjoyed activities, alterations in hunger, sleep disturbances, decreased vitality, and challenges with focus. The impact of depression extends beyond the individual, affecting society at large through decreased productivity and higher healthcare costs. In the realm of social media, users often express their thoughts and emotions through posts, which can provide insightful data for identifying patterns of depression. This research aims to detect depression early by analyzing social media user content with machine learning techniques. We have built advanced machine learning models using a benchmark depression database containing 20,000 tagged tweets from user profiles identified as depressed or non-depressed. We are introducing an innovative BERT-RF feature engineering method that extracts Contextualized Embeddings and Probabilistic Features from textual input. The Bidirectional Encoder Representations from Transformers (BERT) model, based on the Transformer architecture, is used to extract Contextualized Embedding features. These features are then fed into a random forest model to generate class probabilistic features. These prominent features aid in enhancing the identification of depression from social media. In order to classify tweets using the features derived from the BERT-RF features selection step, we have used five popular classifiers: Random Forest (RF), Multilayer Perceptron (MLP), K-Neighbors Classifier (KNC), Logistic Regression (LR), and Long Short-Term Memory (LSTM). Evaluation experiments show that our approach, using BERT-RF for feature engineering, enables the Logistic Regression model to outperform state-of-the-art methods with a high accuracy score of 99%. We have validated the results through k-fold cross-validation and statistical T-tests. We achieved 99% k-fold accuracy during the validation of the proposed approach. This research contributes significantly to computational linguistics and mental health analytics by providing a robust approach to the early detection of user depression from social media content.

In the Australian construction industry, effective supply chain risk management (SCRM) is critical due to its complex networks and susceptibility to various risks. This study explores the application of transformer models like BERT, RoBERTa, DistilBERT, ALBERT, and ELECTRA for Named Entity Recognition (NER) in this context. Utilizing these models, we analyzed news articles to identify and classify entities related to supply chain risks, providing insights into the vulnerabilities within this sector. Among the evaluated models, RoBERTa achieved the highest average F1 score of 0.8580, demonstrating its superior balance in precision and recall for NER in the Australian construction supply chain context. Our findings highlight the potential of NLP-driven solutions to revolutionize SCRM, particularly in geo-specific settings.

Deep learning models have demonstrated their effectiveness in capturing complex relationships between input features and target outputs across many different application domains. These models, however, often come with considerable memory and computational demands, posing challenges for deployment on resource-constrained edge devices. Knowledge distillation is a prominent technique for transferring the expertise from an advanced yet heavy teacher model to a more efficient leaner student model. As ensemble methods have exhibited notable enhancements in model generalization and have achieved state-of-the-art performance in various machine learning tasks, we adopt ensemble techniques to perform knowledge distillation from BERT using multiple lightweight student models. Our approach applies lean architectural paradigms of spatial and sequential networks including LSTM, CNN and their fusion to perform data processing from distinct perspectives. Instead of using contextual word representations which require more space in natural language processing applications, we take advantage of a single static pre-trained and low-dimensional word embedding space to be shared among student models. Empirical studies are conducted on the sentiment classification problem and our model outperforms not only other existing techniques but also the teacher model.

Security attacks are becoming more sophisticated and common as connected devices rapidly exchange personal, sensitive, and important data. Security solutions are therefore required for Internet of Things (IoT) environments. System administrators receive alerts through an automatic Network Intrusion Detection (NID) system when security breaches occur. An automatic NID can be an effective tool to protect IoT networks against various attacks. It is possible to detect intrusions using a variety of intrusion detection techniques, but the performance and class imbalance in the dataset make this a difficult process. To improve detection rates and decrease false alarms, intrusion detection accuracy must be improved. In this paper, an automatic NID system is proposed leveraging a renowned machine learning model named Random Forest (RF) on the (UNSW-NB15) dataset collected from Kaggle. The experimental results indicate that the proposed model not only has higher accuracy at 90.17% surpassing the baseline approach by 7.34%, but also has precision, recall, and F1 scores up to 90.14%, 90.17, and 90.14%, respectively. Moreover, 98.83% accuracy is achieved with a balanced class dataset by using random resampling techniques to generate synthetic data of minority attacks.

The field of Natural Language Processing (NLP) is currently undergoing a revolutionary transformation driven by the power of pre-trained Large Language Models (LLMs) based on groundbreaking Transformer architectures. As the frequency and diversity of cybersecurity attacks continue to rise, the importance of incident detection has significantly increased. IoT devices are expanding rapidly, resulting in a growing need for efficient techniques to autonomously identify network-based attacks in IoT networks with both high precision and minimal computational requirements. This paper presents SecurityBERT, a novel architecture that leverages the Bidirectional Encoder Representations from Transformers (BERT) model for cyber threat detection in IoT networks. During the training of SecurityBERT, we incorporated a novel privacy-preserving encoding technique called Privacy-Preserving Fixed-Length Encoding (PPFLE). We effectively represented network traffic data in a structured format by combining PPFLE with the Byte-level Byte-Pair Encoder (BBPE) Tokenizer. Our research demonstrates that SecurityBERT outperforms traditional Machine Learning (ML) and Deep Learning (DL) methods, such as Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), in cyber threat detection. Employing the Edge-IIoTset cybersecurity dataset, our experimental analysis shows that SecurityBERT achieved an impressive 98.2% overall accuracy in identifying fourteen distinct attack types, surpassing previous records set by hybrid solutions such as GAN-Transformer-based architectures and CNN-LSTM models. With an inference time of less than 0.15 seconds on an average CPU and a compact model size of just 16.7MB, SecurityBERT is ideally suited for real-life traffic analysis and a suitable choice for deployment on resource-constrained IoT devices.

Software Defect Prediction (SDP) reduces the overall cost of software development by identifying the code at a higher risk of defects at the initial phase of software development. SDP helps the test engineers to optimize the allocation of testing resources more effectively. Traditional SDP models are built using handcrafted software metrics that ignore the structural, semantic, and contextual information of the code. Consequently, many researchers have employed deep learning models to capture contextual, semantic, and structural information from the code. In this article, we propose the DP-CCL (Defect Prediction using CodeBERT with Contrastive Learning) model to predict the defective code. The proposed model employs supervised contrastive learning using this CodeBERT Language model to capture semantic features from the source code. Contrastive learning extracts valuable information from the data by maximizing the similarity between similar data pairs (positive pair) and meanwhile minimizing the similarity between dissimilar data pairs (negative pair). Moreover, The model combines the semantic features with software metrics to obtain the benefits of both semantic and handcrafted features. The combined features are input to the logistic regression model for code classification as either buggy or clean. In this study, ten PROMISE projects were utilized to conduct the experiments. Results show that the DP-CCL model achieved significant improvement i.e., 4.9% to 14.9% increase in F-Score as compared to existing approaches.

As the number of people using social networks increases, more people are using social media platforms to meet their news needs. Users think that it is easier to follow the agenda by accessing news, especially on Twitter, rather than newspaper news pages. However, fake news is increasingly appearing on social media, and it is not always possible for people to obtain correct news from partial news pages or short Twitter posts. Understanding whether the news shared on Twitter is true or not is an important problem. Detecting fake tweets is of great importance in Turkish as well as in any language. In this study, fake news obtained from verification platforms on Twitter and real news obtained from the Twitter accounts of mainstream newspapers were labeled and, preprocessed using the Zemberek natural language processing tool developed for the Turkish language, and a dataset named TR_FaRe_News was created. Then, the TR_FaRe_News dataset was explored using ensemble methods and BoW, TF-IDF, and Word2Vec vectorization methods for fake news detection. Then a pre-trained BERT deep learning model was fine-tuned, and variations of the model extended with Bi-LSTM and Convolutional Neural Network (CNN) layers with the frozen and unfrozen parameters methods were explored. The performance evaluation was conducted using seven comparable datasets, namely BuzzFeedNews, GossipCop, ISOT, LIAR, Twitter15, and Twitter16, including an LLM-generated fake news dataset. Analyzing Turkish tweets and using fake news datasets generated by LLM is considered an important contribution. Accuracy values between 90 and 94% were obtained with the BERT and BERTurk + CNN models with 94% accuracy.

Identifying compiler toolchain provenance serves as a basis for both benign and malicious binary analyses. A wealth of prior studies mostly focuses on the inference of a popular compiler toolchain for C and C++ languages from stripped binaries that are built with GCC or clang. Lately, the popularity of an emerging compiler is on the rise such as Rust, Go, and Nim programming languages that complement the downsides of C and C++ (e.g., security), which little has been explored on them. The main challenge arises when applying previous inference techniques for toolchain provenance because some emerging compilation toolchains adopt the same backend of traditional compilers. In this paper, we propose ToolPhet, an effective end-to-end BERT-based system for deducing the provenance of both traditional and emerging compiler toolchains. To this end, we thoroughly study the characteristics of both an emerging toolchain and an executable binary that is generated by that toolchain. We introduce two separate downstream tasks for the compiler toolchain inference with a (BERT-based) fine-tuning process, which produces 1) a toolchain classification model; and 2) a binary code similarity detection model. Our findings show that the classification model 1) may not suffice when producing a binary with the existing backend like Nim, which we adopt the detection model 2) that can infer underlying code semantics. We evaluate ToolPhet with the previous work including one signature-based tool and four machine-learning-based approaches, demonstrating its effectiveness by achieving higher F1 score s with the binaries compiled with emerging compilation toolchains.

In this study, a steganography method based on BERT transformer model is proposed for hiding text data in cover text. The aim is to hide information by replacing specific words within the text using BERT’s masked language modeling (MLM) feature. In this study, two models, fine-tuned for English and Turkish, are utilized to perform steganography on texts belonging to these languages. Furthermore, the proposed method can work with any transformer model that supports masked language modeling. While traditionally the hidden information in text is often limited, the proposed method allows for a significant amount of data to be hidden in the text without distorting its meaning. In this study, the proposed method is tested by hiding stego texts of varying lengths in cover text of different lengths in two different language scenarios. The test results are analyzed in terms of perplexity, KL divergence and semantic similarity. Upon examining the results, the proposed method has achieved the best results compared to other methods found in the literature, with KL divergence of 7.93 and semantic similarity of 0.99. It can be observed that the proposed method has low detectability and demonstrates success in the data hiding process.

Recommendation systems play a pivotal role in delivering user preference information. However, they often face the challenge of information cocoons due to repeated content delivery, particularly prevalent in stock recommendations that are susceptible to investor sentiment. In response to the information cocoons, we propose the Sentiment and Price Combined Model (SPCM), which leverages sentiment features and price factors to predict stock price movements. This novel framework combines collective sentiment analysis with state-of-the-art BERT transformer models and advanced machine learning techniques. Over a three-year period, we collected 40 million stock comments from the Guba platform, extracting investor sentiment conveyed in text information and investigating the impact of metrics such as homophily on stock recommendations. Experimental results indicate that both the volume of posts and the agreement index affect the effectiveness of investor sentiment, while homophily reduces the accuracy of participants’ stock price judgments. The recognition accuracy of the BERT-based sentiment analysis model reaches an impressive 84.12%, and the portfolio constructed by SPCM yields a cumulative return four times that of the industry benchmark. Furthermore, homogeneous quantitative metrics also enhance diversification in stock selection.

Large language models provide high-accuracy solutions in many natural language processing tasks. In particular, they are used as word embeddings in sentiment analysis models. However, these models pick up on and amplify biases and social stereotypes in the data. Causality theory has recently driven the development of effective algorithms to evaluate and mitigate these biases. Causal mediation was used to detect biases, while counterfactual training was proposed to mitigate bias. In both cases, counterfactual sentences are created by changing an attribute, such as the gender of a noun, for which no change in the model output is expected. Biases are detected and eventually corrected each time the model behavior differs between the original and the counterfactual sentence. We propose a new method for de-biasing sentiment analysis models that leverages the causal mediation analysis to identify the parts of the model primarily responsible for the bias and apply targeted counterfactual training for model de-biasing. We validated the methodology by fine-tuning the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model for sentiment prediction. We trained two sentiment analysis models using the Stanford Sentiment Treebank dataset and the Amazon Product Reviews, respectively, and we evaluated the fairness and prediction performances using the Equity Evaluation Corpus. We illustrated the causal patterns in the network and showed that our method achieves both high fairness and more accurate sentiment analysis than the state-of-the-art approach. Contrary to state-of-the-art models, we achieved a noticeable improvement in gender fairness without hindering sentiment prediction accuracy.

Sustainability has become the greatest aspiration on the planet, as every organization has begun reorienting itself to become one. To accomplish this, organizations must strike a balance between their Environmental, Social, and Governance (ESG) policies. It is essential to examine the organizations that currently exist. For this reason, the study conducts a thematic analysis of the sustainability reports of Forbes’ India’s top companies in 2021 in order to determine their current ESG focus using BERT tokenization and YAKE technique along with MLM and NSP methods of natural language processing (NLP). The experiment results in prediction with 98% accuracy. The study addresses the question, “What must organizations do to embark on the path towards sustainability?”

Text, visual, and acoustic are usually complementary in the Multimodal Sentiment Analysis (MSA) task. However, current methods primarily concern shared representations while neglecting the critical private aspects of data within individual modalities. In this work, we propose shared-private memory networks based on the recent advances in the attention mechanism, called SPMN, to decouple multimodal representation from shared and private perspectives. It contains three components: a) a shared memory to learn the shared representations of multimodal data; b) three private memories to learn the private representations of individual modalities, respectively; c) and adaptive fusion gates to fuse multimodal private and shared representations. To evaluate the effectiveness of SPMN, we integrate it into different pre-trained language representation models, such as BERT and XLNET, and conduct experiments on two public datasets, CMU-MOSI and CMU-MOSEI. Experimental results indicate that the performances of pre-trained language representation models are significantly improved because of SPMN and demonstrate the superiority of our model compared to the state-of-the-art methods. SPMN's source code is publicly available at: https://github.com/xiaobaicaihhh/SPMN. [ABSTRACT FROM AUTHOR]

This article presents a novel log abstraction framework based on neural open information extraction (OpenIE) and dynamic word embedding principles. Though various log parsing frameworks are proposed in the literature, the existing frameworks are modeled on predefined heuristics or auto-regressive methodologies that work well in offline scenarios. However, these frameworks are less suitable for dynamic self-adaptive systems, such as the Internet of Things (IoT), where the log outputs have diverse contextual variations and disparate time irregularities. Therefore, it is essential to move away from these traditional approaches and develop a systematic model that can effectively analyze log outputs in real-time and increase the system up-time of IoT networks so that they are almost always available. To address these needs, the proposed framework used OpenIE along with term frequency/inverse document frequency (TF/IDF) vectorization for constructing a set of relational triples (aka triple-sets). Additionally, a dynamic pretrained encoder–decoder architecture is utilized to imbibe the positional and contextualized information in its resultant outputs. The adopted methodology has enabled the proposed framework to extract richer word representations with dynamic contextualization of time-sensitive event logs to enhance further downstream activities, such as failure prediction and prognostic analysis of IoT networks. The proposed framework is evaluated on the system event log traces accumulated from a long range wide-area network (LoRaWAN) IoT gateway to proactively determine the probable causes of its various failure scenarios. Additionally, the study also provided a comparative analysis of its mathematical representations with that of the current state-of-the-art (SOTA) approaches to project the advantages and benefits of the proposed model, particularly from its data analytics standpoint.

Responding to the challenge of efficiently leveraging Open-source Threat Intelligence Feeds (OTIFs) to enhance organizational security, this paper presents an innovative approach to automated threat identification using machine learning. Current machine learning models require large volumes of high-quality annotated data, which is a major obstacle given the inherent scarcity of such data in OTIFs. Bridging this research gap, we propose a novel semi-supervised learning strategy that capitalizes on both labeled and unlabeled data to automate threat identification. Our unique contribution is an advanced iteration of the GAN-BERT framework, which incorporates a self-supervised contrastive objective function to fine-tune the BERT language model. Our experiments show that this method outperforms the original BERT and GAN-BERT, achieving a 3-12% F1-score improvement on various OTIFs datasets. Furthermore, we present an efficient method for selecting hard negatives during training, which further enhances our model’s performance. This innovative approach significantly advances the field of automated threat detection, reducing reliance on human supervision and effectively addressing the issue of limited annotated data. Thus, it offers a robust solution for strengthening security posture through proactive decision making in Cyber Threat Intelligence.

The unstructured-textual crash descriptions recorded by police officers is rarely utilized, despite containing detailed information on traffic situations. This lack of utilization is mainly due to the difficulty in analyzing text data, as there is currently no innovative methodology for extracting meaningful information from it. Given limitations and challenges in analyzing traffic crash descriptions, this study developed a methodology to classify significant words in unstructured data that describe traffic crash scenarios into standardized data. Ultimately, a natural language processing technique, specifically a bidirectional encoder representation from transformer (BERT), was used to extract meaningful information from crash descriptions. This BERT-based model effectively extracts information on the exact collision point and the pre-crash vehicle maneuver from crash descriptions. Its practical approach allows for the interpretation of traffic crash descriptions and outperforms other natural language processing models. Importantly, this method of extracting crash scene information from traffic crash descriptions can aid in better comprehending the unique characteristics of traffic crashes. This comprehension can ultimately aid in the development of appropriate countermeasures, leading to the prevention of future traffic crashes.

Reviews left by consumers on e-commerce platforms provide crucial marketing information as they are a publicly available source of information providing insight into consumers’ thoughts and opinions. However, it is physically impossible to read hundreds of reviews per product. Therefore, algorithmically extracting the necessary insights from a large volume of review data can provide more advanced information while reducing time and costs. This study aims to automate the process of extracting related products, pros and cons of products, and trend forecasting from review data using clustering algorithms. The review dataset for pros and cons of products and related products was constructed by selecting 17 products on the Naver Shopping platform and crawling them and the product keyword and search volume dataset for trend forecasting was constructed by crawling the data from Itemscout and Naver datalab platform. Various clustering-based algorithms such as Deep Clustering Network, BERTopic, and a Transformer-based forecasting model were used to conduct the research. It is expected that this will allow for a more accurate understanding of consumer thoughts, which can be utilized in marketing for various products and services.