Search

Over the past decade, focused ultrasound (FUS) has emerged as a promising therapeutic modality for various medical conditions. However, the exponential growth in the published literature on FUS therapies has made the literature review process increasingly time-consuming, inefficient, and error-prone. Machine learning approaches offer a promising solution to address these challenges. Therefore, the purpose of our study is to (1) explore and compare machine learning techniques for the text classification of scientific abstracts, and (2) integrate these machine learning techniques into the conventional literature review process. A classified dataset of 3588 scientific abstracts related and unrelated to FUS therapies sourced from the PubMed database was used to train various traditional machine learning and deep learning models. The fine-tuned Bio-ClinicalBERT (Bidirectional Encoder Representations from Transformers) model, which we named FusBERT, had comparatively optimal performance metrics with an accuracy of 0.91, a precision of 0.85, a recall of 0.99, and an F1 of 0.91. FusBERT was then successfully integrated into the literature review process. Ultimately, the integration of this model into the literature review pipeline will reduce the number of irrelevant manuscripts that the clinical team must screen, facilitating efficient access to emerging findings in the field.

One of the most underdiagnosed medical conditions worldwide is depression. It has been demonstrated that the current classical procedures for early detection of depression are insufficient, which emphasizes the importance of seeking a more efficient approach to overcome this challenge. One of the most promising opportunities is arising in the field of Artificial Intelligence as AI-based models could have the capacity to offer a fast, widely accessible, unbiased and efficient method to address this problem. In this paper, we compared three natural language processing models, namely, BERT, GPT-3.5 and GPT-4 on three different datasets. Our findings show that different levels of efficacy are shown by fine-tuned BERT, GPT-3.5, and GPT-4 in identifying depression from textual data. By comparing the models on the metrics such as accuracy, precision, and recall, our results have shown that GPT-4 outperforms both BERT and GPT-3.5 models, even without previous fine-tuning, showcasing its enormous potential to be utilized for automated depression detection on textual data. In the paper, we present newly introduced datasets, fine-tuning and model testing processes, while also addressing limitations and discussing further considerations for future research.

Bidirectional Encoder Representations from Transformers (BERT), a revolutionary model in natural language processing (NLP), has significantly impacted text-related tasks including text classification. Several BERT models were developed for Arabic language. While many studies have compared their overall performance for text classification tasks, none has dug deeper and investigated the relationship between their pretraining data and their performance. This study investigates this relationship by utilizing ten models and evaluating them on eight diverse classification tasks using metrics such as accuracy and F1 score. Results revealed variations in performance across tasks which was mainly due to the models pretraining corpora. The study emphasizes the impact of pretraining data size, quality, and diversity on model adaptability. Models pretrained on specific corpora, despite larger sizes, may not outperform those pretrained on more diverse datasets. Notably, domain-specific tasks, such as medical and poetry classification, unveiled performance gaps compared to the original English BERT. The findings suggest the necessity of reevaluating the pretraining approach for Arabic BERT models. Balancing quantity and quality in pretraining corpora, spanning various domains, is identified as crucial. The study provides insights into optimizing pretraining strategies for enhanced performance and adaptability of Arabic BERT models in diverse text classification tasks, offering valuable guidance for researchers and practitioners in the field of NLP. [ABSTRACT FROM AUTHOR]

The increasing volume of electronic text, especially in the biomedical domain, requires automatic text summarization (ATS) to help researchers navigate and find relevant information. This study proposes an unsupervised extractive ATS method to improve the quality of biomedical document summaries by focusing on subtopic diversity. The method integrates topic modeling and clustering with bidirectional encoder representation from transformers (BERT). To evaluate the effectiveness of the proposed study, it tested on a new corpus of 200 biomedical research papers from Biomed Central. The results were evaluated using the ROUGE metric and qualitative evaluation by medical experts. The ROUGE metric yielded scores of 0.4838 (Rouge-1), 0.2174 (Rouge-2), and 0.2206 (Rouge-L), while the qualitative evaluation achieved an average score of 4.10, 4.06, 3.55, 4.0, and 4.0 for completeness, relevance, conciseness, informativity, and readability, respectively. The results demonstrate the effectiveness of the proposed method in summarizing long medical documents. [ABSTRACT FROM AUTHOR]

Pre-trained language models (LMs) have shown remarkable success in diverse tasks and domains. An LM trained on the document of a specific area (e.g., biomedicine, education, and finance) provides expert-level knowledge about that domain, and there have been many efforts to develop such domain-specific LMs. Despite its potential benefits, however, developing LM in the technology commercialization (TC) domain has not been investigated. In this study, we build a TC-specialized large LM pre-trained on the Korean TC corpus. Firstly, we collect a large-scale dataset containing 199,857,586 general Korean sentences and 17,562,751 TC-related Korean sentences. Second, based on this large dataset, we pre-train a Transformer-based language model resulting in TC-BERT. Third, we validate TC-BERT on three practical applications: document classification, keyword extraction, and recommender system. For this, we devise a new keyword extraction algorithm and propose a document recommender algorithm based on TC-BERT’s document embedding. Through various quantitative and qualitative experiments, we comprehensively verify TC-BERT’s effectiveness and its application. [ABSTRACT FROM AUTHOR]

This research presents a novel approach to multilabel Thai text classification within the banking sector using a hybrid Transformer-RNN model. The study focuses on enhancing the accuracy and efficiency of document categorization in Thai language, addressing the complexities inherent to diverse banking documents. The model integrates BERT for pre-training to leverage contextual embeddings and incorporates an RNN to capture sequential dependencies in the text data. Evaluation of the model's performance was conducted using precision, recall, and F1 score metrics over a 10-fold cross-validation setup. The hybrid Transformer-RNN model demonstrated robust performance across multiple evaluation metrics. Specifically, it achieved an average precision of 0.823, recall of 0.817, and F1 score of 0.815. These results indicate the model's efficacy in accurately predicting multiple labels associated with various types of Thai banking documents. Comparative analysis against LSTM, CNN, BERT, and Transformer (Encoder-Only) models further validates the superiority of the proposed approach in handling complex multilabel classification tasks in the Thai banking domain. This research underscores the potential of hybrid Transformer-RNN models in advancing multilabel text classification capabilities, particularly in specialized domains like Thai banking. The findings highlight significant improvements in classification accuracy and model robustness, contributing to enhanced document management, regulatory compliance, and customer service within the banking industry. Future research directions could explore ensemble learning techniques, domain adaptation strategies, and the integration of domain-specific knowledge bases to further enhance the model's performance and applicability in real-world scenarios. [ABSTRACT FROM AUTHOR]

Sentiment analysis (SA) as a research field has gained popularity among the researcher throughout the globe over the past 10 years. Deep neural networks (DNN) and word vector models are employed nowadays and perform well in sentiment analysis. Among the different deep neural networks utilized for SA globally, Bi-directional long short-term memory (Bi-LSTM), BERT, and CNN models have received much attention. Even though these models can process a wide range of text types, Because DNNs treat different features the same, using these models in the feature learning phase of a DNN model leads to the creation of a feature space with very high dimensionality. We suggest an attention-based, stacked, two-layer CNN-Bi-LSTM DNN to overcome these glitches. After local feature extraction, by applying stacked two-layer Bi-LSTM, our proposed model extracted coming and outgoing sequences by seeing sequential data streams in backward and forward directions. The output of the stacked two-layer Bi-LSTM is supplied to the attention layer to assign various words with varying values. A second Bi-LSTM layer is constructed atop the initial layer in the suggested network to increase performance. Various experiments have been conducted to evaluate the effectiveness of our proposed model on two Urdu sentiment analysis datasets named as UCSA-21 and UCSA, and an accuracies of 83.12% and 78.91% achieved, respectively. [ABSTRACT FROM AUTHOR]

The efficient use of text data is very important in investor sentiment research and other fields. Through the sentiment classification of text data containing investor sentiment, we can effectively and accurately identify the sentiment contained in the text. This paper takes the futures market forecast text published by 21 futures companies as the data source and constructs a sentiment classification model of the market forecast text based on BERT (Bidirectional Encoder Representations from Transformers) according to the characteristics of the market forecast text. The sentiment classification of the market forecast text is carried out by using the sentiment classification model of the market forecast text based on BERT and a classification model based on the classical classification algorithm. The classification effects of different models are compared. The results show that the optimized BERT model has the best classification effect. This enriches the research methods of investor sentiment measurement in the financial field and improves the accuracy of this kind of sentiment measurement result. [ABSTRACT FROM AUTHOR]

Focusing on a polarized issue—U.S. gun violence—this study examines agenda setting as an antecedent of political expression on social media. A state-of-the-art machine-learning model was used to analyze news coverage from 25 media outlets—mainstream and partisan. Those results were paired with a two-wave panel survey conducted during the 2018 U.S. midterm elections. Findings show mainstream media shape public opinion about gun violence, which then stimulates expression about the issue on social media. The study also reveals that partisan media's gun violence coverage has significant cross-cutting effects. Notably, exposure to conservative media will decrease public salience of gun violence, pivot opinion in a more conservative direction, and discourage social media expression; and all of these effects are stronger among liberals. [ABSTRACT FROM AUTHOR]

The study of bias in language models is a growing area of work, however, both research and resources are focused on English. In this paper, we make a first approach focusing on gender bias in some freely available Spanish language models trained using popular deep neural networks, like BERT or RoBERTa. Some of these models are known for achieving state-of-the-art results on downstream tasks. These promising results have promoted such models' integration in many real-world applications and production environments, which could be detrimental to people affected for those systems. This work proposes an evaluation framework to identify gender bias in masked language models, with explainability in mind to ease the interpretation of the evaluation results. We have evaluated 20 different models for Spanish, including some of the most popular pretrained ones in the research community. Our findings state that varying levels of gender bias are present across these models.This approach compares the adjectives proposed by the model for a set of templates. We classify the given adjectives into understandable categories and compute two new metrics from model predictions, one based on the internal state (probability) and the other one on the external state (rank). Those metrics are used to reveal biased models according to the given categories and quantify the degree of bias of the models under study. [ABSTRACT FROM AUTHOR]

Social media has revolutionized the manner in which our society is interconnected. While this extensive connectivity offers numerous benefits, it is also accompanied by significant drawbacks, particularly in terms of the proliferation of fake news and the vast dissemination of hate speech. Identifying offensive comments is a critical task for ensuring the safety of users, which is why industry and academia have been working on developing solutions to this problem. Prior research on hate speech detection has predominantly focused on the English language, with few studies devoted to other languages such as Portuguese. This paper introduces the Offensive Language Identification Dataset for Brazilian Portuguese (OLID-BR), a high-quality NLP dataset for offensive language detection, which we make publicly available. The dataset contains 6,354 (extendable to 13,538) comments labeled using a fine-grained three-layer annotation schema compatible with datasets in other languages, which allows the training of multilingual/cross-lingual models. The five NLP tasks available in OLID-BR allow the detection of offensive comments, the classification of the types of offenses such as racism, LGBTQphobia, sexism, xenophobia, and so on, the identification of the type and the target of offensive comments, and the extraction of toxic spans of offensive comments. All those tasks can enhance the capabilities of content moderation systems by providing deep contextual analysis or highlighting the spans that make a text toxic. We further experiment with and evaluate the dataset using state-of-the-art BERT-based and NER models, which demonstrates the usefulness of OLID-BR for the development of toxicity detection systems for Portuguese texts. [ABSTRACT FROM AUTHOR]

This study addresses the critical need for an accurate aspect-based sentiment-analysis (ABSA) model to understand sentiments effectively. The existing ABSA models often face challenges in accurately extracting aspects and determining sentiment polarity from textual data. Therefore, we propose a novel approach leveraging latent-Dirichlet-allocation (LDA) for aspect extraction and transformer-based bidirectional-encoder-representations from transformers (TF-BERT) for sentiment-polarity evaluation. The experiments were carried out on SemEval 2014 laptop and restaurant datasets. Also, a multi-domain dataset was generated by combining SemEval 2014, Amazon, and hospital reviews. The results demonstrate the superiority of the LDA-TF-BERT model, achieving 82.19% accuracy and 79.52% Macro-F1 score for the laptop task and 86.26% accuracy of 87.26% and 81.27% for Macro-F1 score for the restaurant task. This showcases the model's robustness and effectiveness in accurately analyzing textual data and extracting meaningful insights. The novelty of our work lies in combining LDA and TF-BERT, providing a comprehensive and accurate ABSA solution for various industries, thereby contributing significantly to the advancement of sentiment analysis techniques. [ABSTRACT FROM AUTHOR]

In recent years, there has been a noticeable surge in research activities focused on sentiment analysis within the Kannada language domain. The existing research highlights a lack of labelled datasets and limited exploration in feature selection for Kannada sentiment analysis, hindering accurate sentiment classification. To address this gap, the study aims to introduce a novel Kannada dataset and develop an effective classifier for improved sentiment analysis in Kannada texts. The study presents a new Kannada dataset from SemEval 2014 Task4 using Google Translate. It then introduces a modified bidirectional encoder representation from transformers BERT for Kannada dataset called as Kannada-BERT (K-BERT). Further, a probability-clustering (PC) approach is presented to extract the topics and its related aspects. Both the K-BERT classifier and PC approach were merged to attain a K-BERT-PC classifier, integrating a modified BERT model and probability clustering approach for achieving better results. Experimental results demonstrate that K-BERT-PC achieves superior performance in polarity and sentiment analysis accuracy, with an impressive accuracy rate of 91%, surpassing existing classifiers. This work contributes by providing a solution to the scarcity of labelled datasets for Kannada sentiment analysis and introduces an effective classifier, K-BERT-PC, for enhanced sentiment analysis outcomes in Kannada texts. [ABSTRACT FROM AUTHOR]

Agriculture food (agri-food) safety is closely related to all aspects of people's lives. In recent years, with the emergence of deep learning technology based on big data, the extraction of information relations in the field of agri-food safety supervision has become a research hotspot. However, most of the current work only expands the relationship recognition based on the traditional named entity recognition task, which makes it difficult to establish a true 'connection' between entities and relationships. The pipelined and federated extraction architectures that have emerged in this area are problematic in practice. In addition, the contextual information of the text corpus in the agri-food safety regulatory domain has not been fully utilized. To address the above issues, this paper proposes a semi-joint entity relationship extraction model (EB-SJRE) based on contextual entity boundary features. Firstly, a Token pair subject-object correspondence matrix label is designed to intuitively model the subject-object boundary, which is more friendly to complex entities in the field of agri-food safety regulation. Secondly, the dynamic fine-tuning of Bert makes the text embedding more relevant to the textual context of the agri-food safety regulation domain. Finally, we introduce an attention mechanism in the Token pair tagging framework to capture deep semantic subject-object boundary association information, which cleverly solves the problem of bias exposure due to the pipeline structure and the dimensional explosion due to the joint extraction structure. The experimental results show that our model achieves the best F1-score of 88.71% on agri-food safety regulation domain data and F1-scores of 92.36%, 92.80%, 88.91%, and 92.21% on NYT, NYT-star, WebNLG, and WebNLG-star, respectively. This indicates that EB-SJRE has excellent generalization ability in both the agri-food safety regulatory and public sectors. [ABSTRACT FROM AUTHOR]

Phishing attacks present a serious threat to enterprise systems, requiring advanced detection techniques to protect sensitive data. This study introduces a phishing email detection framework that combines Bidirectional Encoder Representations from Transformers (BERT) for feature extraction and CNN for classification, specifically designed for enterprise information systems. BERT's linguistic capabilities are used to extract key features from email content, which are then processed by a convolutional neural network (CNN) model optimized for phishing detection. Achieving an accuracy of 97.5%, our proposed model demonstrates strong proficiency in identifying phishing emails. This approach represents a significant advancement in applying deep learning to cybersecurity, setting a new benchmark for email security by effectively addressing the increasing complexity of phishing attacks. [ABSTRACT FROM AUTHOR]

Transformer-based language models (TLMs) have widely been recognized to be a cutting-edge technology for the successful development of deep-learning-based solutions to problems and applications that require natural language processing and understanding. Like for other textual domains, TLMs have indeed pushed the state-of-the-art of AI approaches for many tasks of interest in the legal domain. Despite the first Transformer model being proposed about six years ago, there has been a rapid progress of this technology at an unprecedented rate, whereby BERT and related models represent a major reference, also in the legal domain. This article provides the first systematic overview of TLM-based methods for AI-driven problems and tasks in the legal sphere. A major goal is to highlight research advances in this field so as to understand, on the one hand, how the Transformers have contributed to the success of AI in supporting legal processes, and on the other hand, what are the current limitations and opportunities for further research development. [ABSTRACT FROM AUTHOR]

Analysis of the impact of the contextual embeddings usage on the text classification accuracy

Transformer-based language models have revolutionized Natural Language Processing tasks, with advancements in language modeling techniques. Current transformer architectures utilize attention mechanisms to model text dependencies effectively. Studies have shown that these models embed syntactic structures and knowledge, explaining their performance in tasks involving syntactic and semantic elements. However, transformer-based models are prone to hallucination where incorporated knowledge is not utilized effectively. To address this, methods are emerging to mitigate hallucination and integrate external knowledge sources like knowledge graphs (e.g., Freebase, WordNet, ConceptNet, ATOMIC). Knowledge graphs represent real-world knowledge through entities and relationships offering a potential injection point to enhance model performance in inference tasks. Various injection approaches, including input, architectural, and output injections, aim to incorporate knowledge from graphs into transformer models. Input injections modify data preprocessing, architectural injections add layers for knowledge integration, and output injections adjust error functions to correct knowledge incorporation during training. Despite ongoing research, a universal solution to hallucination remains elusive, and a standardized benchmark for comparing injection methods is lacking. This study investigates knowledge graphs as one of the methods to mitigate hallucination and their possible integration into Large Language Models. Comparative experiments across General Language Understanding Evaluation benchmark tasks demonstrated that ERNIE 3.0 and XLNet outperform other injection methods with the average scores of 91.1 % and 90.1 %.

Magnesium (Mg) based materials hold immense potential for various applications due to their lightweight and high strength-to-weight ratio. However, to fully harness the potential of Mg alloys, structured analytics are essential to gain valuable insights from centuries of accumulated knowledge. Efficient information extraction from the vast corpus of scientific literature is crucial for this purpose. In this work, we introduce MagBERT, a BERT-based language model specifically trained for Mg-based materials. Utilizing a dataset of approximately 370,000 abstracts focused on Mg and its alloys, MagBERT is designed to understand the intricate details and specialized terminology of this domain. Through rigorous evaluation, we demonstrate the effectiveness of MagBERT for information extraction using a fine-tuned named entity recognition (NER) model, named MagNER. This NER model can extract mechanical, microstructural, and processing properties related to Mg alloys. For instance, we have created an Mg alloy dataset that includes properties such as ductility, yield strength, and ultimate tensile strength (UTS), along with standard alloy names. The introduction of MagBERT is a novel advancement in the development of Mg-specific language models, marking a significant milestone in the discovery of Mg alloys and textual information extraction. By making the pre-trained weights of MagBERT publicly accessible, we aim to accelerate research and innovation in the field of Mg-based materials through efficient information extraction and knowledge discovery.

Abstract The notion of discerning political affiliations from users’ social media behavior instills a sense of unease in many. Democracy necessitates that individuals’ political affiliations remain private, and social media challenges this foundational principle of democracy. This study uses BERT, a pre-trained language model to analyze X’s (formally Twitter) users and their political affiliations to understand that how much it is easy now to find the political affiliation of people. We collect posts in both English and Urdu languages from different political leaders and their followers, which are used to fine-tune the BERT model. The model classifies the users’ profiles into Pro, Neutral, or Anti-government classes. To assess the performance of the proposed method, experiments are conducted to evaluate its accuracy, efficiency, and effectiveness. The findings of this study confirm the hypothesis that it is easy to detect the political affiliation of individuals using social media with high accuracy (69% for English and 94% for Urdu language) and it can undermine democracy.

Cancer immunology offers a new alternative to traditional cancer treatments, such as radiotherapy and chemotherapy. One notable alternative is the development of personalized vaccines based on cancer neoantigens. Moreover, Transformers are considered a revolutionary development in artificial intelligence with a significant impact on natural language processing (NLP) tasks and have been utilized in proteomics studies in recent years. In this context, we conducted a systematic literature review to investigate how Transformers are applied in each stage of the neoantigen detection process. Additionally, we mapped current pipelines and examined the results of clinical trials involving cancer vaccines.

Attention to mental health is increasing in Indonesia, especially with the recent increase in the number of cases of stress and suicide among students. Therefore, this research aims to provide a solution to overcome mental health problems by introducing a chatbot system based on Deep Neural Networks (DNN) and BiDirectional Encoder Representation Transformers (BERT). The primary objective is to enhance accessibility and offer a more effective solution concerning the mental health of students. This chatbot utilizes Natural Language Processing (NLP) and Deep Learning to provide appropriate responses to mild mental health issues. The dataset, comprising objectives, tags, patterns, and responses, underwent processing using Indonesian language rules within NLP. Subsequently, the system was trained and tested using the DNN model for classification, integrated with the TokenSimilarity model to identify word similarities. Experimental results indicate that the DNN model yielded the best outcomes, with a training accuracy of 98.97%, validation accuracy of 71.74%, and testing accuracy of 71.73%. Integration with the TokenSimilarity model enhanced the responses provided by the chatbot. TokenSimilarity searches for input similarities from users within the knowledge generated from the training data. If the similarity is high, the input is then processed by the DNN model to provide the chatbot response. This integration of both models has proven to enhance the responsiveness of the chatbot in providing various responses even when the user inputs remain the same. The chatbot also demonstrates the capability to recognize various inputs more effectively with similar intentions or purposes. Additionally, the chatbot exhibits the ability to comprehend user inputs although there are many writing errors.

Abstract Background Biomedical Relation Extraction (RE) is essential for uncovering complex relationships between biomedical entities within text. However, training RE classifiers is challenging in low-resource biomedical applications with few labeled examples. Methods We explore the potential of Shortest Dependency Paths (SDPs) to aid biomedical RE, especially in situations with limited labeled examples. In this study, we suggest various approaches to employ SDPs when creating word and sentence representations under supervised, semi-supervised, and in-context-learning settings. Results Through experiments on three benchmark biomedical text datasets, we find that incorporating SDP-based representations enhances the performance of RE classifiers. The improvement is especially notable when working with small amounts of labeled data. Conclusion SDPs offer valuable insights into the complex sentence structure found in many biomedical text passages. Our study introduces several straightforward techniques that, as demonstrated experimentally, effectively enhance the accuracy of RE classifiers.

This paper presents a novel approach for cyberbullying detection and classification in social media text using an ensemble model that combines BERT (Bidirectional Encoder Representations from Transformers) and Support Vector Machine (SVM) with grid search for multiclass classification. We have also compared the performance of our proposed with various machine and deep learning models and the results show that our proposed model outperforms other models achieving an accuracy of 90% on testing data. Further, we have used to used SHapley Additive exPlanations (SHAP) an Explainable (XAI) technique to interpret the predictions of the BERT-SVM ensemble model.

Large power texts contain numerous abbreviations of technical terms， alternative names， and irregular expressions. Existing word segmentation tools often fail to identify specialized vocabulary in the electrical engineering field， significantly hindering the analysis and utilization of unstructured texts. To address this challenge， this paper proposes a set of indexing rules tailored to the characteristics of unstructured texts in electrical engineering. Segmentation based on these rules can significantly enhance segmentation accuracy， laying a solid foundation for feature extraction of power texts. Furthermore， by employing effective long-text segmentation algorithms to preserve the semantic information of the original text， the paper integrates and embeds text feature information extracted by the BERT model with vocabulary feature information extracted by Word2Vec. This combined approach enables the extraction of precise features from large unstructured power texts. Finally， experimental results have demonstrated the effectiveness of the proposed method for extracting features from large unstructured power texts.

Abstract Many state-of-the-art results in natural language processing (NLP) rely on large pre-trained language models (PLMs). These models consist of large amounts of parameters that are tuned using vast amounts of training data. These factors cause the models to memorize parts of their training data, making them vulnerable to various privacy attacks. This is cause for concern, especially when these models are applied in the clinical domain, where data are very sensitive. Training data pseudonymization is a privacy-preserving technique that aims to mitigate these problems. This technique automatically identifies and replaces sensitive entities with realistic but non-sensitive surrogates. Pseudonymization has yielded promising results in previous studies. However, no previous study has applied pseudonymization to both the pre-training data of PLMs and the fine-tuning data used to solve clinical NLP tasks. This study evaluates the effects on the predictive performance of end-to-end pseudonymization of Swedish clinical BERT models fine-tuned for five clinical NLP tasks. A large number of statistical tests are performed, revealing minimal harm to performance when using pseudonymized fine-tuning data. The results also find no deterioration from end-to-end pseudonymization of pre-training and fine-tuning data. These results demonstrate that pseudonymizing training data to reduce privacy risks can be done without harming data utility for training PLMs.

In this work, we evaluated the efficacy of Google’s Pathways Language Model (GooglePaLM) in analyzing sentiments expressed in product reviews. Although conventional Natural Language Processing (NLP) techniques such as the rule-based Valence Aware Dictionary for Sentiment Reasoning (VADER) and the long sequence Bidirectional Encoder Representations from Transformers (BERT) model are effective, they frequently encounter difficulties when dealing with intricate linguistic features like sarcasm and contextual nuances commonly found in customer feedback. We performed a sentiment analysis on Amazon’s fashion review datasets using the VADER, BERT, and GooglePaLM models, respectively, and compared the results based on evaluation metrics such as precision, recall, accuracy correct positive prediction, and correct negative prediction. We used the default values of the VADER and BERT models and slightly finetuned GooglePaLM with a Temperature of 0.0 and an N-value of 1. We observed that GooglePaLM performed better with correct positive and negative prediction values of 0.91 and 0.93, respectively, followed by BERT and VADER. We concluded that large language models surpass traditional rule-based systems for natural language processing tasks.

Abstract Among the various molecular properties and their combinations, it is a costly process to obtain the desired molecular properties through theory or experiment. Using machine learning to analyze molecular structure features and to predict molecular properties is a potentially efficient alternative for accelerating the prediction of molecular properties. In this study, we analyze molecular properties through the molecular structure from the perspective of machine learning. We use SMILES sequences as inputs to an artificial neural network in extracting molecular structural features and predicting molecular properties. A SMILES sequence comprises symbols representing molecular structures. To address the problem that a SMILES sequence is different from actual molecular structural data, we propose a pretraining model for a SMILES sequence based on the BERT model, which is widely used in natural language processing, such that the model learns to extract the molecular structural information contained in the SMILES sequence. In an experiment, we first pretrain the proposed model with 100,000 SMILES sequences and then use the pretrained model to predict molecular properties on 22 data sets and the odor characteristics of molecules (98 types of odor descriptor). The experimental results show that our proposed pretraining model effectively improves the performance of molecular property prediction Scientific contribution The 2-encoder pretraining is proposed by focusing on the lower dependency of symbols to the contextual environment in a SMILES than one in a natural language sentence and the corresponding of one compound to multiple SMILES sequences. The model pretrained with 2-encoder shows higher robustness in tasks of molecular properties prediction compared to BERT which is adept at natural language.

Typhoons are extreme weather phenomena that seriously affect the daily lives of residents and regular functioning of society. As one of the most typhoon-prone countries in the world, China is constantly affected by typhoons and their secondary disasters, which can cause significant casualties and economic losses. The extent of damage caused by typhoons is inversely proportional to the effectiveness of the emergency response. Therefore, accurate and comprehensive access to damage information is critical for rescue and recovery. Social media, which is characterized by low collection costs and rich content, is an important means of collecting disaster information. With the development of social media, it has become increasingly important to accurately and comprehensively identify social media texts related to typhoons. In this study, by combining typhoon attribute data and a multi-label classification method with Bidirectional Encoder Representations from Transformers (BERT) and Bidirectional Long Short-Term Memory (BiLSTM) models, a typhoon damage identification method based on Weibo texts and deep learning is proposed to identify the damage caused by severe and super typhoons that made landfall in Guangdong Province from 2010 to 2019. First, texts related to typhoon damage were identified from the massive Weibo texts and further classified into five damage categories: transportation, public, electricity, forestry, and waterlogging. The typhoon damage characteristics were comparatively analyzed using spatial distribution, time curves, and quantity curves. The results showed that the accuracy of typhoon damage classification was high, with an F1 score of 0.907 for identifying typhoon damage-related texts and 0.814 for further classifying them into five damage categories. Typhoon attribute data and multi-label classification methods have improved the accuracy and comprehensiveness of typhoon damage identification. Compared to the use of Weibo texts only and the single-label classification method, typhoon attribute data provide information on the geographic context of the typhoon at the time of the texts' release, and the multi-label classification method allows the texts to belong to more than one damage category. This study shows that there are differences in the proportion of damage caused by different typhoons, which are related to the intensity and track of the typhoon, as well as the development level of the affected areas. In addition, before the typhoon makes landfall, precautions lead to transportation and public-related damage. After the typhoon makes landfall, the typhoon damage shows single and double-peak characteristics, and the different characteristics reflect the changing trends and features of typhoon damage. This study provides a scientific basis for typhoon damage identification and disaster relief in Guangdong Province.

Abstract The advent of the fourth industrial revolution, characterized by artificial intelligence (AI) as its central component, has resulted in the mechanization of numerous previously labor-intensive activities. The use of in silico tools has become prevalent in the design of biopharmaceuticals. Upon conducting a comprehensive analysis of the genomes of many organisms, it has been discovered that their tissues can generate specific peptides that confer protection against certain diseases. This study aims to identify a selected group of neuropeptides (NPs) possessing favorable characteristics that render them ideal for production as neurological biopharmaceuticals. Until now, the construction of NP classifiers has been the primary focus, neglecting to optimize these characteristics. Therefore, in this study, the task of creating ideal NPs has been formulated as a multi-objective optimization problem. The proposed framework, NPpred, comprises two distinct components: NSGA-NeuroPred and BERT-NeuroPred. The former employs the NSGA-II algorithm to explore and change a population of NPs, while the latter is an interpretable deep learning-based model. The utilization of explainable AI and motifs has led to the proposal of two novel operators, namely p-crossover and p-mutation. An online application has been deployed at https://neuropred.anvil.app for designing an ideal collection of synthesizable NPs from protein sequences.

Abstract Relation prediction is a critical task in knowledge graph completion and associated downstream tasks that rely on knowledge representation. Previous studies indicate that both structural features and semantic information are meaningful for predicting missing relations in knowledge graphs. This has led to the development of two types of methods: structure-based methods and semantics-based methods. Since these two approaches represent two distinct learning paradigms, it is difficult to fully utilize both sets of features within a single learning model, especially deep features. As a result, existing studies usually focus on only one type of feature. This leads to an insufficient representation of knowledge in current methods and makes them prone to overlooking certain patterns when predicting missing relations. In this study, we introduce a novel model, RP-ISS, which combines deep semantic and structural features for relation prediction. The RP-ISS model utilizes a two-part architecture, with the first component being a RoBERTa module that is responsible for extracting semantic features from entity nodes. The second part of the system employs an edge-based relational message-passing network designed to capture and interpret structural information within the data. To alleviate the computational burden of the message-passing network on the RoBERTa module during the sampling process, RP-ISS introduces a node embedding memory bank, which updates asynchronously to circumvent excessive computation. The model was assessed on three publicly accessible datasets (WN18RR, WN18, and FB15k-237), and the results revealed that RP-ISS surpasses all baseline methods across all evaluation metrics. Moreover, RP-ISS showcases robust performance in graph inductive learning.

Abstract Background BERT models have seen widespread use on unstructured text within the clinical domain. However, little to no research has been conducted into classifying unstructured clinical notes on the basis of patient lifestyle indicators, especially in Dutch. This article aims to test the feasibility of deep BERT models on the task of patient lifestyle classification, as well as introducing an experimental framework that is easily reproducible in future research. Methods This study makes use of unstructured general patient text data from HagaZiekenhuis, a large hospital in The Netherlands. Over 148 000 notes were provided to us, which were each automatically labelled on the basis of the respective patients’ smoking, alcohol usage and drug usage statuses. In this paper we test feasibility of automatically assigning labels, and justify it using hand-labelled input. Ultimately, we compare macro F1-scores of string matching, SGD and several BERT models on the task of classifying smoking, alcohol and drug usage. We test Dutch BERT models and English models with translated input. Results We find that our further pre-trained MedRoBERTa.nl-HAGA model outperformed every other model on smoking (0.93) and drug usage (0.77). Interestingly, our ClinicalBERT model that was merely fine-tuned on translated text performed best on the alcohol task (0.80). In t-SNE visualisations, we show our MedRoBERTa.nl-HAGA model is the best model to differentiate between classes in the embedding space, explaining its superior classification performance. Conclusions We suggest MedRoBERTa.nl-HAGA to be used as a baseline in future research on Dutch free text patient lifestyle classification. We furthermore strongly suggest further exploring the application of translation to input text in non-English clinical BERT research, as we only translated a subset of the full set and yet achieved very promising results.

Visual question answering is a challenging task that necessitates for sophisticated reasoning over the visual elements to provide an accurate answer to a question. Majority of the state-of-the-art VQA models are only applicable to English questions. However, applications such as visual assistance and tourism necessitate the incorporation of multilingual VQA systems. This paper presents an effective deep learning framework for Malayalam visual question answering (MVQA), which can answer a specific natural language question about an image in Malayalam. As there is no available dataset in English–Malayalam VQA, a MVQA dataset was created by translating English question–answer pairs from the visual genome dataset. The paper proposes an attention-driven MVQA model on the developed dataset. The proposed MVQA model uses a deep learning-based co-attention mechanism to jointly learn the attention for images and Malayalam questions. A second-order multimodal factorized high-order pooling is used for multi modal feature fusion. Different VQA models using combinations of classical CNNs and RNNs were experimented on the developed MVQA dataset, and the performance was compared against the proposed attention-driven model. Experimental results show that the proposed attention-driven MVQA model achieves state-of-the-art results as compared to other models for MVQA on the custom Malayalam VQA dataset. [ABSTRACT FROM AUTHOR]

Social media has become a significant source of essential facts and alarming falsehoods, including rumours. A significant increase in rumour spreading has occurred due to the lack of an autonomous rumour detection mechanism, causing widespread and severe social repercussions. To address this challenge, we present a ground-breaking method for developing an automatic rumour detection system, focusing on the fundamental problem of class imbalance in rumour detection. Our method selectively uses oversampling to obtain a uniformly distributed dataset by leveraging contextualised data augmentation techniques to generate synthetic samples for underrepresented classes. Furthermore, we effectively recreate non-linear dialogues inside a thread using two novel graph neural networks (GNNs), which improves the system's capacity to understand complex links between postings. Our method employs a distinctive feature selection mechanism to enhance further Twitter representations based on the state-of-the-art BERTweet model. The thorough analysis of our methodology using three publicly accessible datasets yielded compelling results: (1) our GNN models outperformed the most state-of-the-art classifiers in F1-score by more than 20%. Emphasizing the importance of our approach to developing sophisticated rumour detection systems. (2) By utilizing our oversampling method, we significantly improve the F1-score by 9%, highlighting the practical implications of resolving class imbalance. (3) Our technique delivers further performance increases through non-random selection criteria for data augmentation, with the selection of relevant tweets highlighting the significance of our novel augmentation strategy. (4) Notably, our approach captures rumours in their early stages more effectively than previous classifiers, establishing a baseline for future works. The innovative aspects of our proposed method lie in its ability to solve class imbalance effectively, outperform existing classifiers in terms of performance, and drastically reduce the propagation of rumours and false information on social media platforms. Our study lays the way for developments in rumour detection by offering a comprehensive solution, eventually helping to ensure the veracity of information flowing online. We are confident that our findings have an influence on the broader field of rumour detection systems and provide fresh directions for further study. [ABSTRACT FROM AUTHOR]

Building task-oriented dialogue systems has become a topic of interest in the research community and industry. The task-oriented dialogue system is a closed-domain dialogue system that can perform specific tasks for users. The natural language understanding module of a task-oriented dialogue system is crucial because it is related to a task-oriented dialogue system that provides correctional services for users. The natural language understanding module of a task-oriented dialogue system performs two tasks: intent detection and slot filling. The intent detection task can be regarded as a text classification task; a classification model is trained to predict the intention of the user from the user's input information. The slot filling task can be regarded as a sequence analysis task; a sequence analysis model is trained to predict the details of the user's intention. In this paper, we proposed a novel model based on a transformer encoder for intent detection and slot filling. It follows the encoder-decoder structure, including a vanilla Transformer encoder, a bidirectional LSTM encoder, a linear classification decoder for intent detection, and a conditional random field decoder for slot filling. The experimental results on two public datasets show that our proposed model outperforms the existing methods based on the Transformer and can be combined with BERT to achieve better intent detection and slot filling results. [ABSTRACT FROM AUTHOR]

Copyright of Food & Fermentation Industries is the property of Food & Fermentation Industries and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The increasing volume of medical information available in digital format presents a significant challenge for researchers seeking to extract relevant information. Manually analyzing voluminous data is a time-consuming process that constrains researchers' productivity. In this context, innovative and intelligent computational approaches to information search, such as large language models (LLMs), offer a promising solution. LLMs understand natural language questions and respond accurately to complex queries, even in the specialized domain of medicine. This paper presents MédicoBERT, a medical language model in Spanish developed by adapting a general domain language model (BERT) to medical terminology and vocabulary related to diseases, treatments, symptoms, and medications. The model was pre-trained with 3 M medical texts containing 1.1 B words. Furthermore, with promising results, MédicoBERT was adapted and evaluated to answer medical questions in Spanish. The question-answering (QA) task was fine-tuned using a Spanish corpus of over 34,000 medical questions and answers. A search was then conducted to identify the optimal hyperparameter configuration using heuristic methods and nonlinear regression models. The evaluation of MédicoBERT was carried out using metrics such as perplexity to measure the adaptation of the language model to the medical vocabulary in Spanish, where it obtained a value of 4.28, and the average F1 metric for the task of answering medical questions, where it obtained a value of 62.35%. The objective of MédicoBERT is to provide support for research in the field of natural language processing (NLP) in Spanish, with a particular emphasis on applications within the medical domain. [ABSTRACT FROM AUTHOR]

NOTAM is an important way to notify changes in aeronautical intelligence information, and there are problems of non-correspondence between Q-code and E-item free text as well as non-standardization of expression in NOTAM. Aiming at the above problems, the bi-directional encoder representation technique-deep pyramidal convolutional neural network (BERT-DPCNN) NOTAM element similarity matching identification method was proposed. To address the above problems, based on the flight information Centre's 105 797 sample NOTAM from September 2020 to April 2023, the improved BERT-DPCNN navigational notice element similarity matching identification method was proposed to construct a text dataset with two types of data, E-code and Q-item, and to annotate the correctness of the text as well as the content of the wrong text correction. the E-item text was preprocessed by regularizing the E-item text using the bi-directional encoder representation technique (BERT) based on transformer, the global text features are extracted, and at the same time, the text encoding was carried out on the Q-code, and the pre-training text vector set was generated. The trained vector set was input into the deep pyramidal convolutional neural network (DPCNN) model as a word embedding layer, and 60% of the training set, 20% of the testing set and 20% of the validation set are randomly selected for model training, and then the trained model is used to discriminate the text similarity, and the results of the model evaluation metrics show that the average precision of all types of navigational notices is 88. 77%, the recall rate is 88. 74%, the F1-score is 89. 50%, the recognition effect is better than Text CNN, BERT, ERNIE, BERT-CNN models. [ABSTRACT FROM AUTHOR]

In order to solve the problem of poor question-answering performance caused by complex Chinese characters and semantic information, a method called Chinesebert-KBQA was proposed. Chinese pre-trained language model ChineseBERT was used as the semantic embedding layer of the text and the font and pinyin information were integrated. It improves the performance of traditional semantic parsing methods in the sub-tasks of entity reference recognition and relationship prediction. Specifically, the two models proposed were ChineseBERT-CRF based entity reference recognition model and ChineseBERT-TextCNN-Softmax based relationship prediction model, which comprehensively improved the semantic understanding ability of Chinese text. Finally, combined with the relevant information among the sub-tasks, the final answer prediction was carried out. The MOOC Q&A and NLPCC2018, both educational question-answering data sets, were used for experiments. The experimental results show the effectiveness and accuracy of the proposed method in Chinese knowledge graph question-answering. [ABSTRACT FROM AUTHOR]

Entity recognition and extraction from contracts play a crucial role in automating contract analysis and extracting valuable information. Named Entity Recognition (NER) techniques are used for identifying and classifying specific entities such as parties, dates, amounts, and clauses within contracts. In this study, we create a high-quality NER dataset from various types of English language contracts by considering their structure, and the legal terminology used within these documents. We present a systematic approach to manually annotate contracts with appropriate entity labels, ensuring accuracy and consistency. The resulting NER dataset serves as a valuable resource for training and evaluating NER models for contract analysis tasks. We evaluate the performance of NER on this dataset using a range of methods. These methods include Conditional Random Fields, various Bidirectional LSTM configurations, and BERT models. Each of these models brings different strengths and capabilities to the task of entity recognition, allowing for a comprehensive evaluation and the selection of the best models over the dataset. Among these, the NER model based on Contracts–BERT–base from the Legal–BERT family, which is pre-trained specifically on English contracts, outperformed all others, achieving an impressive overall F1 score of 0.94. [ABSTRACT FROM AUTHOR]

Sentiment analysis (SA) is the study of human perception in any subject of practice. It retrieves data from datasets using natural language processing (NLP) methodologies and algorithms that are either regulation-based, blended, or rely on machine learning approaches. SA is garnering fame for its capacity to fit in a large chunk of data with user evaluations, uncover a trend, and come to a consensus derived from real facts rather than hypotheses established on a limited number of observations. The flexible nature of sentiment gathering has helped in playing a critical role in both commercial and research applications in the last few years. This study presents new sentiment analysis models based on Bidirectional Encoder Representations from Transformers (BERT) for both labeled and unlabeled datasets. The labeled datasets using supervised learning are modeled in a hybrid architecture of fine-tuned BERT and interval Type—2 fuzzy sets. The inclusion of interval Type-2 fuzzy logic for handling reluctance or inaccuracy in data shows commendable results for the labeled datasets. For the prediction of sentiments in unlabeled datasets, they are embedded through a BERT tokenizer with the help of a threshold and activation functions. The coupling of a multi-layer perceptron with the BERT parser substantially decreases the time and complexity compared to supervised learning. Both the models have been implemented on multiple datasets and have outperformed existing state-of-the-art techniques in this field. [ABSTRACT FROM AUTHOR]

In modern times, system energy load forecasting is an extremely important process in a variety of contexts. Moreover, energy load time series fluctuations are influenced by a wide range of factors, ranging, inter alia, from environmental conditions, natural events, and demographics to both regional and global geopolitical contexts, economic conditions, energy sources, policies, and regulations. Given these, this paper examines the integration of news information from the global scene into Greek energy load forecasting schemes through the use of sentiment analysis. Investigating the ways the general emotional footprint of news worldwide affects and can be used in an energy modeling context, we benchmark possible ensemble configurations incorporating a multitude of 31 emotion polarities. Building on our previous work, an ensemble method that exploits specific outputs from a multi-label sentiment classifier and a sentiment analysis procedure under a multivariate forecasting scheme is presented. It is shown, through an empirical evaluation of the results, that the integration of emotion representations related to non-Greek news concerning global current affairs improves the predictions. [ABSTRACT FROM AUTHOR]

Deep learning models have overcome traditional machine learning techniques for text classification domains in the field of natural language processing (NLP). Since, NLP is a branch of machine learning, used for interpreting language, classifying text of interest, and the same can be applied to analyse the medical clinical electronic health records. Medical text consists of lot of rich data which can altogether provide a good insight, by determining patterns from the clinical text data. In this paper, bidirectional‐long short‐term memory (Bi‐LSTM), bi‐LSTM attention and bidirectional encoder representations from transformers (BERT) base models are used to classify the text which are of privacy concern to a person and which should be extracted and can be tagged as sensitive. This text data which we might think not of privacy concern would majorly reveal a lot about the patient's integrity and personal life. Clinical data not only have patient demographic data but lot of hidden data which might go unseen and thus could arise privacy issues. Bi‐LSTM with attention layer is also added on top to realize the importance of critical words which will be of great importance in terms of classification, we are able to achieve accuracy of about 92%. About 206,926 sentences are used out of which 80% are used for training and rest for testing we get accuracy of 90% approx. with Bi‐LSTM alone. The same set of datasets is used for BERT model with accuracy of 93% approx. [ABSTRACT FROM AUTHOR]

Political partisanship constitutes a pivotal group identity that significantly influences individuals' voting behaviors and shapes their ideological and cultural perspectives. While traditional surveys and experimental studies can directly capture political identity by asking the participants, this task has become intricate when employing digital trace data sourced from social media. Previous classification methods, attempting to infer political identity from users' networks or textual content, suffered from limited efficiency or generalizability. In response, this study introduces a two-step method that utilizes deep learning models to enhance classification efficiency, generalizability, and interpretability. In the first step, two deep learning models, trained on 2.5 million tweets from 825 Congressional politicians in the U.S., achieved accuracy rates of 87.71% and 89.54%, respectively, in detecting politicians' partisanships based on their individual tweets. Subsequently, in the second step, by employing a simple machine learning model that leverages the aggregated predicted values derived from the first-step models, accuracy rates of 94.92% and 96.61% were attained for identifying non-politician users' political identities based off their 50 and 200 tweets, respectively. In addition, an attention mechanism was integrated into the deep learning model to assess the contribution of each word in the classification process. [ABSTRACT FROM AUTHOR]

Up to now, the smart contract vulnerabilities detection methods based on sequence modal data and sequence models have been the most commonly used. However, existing state-of-the-art methods disregard the issue of sequence modal data loses structural information and control flow information. Additionally, it is hard for sequence models to extract global features of smart contracts. Moreover, these methods rarely consider the impact of noise data on vulnerabilities detection. To tackle these issues, we propose a smart contract vulnerabilities detection model based on bidirectional encoder representation from transformers (BERT) and control flow graph (CFG). On the one hand, we design a denoising method suitable for control flow graphs to reduce the impact of noisy data on vulnerabilities detection. On the other hand, we design a novel method to parse the control flow graph into a BERT input form that retains control flow information and structural information. The BERT learns the potential vulnerability characteristics of smart contracts to fine-tune itself. Through an empirical evaluation of a large-scale real-world dataset and compare 5 state-of-the-art baseline methods. Our method achieves (1) optimal performance over all baseline methods; (2) 0.6–17.1% higher F1-score than baseline methods; (3) 0.7–16.7% higher accuracy than baseline methods; (4) 0.6–17% higher precision than baseline methods; (5) 0.2–19.5% higher recall than baseline methods. [ABSTRACT FROM AUTHOR]

• Trained a language model (MagBERT) for magnesium data extraction. • Trained on ∼370K abstracts focused on Mg and its alloys. • Ability to extract mechanical, microstructure, and processing properties. • Publicly available model to boost magnesium materials research. Magnesium (Mg) based materials hold immense potential for various applications due to their lightweight and high strength-to-weight ratio. However, to fully harness the potential of Mg alloys, structured analytics are essential to gain valuable insights from centuries of accumulated knowledge. Efficient information extraction from the vast corpus of scientific literature is crucial for this purpose. In this work, we introduce MagBERT, a BERT-based language model specifically trained for Mg-based materials. Utilizing a dataset of approximately 370,000 abstracts focused on Mg and its alloys, MagBERT is designed to understand the intricate details and specialized terminology of this domain. Through rigorous evaluation, we demonstrate the effectiveness of MagBERT for information extraction using a fine-tuned named entity recognition (NER) model, named MagNER. This NER model can extract mechanical, microstructural, and processing properties related to Mg alloys. For instance, we have created an Mg alloy dataset that includes properties such as ductility, yield strength, and ultimate tensile strength (UTS), along with standard alloy names. The introduction of MagBERT is a novel advancement in the development of Mg-specific language models, marking a significant milestone in the discovery of Mg alloys and textual information extraction. By making the pre-trained weights of MagBERT publicly accessible, we aim to accelerate research and innovation in the field of Mg-based materials through efficient information extraction and knowledge discovery. [Display omitted] [ABSTRACT FROM AUTHOR]

E-commerce platforms play a crucial role in facilitating transactions between buyers and sellers, with technological advancements significantly influencing consumer behaviors. Efficiently managing product catalogs is essential, particularly for identifying and matching products across various channels. This paper explores deep learning techniques for product matching, leveraging both text and image modalities to enhance the accuracy and efficiency of this process. We propose a novel approach using a branch neural network embedding space integrated with K-nearest neighbors (KNN), treating image and text as distinct modalities. For text embedding, we utilize pre-trained BERT and CharacterBERT models, while for image embedding, we employ EfficientNet. Our methodology incorporates the ArcFace loss function to enhance intra-class compactness and inter-class discrepancy, thereby improving classification performance. Our results demonstrate that integrating multimodal embeddings with advanced loss functions like ArcFace significantly enhances the performance of product matching systems. This approach offers valuable insights for developing robust e-commerce platforms. [ABSTRACT FROM AUTHOR]

Sentiment analysis is a crucial discipline that focuses on the interpretation of feelings and points of view in textual data. Our study aims to assess the impact of different feature extraction methods on the accuracy of opinion research models. Techniques such as bag-of-words (BoW), term frequencyinverse document frequency (TF-IDF), Word2Vec, global vectors (GloVe) and bidirectional encoder representations from transformers (BERT) were used with three machine learning algorithms and three deep learning networks as classifiers. The IMDB movie review dataset was used for evaluation. The results showed that combining BERT with LSTM, CNN and RNN improved performance, achieving an accuracy rate of 94%, precision of 94.14%, recall of 93.27% and an F1 score of 89.33%. These results highlight the significant contribution of ERTB to model performance, outperforming other feature extraction techniques in text classification. The study concludes that the fusion of BERT and LSTM significantly improves model accuracy for opinion retrieval, recommending BERT as the main feature extraction method for optimizing performance in NLP tasks. [ABSTRACT FROM AUTHOR]

Predicting the directions of financial markets has been performed using a variety of approaches, and the large volume of unstructured data generated by traders and other stakeholders on social media microblog platforms provides unique opportunities for analyzing financial markets using additional perspectives. Pretrained large language models (LLMs) have demonstrated very good performance on a variety of sentiment analysis tasks in different domains. However, it is known that sentiment analysis is a very domain-dependent NLP task that requires knowledge of the domain ontology, and this is particularly the case with the financial domain, which uses its own unique vocabulary. Recent developments in NLP and deep learning including LLMs have made it possible to generate actionable financial sentiments using multiple sources including financial news, company fundamentals, technical indicators, as well social media microblogs posted on platforms such as StockTwits and X (formerly Twitter). We developed a financial social media sentiment analyzer (FinSoSent), which is a domain-specific large language model for the financial domain that was pretrained on financial news articles and fine-tuned and tested using several financial social media corpora. We conducted a large number of experiments using different learning rates, epochs, and batch sizes to yield the best performing model. Our model outperforms current state-of-the-art FSA models based on over 860 experiments, demonstrating the efficacy and effectiveness of FinSoSent. We also conducted experiments using ensemble models comprising FinSoSent and the other current state-of-the-art FSA models used in this research, and a slight performance improvement was obtained based on majority voting. Based on the results obtained across all models in these experiments, the significance of this study is that it highlights the fact that, despite the recent advances of LLMs, sentiment analysis even in domain-specific contexts remains a difficult research problem. [ABSTRACT FROM AUTHOR]