Search

The price of oil is highly complex to predict as it is impacted by global demand and supply, geopolitical events, and market sentiment. The accuracy of such predictions, however, has far-reaching implications for supply chain performance, portfolio management, and expected stock market returns. This paper contributes to the oil price prediction literature by evaluating the predictive impact of the US President's communication on Twitter, while benchmarking various Natural Language Processing (NLP) techniques, including Term Frequency-Inverse Document Frequency (TF-IDF), Word2Vec, Doc2Vec, Global Vectors for Word Representation (GloVe), and Bidirectional Encoder Representations from Transformers (BERT). These techniques are combined with a deep neural network Long Short-Term Memory (LSTM) architecture using a five-day lag for both the oil price and the textual Twitter data. The data was collected during the term of US President Donald Trump, resulting in 1449 days of crude oil price prediction and a total of 16,457 tweets. The study is validated for Brent and West Texas Intermediate blends, using the daily price of a barrel of crude oil as the target variable. The results confirm that including the US President's tweets significantly increases the predictive power of oil price prediction models, and that an LSTM architecture with BERT as NLP technique has the best performance. [ABSTRACT FROM AUTHOR]

Large-scale text data from public sources, including social media or online platforms, can expand urban planners’ ability to monitor and analyze urban conditions in near real-time. To overcome scalability challenges of manual techniques for qualitative data analysis, researchers and practitioners have turned to computer-automated methods, such as natural language processing (NLP) and deep learning. However, the benefits, challenges, and trade-offs of these methods remain poorly understood. How much meaning can different NLP techniques capture and how do their results compare to traditional manual techniques? Drawing on 90,000 online rental listings in Los Angeles County, this study proposes and compares manual, semi-automated, and fully automated methods for identifying context-informed topics in unstructured, user-generated text data. We find that fully automated methods perform best with more-structured text, but struggle to separate topics in free-flow text and when handling nuanced language. Introducing a manual technique first on a small data set to train a semi-automated method, however, improves accuracy even as the structure of the text degrades. We argue that while fully automated NLP methods are attractive replacements for scaling manual techniques, leveraging the contextual understanding of human expertise alongside efficient computer-based methods like BERT models generates better accuracy without sacrificing scalability.

BACKGROUND: Pathology reports contain key information about the patients diagnosis as well as important gross and microscopic findings. These information-rich clinical reports offer an invaluable resource for clinical studies, but data extraction and analysis from such unstructured texts is often manual and tedious. While neural information retrieval systems (typically implemented as deep learning methods for natural language processing) are automatic and flexible, they typically require a large domain-specific text corpus for training, making them infeasible for many medical subdomains. Thus, an automated data extraction method for pathology reports that does not require a large training corpus would be of significant value and utility. OBJECTIVE: To develop a language model-based neural information retrieval system that can be trained on small datasets and validate it by training it on renal transplant-pathology reports to extract relevant information for two predefined questions: (1) What kind of rejection does the patient show?; (2) What is the grade of interstitial fibrosis and tubular atrophy (IFTA)? METHODS: Kidney BERT was developed by pre-training Clinical BERT on 3.4K renal transplant pathology reports and 1.5M words. Then, exKidneyBERT was developed by extending Clinical BERTs tokenizer with six technical keywords and repeating the pre-training procedure. This extended the models vocabulary. All three models were fine-tuned with information retrieval heads. RESULTS: The model with extended vocabulary, exKidneyBERT, outperformed Clinical BERT and Kidney BERT in both questions. For rejection, exKidneyBERT achieved an 83.3% overlap ratio for antibody-mediated rejection (ABMR) and 79.2% for T-cell mediated rejection (TCMR). For IFTA, exKidneyBERT had a 95.8% exact match rate. CONCLUSION: ExKidneyBERT is a high-performing model for extracting information from renal pathology reports. Additional pre-training of BERT language models on specialized small domains does not necessarily improve performance. Extending the BERT tokenizers vocabulary library is essential for specialized domains to improve performance, especially when pre-training on small corpora.

Background: DNA methylation is a covalent bond modification that is observed mainly at cytosine bases in the context of CG pairs. DNA methylation patterns reflect the status of individual tissues, such as cell composition, age, and the local environment, in mammals. Genetic factors also impact DNA methylation, and the genetic diversity among various dog breeds provides a valuable platform for exploring this topic. Compared to those in the human genome, studies on the profiling of methylation in the dog genome have been less comprehensive. Results: Our study provides extensive profiling of DNA methylation in the whole blood of three dog breeds using whole-genome bisulfite sequencing. The difference in DNA methylation between breeds was moderate after removing CpGs overlapping with potential genetic variation. However, variance in methylation between individuals was common and often occurred in promoters and CpG islands (CGIs). Moreover, we adopted contextual awareness methodology to characterize DNA primary sequences using natural language processing (NLP). This method could be used to effectively separate unmethylated CGIs from highly methylated CGIs in the sequences that are identified by the conventional criteria. Conclusions: This study presents a comprehensive DNA methylation landscape in the dog blood. Our observations reveal the similar methylation patterns across dog breeds, while CGI regions showed high variations in DNA methylation level between individuals. Our study also highlights the potential of NLP approach for analyzing low-complexity DNA sequences, such as CGIs. [ABSTRACT FROM AUTHOR]

Text classification is a fundamental task in NLP that is used in several real-life tasks and applications. Large pre-trained language models such as BERT achieve state-of-the-art performance in several NLP tasks including text classification tasks. Although BERT boosts text classification performance, the common way of using it for classification lacks many aspects of its advantages. This work rethinks the way of using BERT final layer and hidden layers embeddings by proposing different aggregation architectures for text classification tasks such as sentiment analysis and sarcasm detection. This research also proposes different approaches for using BERT as a feature extractor without fine-tuning whose performance surpasses its fine-tuning counterpart. It also proposes promising multi-task learning aggregation architectures to improve the performance of the related classification problems. The experiments of the different architectures show that freezing BERT can outperform fine-tuning it for sentiment analysis. The experiments also show that multi-task learning while freezing BERT boosts the performance of yet hard tasks such as sarcasm detection. The best-performing models achieved new state-of-the-art performance on the ArSarcasm-v2 dataset for Arabic sarcasm detection and sentiment analysis. For multi-task learning and freezing BERT, a new SOTA F1-score of 64.41 was achieved for the sarcasm detection with a 3.47% improvement and near SOTA FPN of 75.78 for the sentiment classification. For single-task learning, a new SOTA FPN of 75.26 was achieved for the sentiment with a 1.81% improvement. [ABSTRACT FROM AUTHOR]

Cybersecurity is crucial for protecting the privacy of digital systems, maintaining economic stability, and ensuring national security. This study presents a comprehensive approach to cybersecurity knowledge discovery through topic modelling, using a multi-perspective analysis of academic and industry sources. The datasets include 15,751 articles from the Web of Science (WoS) database and 5,831 articles from Security Magazine, spanning from 2011 to 2023. We employed BERTopic for topic modelling, UMAP for dimensionality reduction, and HDBSCAN clustering algorithm for grouping and analysing distinct article clusters to uncover latent topics, enhancing the understanding of the evolving cybersecurity landscape. This study found 12 micro-clusters and three macro-clusters, namely technology, smart city and education, from the WoS database and 12 more micro-clusters and four macro-clusters, including organization, public security, governance, and education, from magazines. This study reveals key cybersecurity research and practice trends, such as the increasing focus on malware, ransomware, and cyber-attack mitigation. Additionally, temporal analysis indicates a significant rise in cybersecurity interest around 2020, followed by a diversification of topics. The results highlight the importance of integrating diverse data sources to capture a holistic view of cybersecurity developments. Future work will aim to refine the clustering algorithms to further improve topic extraction and analysis and expand the datasets to include more diverse sources and perspectives. This approach helps discover current cybersecurity trends and provides a foundation for more targeted and effective cybersecurity strategies. [ABSTRACT FROM AUTHOR]

Sarcasm is often related to something that has created a mass confusion among the general uninformed public. It is always associated with a mockery tone or trenchancy facial expression or weird language. Existing literatures that are profound in the field of sarcasm detection mainly focused on text-based input with sarcastic comments or facial expression-based analysis, i.e., image input. But both text and image input are not sufficient to analyze the underlying sarcasm behind the scene. This kind of analysis can also be misleading sometimes as the emotional expression can change with social circumstances (i.e., audio tone) over time. Hence to address these challenges, "A Smart Video Analytical framework for Sarcasm Detection using Deep Learning" is introduced where sarcasm detection is done by considering video modality. Proposed model extracts three important features from the video, i.e., text using proposed Enhanced-BERT, image using ImageNet and audio using Librosa. After extraction, each modality is addressed individually and is finally fused using proposed adaptive early fusion approach. The final task prediction of classification is done using novel deep neural network called "SarcasNet-99" to detect sarcasm in video over distributed framework called Apache Storm. TedX and GIF Reply datasets are used for model training and testing with around 10,000 + video clips. When compared against existing state-of-the-art techniques such as AlexNet, DenseNet, SqueezeNet and ResNet, the proposed model predicted accuracy 99.005% with LeakyReLU activation function. [ABSTRACT FROM AUTHOR]

Transformer-based models for transfer learning have the potential to achieve high prediction accuracies on text-based supervised learning tasks with relatively few training data instances. These models are thus likely to benefit social scientists that seek to have as accurate as possible text-based measures, but only have limited resources for annotating training data. To enable social scientists to leverage these potential benefits for their research, this article explains how these methods work, why they might be advantageous, and what their limitations are. Additionally, three Transformer-based models for transfer learning, BERT, RoBERTa, and the Longformer, are compared to conventional machine learning algorithms on three applications. Across all evaluated tasks, textual styles, and training data set sizes, the conventional models are consistently outperformed by transfer learning with Transformers, thereby demonstrating the benefits these models can bring to text-based social science research. [ABSTRACT FROM AUTHOR]

Transfer learning models have proven superior to classical machine learning approaches in various text classification tasks, such as sentiment analysis, question answering, news categorization, and natural language inference. Recently, these models have shown exceptional results in natural language understanding (NLU). Advanced attention‐based language models like BERT and XLNet excel at handling complex tasks across diverse contexts. However, they encounter difficulties when applied to specific domains. Platforms like Facebook, characterized by continually evolving casual and sophisticated language, demand meticulous context analysis even from human users. The literature has proposed numerous solutions using statistical and machine learning techniques to predict the sentiment (positive or negative) of online customer reviews, but most of them rely on various business, review, and reviewer features, which leads to generalizability issues. Furthermore, there have been very few studies investigating the effectiveness of state‐of‐the‐art pre‐trained language models for sentiment classification in reviews. Therefore, this study aims to assess the effectiveness of BERT, RoBERTa, ALBERT, DistilBERT, and XLNet in sentiment classification using the Yelp reviews dataset. The models were fine‐tuned, and the results obtained with the same hyperparameters are as follows: 98.30 for RoBERTa, 98.20 for XLNet, 97.40 for BERT, 97.20 for ALBERT, and 96.00 for DistilBERT. [ABSTRACT FROM AUTHOR]

This work addresses the gap in aspect extraction techniques for Portuguese by adapting methods originally designed for English. It focuses on TV devices and literary reviews in the TV and ReLi datasets. For this, models based on the BERT architecture were employed, including pre-trained general domain (BERTimbau) and specific domain models (BERTtv and BERTreli). Also, this paper contributes with a novel double embedding technique that merges these models. We further explored the potential of large language models (LLMs) with a Portuguese-trained LLaMa variant, Cabrita. Efficient fine-tuning techniques such as LoRA (low-rank adaptation) for BERTimbau and QLoRA (quantized low-rank adaptation) for Cabrita were applied to optimize resource demands. The BERTimbau model, adjusted with LoRA, achieved the highest F1 scores (0.846 for TV and 0.615 for ReLi), while Cabrita showed lower performance (0.68 for TV and 0.46 for ReLi). This study underscores the potential of adapting and optimizing existing techniques for aspect extraction in Portuguese, marking a significant advancement in the field [ABSTRACT FROM AUTHOR]

With the development of technology, the popularity of online medical treatment is becoming more and more widespread. However, the accuracy and credibility of online medical treatment are affected by model design and semantic understanding. In particular, there are still some problems in the accurate understanding of complex structured texts, which affects the accuracy of judging users' intentions and needs. Therefore, this paper proposes a new method for medical text parsing, which realizes core tasks such as named entity recognition, intention recognition, and slot filling through a multi-task learning framework; uses BERT to obtain contextual semantic information; and combines BiGRU and BiLSTM neural networks, and uses CRF to realize sequence annotation and DPCNN to realize classification prediction. Thus, the task of entity recognition and intention recognition can be accomplished. On this basis, this paper builds a multi-task learning model based on BiGRU-BiLSTM, and uses CBLUE and CMID databases to verify the method. The verification results show that the accuracy of named entity recognition and intention recognition reaches 86% and 89%, respectively, which improves the performance of various tasks. The ability of the model to process complex text is enhanced. This method can improve the text generalization ability and improve the accuracy of online medical intelligent dialogue when it is used to analyze medical texts. [ABSTRACT FROM AUTHOR]

This paper presents an approach to identifying political fake news in Spanish using Transformer architectures. Current methodologies often overlook political news due to the lack of quality datasets, especially in Spanish. To address this, we created a synthetic dataset of 57,231 Spanish political news articles, gathered via automated web scraping and enhanced with generative large language models. This dataset is used for fine-tuning and benchmarking Transformer models like BERT and RoBERTa for fake news detection. Our fine-tuned models showed outstanding performance on this dataset, with accuracy ranging from 97.4% to 98.6%. However, testing with a smaller, independent hand-curated dataset, including statements from political leaders during Spain's July 2023 electoral debates, revealed a performance drop to 71%. Although this suggests that the model needs additional refinements to handle the complexity and variability of real-world political discourse, achieving over 70% accuracy seems a promising result in the under-explored domain of Spanish political fake news detection. [ABSTRACT FROM AUTHOR]

With the development of the Internet, services such as catering, beauty, accommodation, and entertainment can be reserved or consumed online. Therefore, consumers increasingly rely on online information to choose merchants, products, and services, with reviews becoming a crucial factor in their decision making. However, the authenticity of reviews is highly debated in the field of Internet-based process-of-life service consumption. In recent years, due to the rapid growth of these industries, the detection of fake reviews has gained increasing attention. Fake reviews seriously mislead customers and damage the authenticity of online reviews. Various fake review classifiers have been developed, taking into account the content of the reviews and the behavior involved in the reviews, such as rating, time, etc. However, there has been no research considering the credibility of reviewers and merchants as part of identifying fake reviews. In order to improve the accuracy of existing fake review classification and detection methods, this study utilizes a comment text processing module to model the content of reviews, utilizes a reviewer behavior processing module and a reviewed merchant behavior processing module to model consumer review behavior sequences that imply reviewer credibility and merchant review behavior sequences that imply merchant credibility, respectively, and finally merges the two features for fake review classification. The experimental results show that, compared to other models, the model proposed in this paper improves the classification performance by simultaneously modeling the content of reviews and the credibility of reviewers and merchants. [ABSTRACT FROM AUTHOR]

Phishing websites pose a significant threat to online security, necessitating robust detection mechanisms to safeguard users' sensitive information. This study explores the efficacy of various deep learning architectures for phishing website detection. Initially, traditional sequential models, including recurrent neural networks (RNN), long short-term memory (LSTM), and gated recurrent unit (GRU), achieve accuracies of 95%, 96%, and 96.5%, respectively, on a curated dataset. Building upon these results, hybrid architectures that combine the strengths of traditional sequential models with state-of-the-art language representation models, bidirectional encoder representations from transformers (BERT) and XLNet, are investigated. Combinations such as RNN with BERT, BERT with LSTM, BERT with GRU, RNN with XLNet, XLNet with LSTM, and XLNet with GRU are evaluated. Through experimentation, accuracies of 94.5%, 96.5%, 96.1%, 95.7%, 97.4%, and 97%, respectively, are achieved, demonstrating the effectiveness of hybrid deep learning architectures in enhancing phishing detection performance. These findings contribute to advancing the state-of-the-art in cybersecurity practices and underscore the importance of leveraging diverse model types to combat online threats effectively. [ABSTRACT FROM AUTHOR]

This paper proposes an approach to sentiment analysis of student evaluation feedback using transformer-based language models. The primary objective of this study is to conduct an in-depth analysis of sentiment expressed in student evaluation feedback, with a focus on introducing contextual understanding into the sentiment classification process. In this research, four different variants of transformer language models were assessed, namely multilingual bidirectional encoder representations from transformers (MBERT), IndoBERT, RoBERTa Indonesia, and generative pre-trained transformer (GPT-2 Indonesia). Additionally, we also compared the performance of transformer models with two traditional models, namely support vector machine (SVM) and Naive Bayes (NB). The evaluation was conducted using feedback data collected from the Evaluasi Dosen oleh Mahasiswa (EDOM) system at Riau University, which had been categorized as either positive or negative. The outcomes indicate that IndoBERT base uncased exhibits the highest performance, with precision, accuracy, and recall values of 0.858, 0.929, and 0.911, respectively. This observation highlights the effectiveness of transformer-based language models in sentiment analysis of student evaluation feedback and provides insights for improving educational assessment practices. [ABSTRACT FROM AUTHOR]

In communication, emotional information is crucial, yet its preservation in tweet embeddings remains a challenge. This study aims to address this gap by exploring three distinct methods for generating embedding vectors of tweets: word2vec models, pre-trained BERT models, and fine-tuned BERT models. We conducted an analysis to assess the degree to which emotional information is conserved in the resulting embedding vectors. Our findings indicate that the fine-tuned BERT model exhibits a higher level of preservation of emotional information compared to other methods. These results underscore the importance of utilizing advanced natural language processing techniques for preserving emotional context in text data, with potential implications for enhancing sentiment analysis and understanding human communication in social media contexts. [ABSTRACT FROM AUTHOR]

As a critical business activity in the aircraft life cycle, maintenance processes are highly complex and require multidisciplinary knowledge. Knowledge integration and representation oriented toward aircraft maintenance processes are necessary to improve work efficiency. Nonetheless, conventional approaches lack effective unified management, which obstructs domain knowledge sharing and ultimately impedes maintenance work. In this context, this paper proposes a knowledge representation framework based on the benefits of ontology, which formalizes multidisciplinary knowledge for aircraft maintenance processes. An ontology of aircraft maintenance processes is developed for knowledge conceptualization and reuse. On this basis, a domain knowledge extraction model based on the bidirectional encoder representation from transformers (BERT) is constructed to automatically extract entities and relationships related to maintenance processes. With a series of Semantic Web Rule Language (SWRL) rules, a knowledge reasoning method is proposed based on the aircraft maintenance process ontology to mine hidden knowledge. We evaluate the developed ontology and demonstrate the feasibility and usefulness of the proposed knowledge reasoning method in a case study. The results show that the proposed knowledge representation framework provides an effective knowledge formalization method for complex knowledge in aircraft maintenance processes to support work optimization. [ABSTRACT FROM AUTHOR]

The purpose of this article is to conceive an advanced decision support environment that assists a company in selecting the best service requiring several constraints (expert or machine). The proposed contribution integrates the recommendation in deep learning technique deployed in Cloud. The first step involves preprocessing data using deep learning techniques, particularly NMT and BERT encoders, while the second step uses the neural network to generate recommendations. The neural network's architecture includes two hidden layers consisting of 16 and 8 neurons, configured with the "ReLU" activation function, while the output layer uses the "Softmax" function. The experiments have been conducted on a dataset of 20, 000 services. Results demonstrate that migrating the DL-based approach to cloud computing significantly reduces response time and memory consumption by approximately 15% and 10% for classification tasks, and 50% and 30% for recommendation tasks, respectively. Compared to an earlier version of the proposed approach based on machine learning, the DL-approach improves recommendation accuracy by approximately 3%. However, the results in terms of response time and memory usage remain variable, suggesting that deep learning requires considerable computing resources. The proposed method was benchmarked using evaluation metrics such as accuracy, MAE, response time, and user satisfaction, demonstrating its practical superiority and applicability across various industries. [ABSTRACT FROM AUTHOR]

Elongated words such as "Wiiiiiin" or "allloooo" are common in oral communication and are often used to emphasize or exaggerate the hidden message of the root word. While elongated words are rarely found in written languages and dictionaries, they are prevalent in social media networks. Considering elongation in sentiment analysis can provide valuable insights into user sentiments. In this article, we analyze the impact of elongation on sentiment classification, along with an in-depth study of lexical forms of elongation. We propose a method to enhance sentiment classification accuracy by incorporating elongation-based features using BERT (bidirectional encoder representations from transformers) approaches. Experimental results conducted on Twitter data demonstrate that our model achieves an average accuracy of 87% through 10-fold cross-validation experiments. [ABSTRACT FROM AUTHOR]

Background: Relation extraction (RE) plays a crucial role in biomedical research as it is essential for uncovering complex semantic relationships between entities in textual data. Given the significance of RE in biomedical informatics and the increasing volume of literature, there is an urgent need for advanced computational models capable of accurately and efficiently extracting these relationships on a large scale. Results: This paper proposes a novel approach, SARE, combining ensemble learning Stacking and attention mechanisms to enhance the performance of biomedical relation extraction. By leveraging multiple pre-trained models, SARE demonstrates improved adaptability and robustness across diverse domains. The attention mechanisms enable the model to capture and utilize key information in the text more accurately. SARE achieved performance improvements of 4.8, 8.7, and 0.8 percentage points on the PPI, DDI, and ChemProt datasets, respectively, compared to the original BERT variant and the domain-specific PubMedBERT model. Conclusions: SARE offers a promising solution for improving the accuracy and efficiency of relation extraction tasks in biomedical research, facilitating advancements in biomedical informatics. The results suggest that combining ensemble learning with attention mechanisms is effective for extracting complex relationships from biomedical texts. Our code and data are publicly available at: https://github.com/GS233/Biomedical. [ABSTRACT FROM AUTHOR]

Drug‐target binding affinity (DTA) prediction is vital for drug repositioning. The accuracy and generalizability of DTA models remain a major challenge. Here, we develop a model composed of BERT‐Trans Block, Multi‐Trans Block, and DTI Learning modules, referred to as Molecular Representation Encoder‐based DTA prediction (MREDTA). MREDTA has three advantages: (1) extraction of both local and global molecular features simultaneously through skip connections; (2) improved sensitivity to molecular structures through the Multi‐Trans Block; (3) enhanced generalizability through the introduction of BERT. Compared with 12 advanced models, benchmark testing of KIBA and Davis datasets demonstrated optimal performance of MREDTA. In case study, we applied MREDTA to 2034 FDA‐approved drugs for treating non‐small‐cell lung cancer (NSCLC), all of which act on mutant EGFRT790M protein. The corresponding molecular docking results demonstrated the robustness of MREDTA. [ABSTRACT FROM AUTHOR]

Nowadays, it is very tough to differentiate between real news and fake news due to fast-growing social networks and technological progress. Manipulative news is defined as calculated misinformation with the aim of creating false beliefs. This kind of fake news is highly detrimental to society since it deepens political division and weakens trust in authorities and institutions. Therefore, the identification of fake news has emerged as a major field of research that seeks to validate content. The proposed model operates in two stages: First, TF-IDF is applied to an entire document to obtain its global features, and its spatial and temporal features are simultaneously obtained by employing Bidirectional Encoder Representations from Transformers and Bidirectional Long Short-Term Memory with a Gated Recurrent Unit. The Fast Learning Network efficiently classifies the extracted features. Comparative experiments were conducted on three easily and publicly obtainable large-scale datasets for the purposes of analyzing the efficiency of the approach proposed. The results also show how well the model performs compared with past methods of classification. [ABSTRACT FROM AUTHOR]

CRISPR/Cas9 has been applied to edit the genome of various organisms, but our understanding of editing outcomes at specific sites after Cas9-mediated DNA cleavage is still limited. Several deep learning-based methods have been proposed for repair outcome prediction; however, there is still room for improvement in terms of performance regarding frameshifts and model interpretability. Here, we present DeepIndel, an end-to-end multi-label regression model for predicting repair outcomes based on the BERT-base module. We demonstrate that our model outperforms existing methods in terms of accuracy and generalizability across various metrics. Furthermore, we utilized Deep SHAP to visualize the importance of nucleotides at various positions for DNA sequence and found that mononucleotides and trinucleotides in DNA sequences surrounding the cut site play a significant role in repair outcome prediction. [ABSTRACT FROM AUTHOR]

Pre-trained language models perform well in various natural language processing tasks. However, their large number of parameters poses significant challenges for edge devices with limited resources, greatly limiting their application in practical deployment. This paper introduces a simple and efficient method called Autocorrelation Matrix Knowledge Distillation (AMKD), aimed at improving the performance of smaller BERT models for specific tasks and making them more applicable in practical deployment scenarios. The AMKD method effectively captures the relationships between features using the autocorrelation matrix, enabling the student model to learn not only the performance of individual features from the teacher model but also the correlations among these features. Additionally, it addresses the issue of dimensional mismatch between the hidden states of the student and teacher models. Even in cases where the dimensions are smaller, AMKD retains the essential features from the teacher model, thereby minimizing information loss. Experimental results demonstrate that BERTTINY-AMKD outperforms traditional distillation methods and baseline models, achieving an average score of 83.6% on GLUE tasks. This represents a 4.1% improvement over BERTTINY-KD and exceeds the performance of BERT4-PKD and DistilBERT4 by 2.6% and 3.9%, respectively. Moreover, despite having only 13.3% of the parameters of BERTBASE, the BERTTINY-AMKD model retains over 96.3% of the performance of the teacher model, BERTBASE. [ABSTRACT FROM AUTHOR]

Microblogs and social media sites have gained a central place and people use these platforms to express their opinions, sentiments, and thoughts about products, news, events, blogs, etc. Sentiment analysis is the process of exploring opinions and sentiments in user reviews and tweets. This area is still in its early developmental phase and requires imperative improvements on various issues. One of the main issues is multilingual tweets and reviews. Earlier sentiment analysis techniques only classified the text of a specific language, i.e., English, Turkish, etc. The accuracy of these techniques decreases in the presence of multilingual text. Existing methods are domain oriented. Using BERT and a lexicon, we propose a method for sorting out multilingual text and improving the polarity calculation of reviews. Experimental results reveal that our proposed technique achieved 90.14% accuracy and outperformed existing aspect-based sentiment analysis techniques. [ABSTRACT FROM AUTHOR]

This paper presents a mathematical analysis of semantic convergence in transformer-based language models, drawing inspiration from the concept of fractal self-similarity. We introduce and prove a novel theorem characterizing the gradient of embedding similarity across layers. Specifically, we establish that there exists a monotonically increasing function that provides a lower bound on the rate at which the average cosine similarity between token embeddings at consecutive layers and the final layer increases. This establishes a fundamental property: semantic alignment of token representations consistently increases through the network, exhibiting a pattern of progressive refinement, analogous to fractal self-similarity. The key challenge addressed is the quantification and generalization of semantic convergence across diverse model architectures and input contexts. To validate our findings, we conduct experiments on BERT and DistilBERT models, analyzing embedding similarities for diverse input types. While our experiments are limited to these models, we empirically demonstrate consistent semantic convergence within these architectures. Quantitatively, we find that the average rates of semantic convergence are approximately 0.0826 for BERT and 0.1855 for DistilBERT. We observe that the rate of convergence varies based on token frequency and model depth, with rare words showing slightly higher similarities (differences of approximately 0.0167 for BERT and 0.0120 for DistilBERT). This work advances our understanding of transformer models' internal mechanisms and provides a mathematical framework for comparing and optimizing model architectures. [ABSTRACT FROM AUTHOR]

Simple Summary: Metagenomic binning is a part of the metagenomic analysis process that can classify the genomes in complex microbial communities into possible source species. It is dedicated to restoring more complete genomes and is conducive to studying microbial community structure and function. Obtaining more effective feature representations of genome sequences to improve binning is still a major challenge. This study proposes a new metagenomic binning method called CedtBin, which uses an improved BERT model to obtain the embedded representation of contigs and splices it with the decomposed tetranucleotide frequencies into a new feature representation. Then, the Annoy-DBSCAN clustering algorithm is proposed, which can adaptively determine the parameters of the DBSCAN clustering algorithm for binning. The results show that CedtBin can achieve good binning effects on both simulated and real datasets, and can reconstruct more genomes. Metagenomic binning is a crucial step in metagenomic research. It can aggregate the genome sequences belonging to the same microbial species into independent bins. Most existing methods ignore the semantic information of contigs and lack effective processing of tetranucleotide frequency, resulting in insufficient and complex feature information extracted for binning and poor binning results. To address the above problems, we propose CedtBin, a metagenomic binning method based on contig embedding and decomposed tetranucleotide frequency. First, the improved BERT model is used to learn the contigs to obtain their embedding representation. Secondly, the tetranucleotide frequencies are decomposed using a non-negative matrix factorization (NMF) algorithm. After that, the two features are spliced and input into the clustering algorithm for binning. Considering the sensitivity of the DBSCAN clustering algorithm to input parameters, in order to solve the drawbacks of manual parameter input, we also propose an Annoy-DBSCAN algorithm that can adaptively determine the parameters of the DBSCAN algorithm. This algorithm uses Approximate Nearest Neighbors Oh Yeah (Annoy) and combines it with a grid search strategy to find the optimal parameters of the DBSCAN algorithm. On simulated and real datasets, CedtBin achieves better binning results than mainstream methods and can reconstruct more genomes, indicating that the proposed method is effective. [ABSTRACT FROM AUTHOR]

With the increasing popularity of watching baseball videos, there is a growing desire among fans to enjoy the highlights of these videos. However, the extraction of the highlights from lengthy baseball videos faces a significant challenge due to its time-consuming and labor-intensive nature. To address this challenge, this paper proposes a novel mechanism, called Hierarchical Multimodal Transformer for Video query (HMTV). The proposed HMTV incorporates a two-phase involving Coarse-Grained clipping for candidate videos and Fine-Grained identification for highlights. In the Coarse-Grained phase, a pitching detection model is employed to extract relevant candidate videos from baseball videos, encompassing the features of pitch deliveries and pitching. In the Fine-Grained phase, Transformer encoder and pre-trained Bidirectional Encoder Representations from Transformers (BERT) are utilized to capture relationship features between frames of candidate videos and words from users’ questions, respectively. These relationship features are then fed into the Video Query (VideoQ) model, implemented by the Text Video Attention (TVA). The VideoQ model identifies the start and end positions of the highlights mentioned in the query within the candidate videos. Simulation results demonstrate that the proposed HMTV significantly improves accuracy of highlights identification in terms of precision, recall, and F1-score. [ABSTRACT FROM AUTHOR]

Patents are documents that help protect innovations in information technologies and grant special rights to the creator of these innovations for a certain period of time. While these rights give the patent owner the right to use the innovation commercially, they prevent others from using the innovation without permission. Radical innovations and ground-breaking technological advances are derived from technical information contained in existing patents. Using an automatic classification system, patents assigned to the technical class to which they belong can pave the way for researchers and provide an environment in which they can create new inventions. This study presents an automatic patent classification analysis using the BERT algorithm. Hyperparameter analyses are also preferred in this study in order to achieve more successful prediction accuracy in automatic patent classification problems. The obtained results were at a level that competed with those in the literature. An accuracy of 58% was achieved at the subclass level. [ABSTRACT FROM AUTHOR]

User identification is an essential technical support for downstream tasks such as recommendation systems, information retrieval, and collaborative filtering. Computing the similarity between user display names through classifiers is an effective solution for user identification across social networks. However, there are two problems with existing methods. Applying expert domain knowledge to extract handcrafted features of display names ignores the semantic information, resulting in poor performance of these methods. Selecting influential display names, and handcrafted features in user identification problems are also one of the difficulties. To solve these two problems, we propose a method based on the multi-feature fusion of display names using gated units. First, we extract the deep semantic features of display names through the BERT pre-trained multi-language model. Then, the gated mechanism is applied to select the handcrafted features we extracted to retain the essential features. Then, the adaptive factors are used to fuse handcrafted features and deep features to obtain user identification results across social networks. Finally, the efficiency of our model is verified on three constructed real-world multilingual display names datasets across multiple online social networks and compared with existing state-of-the-art methods. Experimental results show that the proposed algorithm outperforms the compared algorithms. [ABSTRACT FROM AUTHOR]

Learning-based data compression methods have gained significant attention in recent years. Although these methods achieve higher compression ratios compared to traditional techniques, their slow processing times make them less suitable for compressing large datasets, and they are generally more effective for short texts rather than longer ones. In this study, MLMCompress, a word-based text compression method that can utilize any BERT masked language model is introduced. The performance of MLMCompress is evaluated using four BERT models: two large models and two smaller models referred to as "tiny". The large models are used without training, while the smaller models are fine-tuned. The results indicate that MLMCompress, when using the best-performing model, achieved 3838% higher compression ratios for English text and 42% higher compression ratios for multilingual text compared to NNCP, another learning-based method. Although the method does not yield better results than GPTZip, which has been developed in recent years, it achieves comparable outcomes while being up to 35 times faster in the worst-case scenario. Additionally, it demonstrated a 20% improvement in compression speed and a 180% improvement in decompression speed in the best case. Furthermore, MLMCompress outperforms traditional compression methods like Gzip and specialized short text compression methods such as Smaz and Shoco, particularly in compressing short texts, even when using smaller models. [ABSTRACT FROM AUTHOR]

Lexicon Enhanced Bidirectional Encoder Representations from Transformers (LEBERT) has achieved great success in Chinese Named Entity Recognition (NER). LEBERT performs lexical enhancement with a Lexicon Adapter layer, which facilitates deep lexicon knowledge fusion at the lower layers of BERT. However, this method is likely to introduce noise words and does not consider the possible conflicts between words when fusing lexicon information. To address this issue, we advocate for a novel lexical enhancement method, Sequential Lexicon Enhanced BERT (SLEBERT) for the Chinese NER, which builds sequential lexicon to reduce noise words and resolve the problem of lexical conflict. Compared with LEBERT, it leverages the position encoding of sequential lexicon and adaptive attention mechanism of sequential lexicon to enhance the lexicon feature. Experiments on the four available datasets identified that SLEBERT outperforms other lexical enhancement models in performance and efficiency. [ABSTRACT FROM AUTHOR]

Conventional approaches fall short of managing the current scale of data, underscoring the significance of big data analytics in the field of data science. The study highlights the importance of big data analytics in data science, focusing on sentiment analysis and developing a machine learning framework for identifying sentiments in product reviews and assessing Massive Open Online Courses (MOOCs). Online education proved its value to the academic community during the COVID-19 pandemic, when much of the world was at a standstill. Since the coronavirus outbreak, Massive Open Online Courses (MOOCs) have taken over many public schools and undergraduate degree programs as the new norm. Nowadays, online courses are being incorporated into traditional educational programs. Therefore, it has become crucial to develop a framework that both government and nongovernment organizations could use to assess public opinion before implementing online education programs. This article offers a methodology for utilizing deep learning and natural language processing techniques to examine public sentiments toward online education and courses such as MOOCs. Public reviews from Coursera and Udemy are utilized to assess the quality of online courses. Using sentiment analysis, the proposed methodology employs the Bidirectional Encoder Representations from Transformers (BERT) model to classify public reviews into positive and negative categories. Positive and negative reviews are clustered using the k-means algorithm to evaluate the quality of MOOCs. The study's findings indicate that MOOCs with qualified professors are beneficial for learning basic concepts. But they may lack advanced knowledge, real-world experience, and enough examples. These courses are lengthy, and students request more questions after lessons to comprehend the concept. The proposed framework assists in assessing public opinion before implementing online education and may provide valuable insight to the developers of the courses. [ABSTRACT FROM AUTHOR]

Crowdsourcing Software Development (CSD) platforms, i.e., TopCoder, function as intermediaries connecting clients with developers. Despite employing systematic methodologies, these platforms frequently encounter high task abandonment rates, with approximately 19% of projects failing to meet satisfactory outcomes. Although existing research has focused on task scheduling, developer recommendations, and reward mechanisms, there has been insufficient attention to the support of platform moderators, or copilots, who are essential to project success. A critical responsibility of copilots is estimating project duration; however, manual predictions often lead to inconsistencies and delays. This paper introduces an innovative machine learning approach designed to automate the prediction of project duration on CSD platforms. Utilizing historical data from TopCoder, the proposed method extracts pertinent project attributes and preprocesses textual data through Natural Language Processing (NLP). Bidirectional Encoder Representations from Transformers (BERT) are employed to convert textual information into vectors, which are then analyzed using various machine learning algorithms. Zero-shot learning algorithms exhibit superior performance, with an average accuracy of 92.76%, precision of 92.76%, recall of 99.33%, and an f-measure of 95.93%. The implementation of the proposed automated duration prediction model is crucial for enhancing the success rate of crowdsourcing projects, optimizing resource allocation, managing budgets effectively, and improving stakeholder satisfaction. [ABSTRACT FROM AUTHOR]

Identifying and categorizing proper nouns in text, known as named entity recognition (NER), is crucial for various natural language processing tasks. However, developing effective NER techniques for low-resource languages like Urdu poses challenges due to limited training data, particularly in the nastaliq script. To address this, our study introduces a novel data augmentation method, "contextual word embeddings augmentation" (CWEA), for Urdu, aiming to enrich existing datasets. The extended dataset, comprising 160,132 tokens and 114,912 labeled entities, significantly enhances the coverage of named entities compared to previous datasets. We evaluated several transformer models on this augmented dataset, including BERT-multilingual, RoBERTa-Urdu-small, BERT-base-cased, and BERT-large-cased. Notably, the BERT-multilingual model outperformed others, achieving the highest macro F1 score of 0.982%. This surpassed the macro f1 scores of the RoBERTa-Urdu-small (0.884%), BERT-large-cased (0.916%), and BERT-base-cased (0.908%) models. Additionally, our neural network model achieved a micro F1 score of 96%, while the RNN model achieved 97% and the BiLSTM model achieved a macro F1 score of 96% on augmented data. Our findings underscore the efficacy of data augmentation techniques in enhancing NER performance for low-resource languages like Urdu. [ABSTRACT FROM AUTHOR]