Search

As intelligent education advances and online learning becomes more prevalent, Knowledge Tracing (KT) has become increasingly important. KT assesses students’ learning progress by analysing their historical performance in related exercises. Despite significant advances in the field, there are still shortcomings in two aspects: first, a lack of effective integration between exercises and knowledge points; second, an overemphasis on nodal information, neglecting deep semantic relationships. To address these, we propose a self-supervised learning approach that uses an enhanced heterogeneous graph attention network to represent and analyse complex relationships between exercises and knowledge points. We introduce an innovative surrogate view generation method to optimise the integration of local structural information and global semantics within the graph, addressing relational inductive bias. In addition, we incorporate the improved representation algorithm into the loss function to handle data sparsity, thereby improving prediction accuracy. Experiments on three real-world datasets show that our model outperforms baseline models.

In the realm of Visual Question Answering, accurate answers often hinge on the harmonious fusion of textual and visual elements. While these complex architectures are effective, they typically come with a hefty price tag: a large number of parameters that demand significant processing power and lengthy training times. In contrast, traditional Dual-stream approaches prioritize accuracy above all else, neglecting the memory requirements of GPU processing and training time. This paper presents a novel Dual-stream architecture for VQA, whose parameters have been rigorously tested and evaluated not only for performance, but also for GPU memory consumption and training time. The results show that it’s possible to achieve competitive performance while significantly reducing the computational burden typically associated with complex VQA models.

Time series forecasting plays a crucial role in various industries, particularly in predicting product demand for effective supply chain management. This paper presents a novel approach to time series forecasting by leveraging advanced deep learning techniques, specifically focusing on hybrid models that combine attention mechanisms with traditional recurrent neural networks. Our proposed method, the Hybrid Attention-based Long Short-Term Memory (HA-LSTM) network, integrates multi-head self-attention modules with LSTM layers to capture both long-term dependencies and local temporal patterns in time series data. We evaluate our model on a large-scale retail dataset from the “Predict Future Sales” competition, demonstrating its effectiveness in handling complex, real-world time series with multiple seasonalities and trends. Experimental results show that the HA-LSTM outperforms state-of-the-art baselines, including ARIMA, Prophet, and vanilla LSTM models, achieving a 15% improvement in Mean Absolute Percentage Error (MAPE) and a 12% reduction in Root Mean Square Error (RMSE). Furthermore, we conduct an extensive ablation study to analyze the contribution of each component in our hybrid model. The proposed approach not only enhances forecasting accuracy but also provides interpretable attention weights, offering insights into the relative importance of different time steps in making predictions. This research contributes to the growing body of work on deep learning for time series analysis and offers practical implications for improving demand forecasting in retail and supply chain management.

With the rise of digital payment methods and the growth in financial transactions, the issue of credit card fraud has become increasingly severe. Traditional fraud detection methods are currently facing challenges such as poor model performance, difficulty in obtaining accurate results, and limitations in distributed deployment. These challenges stem from constantly evolving fraud strategies, higher volumes of transactions, and the complexity of the financial environment. This study proposes a credit card fraud detection algorithm based on Structured Data Transformer (SDT) and federated learning, which leverages the advanced capabilities of the Transformer model in deep learning. First, we organize credit card data into sequences and introduce a special, learnable token at the beginning of each sequence for classification purposes. Thanks to the attention mechanism of the Transformer, the model can automatically highlight important features in the data, significantly improving the accuracy of fraud detection. Second, addressing the complex financial environment and concerns about financial data privacy, we introduce a federated learning architecture to deploy the SDT model across different banks in a distributed manner. Momentum updates are used for model parameter updates during training, which enhance model performance and ensure data privacy between banks. Lastly, we conducted experimental validation on two financial datasets of different scales. The results on Dataset 1 and Dataset 2 show that our proposed SDT model surpasses traditional detection methods in terms of AUC-PR values (0.882, 0.816) and AUC-ROC values (0.982, 0.994). By integrating federated learning and deploying and testing the two datasets in a distributed environment, the AUC-PR values (0.884, 0.892) and AUC-ROC values (0.963, 0.998) can be further improved.

Light Field (LF) processing is a critical step in advancing immersive experiences beyond two-dimensional (2D) images. While LF-based technology facilitates a smooth transition from 2D to three-dimensional (3D) media, processing LFs composed of numerous images remains challenging. Most existing research has provided only limited processing capabilities for low-resolution LFs. For instance, LF editing typically offers basic functions like color changes and object removal. This paper proposes micro-cell-based processing for spatially and angularly large-scale LFs. By partitioning and rearranging the LF into micro-cells, which are then used as input for a vision transformer (ViT), we enable spatio-angular joint attention that significantly enhances performance. The local attention applied to small-sized micro-cells allows for high-resolution LF processing without excessive computational costs. This paper utilizes this combination of micro-cells and the ViT for LF editing applications. Experimental results demonstrate that the proposed LF editing system delivers exceptional performance both quantitatively and qualitatively, enabling high-resolution LF processing with a variety of editing functions.

The rapid increase in telecommunication fraud has led to substantial financial losses. A critical challenge in detecting such fraud based on call content lies in accurately modeling and interpreting the dual-role interactions between fraudsters and victims, often manifested in complex dialog structures. To address this issue, we propose the Hierarchical Dual-Role Interaction Network (HDRIN), a novel approach for telecom fraud detection. Our method categorizes raw text into two types: two-person dialog text and single text. For two-person conversational texts, we introduce DialogBert, which performs word embedding transformations followed by local feature extraction using 1DCNN. For single texts, we develop the Multi-Level Dynamic TextRank (MDTR) algorithm to extract concise summaries from lengthy single dialogs, while Hierarchical Attention Networks (HAN) are employed to enhance global features. Additionally, a synergistic attention mechanism is proposed to integrate features from both text types. We constructed the ZhCCVi dataset, using real Chinese call speech from platforms like Jitterbug and YouTube, to evaluate our approach. Extensive experiments on ZhCCVi and the publicly available KoCCVi2 dataset demonstrate that our approach achieves an accuracy of 99.88% and an F1 score of 99.88%, surpassing the performance of existing state-of-the-art models. The code is available at https://github.com/Gaopayvin/HDRIN.

Existing traffic sign detection algorithms suffer from high computational complexity and large parameter sizes, limiting their deployability. The YOLO-TS model integrates the Normalized Wasserstein Distance (NWD) with the Complete Intersection over Union (CIoU) loss function, significantly enhancing the detection of small traffic signs. The integration of StarBlock from StarNet into the C2f module forms the C2f-Star configuration, which simplifies the architecture. In addition, the Slimneck design paradigm is introduced into the neck network to further reduce computational demands while maintaining model accuracy. Subsequently, a Squeeze and Excitations Shared Detection Head (SESDH) is developed to integrate a squeezing and excitation attention mechanism. This design helps diminish the intricacy of the network architecture, increase attention on areas where traffic signs are presence and improve the ability of the model to indicate objects in complex surroundings. Experimental results on the CCTSDB traffic sign dataset reveal that the updated algorithm decreased the number of parameters by 35.33%, reduced the computation by 35.80% and shrunk the model size by 32.94% of the baseline YOLOv8n model while improving the mAP@50 by 1%, and the F1 score by 1.32%. Compared to other algorithms, here, a good balance between accuracy and lightweight design is achieved.

Scientific and accurate prediction of the consumption of equipment maintenance spare parts is crucial to improve the efficiency of equipment failure disposal and enhance the competitiveness of enterprises. However, the consumption of some maintenance spare parts is usually affected by a variety of factors, with strong randomness and uncertainty, and it is difficult to accurately predict them. In this paper, a multi-level migration learning CNN-ISE-Attention-BiLSTM prediction model for maintenance spare parts is designed through the study of neural networks, time series and attention mechanisms. Through several sets of experiments, it is proved that the model can better solve the problems of insufficient mining of key information due to insufficient data samples of maintenance spare parts consumption and insufficient mining of spare parts consumption law by ordinary neural networks, and better achieve the prediction of non-stationary maintenance spare parts, which is of great significance for the implementation of the scientific management of maintenance spare parts in enterprises.

O2O (Online to Offline) can analyze users’ behaviors according to data mining, realize personalized marketing and improve marketing effect. The “push” delivery method of O2O coupons ignores the active participation and user experience of users, and the pertinence and effectiveness of delivery are greatly affected. Aiming at the simple structure of a single network and mainly relying on artificial construction to extract features, in order to improve the utilization efficiency of deep-seated features in the model and effectively extract multi-level features, this paper introduces Bi-GRU according to the time-series characteristics of O2O consumption behavior, and proposes a new multi-grained attention mechanism. First, build a multi-dimensional consumer behavior feature project; Secondly, using convolutional neural network (CNN) and Bi-GRU to extract local and global features; Finally, multi-level and multi-grained information is extracted by using multi-grained attention to avoid the loss of hierarchical structure information, enrich feature vectors and further improve model performance. Using real O2O coupons and data sets, the CB-MA model proposed in this paper achieves 93.29% accuracy, 91.72% AUC and 0.0332 Loss. The results show that multi-grained attention mechanism can extract multi-level features more effectively than traditional attention mechanism. At the same time, CNN and Bi-GRU are combined to learn local and global features at the same time, and the correlation information of time and space is extracted.

Current research on personal protective equipment (PPE) detection has mainly focused on hard hats, overlooking the detection of reflective clothing. Therefore, this study aims to address this research gap comprehensively. We achieve this by creating a novel dataset using semi-automatic labeling techniques and enhancing the YOLOv5 model. The dataset consists of four categories, assessing the presence of hard hats and reflective clothing. Additionally, we introduce an attention mechanism and an improved loss function to tackle challenges related to detecting reflective clothing and overlapping detection frames. Through extensive multi-model comparison experiments, our improved model, AL-YOLOv5, outperforms the baseline model with remarkable advancements of 0.9 AP in the data-limited category and 0.4 mAP overall. Notably, our improved model shows substantial progress in detecting reflective clothing, significantly reducing false detections, and improving overlapping bounding frames. In conclusion, this study contributes to PPE detection accuracy through a novel dataset and an improved model.

The recommendation accuracy of traditional product recommendation systems is insufficient. Therefore, an improved deep factor decomposition machine algorithm combining adaptive regularization and attention mechanisms is proposed, and big data components are integrated to enable the algorithm to support more data input types. The publicly available datasets Criteo and Avazu from the Kaggle competition were selected for testing experiments in the study. The experimental results are as follows. During the training phase, after the convergence of each model, the loss function value of the model designed in this study is 1.26, which is the lowest among all comparison models. Moreover, when the number of recommended products is 7, the overall recommendation effect of each model is the best. The area under the curve of the subject operation characteristic curve of the model designed in this study on the Criteo dataset is 0.809, which is significantly higher than all comparison models. It is believed that this model has higher recommendation accuracy and can be used in application scenarios that require higher recommendation quality.

Distracted driving, a leading cause of traffic accidents with severe consequences, still faces numerous technical challenges in practical implementation for recognizing unsafe driving behavior. These challenges include the complexity of feature extraction using traditional convolutional neural networks (CNNs) for driver behavior analysis and the lack of real-time perception during driving. To address these issues, this study proposes an improved method for distracted driving behavior recognition by combining the Bi-LSTM model with an attention mechanism based on Dilated Convolutional Neural Networks (ID-CNN). Firstly, we employ a dilated convolution model to extract features efficiently with fewer parameters while enhancing multi-scale feature extraction capabilities and widening the receptive field. Subsequently, we integrate the attention mechanism into the Bi-LSTM model to enhance its effectiveness in solving the driving behavior classification problem. The integrated Bi-LSTM model with attention mechanism calculates correlation between intermediate and final states to obtain a probability distribution of attention weights at each moment, thereby reducing information redundancy while preserving useful information effectively. Furthermore, image feature vectors are enhanced to further improve accuracy in image classification tasks. Compared to other methods, the proposed approach exhibits faster convergence rates and more stable model accuracy. Specifically, on both the StateFarm dataset and our own collected Drive&Act-Distracted data, we achieved accuracies of 95.8367% and 97.8911%, respectively. This indicates that incorporating dilated convolution and attention mechanisms strengthens sequence data learning and feature weighting within our network model, resulting in significantly improved accuracy for driving behavior recognition.

Multimodal Sentiment Analysis (MSA) is gaining attention, but faces two main challenges: efficient extraction of cross-modal features without redundancy and removing spurious correlations between sentiment labels and multimodal features. In this paper, we propose a novel multimodal learning debiasing model, named Bilateral Cross-modal Debias Multimodal sentiment analysis Model (BCD-MM), to address these issues. Specifically, BCD-MM ultimately enhances the generalisation of the model to out-of-distribution (OOD) situations by improving the ability of cross-modal low-redundancy feature extraction and reducing the reliance on non-causal correlations. First, BCD-MM utilizes an attention score-based method to preserve critical information and eliminate redundancy within modalities. It also employs a gated crossmodal attention mechanism to filter inconsistencies through modal interaction, thereby enhancing the extraction of cross-modal specific features. Second, BCD-MM incorporates a debiasing approach with double bias extraction, using a Tanh-based Mean Absolute Error (TMAE) loss function and inverse probability weighting to mitigate spurious correlations. Finally, extensive testing on three public datasets (MOSI, MOSEI, and SIMS) and two OOD datasets (OOD MOSI and OOD MOSEI) demonstrates our model’s effectiveness in both MSA and debiasing tasks.

Real-world applications of word embeddings to downstream clustering tasks may experience limitations to performance, due to the high degree of dimensionality of the embeddings. In particular, clustering algorithms do not scale well when applied to highly dimensional data. One method to address this is through the use of dimensionality reduction algorithms (DRA). Current state of the art algorithms for dimensionality reduction (DR) have been demonstrated to contribute to improvements in clustering accuracy and performance. However, the impact that a neural network architecture can have on the current state of the art Parametric Uniform Manifold Approximation and Projection (UMAP) algorithm is yet unexplored. This work investigates, for the first time, the effects of using attention mechanisms in neural networks for Parametric UMAP, through the application of network architectures that have had considerable effect upon the wider machine learning and natural language processing (NLP) fields - namely, the transformer-encoder, and the bidirectional recurrent neural network. We implement these architectures within a semi-supervised metric learning pipeline, with results demonstrating an improvement in the clustering accuracy, compared to conventional DRA techniques, on three out of four datasets, and comparable SoA accuracy on the fourth. To further support our analysis, we also investigate the effects of the transformer-encoder metric-learning pipeline upon the individual class accuracy of downstream clustering, for highly imbalanced datasets. Our analyses indicate that the proposed pipeline with transformer-encoder for parametric UMAP confers a significantly measurable benefit to the accuracy of underrepresented classes.

During online communication, users often use irregular and ambiguous words, and sometimes use irony to express sarcasm. These words are difficult to analyze through text analysis, which poses a significant challenge for text sentiment analysis. As a novel communication method, emojis have a significant correlation with user emotions. In this paper, we use emojis to analyze the sentiment of short texts. Firstly, we validate that user information can help reduce the uncertainty of some emojis and use this information to identify the polarity of emojis. Then, we generate emoji representations by merging positional information, semantic information, emotional information, and frequency of appearance. Furthermore, we propose TaneNet, a two-level attention network based on emojis, which combines clause vectors and emoji representations to study the impact of emojis on the emotions of each clause in the text. Empirical results on two real-world datasets demonstrate that TaneNet outperforms existing state-of-the-art methods.

Pavement distress detection is crucial in road health assessment and monitoring. However, there are still some challenges in extracting pavement distress based on deep learning: such as insufficient segmentation, extraction errors and discontinuities. In this paper, we propose DARNet, a network for pavement distress extraction. A Distress Aware Attention Module (DAAM) is proposed to solve the problem of discontinuity in distress extraction due to inaccurate recovery of distress pixels during upsampling. Based on the characteristics of distress morphology, a Refinement Extraction Module (REM) is designed to effectively capture horizontal and vertical pavement damage features by fusing high-level and low-level features, which improves the accuracy of the model in extracting details of pavement damage information. Finally, a Weighted Cross-Entropy Loss function (WCEL) is introduced to assign weights according to the distance of the pixel point to the boundary of the distress, which solves the problem that the traditional cross entropy function treats each pixel point equally. We also propose a set of pavement distress datasets LNTU_RDD_GS, and the experimental results show that DARNet can reach 82.68% mIoU and 90.13% F score in the datasets in this paper, 80.63% mIoU and 88.35% F score in the four public datasets.

To meet the needs of embedded devices for model lightweight and high-precision recognition, this paper proposes a lightweight YOLO-DLHS-P model for pig behavior recognition based on the improved YOLOv8n model. Firstly, the C2f-DRB structure is introduced at the Backbone position, and the sizeable convolutional kernel is used to extend the receptive field to enhance the spatial perception ability of the model, and to enhance the network’s ability to capture spatial information while maintaining the number of learnable parameters and computational efficiency; The LSKA attention mechanism is then introduced to be integrated into the SPPF module to construct the SPPF-LSKA structure, which significantly improves the ability of the SPPF module to aggregate features at multiple scales; Then, the downsampling at the Neck position is optimised to the HWD algorithm, which reduces the spatial resolution of the feature map while retaining more useful information and reduces the uncertainty of the information compared with the downsampling method of the baseline model; finally, the Shape-IoU is used to replace the original CIoU, which significantly improves the detection efficiency and accuracy of the model without increasing the extra computational burden. After constructing the improved YOLO-DLHS model, the improved model is then pruned using the LAMP pruning scoring algorithm to obtain a lightweight YOLO-DLHS-P model. The experimental results show that the YOLO-DLHS model improves P, mAP@0.5, and mAP@0.5-0.95 by 4.39%, 1.68%, and 3.97%, respectively, compared to the YOLOv8n model. The YOLO-DLHS-P model improves P, mAP@0.5, and mAP@0.5-0.95 by 3.37%, 1.16%, and 2.11%, and the number of parameters, computation, and model occupancy are substantially reduced by 52.49%, 54.32%, and 49.33%, respectively. Moreover, the FPS of the YOLO-DLHS-P model reaches 79 frames, which has good real-time performance for pig behavior recognition. Therefore, the improved YOLO-DLHS-P in this paper is able to reduce the demand for hardware at the time of deployment under the premise of guaranteed accuracy and provides a lightweight behavioral recognition solution for the intelligent farming of captive pigs.

Manually checking for defects on industrial parts such as steel surfaces is ineffective, error-prone, and can damage a company’s reputation. Current automated methods often lack accuracy or real-time detection capabilities. Early detection allows for timely corrective action, such as removing defective parts or adjusting production parameters which can streamline the manufacturing process and improve brand reputation, customer satisfaction, and compliance with industry standards. This paper presents MoL-YOLOv7 (MobileNet integrated with attention and loss function to You Only Look Once version 7), a deep learning model for accurate and real-time detection of steel defects. MoL-YOLOv7 modifies the YOLOv7 model by inserting a MobileNet block that reduces computational complexity while maintaining accuracy, allowing for faster detection. Adding the SimAm (Simple Parameter free Attention module) attention mechanism to the MobileNet block refines feature representations for complex tasks such as steel defect detection. Finally, replacing loss functions with EIoU (Effective IoU), WIoU (Wise-IoU), and SIoU (Scylla-IoU) improves the localization accuracy and addresses the class imbalance in the data. The modified model achieves high accuracy and real-time detection, enabling a streamlined defect detection process. Experimental results show that the modified model involving different loss functions used in this work achieves high accuracy, i.e. 0.5% to 3.5% higher than the original model YOLOv7. The superiority and validity of our modified model are demonstrated by comparison with other attention mechanisms and loss functions integrated into YOLOv7, and also on different texture datasets, putting forward a modified method to detect surface defects on steel strips in daily operations.

This paper introduces the Multi-Dimensional Spatio-Temporal Fusion Transformer (MDSTFT), a state-of-the-art deep learning framework designed to enhance multi-variate time series forecasting. The MDSTFT diverges from traditional models by integrating Convolutional Neural Networks (CNNs), Attention-Enhanced Long Short-Term Memory (LSTM) networks, and advanced Transformer models into a cohesive architecture. This framework intricately captures the spatial-temporal dynamics in complex datasets, significantly improving forecasting accuracy. At its core, MDSTFT employs a multi-dimensional spatio-temporal encoder to meticulously unravel spatial correlations among variables, followed by the dynamic processing of temporal patterns through LSTM networks augmented with a sophisticated attention mechanism. This dual-phase processing is intricately designed to capture the essence of time series data, which is then refined through our novel Fusion Transformer Layers. These layers, incorporating advanced self-attention mechanisms, elevate the model’s capacity to discern long-range dependencies and subtle temporal nuances, thus offering a profound understanding of temporal sequences. Evaluated on the “Bank Central Asia Stock Historical Price” dataset, MDSTFT demonstrates unparalleled forecasting accuracy, significantly outperforming both traditional models and contemporary deep learning approaches. This empirical validation not only highlights MDSTFT’s efficacy in financial market forecasting but also its broader applicability across various domains requiring nuanced multi-variate time series analysis. Through integrating multiple deep learning paradigms, MDSTFT not only advances the field of time series forecasting but also sets a new benchmark for predictive analytics, promising enhanced decision-making capabilities in finance, environmental studies, supply chain management, and beyond.

Arabic language processing presents significant challenges due to its complex linguistic patterns and shortage of resources. This study describes MSG-ATS, a new technique to abstractive text summarization in Arabic that aims to overcome these issues. The key challenge is producing coherent and high-quality summaries given the Arabic language’s rich syntactic, semantic, and contextual elements. Traditional approaches, such as word2vec, frequently fail to capture these subtleties well. MSG-ATS uses multilevel semantic graphs and deep learning techniques to create a more thorough representation of Arabic text. This approach improves traditional text generation and embedding approaches by collecting syntactic, semantic, and contextual information fully. MSG-ATS uses a deep neural network to create high-quality summaries that are coherent and contextually appropriate. To verify MSG-ATS, we performed rigorous assessments that compared its performance to word2vec, a fundamental word embedding approach. These assessments employed a unique dataset created expressly for this study and included automated assessment using the ROUGE measure. The results are compelling: MSG-ATS outperformed the baseline model by 42.4% in precision, 23.8% in recall, and 38.3% overall. The outcomes of this study highlight MSG-ATS’s potential to considerably increase Arabic text summarization by providing a strong framework that solves the constraints of existing models while also laying the groundwork for future developments in the area.

Various tasks in robotics such as pick and place and catching flying/rolling objects have been studied in the literature. Previously, to accomplish such tasks, it was necessary to detect the position of the object using a Sobel detector, a marker, or a stereo method and then predict the trajectory of the object through the model-based Kalman filter. However, these existing studies are not practical, since with this detection method, only one type of object can be detected or additional equipments are required. In addition, to compute the Kalman filter, a measurement of the object’s position is essentially required, which may not be precise in various situations due to unmodeled noise. In this paper, we study the new framework of catching a rolling ball task in robotics using only machine learning techniques. Unlike previous approaches that rely on specified markers or stereo camera systems, our method uses a machine learning algorithm that can learn object positions and detect various sizes of balls using only one RGB camera without any markers. In our method, Convolutional Neural Network (CNN)-based models are applied to detect objects and the transformer model with an attention mechanism is applied for end-to-end trajectory prediction. We use the robotics simulator to efficiently train models and evaluate their performance directly in the real world. The experimental results of catching a rolling ball show that our framework is practical, and performs well in various sizes of balls. By using the proposed framework, the performance of the $Gripper \, vicinity$ is 93.3% and the Catching success rate is 73.3%. In contrast, other baselines, such as CNN and long short-term memory (LSTM), show poor Gripper vicinity and success rates, with all criteria falling below 30%.

This paper introduces OVANet, a new deep learning framework for the diagnosis and classification of ovarian cancer subtypes from histopathological images. OVANet has been designed by integrating modified VGG19 and InceptionV3 blocks, incorporating dual attention mechanism (Squeeze-and-Excitation and custom spatial attention) to capture rich, multi-scale, and context-aware feature representations. This unique architecture combines intermediate layers from both base models, along with additional convolutional and max-pooling layers, dual attention mechanisms and global average pooling to extract diverse image features crucial for accurate subtype differentiation. OVANet is trained and evaluated on an original dataset of 508 histopathological images, augmented to approximately 13,024 images through preprocessing and augmentation techniques like rotations, shifts, zooms, and flips. This approach significantly enhances the generalization capabilities of the model, enabling it to achieve superior performance compared to well-known pretrained models and existing research outputs. The propose model shows accuracy, precision and AUC of 99.01% along with a recall of 99.87%. Using feature map visualization of all layers of the model and a model explainibility technique GRADCam, visual representation of classification have been shown which will help to establish transparency in clinical aspect. This architecture can be considered as a significant advancement in medical imaging, especially for the diagnosis and treatment planning of ovarian cancer. To reduce the death rate of cancer, proper detection is very essential and this model exhibits so promising results that can easily implemented clinically.

The increasing popularity of ride-hailing services has led to a growing need for optimal utilization of resources and minimization of passenger wait times. A major prerequisite for this goal is to develop accurate taxi demand prediction models. In this study, we propose a novel approach for district-centric online taxi demand prediction. We first used historical data and Pearson coefficient to identify correlated districts. In order to dynamically weight the importance of related districts, we utilized an architecture which is based on attention mechanism. Next, for demand prediction in each district we used an LSTM network and finally to refine the prediction models we used the Gradient Boosting algorithm. To evaluate the effectiveness of our approach, we leverage the real data of Snapp!, a ride-hailing service in Iran. Results confirm that the proposed model outperforms the state-of-the-art methods in terms of prediction error.

Deep learning-based image processing methods for medical brain tumors are current research hotspots in this field. However, a great deal of research has focused on how to classify and segment brain tumors, while relatively little research has been done on brain tumor detection. This is mainly because brain tumor images are often filled with indistinguishable lesions, which can easily lead to false positives or missed diagnoses, and the YOLO detection algorithm has attracted a lot of attention due to its efficient real-time target detection capability. Based on the YOLO framework, we created a new neural network to accurately identify lesion regions in brain tumor medical images.The core of the approach is to propose a feature extraction network based on reparameterized heterogeneous convolution of large kernels(RGNet), and a incorporating an attention grather and distribute strategy(GDB). RGNet can better cope with significant changes in scale and different contextual feature textures in brain tumor detection, and GDB aggregates high-level and low-level semantic features and spatial details to overcome the information loss problem of the traditional feature pyramid module during feature fusion. information loss problem, and utilizes structured convolution module and channel mixing technique to improve the refinement of multi-dimensional details and enrich the semantic information during the whole feature fusion process. The experimental results show that compared with other object detection models, our model achieves 95.4% precision, 93.9% recall, 96.9% mAP50, and 74.8% mAP50:95 on the Br35H dataset,and 47.1% precision, 86.1% recall, 54.0% mAP50, and 38.7% mAP50:95 on the Ultralytics brain tumor dataset. These numbers not only highlight the effectiveness of the proposed model in brain tumor detection, but also provide new ideas for the subsequent application of object detection in medical imaging and clinical disease diagnosis.

This study introduces an innovative air quality prediction model, TD-CS-Transformer, which fuses time series decomposition and convolutional sparse self-attention Transformer model. It solves inefficient and limited long-distance dependency capture of traditional models in long sequence data. The model simplifies the structure and reduces the amount of computation by decomposing the series into components. Convolutional sparse self-attention enhances long-distance dependent capture and improves accuracy. Experiments on public datasets show that the TD-CS-Transformer outperforms existing methods in PM2.5 and PM10 concentration prediction and has the characteristics of high accuracy, low error, fast training, and minimal memory footprint, showing strong practicality and scalability. In the study of air quality prediction based on time series decomposition and convolutional sparse self-attention mechanism Transformer model, we use this model to conduct an in-depth analysis of air quality data containing more than 180,000 records in the last 5 years. The time series decomposition technique decomposes the complex time series data into trend, season, period and irregular components, which simplifies the data pattern. Subsequently, the Transformer model with convolutional sparse self-attention mechanism is used for processing, which significantly improves the model’s ability to capture long-distance dependencies and reduces the computational complexity through sparse connections, enabling the model to maintain high-accuracy predictions.

Effective drug combination prediction is crucial for the success of drug discovery, but it is a challenging task due to drug-drug interactions and potential adverse drug reactions. In this work, a novel technique to DDI prediction using knowledge graph-based approach called KGAT is proposed, which utilizes attention mechanisms with graph convolution layers to capture important features and correlations between drugs and other entities such as targets and genes. Our model employs attention mechanisms to prioritize significant interactions and aggregates information through sum, mean, and max operations to enhance prediction accuracy. This allows KGAT to effectively mine high-order structures and semantic relationships within the knowledge graph. We evaluate our model on the KEGG dataset and compare its performance with existing state-of-the-art methods. The results show that KGAT outperforms these methods. Additionally, our approach has several advantages, including simplicity, interpretability, and low-dimensional complexity, making it a promising tool for accelerating drug discovery and development. By identifying novel drug combinations with improved efficacy and safety profiles, our approach has the potential to improve patient outcomes and support safer drug development. Our study highlights the potential of attention mechanisms in knowledge graph-based drug combination prediction, and we believe that KGAT can serve as a valuable framework for future research in this field.

Under the guidance of the goal of the “carbon peaking and carbon neutrality”, due to the high proportion of renewable energy and the high proportion of power electronic equipment, the power system will bring strong randomness, low inertia and other characteristics, causing a large number of frequency stability problems. In order to solve the problems of traditional power system frequency prediction methods, such as difficulty in modeling and poor prediction accuracy, and to determine whether the frequency stability problem will occur after the wind power grid-connected system is disturbed, the Lasso algorithm is first used to reduce the dimension of the input data, and then the attention mechanism based long and short memory (attention LSTM) neural network is used to predict the output frequency curve, The network parameters are optimized by the global search algorithm Whale Optimization Algorithm (WOA). Finally, the accuracy of the algorithm is verified by taking the improved wind power grid-connected 39 bus as an example. The results show that this method has a good guiding significance for evaluating the frequency stability of the wind power grid-connected system after interference, and can effectively predict the frequency curve of the system after interference, and evaluate the frequency stability of the system after topology change.

Recently, learned image compression algorithms have shown incredible performance compared to classic hand-crafted image codecs. Despite its considerable achievements, the fundamental disadvantage is not optimized for retaining local redundancies, particularly non-repetitive patterns, which have a detrimental influence on the reconstruction quality. This paper introduces the autoencoder-style network-based efficient image compression method, which contains three novel blocks, i.e., adjacent attention block, Gaussian merge block, and decoded image refinement block, to improve the overall image compression performance. The adjacent attention block allocates the additional bits required to capture spatial correlations (both vertical and horizontal) and effectively remove worthless information. The Gaussian merge block assists the rate-distortion optimization performance, while the decoded image refinement block improves the defects in low-resolution reconstructed images. A comprehensive ablation study analyzes and evaluates the qualitative and quantitative capabilities of the proposed model. Experimental results on two publicly available datasets reveal that our method outperforms the state-of-the-art methods on the KODAK dataset (by around 4dB and 5dB) and CLIC dataset (by about 4dB and 3dB) in terms of PSNR and MS-SSIM.

Road information is an important geographic information. Road information extracted from remote sensing images has been widely used in map, traffic, navigation and many other fields. However, the autonomous extraction of road information from high resolution remote sensing images has some problems such as incoherence, incompleteness and poor connectivity, therefore, a semantic segmentation model for roads in high resolution remote sensing images, called AGD-Linknet, is proposed, which integrates attention mechanisms, gated decoder block, and dilated convolution. This model mainly consists of three parts. Firstly, the stem block is used as the initial convolution layer of the model to reduce the information loss in the convolution stage; Secondly, the series-parallel combined dilated convolution and coordinate attention block into the center of the network, which enlarges the receptive field of the network and improves the feature extraction ability of spatial domain and channel domain information; Finally, in the decoder part, gated convolution is introduced to improve the extraction of road edge. Compared with U-Net, Linknet and D-Linknet on the DeepGlobe dataset, the proposed AGD-Linknet has improved the pixel accuracy, mean intersection over union and F1-Score index of road recognition by 1.41%-11.52%, 0.0077-0.1473, 0.0057-0.1292, and has certain effectiveness and feasibility in many scenarios in rural areas, urban, and suburbs. And can be apply to the tasks of road recognition and extraction in high-resolution remote sensing.

Time series forecasting uses data from the past periods of time to predict future information, which is of great significance in many applications. Existing time series forecasting methods still have problems such as low accuracy when dealing with some non-stationary multivariate time series data forecasting. Aiming at the shortcomings of existing methods, in this paper we propose a new time series forecasting model LSTM-attention-LSTM. The model uses two LSTM models as the encoder and decoder, and introduces an attention mechanism between the encoder and decoder. The model has two distinctive features: first, by using the attention mechanism to calculate the interrelationship between sequence data, it overcomes the disadvantage of the coder-and-decoder model in that the decoder cannot obtain sufficiently long input sequences; second, it is suitable for sequence forecasting with long time steps. In this paper we validate the proposed model based on several real data sets, and the results show that the LSTM-attention-LSTM model is more accurate than some currently dominant models in prediction. The experiment also assessed the effect of the attention mechanism at different time steps by varying the time step.

Object detection for remote sensing images has problems such as complex backgrounds, multi-scale and difficulty in the detection of small targets. Because of the above problems, an improved object detection algorithm for remote-sensing images is proposed. Firstly, fusing the characteristics of shallow and deep networks, the original feature pyramid structure is reconstructed, the improved adaptive feature fusion structure is introduced before the network prediction, and the location and category information of different feature maps are fused. Secondly, the attention mechanism is introduced into the network to reduce the interference caused by complex backgrounds; Finally, the up-sampling module is improved to expand the receptive field and realize up-sampling based on semantic information. Through experiments, we show that our network achieves 93.9% and 98.5% detection accuracy on Levir and RSOD dataset, respectively, with the detection speed, reaching 98.26fps. Compared with the original network, the accuracy is increasing by 4% and 1.2%, respectively. Experimental results show that the detection accuracy of our algorithm is improved while maintaining the light weights and efficiencies of the original network.

Current speaker recognition systems are mainly for the combined application of network architectures and attention mechanisms, however, lightweight networks are not able to extract frame-level features of speaker speech well, and deeper and wider networks also face the problems of slower inference and excessive number of parameters. To this end, we proposes Split-ResNet, a network structure for split residuals, which can obtain a combination of multiple receptive field at a finer-grained level, thus obtaining a variety of feature representations with different scale combinations and producing more informative and comprehensive multi-scale features. In addition we propose a dual time-frequency attention (DTFA) that enhances key features and suppresses unimportant features by focusing on features in the time and frequency domains and learning weights from the time and frequency channels, respectively. We finally tested the speaker recognition system using a combination of Split-ResNet and DTFA against other speaker recognition systems on the Voxceleb1-O test set. The test results show that the speaker recognition system proposed in this paper is 0.98%, 0.39%, 0.69% and 0.47% lower in EER compared with SpeechNAS, RawNet2, Y-vector and CNN+Transformer, respectively, proving that DTFA+Split-ResNet is a speaker recognition system with good speaker audio feature extraction capability and discriminative capability.

To address the problems of low precision and poor real-time performance in the process of part identification and positioning of production line assembly robotic arm, Ghost-SE YOLOv5, an assembly part identification and positioning algorithm integrating lightweight network and attention mechanism is proposed. First, the redundancy of feature map convolution is utilized, which solves the problems of large number of model parameters and floating point operations by using Ghost convolution and Ghost Bottleneck modules. Second, the attention mechanism SE Module is introduced in the backbone network to increase the propensity of feature extraction. Last, the loss function is optimized to speed up the convergence of the model. The results shows that the number of parameters, float operation per second and train time of the proposed algorithm are reduced by 45.98%, 55.99% and 24.07%, respectively. And GPU use was reduced from 7.61G to 6.43G. Furthermore, during the test the precision reached 98.6%, and the recall rate realized 95.3%. The real-time detection performance achieved 97.59 FPS, with an improvement of 34.53%. It can be seen that Ghost-SE YOLOv5 algorithm has better practicality in the part identification and positioning of robotic arm for production line assembly.

Though acoustic speech emotion recognition has been studied for a while, bimodal speech emotion recognition using both acoustic and text has gained momentum since speech emotion recognition doesn’t only involve the acoustic modality. However, there is less review work on the available bimodal speech emotion recognition (SER) research. The review works available mostly concentrate on the use of convolution neural networks (CNNs) and recurrent neural networks (RNNs). However, recent deep learning techniques like attention mechanisms and fusion strategies have shaped the bimodal SER research without explicit analysis of their significance when used singly or in combination with the traditional deep learning techniques. We therefore, review the recently published literature that involves these deep learning techniques in this paper to ascertain the current trends and challenges of bimodal SER research that have hampered it to be fully deployed in the natural environment for off-the-shelf SER applications. In addition, we carried out experiments to ascertain the optimal combination of acoustic features and the significance of the attention mechanisms and their combination with the traditional deep learning techniques. We propose a multi-technique model called the deep learning-based multi-learning model for emotion recognition (DBMER) that operates with multi-learning capabilities of CNNs, RNNs, and multi-head attention mechanisms. We noted that attention mechanisms play a pivotal role in the performance of bimodal dyadic SER systems. However, few publicly available datasets, the difficulty in acquisition of bimodal SER data, cross-corpus and multilingual studies remain open problems in bimodal SER research. Our experiments on the proposed DBMER model showed that though each of the deep learning techniques benefits the task, the results are more accurate and robust when they are used in careful combination with multi-level fusion approaches.

Kidney tumor segmentation from MR images became a pivotal research area in kidney cancer diagnosis and treatment planning. Accurate and efficient segmentation enables precise tumor localization, treatment planning, and monitoring of disease progression. Former studies have demonstrated the remarkable capability of the U-Net architecture in semantic segmentation of kidney tumors. A recent variant of the U-Net architecture, known as 3D-CU-Net, has been specifically designed with fully-connected dense skip connections to tackle the kidney tumor segmentation challenges related to network depth invariance, segmentation errors, and enforced feature fusion. While the 3D-CU-Net model demonstrated improved effectiveness in kidney tumor segmentation, it still exhibits significant limitations, including challenges in precise localization, fixed feature selection, image diversity, limited contextual information, and computational complexity. To address the limitations of the 3D-CU-Net model, this paper introduces the Attention 3D-CU-Net as a novel variant. The attention-based mechanism is seamlessly integrated with the 3D-CU-Net, prioritizing informative features to enhance segmentation accuracy by concentrating on selective regions. This innovative approach serves to significantly improve the model’s performance, particularly in challenging cases. The proposed model is evaluated on the TCGA-KIRC dataset, a widely used benchmark for kidney tumor segmentation. In comparative experiments, we evaluated the results using metrics like IoU, DSC and accuracy. Our Attention 3D-CU-Net model outperforms the baseline 3D-CU-Net and U-Net with notably higher scores: IoU (0.92), DSC (0.94), and accuracy (0.96).

The efficacy of credit risk assessment models is pivotal to the risk management capacity of financial institutions. Traditional credit risk models often suffer from inadequate predictive accuracy due to overlooked feature combinations and weak supervisory signals. Addressing these limitations, we present a novel approach for credit default prediction that integrates feature interactions and contrastive learning. Specifically, we introduce second-order interactions atop standard linear models to achieve low-order feature interplay. Concurrently, the integration of deep neural networks and attention mechanisms facilitates the learning of concealed high-order features, thus enhancing the model’s non-linear modeling capabilities and illuminating latent feature associations. Further, to ameliorate the issues of noise and diminished supervisory signals, we embed slight noise in feature embeddings for data augmentation and construct contrastive views, ultimately refining feature quality. To attest to the effectiveness of our approach, we conducted experiments on two real-world datasets, benchmarking against eight predictive methods including LR, XGBoost, and FiBiNET. The results unequivocally demonstrate the superior performance of our method across various metrics, underscoring its promise and excellence in the realm of credit risk assessment.

Branch Convolutional Neural Nets have become a popular approach for hierarchical classification in computer vision and other areas. Unfortunately, these models often led to hierarchical inconsistency: predictions for the different hierarchy levels do not necessarily respect the class-subclass constraints imposed by the hierarchy. Several architectures to connect the branches have arisen to overcome this limitation. In this paper, we propose a more straightforward and flexible method: let the neural net decide how these branches must be connected. We achieve this by formulating an attention mechanism that dynamically determines how branches influence each other during training and inference. Experiments on image classification benchmarks show that the proposed method can outperform state-of-the-art models in terms of hierarchical performance metrics and consistency. Furthermore, although sometimes we found a slightly lower performance at the deeper level of the hierarchy, the model predicts much more accurately the ground-truth path between a concept and its ancestors in the hierarchy. This result suggests that the model does learn not only local class memberships but also hierarchical dependencies between concepts.

The increasing popularity of attention mechanisms in deep learning algorithms for computer vision and natural language processing made these models attractive to other research domains. In healthcare, there is a strong need for tools that may improve the routines of the clinicians and the patients. Naturally, the use of attention-based algorithms for medical applications occurred smoothly. However, being healthcare a domain that depends on high-stake decisions, the scientific community must ponder if these high-performing algorithms fit the needs of medical applications. With this motto, this paper extensively reviews the use of attention mechanisms in machine learning methods (including Transformers) for several medical applications based on the types of tasks that may integrate several works pipelines of the medical domain. This work distinguishes itself from its predecessors by proposing a critical analysis of the claims and potentialities of attention mechanisms presented in the literature through an experimental case study on medical image classification with three different use cases. These experiments focus on the integrating process of attention mechanisms into established deep learning architectures, the analysis of their predictive power, and a visual assessment of their saliency maps generated by post-hoc explanation methods. This paper concludes with a critical analysis of the claims and potentialities presented in the literature about attention mechanisms and proposes future research lines in medical applications that may benefit from these frameworks.

Bad weather, such as snowfall, can seriously decrease the quality of images and pose great challenges to computer vision algorithms. In view of the negative effect of snowfall, this paper presents a single-image snow removal method based on a generative adversarial network (GAN). Unlike previous GANs, our GAN includes an attention mechanism in the generator component. By injecting attention information, the network can pay increased attention to areas covered by snow and improve its capability to perform local repairs. At the same time, we improve the traditional U-Net network by combining it with the residual network to enhance the effect of the model when removing snowflakes from a single image. Our experiments on both synthetic and real-word images show that our method produces better results than those of other state-of-the-art methods.

Scene classification is a highly useful task in Remote Sensing (RS) applications. Many efforts have been made to improve the accuracy of RS scene classification. Scene classification is a challenging problem, especially for large datasets with tens of thousands of images with a large number of classes and taken under different circumstances. One problem that is observed in scene classification is the fact that for a given scene, only one part of it indicates which class it belongs to, whereas the other parts are either irrelevant or they actually tend to belong to another class. To address this issue, this paper proposes a deep attention Convolutional Neural Network (CNN) for scene classification in remote sensing. CNN models use successive convolutional layers to learn feature maps from larger and larger regions (or receptive fields) of the scene. The attention mechanism computes a new feature map as a weighted average of these original feature maps. In particular, we propose a solution, named EfficientNet-B3-Attn-2, based on the pre-trained EfficientNet-B3 CNN enhanced with an attention mechanism. A dedicated branch is added to layer 262 of the network, to compute the required weights. These weights are learned automatically by training the whole CNN model end-to-end using the backpropagation algorithm. In this way, the network learns to emphasize important regions of the scene and suppress the regions that are irrelevant to the classification. We tested the proposed EfficientNet-B3-Attn-2 on six popular remote sensing datasets, namely UC Merced, KSA, OPTIMAL-31, RSSCN7, WHU-RS19, and AID datasets, showing its strong capabilities in classifying RS scenes.

Existing approaches to solving combinatorial optimization problems on graphs suffer from the need to engineer each problem algorithmically, with practical problems recurring in many instances. The practical side of theoretical computer science, such as computational complexity, then needs to be addressed. Relevant developments in machine learning research on graphs are surveyed for this purpose. We organize and compare the structures involved with learning to solve combinatorial optimization problems, with a special eye on the telecommunications domain and its continuous development of live and research networks.

Session-based recommendations have attracted significant attention because of their broad application scenarios. Recently, graph neural networks (GNNs) have been employed in session-based recommendation because of their superior performance in representation learning compared with recurrent neural networks (RNNs). Although most existing GNN-based methods have made great achievements in this field, none of them emphasizes the importance of repeat recommendations, which has been an important component in session-based recommendation (e.g., people tend to browse product information repeatedly or revisit websites in a period of time). In this paper, we propose a novel model called ReGNN to combine a graph neural network with a repeat-exploration mechanism for better recommendations. Specifically, we dynamically process the item sequence of a session as a graph structure and capture the complex transitions between items by a GNN. Then, we formulate an exact session representation with the attention mechanism. Finally, the repeat-exploration mechanism is incorporated into the ReGNN to model the user’s repeat-exploration behavior patterns and make more accurate predictions. We conduct extensive experiments on two public datasets. The experimental results show that our proposed model ReGNN consistently outperforms other state-of-the-art methods.

Motivated by impressive success of deep recurrent neural networks (RNNs), sequence-to-sequence (seq2seq) architecture has been widely adapted to tackle human motion prediction. However, forecasting in longer time horizons always leads to implausible human poses or converges to mean poses. To address these challenges, we dig into the root causes and lay emphasis on two key principles. First, error can be easily accumulated on seq2seq architecture without modifications and thus RNNs cannot recover from its own mistakes in longer time horizons. Second, all the frames or joints are treated equally, whereas both of them often have different levels of importance in human motion. To mitigate this gap, we propose to retrospect human dynamics with attention. We design a retrospection module built upon seq2seq architecture to recollect previous subsequences and correct mistakes in time which enables a self-correction ability. This assists the original seq2seq architecture to eliminate error accumulation which improves significantly both short-term and long-term performances. Besides, we present two attention techniques to explore correlations among different joints as well as different frames in both spatial and temporal domains, which successfully capture key properties of different actions and enable our model to generate more realistic human poses. Quantitative and qualitative experiments have been both conducted to evaluate the our proposed model. Experimental results clearly demonstrate the superiority of proposed model over other baselines.

Malware and its variants continue to pose a threat to network security. Machine learning has been widely used in the field of malware classification, but some emerging studies, such as attention mechanisms, are rarely applied in this field. In this paper, we analyze the correspondence between bytecode and disassembly of malware, and propose a new feature extraction method based on multi-dimensional sequence. Also, we construct a new classification framework based on attention mechanism and Convolutional Neural Networks mechanism. Furthermore, we also compare the different architectures based on the attention mechanisms. Experiments on open datasets show that our feature extraction method and our framework have a good classification effect, and the accuracy rate is 0.9609.

Methods based on stacked hourglass networks (SHNs) have achieved great progress in face alignment tasks. However, most of these algorithms have met with limited success in modeling correlations among features. Visual attention mechanisms have shown promise in terms of effectively understanding scenes in various computer vision tasks. In this paper, an attention-guided coarse-to-fine network (AGCFN) based on an attention mechanism is proposed for robust face alignment. Thus, the network is guided to emphasize key information while suppressing less important information. Meanwhile, the fusion of features from different levels is adopted to improve the information flow through the proposed network. Additionally, conditional random fields (CRFs) are introduced to model the spatial interactions between landmarks in the prediction maps. Experimental results obtained on the 300-W dataset, the 300-W private test set, and the WFLW dataset demonstrate the superiority of the proposed method in terms of accuracy and robustness.