Search

Transformers are Deep Neural Networks (DNN) that utilize a self-attention mechanism to capture contextual relationships within sequential data. Unlike traditional neural networks and variants of Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM), Transformer models excel at managing long dependencies among input sequence elements and facilitate parallel processing. Consequently, Transformer-based models have garnered significant attention from researchers in the field of artificial intelligence. This is due to their tremendous potential and impressive accomplishments, which extend beyond Natural Language Processing (NLP) tasks to encompass various domains, including Computer Vision (CV), audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published, spotlighting the Transformer's contributions in specific fields, architectural disparities, or performance assessments, there remains a notable absence of a comprehensive survey paper that encompasses its major applications across diverse domains. Therefore, this paper addresses this gap by conducting an extensive survey of proposed Transformer models spanning from 2017 to 2022. Our survey encompasses the identification of the top five application domains for Transformer-based models, namely: NLP, CV, multi-modality, audio and speech processing, and signal processing. We analyze the influence of highly impactful Transformer-based models within these domains and subsequently categorize them according to their respective tasks, employing a novel taxonomy. Our primary objective is to illuminate the existing potential and future prospects of Transformers for researchers who are passionate about this area, thereby contributing to a more comprehensive understanding of this groundbreaking technology. • The paper presents a comprehensive survey on transformers for deep learning tasks. • The paper conducts a thorough analysis on highly effective models in five domains. • The paper classifies the models based on respective tasks using a proposed taxonomy. • The characteristics of the surveyed models are deeply explored and analyzed. • Future directions and challenges for transformer-based models are deciphered. [ABSTRACT FROM AUTHOR]

Automatic license plate recognition plays an important role in vehicle management. The performance of current license plate recognition systems is significantly affected by large-angle deflection in unconstrained scenes. For addressing this issue, this paper develops a license plate recognition system via a new image correction scheme and an improved CRNN (Convolutional Recurrent Neural Network). In which, the yolov5l is firstly introduced for license plate detection, and it is trained by transfer learning in order to address insufficient data. Then, a new license plate correction scheme and AFF-Net (Adaptive Fusion Feature Segmentation Network) is proposed and applied, which uses the segmentation result of license plate and its original image area for perspective transformation to improve the correction effect. Finally, the channel attention mechanism is added to the CRNN model for license plate recognition, so that a single grid cell of the feature network can obtain more spatial information. The recognition accuracy of the developed license plate recognition system is 98.8 % on CTPFSD (China Temporary Parking Fee System Data), which is much higher than current mainstream license plate recognition systems. More importantly, the developed license plate recognition system has been deployed in the cloud and applied in outdoor parking toll in practice. [ABSTRACT FROM AUTHOR]

Nowadays, video surveillance systems are commonly found in most public and private spaces. These systems typically consist of a network of cameras that feed into a central node. However, the processing aspect is evolving towards distributed approaches, leveraging edge-computing. These distributed systems are capable of effectively addressing the detection of people or events at each individual node. Most of these systems, rely on the use of deep-learning and segmentation algorithms which enable them to achieve high performance, but usually with a significant computational cost, hindering real-time execution. This paper presents an approach for people detection and action recognition in the wild, optimized for running on the edge, and that is able to work in real-time, in an embedded platform. Human Action Recognition (HAR) is performed by using a Recurrent Neural Network (RNN), specifically a Long Short-Term Memory (LSTM). The input to the LSTM is an ad-hoc, lightweight feature vector obtained from the bounding box of each detected person in the video surveillance image. The resulting system is highly portable and easily scalable, providing a powerful tool for real-world video surveillance applications (in the wild and real-time action recognition). The proposal has been exhaustively evaluated and compared against other state-of-the-art (SOTA) proposals in five datasets, including four widely used (KTH, WEIZMAN, WVU, IXMAX) and a novel one (GBA) recorded in the wild, that includes several people performing different actions simultaneously. The obtained results validate the proposal, since it achieves SOTA accuracy within a much more complicated video surveillance real scenario, and using a lightweight embedded hardware. • Real-time action recognition based on LSTMs with an ultralight feature vector. • Experimental results on for widely used dataset and a novel dataset in-the-wild. • Comparison to other state-of-the-art methods. • Achieves accuracy close to state-of-the-art while reducing computational cost. • Proposal runs in real time in an embedded platform. [ABSTRACT FROM AUTHOR]

The irregularity of the time interval between observations in and across the stream is a key factor that leads to a drop in performance when classical machine learning or deep learning models are used for a downstream task requiring multivariate time series. Indeed, irregular multivariate time series not only increase the rate of missing values but also lead to data sparsity, which consequently makes the data almost unleverageable and/or ineffective for models. To tackle this scorching challenge, most of the pioneering approaches apply imputation or interpolation in their core, which might lead to embedding data with noise. To especially address this irregular multivariate time series issue, we introduce, in this paper, a new deep neural network model called ALignment-driven Neural Network. The innovative idea of our model is to transform the irregular multivariate time series into pseudo-aligned (or pseudo-regular) latent values. The latter are shown as a matrix, where the coefficients are the latent values of each feature at user-defined reference time points that are evenly spaced. They are obtained through a duplication process driven by an exponential decay mechanism. The obtained output is then passed to a Recurrent Neural Network model, which is undoubtedly the must-use model for regular time series data. To show that our model added value, we looked at the Intensive Care Unit mortality prediction task. In this unit, the physiological measurements used to make decisions have a problem with time irregularity. Leveraging the publicly available MIMIC-III, we compare the performance of our model to that of flagship models. In addition, we also performed extensive ablation studies to highlight the importance of specific components in our model. Interestingly enough, whenever data is collected 24 and 48 h after a patient's admission, we outperform our pioneering competitors, i.e., + 1. 1 % and + 1. 5 % for the AUC score, + 2. 3 % and + 2. 4 % for the AUPRC score and + 0. 6 % and + 1. 7 % for the F1-score. • Irregular Multivariate Time Series Transformation for Recurrent Neural Networks. • Transformation by a duplication process. • Duplication process driven by an exponential decay mechanism. • Predicting Intensive Care Unit Mortality. [ABSTRACT FROM AUTHOR]

Electric Health Records (EHR) have been widely adopted by many hospitals to improve clinical decision making and re-admission prediction. Each patient admission usually contains both multiple medical codes as well as clinical notes. Accurate learning of the representations (also called embeddings) of medical concepts from EHR is a key strategy to improve prediction performance in healthcare. Existing works employ medical ontologies to improve the quality of representations but focus solely on the relationships amongst the medical codes, ignoring textual data such medical code descriptions, clinical notes, and patient demographics. In this paper, we propose a new model called Semantic-based Attention model using Textual data for Medical Concept Embedding (SATexMCE). SATexMCE consists of three parts: medical codes embedding, admission embedding, and prediction model. First, we generate representations of medical codes by using both the textual description of medical codes and also the relationships among medical codes. Then, we generate the admission representation based on the representation of medical codes in the admission, the clinical notes and the demographic information associated with the admission via several attention mechanisms. The admission embeddings are used to construct a Recurrent neural network model which is used to predict patients' readmission and a disease in the next admission based on patient admission data. Experimental results show that the proposed SATexMCE model improves not only the performance of readmission prediction but also the quality of medical concept representations. Attention mechanism helps us measure the importance of different medical codes and understand the meaning of the words in clinical notes for predictions. • A attention model to learn medical code representations using code descriptions. • A method to learn medical code and admission representations using clinical notes. • A RNN-based classification model for readmission prediction and CKD prediction. • Analyses of the proposed medical code representations for the prediction tasks. [ABSTRACT FROM AUTHOR]

Excessive stress generally leads to degraded driving performance, which increases the risk of road accidents. Therefore, real-time driver stress detection is of great significance to improve road safety. To this end, this paper proposes a deep convolutional recurrent neural network with Wasserstein generative adversarial network-based data augmentation (WGAN-DCRNN) for real-time driver stress detection, where only the pupillary response data are used. The proposed model can learn the sequence features from the input data more effectively and address the class-imbalance problem of the dataset. First, two types of DCRNN models (i.e., CNN-LSTM and CNN-GRU) are established and compared with baseline models (i.e., CNN, LSTM and GRU). The best baseline model (i.e., CNN) achieves 93.02% accuracy on the overall samples of the test set, while the CNN-LSTM and CNN-GRU improve the accuracy to 94.68% and 94.71%, respectively. This suggests that the performance of the CNN can be improved by integrating the RNN to consider the temporal dependencies of the pupillary response data. Second, deep generative model-based data augmentation is used to address the class-imbalance problem of the dataset. Specifically, the WGAN is trained to generate new samples, which are used to expand and balance the training set. To validate the effect of the proposed data augmentation method, comparative experiments are further conducted. The results show that the accuracies of CNN-LSTM and CNN-GRU are further improved to 95.39% and 95.30%, respectively. More importantly, the recall rates of minority categories achieve significant improvement. This indicates that the proposed model overcomes the class-imbalance problem and that the generalization ability of the recognition model has been further improved by using data augmentation. In addition, the WGAN-DCRNN outperforms models in existing driver stress detection studies. The proposed model can be used for different classification tasks to reduce road accident risks caused by drivers. [ABSTRACT FROM AUTHOR]

The application of Machine learning and deep learning techniques for time series forecasting has gained significant attention in recent years. Numerous endeavors have been devoted to automating forecasting through the utilization of cutting-edge neural networks. Notably, the recurrent neural network (LSTM – Long Short-Term Memory) has emerged as a central concept in most research endeavors. Although LSTM was initially introduced in 1997 for sequence modeling, subsequent updates have primarily focused on language learning tasks. These updates have introduced various computational mechanisms within the LSTM cell, including the forget gate, input gate, and output gate. In this study, we investigate the impact of each computational component in isolation to analyze their effects on time series forecasting tasks. Our experiments utilize the Jena weather dataset and Appliance Energy Usage time series for evaluation. The experimental results reveal that variations of the LSTM model outperform the most popular LSRM cell format in terms of error rate and training time. Specifically, the variations identified in this paper demonstrate superior generalization capabilities and yield reduced forecasting errors. [ABSTRACT FROM AUTHOR]

Accurate and reliable load forecasting can ensure the safety and economy of power system operation. To improve the accuracy of short-term power load forecasting, this paper adopts feature extraction combined with a variety of machine learning methods,(Empirical wavelet decomposition (EWT), Convolutional neural network (CNN) model, Recurrent neural network (RNN) model, Long short-term memory network (LSTM) model, Bayesian optimization (BOA) algorithm). A new hybrid forecasting model, the EWT-CNN-S-RNN + LSTM model is proposed to predict power consumption. In this method, the fixed mode of EWT decomposition is used to extract statistical features, and the LSTM/RNN model is selected according to the statistical features. This method uses the Bayesian optimization (BOA) algorithm to optimize the parameters to solve the model gradient explosion problem. Based on the electricity consumption data of 2007 (Australian Energy Market Operator) and a total of 336 power load data, a good load forecasting result is obtained, which can provide indirect support for power regulation and sustainable development of electricity. [ABSTRACT FROM AUTHOR]

Radar high-resolution range profile (HRRP) plays a crucial role in ballistic target recognition due to its simplicity and fast computation. Convolutional neural networks (CNNs), a popular deep learning approach, are widely used in radar automatic target recognition (RATR) based on HRRP, thanks to their excellent feature extraction capabilities. However, current research mainly focuses on integrating branch networks like recurrent neural networks and attention mechanisms into vanilla CNNs to enhance feature extraction. Limited research exists on reducing the computational complexity of vanilla CNNs when applied to HRRP-based RATR, as well as target recognition problems with limited samples. To address these issues, this paper proposes a lightweight depth-wise separable fusion CNN (DSFCNN) for ballistic target HRRP recognition. The DSFCNN reduces the computational complexity of vanilla CNNs while improving recognition accuracy. Furthermore, we introduce the sample-level fitting and class-level distinguishability (SFCD) loss for limited-sample ballistic target HRRP recognition. The SFCD loss ensures better fitting performance of all samples, as well as higher intra-class compactness and inter-class separability, enhancing the distinguishability of embedded features in limited-sample conditions. During the training phase, the SFCD loss is equally applied to four DSFCNNs with shared structures and parameters, resulting in a model known as Quadruplet DSFCNN (QDSFCNN). However, during the testing phase, only one DSFCNN is utilized. Additionally, we propose the parameter quantity shifting-fitting performance (PQS-FP) coordinate system to elucidate the performance disparities between DSFCNN and vanilla CNNs in our experiments. PQS-FP offers inspirational insights into the relations among parameter quantity, fitting performance, and recognition performance. Extensive experimental results demonstrate the robustness and effectiveness of QDSFCNN for ballistic target recognition in various limited-sample scenarios. [ABSTRACT FROM AUTHOR]