Search

To improve the accuracy of user implicit rating prediction, we combine the traditional latent factor model (LFM) and bidirectional gated recurrent unit neural network (BiGRU) model to propose a hybrid model that deeply mines the latent semantics in the unstructured content of the text and generates a more accurate rating matrix. First, we utilize the user’s historical behavior (favorites records) to build a user rating matrix and decompose the matrix to obtain the latent factor vectors of users and literature. We also apply the BERT model for word embedding of the research papers to obtain the sequence of word vectors. Then, we apply the BiGRU with the user attention mechanism to mine the research paper textual content and to generate the new literature latent feature vectors that are used to replace the original literature latent factor vectors decomposed from the rating matrix. Finally, a new rating matrix is generated to obtain users’ ratings of noninteractive research papers and to generate the recommendation list according to the user latent factor vector. We design experiments on the real datasets and verify that the research paper recommendation model is superior to traditional recommendation models in terms of precision, recall, F1-value, coverage, popularity and diversity.

Most existing studies and public datasets for handwritten Chinese text recognition are based on the regular documents with clean and blank background, lacking research reports for handwritten text recognition on challenging areas such as educational documents and financial bills. In this paper, we focus on examination paper text recognition and construct a challenging dataset named examination paper text (SCUT-EPT) dataset, which contains 50 000 text line images (40 000 for training and 10 000 for testing) selected from the examination papers of 2 986 volunteers. The proposed SCUT-EPT dataset presents numerous novel challenges, including character erasure, text line supplement, character/phrase switching, noised background, nonuniform word size, and unbalanced text length. In our experiments, the current advanced text recognition methods, such as convolutional recurrent neural network (CRNN) exhibits poor performance on the proposed SCUT-EPT dataset, proving the challenge and significance of the dataset. Nevertheless, through visualizing and error analysis, we observe that humans can avoid vast majority of the error predictions, which reveal the limitations and drawbacks of the current methods for handwritten Chinese text recognition (HCTR). Finally, three popular sequence transcription methods, connectionist temporal classification (CTC), attention mechanism, and cascaded attention-CTC are investigated for HCTR problem. It is interesting to observe that although the attention mechanism has been proved to be very effective in English scene text recognition, its performance is far inferior to the CTC method in the case of HCTR with large-scale character set.

Distress image retrieval for infrastructure maintenance via self-trained deep metric learning using experts’ knowledge is proposed in this paper. Since engineers take multiple images of a single distress part for inspection of road structures, it is necessary to construct a similar distress image retrieval method considering the input of multiple images to support determination of the level of deterioration. Thus, the construction of an image retrieval method while selecting an effective input from multiple images is described in this paper. The proposed method performs deep metric learning by using a small number of effective images labeled by experts’ knowledge with information about their effectiveness and a large number of unlabeled images via a self-training approach. Specifically, an end-to-end learning approach that performs retraining of the model by assigning pseudo-labels to these unlabeled images according to the output confidence of the model is achieved. Thus, the proposed method can select an effective image from multiple images that are input at the retrieval as a query image. This is the main contribution of this paper. As a result, the proposed method realizes highly accurate retrieval of similar distress images considering the actual situation of inspection in which multiple images of a distress part are input.

A human-centric emotion estimation method based on correlation maximization with consideration of changes with time in visual attention and brain activity when viewing images is proposed in this paper. Owing to the recent developments of many kinds of biological sensors, many researchers have focused on multimodal emotion estimation using both eye gaze data and brain activity data for improving the quality of emotion estimation. In this paper, a novel method that focuses on the following two points is introduced. First, in order to reduce the burden on users, we obtain brain activity data from users only in the training phase by using a projection matrix calculated by canonical correlation analysis (CCA) between gaze-based visual features and brain activity-based features. Next, for considering the changes with time in both visual attention and brain activity, we obtain novel features based on CCA-based projection in each time unit. In order to include these two points, the proposed method analyzes a fourth-order gaze and image tensor for which modes are pixel location, color channel and the changes with time in visual attention. Moreover, in each time unit, the proposed method performs CCA between gaze-based visual features and brain activity-based features to realize human-centric emotion estimation with a high level of accuracy. Experimental results show that accurate human emotion estimation is achieved by using our new human-centric image representation.

Machine reading comprehension is a very challenging task, which aims to determine the answer span based on the given context and question. The newly developed pre-training language model has achieved a series of successes in various natural language understanding tasks with its powerful contextual representation ability. However, these pre-training language models generally lack the downstream processing structure for specific tasks, which limits further performance improvement. In order to solve this problem and deepen the model’s understanding of the question and context, this paper proposes S&I Reader. On the basis of the pre-training model, skimming, intensive reading, and gated mechanism modules are added to simulate the behavior of humans reading text and filtering information. Based on the idea of granular computing, a multi-granularity module for computing context granularity and sequence granularity is added to the model to simulate the behavior of human beings to understand the text from words to sentences, from parts to the whole. Compared with the previous machine reading comprehension model, our model structure is novel. The skimming module and multi-granularity module proposed in this paper are used to solve the problem that the previous model ignores the key information of the text and cannot understand the text with multi granularity. Experiments show that the model proposed in this paper is effective for both Chinese and English datasets. It can better understand the question and context and give a more accurate answer. The performance has made new progress on the basis of the baseline model.

This paper tackles the problem of detecting incongruities between headlines and body text, where a news headline is irrelevant or even in opposition to the information in its body. Our model, called the graph-based hierarchical dual encoder (GHDE), utilizes a graph neural network to efficiently learn the content similarity between news headlines and long body paragraphs. This paper also releases a million-item-scale dataset of incongruity labels that can be used for training. The experimental results show that the proposed graph-based neural network model outperforms previous state-of-the-art models by a substantial margin (5.3%) on the area under the receiver operating characteristic (AUROC) curve. Real-world experiments on recent news articles confirm that the trained model successfully detects headline incongruities. We discuss the implications of these findings for combating infodemics and news fatigue.

An intrusion detection system (IDS) is an important protection instrument for detecting complex network attacks. Various machine learning (ML) or deep learning (DL) algorithms have been proposed for implementing anomaly-based IDS (AIDS). Our review of the AIDS literature identifies some issues in related work, including the randomness of the selected algorithms, parameters, and testing criteria, the application of old datasets, or shallow analyses and validation of the results. This paper comprehensively reviews previous studies on AIDS by using a set of criteria with different datasets and types of attacks to set benchmarking outcomes that can reveal the suitable AIDS algorithms, parameters, and testing criteria. Specifically, this paper applies 10 popular supervised and unsupervised ML algorithms for identifying effective and efficient ML-AIDS of networks and computers. These supervised ML algorithms include the artificial neural network (ANN), decision tree (DT), k-nearest neighbor (k-NN), naive Bayes (NB), random forest (RF), support vector machine (SVM), and convolutional neural network (CNN) algorithms, whereas the unsupervised ML algorithms include the expectation-maximization (EM), k-means, and self-organizing maps (SOM) algorithms. Several models of these algorithms are introduced, and the turning and training parameters of each algorithm are examined to achieve an optimal classifier evaluation. Unlike previous studies, this study evaluates the performance of AIDS by measuring the true positive and negative rates, accuracy, precision, recall, and F-Score of 31 ML-AIDS models. The training and testing time for ML-AIDS models are also considered in measuring their performance efficiency given that time complexity is an important factor in AIDSs. The ML-AIDS models are tested by using a recent and highly unbalanced multiclass CICIDS2017 dataset that involves real-world network attacks. In general, the k-NN-AIDS, DT-AIDS, and NB-AIDS models obtain the best results and show a greater capability in detecting web attacks compared with other models that demonstrate irregular and inferior results.

The key to an accurate understanding of terrain is to extract the informative features from the multi-modal data obtained from different devices. Sensors, such as RGB cameras, depth sensors, vibration sensors, and microphones, are used as the multi-modal data. Many studies have explored ways to use them, especially in the robotics field. Some papers have successfully introduced single-modal or multi-modal methods. However, in practice, robots can be faced with extreme conditions; microphones do not work well in crowded scenes, and an RGB camera cannot capture terrains well in the dark. In this paper, we present a novel framework using the multi-modal variational autoencoder and the Gaussian mixture model clustering algorithm on image data and audio data for terrain type clustering by forcing the features to be closer together in the feature space. Our method enables the terrain type clustering even if one of the modalities (either image or audio) is missing at the test-time. We evaluated the clustering accuracy with a conventional multi-modal terrain type clustering method and we conducted ablation studies to show the effectiveness of our approach.

Deep convolutional neural networks (DCNNs) have attracted extensive attention in synthetic aperture radar (SAR) image target detection. However, there is usually a lack of labeled data in SAR images which results in difficulties to improve the detection performance with DCNN based methods. Transfer learning borrows generic knowledge from other tasks for the target task and can be an effective solution to the small sample problem, but the features’ generality and specificity across various models are not discussed in target detection. In addition, conventional DCNN based models have too many parameters related to the prior information of the targets which increase the difficulty of transfer learning between different tasks. In this paper, we mainly conduct research in three folds to address these mentioned issues. First, the generality and specificity of features in different network models are analyzed in the field of SAR target detection. Second, a transitive transfer framework is developed for detection under small sample condition. Third, an anchor free rotatable detector with flexible structure is designed for SAR images detection to suit the transfer detection framework proposed in this paper and a reasonable explanation is given for its upper hand with small samples. Experimental results demonstrate that: 1) transitive transfer framework is valid in SAR target detection; 2) the anchor free detector in this framework is superior to the anchor based one; 3) the proposed detector achieves better performance under small sample condition by comparison with state-of-the-art methods.

With the rapid development of remote-sensing imaging technology, remote-sensing images have become increasingly diverse, and people are paying more attention to ocean remote-sensing research. Because ocean remote-sensing data are complex, and the ocean environment is diverse, results will differ, even if the same target is detected at different times in the same scene. To obtain more semantic features and better pixel-level prediction capabilities, this paper proposes a pixel-level ocean remote-sensing image algorithm (GLPO-Net) that combines local and global features. First, texture features, color features, and spatial relationship features are extracted. Second, the algorithm constructs a multiscale local cross-attention mechanism strategy to obtain feature weight information in different directions to fully mine the local features of ocean remote-sensing images. Concurrently, an algorithm constructs a multiscale global cross-attention mechanism strategy to obtain global features. Then, the fusion of global features and local features is described in each submodule to obtain more representative deep features. Finally, small-sample ocean remote-sensing is described via image pixel-level prediction. The algorithm proposed in this paper has been tested with three public ocean remote-sensing datasets. The experimental results show that the proposed GLPO-Net algorithm can learn features from small samples of ocean remote-sensing images. Compared to the prediction results of other remote-sensing image algorithms, GLPO-Net exhibits better prediction capabilities.

Near-infrared spectroscopy (NIR) has become one of the most important methods for counterfeit drugs identification for its low cost, non-destructive, and on-site detection. However, it is often invalid for unknown samples beyond the scope of modeling samples, as well as it is not efficient in conditions of insufficient samples (insufficient number of samples within the class), unbalanced samples (large difference in the number of samples between classes), and sensitivity of identification results (different tolerance for different errors in application scenarios). To solve these problems, this paper proposes a general method for on-site identification of counterfeit drugs based on Siamese-network modeling with near-infrared spectroscopy, which especially constructs the train set and test set, learning the general knowledge of spectral differences to identify the different drugs by a costumed one-dimensional convolution neural network (1D-CNN), and finally answered the question of whether the on-site two spectra are pointed to the same drug. Based on experimental modeling samples of 1314 spectra, which are involved 9 drugs produced by 25 manufacturers, this paper has constructed and fully trained its model. Then, not known at the time of modeling, 4 drugs produced by 9 manufacturers are used for testing in the on-site application, and the accuracy rate amounts to 97.3%. For generalizing consideration, randomly divided into training and testing categories, the 32015 spectra of 135 drugs produced by 391 manufacturers in the spectral library are handled by the same processing. The generalization model is equally applicable, and the accuracy is above 97%. Compared with traditional binary classification identification methods such as SVM, PLS-DA, Auto-encoding, and one class (OC) threshold identification algorithms such as SVM-OC, SIMCA, conformity test (CT), the proposed method has the best identification ability for unknown samples in modeling.

Handwritten mathematical expressions are an essential part of many domains, including education, engineering, and science. The pervasive availability of computationally powerful touch-screen devices, similar to the recent emergence of deep neural networks as high-quality sequence recognition models, result in the widespread adoption of online recognition of handwritten mathematical expressions. Also, a deeper study and improvement of such technologies is necessary to address the current challenges posed by the extensive usage of distance learning, and remote work due to the world pandemic. This paper delineates the state-of-the-art recognition methods along with the user's experience in pen-centric applications for operating with handwritten mathematical expressions. Recognition methods have been categorized into classes, with a description of their merits and limitations. Particular attention is paid to end-to-end approaches based on encoder-decoder architecture and multi-modal input. Evaluation protocols and open benchmark datasets are considered as well as the comparison of the recognition performance, based on open competition results. The use of handwritten math recognition is illustrated by examples of applications for various fields and platforms. A distinctive part of the survey is that we also considered how UI design relies on the use of different recognition approaches, which is aimed at helping potential researchers improve the performance of the introduced approaches toward the best responses in practical applications. Finally, this paper presents the prospective survey of future research directions in handwritten mathematical expression recognition and their applications.

The Internet of Things is based on a communication network to automatically receive various information from the natural world. It uses intelligent objects with perception, communication and computers to automatically receive various information from nature. All independently managed physical objects are connected to each other to achieve integrated perception, reliable transmission and intelligent processing, and to establish intelligent information between people and objects, and objects and object service systems. This article mainly introduces the impact of industrial agglomeration on the regional economy in a simulated intelligent environment based on machine learning. This paper proposes a method to detect industrial agglomeration index to analyze industrial agglomeration. Through the establishment of the industrial agglomeration index system, the level of integration of the manufacturing industry in our city was objectively analyzed, and the impact of the integration of manufacturing industry on our city was tested empirically. Finally, the relationship between industrial integration and regional economic development was tested. The experimental results in this paper show that industrial integration in a smart environment based on simulation learning has a significant positive impact on regional economic development. Among them, the level of cooperation between the manufacturing service industry and the manufacturing industry increased by 1%, and the level of regional economic development increased by 0.025%.

Technology forecasting not only helps business managers to make the right decisions but also helps researchers to grasp the direction of technology development. Technology forecasting, which facilitates the identification of the development technologies with high potential, can be an effective tool to support the management and plan for the future research activities. For this purpose, this paper firstly constructs Multi-modal input based on deep learning (MIDL) text classification model to extract relevant SCI papers from Web of Science database from 1996 to 2019 for topic classification, and then apply the Ensemble Empirical Mode Decomposition (EEMD) and Long Short-Term Memory (LSTM) neural networks to build an EEMD-LSTM technology forecasting model to predict the future development of each research field. Besides, we verify the validity of the method by taking robotics as an example in this paper. The results show that the accuracy, recall and F1 of MIDL text classification model are 0.826, 0.822 and 0.824 respectively. As compared with the optimal results of other classification models, the accuracy, recall and F1 are improved by 4.1%, 3.5%, and 3.8% respectively. The mean MAPE of the EEMD-LSTM model is 7%, which is 11% lower than ARIMA, 8% lower than LSTM, and 10% lower than 2-layer-LSTM.

Autonomous driving (AD) provides a reliable solution for safe driving by replacing human drivers responsible for the majority of accidents. The emergence of Machine Learning, specifically Deep Reinforcement Learning (DRL), and its ability to solve complex games proved its potential to address AD challenges. However, model-free methods still suffer from safety-related issues that can be resolved using safe-DRL approaches. The addition of model-based safety filters to the learning-based algorithms provides safety bounds on their performance and constraint satisfaction. In this paper, we investigate the addition of a safety filter based on Model Predictive Control and show an increase in mean testing episode reward by 110% from −75 mean episode reward during testing for 50 episodes for Deep Deterministic Policy Gradient ( $DDPG$ ) to 7.758. We study the impacts of safety filters (7.758 mean reward), heuristic rules, bounded additive noises (0.49% performance increase comparing to noise-free case), and exploration (3.425 mean reward) on the learning algorithm. We compare the effects of filters in the context of simulated exploration and bounded exploration and prove that bounded exploration results in 9.86% increase in mean reward and 12.95% decrease in std comparing to the other method. Additionally, inspired by Deep Internal Learning and biological mechanisms like brain plasticity, we investigate the idea of using each sample for training only once instead of utilizing stochastic batches which increases the mean testing accumulated reward by 1.87% and leads to the best performance (7.942 mean reward and 0.048 std). Finally, the results demonstrate better automotive results for our proposed method than DDPG. Our proposed method, DDPG with safety filter in bounded exploration and adaptive learning under noisy input conditions, has a success rate of 100% under different traffic densities for the simulation environment used in this paper and our assumptions. The proposed method’s automotive results are shown for a braking scenario to avoid collision with other road users.

Anomaly detection has become an indispensable tool for modern society, applied in a wide range of applications, from detecting fraudulent transactions to malignant brain tumours. Over time, many anomaly detection techniques have been introduced. However, in general, they all suffer from the same problem: a lack of data that represents anomalous behaviour. As anomalous behaviour is usually costly (or dangerous) for a system, it is difficult to gather enough data that represents such behaviour. This, in turn, makes it difficult to develop and evaluate anomaly detection techniques. Recently, generative adversarial networks (GANs) have attracted a great deal of attention in anomaly detection research, due to their unique ability to generate new data. In this paper, we present a systematic literature review of the applications of GANs in anomaly detection, covering 128 papers on the subject. The goal of this review paper is to analyze and summarize: (1) which anomaly detection techniques can benefit from certain types of GANs, and how, (2) in which application domains GAN-assisted anomaly detection techniques have been applied, and (3) which datasets and performance metrics have been used to evaluate these techniques. Our study helps researchers and practitioners to find the most suitable GAN-assisted anomaly detection technique for their application. In addition, we present a research roadmap for future studies in this area., Comment: 34 pages,5 figures

This paper discusses impact of recovering missing Electroencephalography (EEG) data on classification accuracy of hand movements using tensor-based methods. Improvement in classification accuracy is important for efficient performance of prosthesis. Classification accuracy relies on quality of the observed data. In practice, observed data is usually incomplete because of disconnection of electrodes and other artefacts, which negatively affect the classification accuracy. In this paper, we employ tensor-based imputation methods (canonical/polyadiac decomposition (CPD), weighted optimization version of CPD (CP-WOPT and Nonnegative Matrix Factorization (NMF)) to recover missing data in EEG signals and apply various classifiers to classify hand movements on recovered data. In particular, structured missing data was considered because that is how data gets missed in real-life data acquisition. Percentage of missing data was changed from 10% to 50%. Classifiers are applied on complete, missing and recovered data explicitly to test the performance of our framework. Results show that mean classification accuracy on complete data, missing data and recovered data was 71%, 53% and 64% respectively which shows applicability of our framework.

The new coronavirus, which has become a global pandemic, has confirmed more than 88 million cases worldwide since the first case was recorded in December 2019, causing over 1.9 million deaths. Since COIVD-19 lesions have clear imaging features on CT images, it is suitable for the auxiliary diagnosis and treatment of COVID-19. Deep learning can be used to segment the lesions areas of COVID-19 in CT images to help monitor the epidemic situation. In this paper, we propose a multi-point supervision network (MPS-Net) for segmentation of COVID-19 lung infection CT image lesions to solve the problem of a variety of lesion shapes and areas. A multi-scale feature extraction structure, a sieve connection structure (SC), a multi-scale input structure and a multi-point supervised training structure were implemented into MPS-Net. In order to increase the ability to segment various lesion areas of different sizes, the multi-scale feature extraction structure and the sieve connection structure will use different sizes of receptive fields to extract feature maps of various scales. The multi-scale input structure is used to minimize the edge loss caused by the convolution process. In order to improve the accuracy of segmentation, we propose a multi-point supervision training structure to extract supervision signals from different up-sampling points on the network. Experimental results showed that the dice similarity coefficient (DSC), sensitivity, specificity and IOU of the segmentation results of our model are 0.8325, 0.8406, 09988 and 0.742, respectively. The experimental results demonstrated that the network proposed in this paper can effectively segment COVID-19 infection on CT images. It can be used to assist the diagnosis and treatment of new coronary pneumonia.

In recent years, a variety of supervised manifold learning techniques have been proposed to outperform their unsupervised alternative versions in terms of classification accuracy and data structure capturing. Some dissimilarity measures have been used in these techniques to guide the dimensionality reduction process. Their good performance was empirically demonstrated; however, the relevant analysis is still missing. This paper contributes to a theoretical analysis on a) how dissimilarity measures affect maintaining manifold neighbourhood structure and b) how supervised manifold learning techniques could contribute to the reduction of classification error. This paper also provides a cross-comparison between supervised and unsupervised manifold learning approaches in terms of structure capturing using Kendall’s Tau coefficients and co-ranking matrices. Four different metrics (including three dissimilarity measures and Euclidean distance) have been considered along with manifold learning methods such as Isomap, ${t}$ -Stochastic Neighbour Embedding ( ${t}$ -SNE), and Laplacian Eigenmaps (LE), in two datasets: Breast Cancer and Swiss-Roll. This paper concludes that although the dissimilarity measures used in the manifold learning techniques can reduce classification error, they do not learn well or preserve the structure of the hidden manifold in the high dimensional space, but instead, they destroy the structure of the data. Based on the findings of this paper, it is advisable to use supervised manifold learning techniques as a pre-processing step in classification. In addition, it is not advisable to apply supervised manifold learning for visualization purposes since the two-dimensional representation using supervised manifold learning does not improve the preservation of data structure.

It is expected that peer to peer energy trading will constitute a significant share of research in upcoming generation power systems due to the rising demand of energy in smart microgrids. However, the on-demand use of energy is considered a big challenge to achieve the optimal cost for households. This paper proposes a blockchain-based predictive energy trading platform to provide real-time support, day-ahead controlling, and generation scheduling of distributed energy resources. The proposed blockchain-based platform consists of two modules; blockchain-based energy trading and smart contract enabled predictive analytics modules. The blockchain module allows peers with real-time energy consumption monitoring, easy energy trading control, reward model, and unchangeable energy trading transaction logs. The smart contract enabled predictive analytics module aims to build a prediction model based on historical energy consumption data to predict short-term energy consumption. This paper uses real energy consumption data acquired from the Jeju province energy department, the Republic of Korea. This study aims to achieve optimal power flow and energy crowdsourcing, supporting energy trading among the consumer and prosumer. Energy trading is based on day-ahead, real-time control, and scheduling of distributed energy resources to meet the smart grid’s load demand. Moreover, we use data mining techniques to perform time-series analysis to extract and analyze underlying patterns from the historical energy consumption data. The time-series analysis supports energy management to devise better future decisions to plan and manage energy resources effectively. To evaluate the proposed predictive model’s performance, we have used several statistical measures, such as mean square error and root mean square error on various machine learning models, namely recurrent neural networks and alike. Moreover, we also evaluate the blockchain platform’s effectiveness through hyperledger calliper in terms of latency, throughput, and resource utilization. Based on the experimental results, the proposed model is effectively used for energy crowdsourcing between the prosumer and consumer to attain service quality.

The epidemic increase in online reviews’ growth made the sentiment classification a fascinating domain in academic and industrial research. The reviews assist several domains, which is complicated to gather annotated training data. Several sentiment classification methodologies are devised for performing the sentiment analysis, but retrieval of information is not accurately performed, less effective, and less convergence speed. In this paper, we propose a sentiment paper proposes a sentiment classification model, namely Spider Monkey Crow Optimization algorithm (SMCA), for training the deep recurrent neural network (DeepRNN). In this method, the telecom review is employed to remove stop words and stemming to eliminate inappropriate data to minimize user’s seeking time. Meanwhile, the feature extraction is performed using SentiWordNet to derive the sentiments from the reviews. The extracted SentiWordNet features and other features, like elongated words, punctuation, hashtag, and numerical values, are employed in the DeepRNN for classifying sentiments. To retrieve the required review, the Fuzzy K-Nearest neighbor (Fuzzy-KNN) is employed to retrieve the review based on a distance measure. With rigorous assessments and experimentation, it is observed that the proposed SMCA-based DeepRNN performs better in terms of accuracy of 97.7%, precision of 95.5%, recall of 94.6%, and F1-score 96.7%, respectively.

In order to solve the problems of unbalanced sample data and the lack of consideration of temporal information in existing Siamese-based trackers, this paper proposes a Siamese recurrent neural network and region proposal network (Siamese R-RPN), which can be trained in an end-to-end manner. Siamese R-RPN is consisted of Siamese network, recurrent neural network and region proposal network. Image features extracted by the Siamese network are strengthened by the channel and spatial attention mechanisms, and are sent to the RPN for classification and regression. Temporal information is processed by a recurrent neural network-based Long Short-Term Memory (LSTM) to predict the rough location of the target, it is mapped to the anchor feature map of the RPN for anchor selection. This makes the positive and negative samples participating in the training procedure to become more balanced and representative. Because of the collaborative use of temporal and spatial information, the tracker proposed in this paper has achieved state-of-the-art performance on three large tracking benchmarks—OTB 2015, VOT2016 and VOT 2018—where this verifies its effectiveness.

Hyperspectral and multispectral information processing systems and technologies have demonstrated its usefulness for the improvement of agricultural productivity and practices by providing useful information to farmers and crop managers on the factors affecting crop status and growth. These technologies are widely used in a range of agriculture applications such as crop management, crop yield forecasting, crop disease detection, and the monitoring of agriculture land usage, water, and soil conditions. Hyperspectral information sensing can acquire several hundred spectral bands that cover the electromagnetic spectrum of an observational scene in a single acquisition. The resulting hyperspectral data cube contains a large volume of spatial and spectral information. The hyperspectral sequence of images or video further increases the data generation velocity and volume which lead to the Big data challenges particularly in agricultural remote sensing applications. This paper is structured to first give a comprehensive review of representative studies to provide insights into significant research efforts in agriculture using Big data, machine learning and deep learning with the focus on frameworks or architectures, information processing and analytics with hyperspectral and multispectral data. The potential for utilizing Big data, machine learning and deep learning for hyperspectral and multispectral data in agriculture is very promising. The paper then further explores the potential of using ensemble machine learning and scalable parallel discriminant analysis which takes into consideration the spatial and spectral components for Big data in agriculture. To the best of our knowledge, no similar review study on agriculture with Big data, machine learning and deep learning for hyperspectral and multispectral information processing has been reported. Furthermore, the potential of ensemble machine learning and scalable parallel discriminant analysis has not been explored in agriculture information processing. Experiments and data analytics have been performed on hyperspectral data from agriculture for validation. The results have shown the good performance of our approach.

Dyslexia is a neurological disorder that is characterized by imprecise comprehension of words and generally poor reading performance. It affects a significant population of school-age children, with more occurrences in males, thus, putting them at risk of poor academic performance and low self-esteem for a lifetime. The long-term hope is to have a dyslexia diagnostic method that is informed by neural-biomarkers. In this regard, large numbers of machine learning methods and, more recently, deep learning methods have been implemented across various types of dataset with the above-chance classification accuracy. However, attainment of clinical acceptability of these state-of-the-art methods is bedeviled by certain challenges including lack of biologically-interpretable biomarkers, privacy of dataset and classifiers, hyper-parameter selection/optimization, and overfitting problem among others. This review paper critically analyzes recent machine learning methods for detecting dyslexia and its biomarkers and discusses challenges that require proper attentions from the users of deep learning methods in order to enable them to attain clinically relevance and acceptable level. The review is conducted within the premise of implementation and experimental outcomes for each of the 22 selected articles using the Preferred Reporting Items for Systematic review and Meta-Analyses (PRISMA) protocol, with a view to outlining some critical challenges for achieving high accuracy and reliability of the state-of-the-art machine learning methods. As an evidence-based protocol for reporting in systematic reviews and meta-analyses, PRISMA helps to ensure clarity and transparency of this paper by showing a four-phase flow diagram of the selection process for articles used in this review. It is therefore, envisaged that higher classification performance of clinical relevance can be achieved using deep learning models for dyslexia and its biomarkers by addressing identified potential challenges.

Several hypovigilance detection systems (HDx) were developed to avoid road-side accidents due to driver fatigue. They have suffered from several limitations. Notably many of these are focused on center-head position to define an area of interest (often referred to as PERCLOS (percentage eye closure)) without considering the face occlusion problem, light illumination, and suffer poor response time. These HDx systems mostly depend on image processing, vision-based, and multisensor-based features. To address these problems, the author utilized vision, sensors, environmental, and vehicular-based features that integrated together by fusion to predict multistage of HDx. Lately, few studies have utilized the combination of multimodal features and deep learning (DL) architectures. Those multimodal-based features (M-HDx) were feasible to predict stages of driver fatigue (multi-stage). However, there is a need to critically measure the performance of these M-HDx by carrying out a comparative analysis to recognize multi-stage of fatigue in terms of hardware-based benchmarks. Moreover, it is important to evaluate the M-HDx systems using different features-set with respect to traditional and advanced machine learning techniques. Therefore, the primary aim of this work is in algorithm and feature modeling, then compare the advantages and differences with other work. In this paper, a different study is conducted compare to state-of-the-art survey articles by statistically measuring the performance. After experiments on M-HDx systems, this paper concludes that there is still a research gap to real-time development of multistage M-HDx systems. In the end, the paper summarizes the directions, challenges, and applications in the development of HDx systems to assist other researchers for further research.

The sixth generation (6G) wireless communication network presents itself as a promising technique that can be utilized to provide a fully data-driven network evaluating and optimizing the end-to-end behavior and big volumes of a real-time network within a data rate of Tb/s. In addition, 6G adopts an average of 1000+ massive number of connections per person in one decade (2030 virtually instantaneously). The data-driven network is a novel service paradigm that offers a new application for the future of 6G wireless communication and network architecture. It enables ultra-reliable and low latency communication (URLLC) enhancing information transmission up to around 1 Tb/s data rate while achieving a 0.1 millisecond transmission latency. The main limitation of this technique is the computational power available for distributing with big data and greatly designed artificial neural networks. The work carried out in this paper aims to highlight improvements to the multi-level architecture by enabling artificial intelligence (AI) in URLLC providing a new technique in designing wireless networks. This is done through the application of learning, predicting, and decision-making to manage the stream of individuals trained by big data. The secondary aim of this research paper is to improve a multi-level architecture. This enables user level for device intelligence, cell level for edge intelligence, and cloud intelligence for URLLC. The improvement mainly depends on using the training process in unsupervised learning by developing data-driven resource management. In addition, improving a multi-level architecture for URLLC through deep learning (DL) would facilitate the creation of a data-driven AI system, 6G networks for intelligent devices, and technologies based on an effective learning capability. These investigational problems are essential in addressing the requirements in the creation of future smart networks. Moreover, this work provides further ideas on several research gaps between DL and 6G that are up-to-date unknown.

Terrain generation aims to automatize the procedure of creating landscapes using a computer system. The generation models must follow different terrains’ topographical features, such as areas with river deltas and other regions where water bodies affect the natural landscapes. It is possible to generate more realistic terrains thanks to improvements in computer graphics techniques and deep learning models that use specific hardware. However, the advance on the generation of terrains that include river deltas, fjords, and waterfalls has not had the same pace as other more studied landscapes. Therefore, as a contribution to the advance of the research of terrain generation with water bodies using generative models, this paper presents the DRCA2020 dataset, which is useful for supervised training. The proposed dataset contains eight different types of real-world satellite images. These images are grouped by the same geographical location. There are 13,184 groups; each one has three RGB surface images, a water coverage map, three binarizations of water coverage, and a digital elevation model (DEM). Additionally, this paper proposes the use of a cGAN composite model, trained with the DRCA2020 dataset, for generating synthetic DEMs from water coverage images and therefore, to create realistic surface texture images with promising validation results.

DNA microarray data analysis is infamous due to a massive number of features, imbalanced class distribution, and limited available samples. In this paper, we focus on high-dimensional multi-class imbalanced problems. The high dimensional and multi-class imbalanced problem has posed acute challenges for the conventional classifiers to effectively perform classification tasks on both the minority and majority classes. Numerous efforts have been devoted to addressing either high dimensionality dataset or class imbalance problems. Nonetheless, few methods have been proposed to address the intersection of multi-class imbalanced and high-dimensional problems concurrently due to their intricate interactions. This paper presents novel hybrid algorithms for feature selection with the high dimensional multi-class imbalanced problem using multiple filter-based rankers (MFR) and hybrid Grasshopper optimization algorithm (GOA). The Simulated Annealing (SA) algorithm is incorporated into GOA. SA is used to enhance the best solution found by the GOA algorithm. The aim of using the SA here is to tackle the slow convergence and improve the exploitation by searching the high-quality regions found by the GOA. The experimental results confirm the effectiveness of the proposed methods in improving the classification performance in terms of area under the curve (AUC) compared to other well-known methods, which guarantees the ability of the proposed methods in searching the feature space and identifying very robust and discriminative features that best predict the minority class.

Neuromorphic vision sensors provide low power sensing and capture salient spatial-temporal events. The majority of the existing neuromorphic sensing work focus on object detection. However, since they only record the events, they provide an efficient signal domain for privacy aware surveillance tasks. This paper explores how the neuromorphic vision sensor data streams can be analysed for human action recognition, which is a challenging application. The proposed method is based on handcrafted features. It consists of a pre-processing step for removing the noisy events followed by the extraction of handcrafted local and global feature vectors corresponding to the underlying human action. The local features are extracted considering a set of high-order descriptive statistics from the spatio-temporal events in a time window slice, while the global features are extracted by considering the frequencies of occurrences of the temporal event sequences. Then, low complexity classifiers, such as, support vector machines (SVM) and K-Nearest Neighbours (KNNs), are trained using these feature vectors. The proposed method evaluation uses three groups of datasets: Emulator-based, re-recording-based and native NVS-based. The proposed method has outperformed the existing methods in terms of human action recognition accuracy rates by 0.54%, 19.3%, and 25.61% for E-KTH, E-UCF11 and E-HMDB51 datasets, respectively. This paper also reports results for three further datasets: E-UCF50, R-UCF50, and N-Actions, which are reported for the first time for human action recognition on neuromorphic vision sensor domain.

In a smart urban environment, providing accurate information about human activities is an important task. It is a trend to implement human activity recognition (HAR) algorithms and applications on smart phones, including health monitoring, self-management system, health tracking and so on. However, human activity recognition(HAR) is very complex, and it is important to use the best technology and machine learning to understand human activities. One of the main problems of the existing HAR strategies is that the classification accuracy is relatively low, and in order to improve the accuracy, it needs high computational overhead. The purpose of this paper is to use an exploratory data analysis method to deal with HAR, after the implementation of different data mining techniques, the results of dimensionality reduction and visualization are obtained. The HAR method based on smart phone and EDA proposed in this paper is a high precision method. Compared with other classifiers, its accuracy is 96.56%. This article will discuss computational prediction activities and the computational limitations of using exploratory data to analyze (EDA) on 564 feature data frames. The experimental results show that HAR-based exploratory data analysis is a common sensory signal processing technology. The GridSearchCV and LinearSVC algorithm can provide accurate automatic human activity recognition (HAR), for elderly and disabled patients in need of continuous care, and it is a decision-making tool to support sports coaches to make plans.

Photo Response Non-Uniformity (PRNU) is considered the most effective trace for the image source attribution task. Its uniqueness ensures that the sensor pattern noises extracted from different cameras are strongly uncorrelated, even when they belong to the same camera model. However, with the advent of computational photography, most recent devices heavily process the acquired pixels, possibly introducing non-unique artifacts that may reduce PRNU noise's distinctiveness, especially when several exemplars of the same device model are involved in the analysis. Considering that PRNU is an image forensic technology that finds actual and wide use by law enforcement agencies worldwide, it is essential to keep validating such technology on recent devices as they appear. In this paper, we perform an extensive testing campaign on over 33.000 Flickr images belonging to 45 smartphone and 25 DSLR camera models released recently to determine how widespread the issue is and which is the plausible cause. Experiments highlight that most brands, like Samsung, Huawei, Canon, Nikon, Fujifilm, Sigma, and Leica, are strongly affected by this issue. We show that the primary cause of high false alarm rates cannot be directly related to specific camera models, firmware, nor image contents. It is evident that the effectiveness of \prnu based source identification on the most recent devices must be reconsidered in light of these results. Therefore, this paper is intended as a call to action for the scientific community rather than a complete treatment of the subject. Moreover, we believe publishing these data is important to raise awareness about a possible issue with PRNU reliability in the law enforcement world., Comment: Final paper in : https://ieeexplore.ieee.org/document/9393356

The increasing significance of technology in daily lives led to the need for the development of convenient methods of human-computer interaction (HCI). Given that the existing HCI approaches exhibit various limitations, hand gesture recognition-based HCI may serve as a more intuitive mode of human-machine interaction in many situations. In addition, the system has to be deployable on low-power devices for applicability in broadly defined Internet of Things (IoT) and smart home solutions. Recent advances exhibit the potential of deep learning models for gesture classification, whereas they are still limited to high-performance hardware. Embedded neural network accelerators are constrained in terms of available memory, central processing unit (CPU) clock speed, graphics processing unit (GPU) performance, and a number of supported operations. The aforementioned problems are addressed in this paper by namely two approaches - simplifying the signal processing pipeline to avoid recurrent structures and efficient topological design. This paper employs an intuitive scheme allowing for the generation of the data in the compressed form from the sequence of range-Doppler images (RDI). Thus, it allows for the design of a neural classifier avoiding the usage of recurrent layers. The proposed framework has been optimized for Intel® Neural Compute Stick 2 (Intel® NCS 2), at the same time achieving promising classification accuracy of 97.57%. To confirm the robustness of the proposed algorithm, five independent persons have been involved in the algorithm testing process.

Research on the particle size of blast piles has always been an essential issue in mining engineering. Reasonable blasting parameters can reduce mining costs and reduce the workload of secondary crushing, which can significantly improve mining efficiency. The usual particle size analysis methods include the sieving method, the large particle size statistical method and other manual measurement methods. Nevertheless, these methods have the disadvantages of a high labor intensity, low efficiency and low precision. This paper analyzes UAV image information based on the single-picture photogrammetry method of computer image processing technology. A two-dimensional empirical wavelet transform (EWT) is used for image noise reduction. The nonlocal multiscale fractional differential (NMFD) enhances the texture of dark images and uses superpixel image segmentation technology so that the processed image can meet the granularity statistical study requirements of blast piles. The research results show that the accuracy of the ore particle size distribution by the method proposed in this paper is more than 90%.

Unsupervised person re-identification (re-id) is an effective analysis for video surveillance in practice, which can train a pedestrian matching model without any annotations, and it is easy to deploy in unseen camera scenarios. The most challenging problem in unsupervised re-id task is the huge distribution-gap among different camera views, and the intrinsic correlations in unlabeled identities are also complicated to sufficiently explored. This paper proposes a Multi-order Cross-view Graph adversarial Network (MCGN) to bridge the cross-view distribution-gap, and mine the inherent discriminative information by multi-order triplet correlations. Specifically, MCGN firstly exploits graph representations by a cross-view graph convolutional network according to intra-view and inter-view graph structure, and then encodes each pedestrian image into a view-shared feature space, which is iteratively trained by a graph generative adversarial learning strategy to deeply bridge the distribution-gap. Finally, this paper proposes a multi-order discriminative learning module for composing reasonable triplet samples according to multi-order similarity correlations among unlabeled pedestrian images. Furthermore, sufficient experiments are conducted in two large scale person re-id datasets (Market-1501 and DukeMTMC-reID). The comparison to state-of-the-art methods and ablation study demonstrate the superiority of MCGN and the contribution of each module proposed in this paper.

Image super-resolution reconstruction has been widely used in remote sensing, medicine and other fields. In recent years, due to the rise of deep learning research and the successful application of convolutional neural networks in the image field, the super-resolution reconstruction technology based on deep learning has also achieved great development. However, there are still some problems that need to be solved. For example, the current mainstream image super-resolution algorithms based on single or multiple frames pursue high performance indicators such as PSNR and SSIM, while the reconstructed image is relatively smooth and lacks many high-frequency details. It is not conducive to application in a real environment. To address such problem, this paper proposes a super-resolution reconstruction model of sequential images based on Generative Adversarial Networks (GAN). The proposed approach combines the registration module to fuse adjacent frames, effectively use the detailed information in multiple consecutive frames, and enhances the spatio-temporality of low-resolution images in sequential images. While the GAN was used to improve the effect of image high-frequency texture detail reconstruction, WGAN was introduced to optimize model training. The reconstruction results not only improved the PSNR and SSIM indexes but also reconstructed more high-frequency detail textures. Finally, in order to further improve the perception effect, an additional registration loss item RLT is introduced in the GAN network perception loss. Through extensive experiments, it shows that the model proposed in this paper effectively obtains the information between the sequence images. When the PSNR and SSIM indicators are improve, it can reconstruct better high-frequency texture details than the current advanced multi-frame algorithms.

Pitting corrosion of rotating machinery is one of the most common faults in industrial engineering. The convolutional neural network (CNN) is increasingly applied to the fault diagnosis. However, the conventional CNN method will reduce the feature dimension of the collected signal and cause the loss of information during the pooling process. In this paper, a new method based on dilated CNN combined with spatial dropout (DCSD) is proposed to diagnose the early faults of rotating machinery. By filling the convolution kernel, the DCSD method can increase the receptive field of the CNN without increasing the number of parameters while retaining more features of the raw vibration signal of the rotating machine. To avoid the dropout method eliminates the adjacent elements with a strong correlation, the Spatial Dropout method is adopted to reduce the overfitting problem of deep networks. The early pitting gears experiment was designed to verify the DCSD method in this paper. The raw vibration signal data of 6 different healthy states were collected to verify the effectiveness of the method. The experimental results show that the DCSD method proposed can effectively distinguish the different early gears pitting, and the diagnostic accuracy is better than other popular deep learning methods.

Wind speed fluctuations and load demand variations represent the big challenges against wind energy conversion systems (WECS). Besides, the inefficient measuring devices and the environmental impacts (e.g. temperature, humidity, and noise signals) affect the system equipment, leading to increased system uncertainty issues. In addition, the time delay due to the communication channels can make a gap between the transmitted control signal and the WECS that causes instability for the WECS operation. To tackle these issues, this paper proposes an adaptive neuro-fuzzy inference system (ANFIS) as an effective control technique for blade pitch control of the WECS instead of the conventional controllers. However, the ANFIS requires a suitable dataset for training and testing to adjust its membership functions in order to provide effective performance. In this regard, this paper also suggests an effective strategy to prepare a sufficient dataset for training and testing of the ANFIS controller. Specifically, a new optimization algorithm named the mayfly optimization algorithm (MOA) is developed to find the optimal parameters of the proportional integral derivative (PID) controller to find the optimal dataset for training and testing of the ANFIS controller. To demonstrate the advantages of the proposed technique, it is compared with different three algorithms in the literature. Another contribution is that a new time-domain named figure of demerit is established to confirm the minimization of settling time and the maximum overshoot in a simultaneous manner. A lot of test scenarios are performed to confirm the effectiveness and robustness of the proposed ANFIS based technique. The robustness of the proposed method is verified based on the frequency domain conditions that are driven from Hermite–Biehler theorem. The results emphases that the proposed controller provides superior performance against the wind speed fluctuations, load demand variations, system parameters uncertainties, and the time delay of the communication channels.

The standardization and digitalization of nine constitutions of traditional Chinese medicine (TCM) have promoted the development of TCM automation. The keys to constitution identification are tongue diagnosis and consultation diagnosis. In this paper, a tongue detection method based on the combination of the histogram of oriented gradients (HOG) and the support vector machine (SVM) is proposed. To separate the tongue body and tongue coating, a k-means segmentation method based on the Lab color space is proposed. Based on the clustering analysis of the difference between the color components of the tongue body and tongue coating in the Lab color space, the separation is realized. Thus, the relationship between the tongue image and constitution can be analyzed. The method for consultation diagnosis is divided into a questionnaire scale and a question answering system. The questionnaire includes 29 quantified items with corresponding weights and answer scores, so the final score of each type of constitution is calculated. The final constitution type is determined on the basis of the scores of both diagnoses. In addition, the key of question answering system is text similarity calculation. First, a sentence similarity calculation method combining the n-gram language model and word2vec is proposed. Then, a multifeature fusion sentence similarity calculation method combining the Word Mover’s Distance (WMD) and editing distance is proposed. Finally, the experimental results show the effectiveness of the proposed method, and a constitution identification application is developed based on the content of this paper.

Pins are standard fasteners in power transmission lines, and the hidden dangers of pins falling off dramatically affects their safe operation. If a pin is missed, it is called pin defects in this paper. As the pin is a small target and has a complex background, traditional detection algorithms were used to identify pin defects from aerial images which suffer from poor accuracy and low efficiency. This paper proposed a target detection method based on cascaded convolutional neural networks. First, a small-scale shallow full convolutional neural network was used to obtain the region of interest; then, a deeper convolutional neural network conducted target classification and positioning on the obtained region of interest. Next, a nonlinear multilayer perceptron was introduced, the convolution kernel was decomposed, and the multi-scale feature maps were fused. At this point, an angle variable was added to the classification cross-entropy loss function. Multi-task learning and offline hard sample mining strategies were used in the training phase. The proposed method was tested on a self-built pin dataset and the remote sensing image RSOD dataset, and the experimental results proved its effectiveness. Our method can accurately identify pin defects in aerial images, thereby solving the engineering application problem of pin defect detection in transmission lines.

Surveillance cameras are everywhere, keeping an eye on pedestrians as they navigate through a scene. With this context, our paper addresses the problem of pedestrian attribute recognition (PAR). This problem entails recognizing attributes such as age-group, clothing style, accessories, footwear style etc. This is a multi-label problem and challenging even for human observers. The problem has rightly attracted attention recently from the computer vision community. In this paper, we adopt trainable Gabor wavelets (TGW) layers and use it with a convolution neural network (CNN). Whereas other researchers are using fixed Gabor filters with the CNN, the proposed layers are learnable and adapt to the dataset for a better recognition. We propose a multi-branch neural network where mixed-layers, a combination of the TGW and convolutional layer, make up the building block of our 3-branch deep neural network. We test our method on publicly available challenging datasets and compare our results with state of the art.

This paper proposes a human gait estimation algorithm using a multiple 360 degree 2D LiDARs system. The system is fixed on the ground at the shin height to scan human legs during gait. The multiple LiDARs system is to overcome the drawbacks of a single LiDAR system, which could lose data due to occlusion between legs during walking and has a short tracking range. The performance of a sensor fusion system strongly depends on the calibration. In this paper, we propose a calibration method using a cylinder with known radius as a specific marker. In contrast to other methods, the calibration parameters and the cylinder center points at different positions are estimated by a proposed iterative algorithm. The measurement noises in the LiDAR output are considered to increase the accuracy of calibration and human leg center points estimation. Instead of using least square fitting of circle algorithm to estimate the leg center point, a new iterative algorithm which includes measurement noises is proposed. Although multiple LiDARs are used, the discontinuities of leg center points could still happen. Therefore, a quadratic optimization based eighth-order splines algorithm is derived to interpolate and smooth the data. Two configurations of three LiDARs are tested in the experiment. The former is the triangle configuration in which the whole walking path is covered by all three LiDARs. This configuration minimizes the occlusion between legs. The maximum RMSE of step length estimation of this configuration compared with the optical camera system is 0.03m. The latter is the line configuration in which each LiDAR covers a certain walking path sequentially. This configuration maximizes the tracking range. The experiment with 20m straight walking has the RMSE of about 0.10m.

Aero-turbines usually work in various non-stationary and harsh operating environments. Its blades easily appear early crack faults in long-time operation, and it is difficult to dig out discriminant features of early crack. To solve these problems, in the paper, a novel intelligent approach for early crack diagnosis of turbine blades using three-dimensional blade tip clearance is presented. In order to improve feature learning ability to obtain better generalization ability, the paper firstly develops a novel deep learning method based on deep belief networks (DBNs). Considering the fact that the feature degradation easily occurs in deeper layers because of the change of the distribution in each layer's outputs with the increase of the layers, it is hard to decide which layer to learn features is useful for fault diagnosis. Accordingly, in the pre-training process, the global back-reconstruction (GBR) mechanism is introduced into DBNs to optimize the feature learning ability. The GBR mechanism can be realized between the input layer and hidden layers by “shortcut connection”, and the layer to learn more discriminant features can be determined automatically without prior knowledge. Moreover, due to three-dimensional blade tip clearance (3-DBTC) acquired from three different directions of turbine blades contains much more useful crack failure multiscale information, it is suitable to be used as an input from which to extract multiscale discriminant features. Eventually, in the supervised training, the softmax regression model is employed to classify the health conditions of turbine blades using these sensitive features learned from 3-DBTC. The experimental results show that the proposed method can effectively identify the crack of turbine blades with fairly high diagnostic accuracies and significantly outperform other methods considered in the paper.

A novel method for detection of important scenes in baseball videos based on correlation maximization between heterogeneous modalities via bidirectional time lag aware deep multiset canonical correlation analysis (BiTl-dMCCA) is presented in this paper. The proposed method enables detection of important scenes by collaboratively using baseball videos and their corresponding tweets. The technical contributions of this paper are twofold. First, since there are time lags between not only “tweets and corresponding multiple previous events” but also “events and corresponding multiple following posted tweets”, the proposed method considers these bidirectional time lags. Specifically, the representation of such bidirectional time lags into the derivation of their covariance matrices is newly introduced. Second, the proposed method adopts textual, visual and audio features calculated from tweets and videos as multi-modal time series features. Important scenes are detected as abnormal scenes via anomaly detection based on a generative adversarial network using multi-modal features projected by BiTl-dMCCA. The proposed method does not need any training data with annotation. Experimental results obtained by applying the proposed method to actual baseball matches show the effectiveness of the proposed method.

Identifying k ey frames is the first and necessary step before solving the variety of other B haratanatyam problems. The paper aims to partition the momentarily stationary frames (key frame s) from this dance video’s motion frames. The proposed key frame s (KFs) localization is novel, simple, and effective compared to the existing dance video analysis methods. It is distinctive from standard KFs detection algorithms as used in other human motion videos. In the dance’s basic structure, the occurrence of KFs during performances is often not completely stationary and varies with the dance form and the performer. Hence, it is not easy to decide a global threshold (on the quantum of motion) to work across dancers and performances. The earlier approaches try to compute the threshold iteratively. However, the novelty of the paper is: (a) formulating an adaptive threshold, (b) adopting Machine Learning (ML) approach and, (c) generating the effective feature by combining three frame differencing and bit-plane technique for the KF detection. In ML, we use Support Vector Machine (SVM) and Convolutional Neural Network (CNN) as the classifiers. The proposed approaches are also compared and analyzed with the earlier approaches. Finally, the proposed ML techniques emerge as a winner with around 90% accuracy.

Partial discharge (PD) detection and diagnosis based on the ultra-high frequency (UHF) signals is one of the most widely adopted methods to evaluate the internal insulation status of high voltage equipment. Benefit from the rapid development of computing hardware and data processing algorithms, the intelligent PD fault diagnosis method based on the UHF data has made considerable progress in the past two decades. This two-part paper aims to give a comprehensive review about the application of signal processing and machine learning technologies in UHF PD detection and diagnosis. These technologies are divided into three categories according to their respective purpose, which are the preprocessing technology, source localization technology and pattern recognition technology. As the first one of the two-part review, we focus on the preprocessing and localization approaches in this paper. Specifically, for the preprocessing topic, the methods for signal denoising, multi-source separation, and pulse segmentation are included. While for the localization topic, the time difference of arrival (TDOA) method, direction of arrival (DOA) method, received signal strength indicator (RSSI) method, and other latest methods are reviewed. For each topic, the basic ideas, recent research progresses, advantages and limitations are discussed in detail. Before the conclusion, we also make a discussion about the application effects of the above technologies and prospect some future directions accordingly. In the second paper, the pattern recognition problems based on the UHF PD data will be concentrated, especially the application of deep learning algorithms.

Camera localization is an essential technique in many applications, such as robot navigation, mixed reality, and unmanned vehicle. We are committed to solving the problem of predicting the 6-DoF pose of cameras from a single color image in a given three-Dimensional (3D) environment. In this paper, we proposed a robust learning model for it. Basically, our proposed methodology consists of two steps: image retrieval and space reprojection. The former is in charge of simultaneous localization and mapping based on pre-captured reference images that rely on the correspondence between pixel points and scene coordinates; whereas the latter carries out camera calibration between the 2D image plane and the 3D scene. Given a two-Dimensional (2D) image, the initial localization is accomplished rapidly by matching a reference image using Siamese networks. More precise localization is achieved by camera calibration between the 2D image and the 3D scene using a fully convolutional network. The experimental results on the public dataset show that our model is more robust and expandable than the previous methods. At the end of this paper, we also apply the system to Unmanned Aerial Vehicle (UAV) localization and achieve good results.

Stress is an escalated psycho-physiological state of the human body emerging in response to a challenging event or a demanding condition. Environmental factors that trigger stress are called stressors. In case of prolonged exposure to multiple stressors impacting simultaneously, a person’s mental and physical health can be adversely affected which can further lead to chronic health issues. To prevent stress-related issues, it is necessary to detect them in the nascent stages which are possible only by continuous monitoring of stress. Wearable devices promise real-time and continuous data collection, which helps in personal stress monitoring. In this paper, a comprehensive review has been presented, which focuses on stress detection using wearable sensors and applied machine learning techniques. This paper investigates the stress detection approaches adopted in accordance with the sensory devices such as wearable sensors, Electrocardiogram (ECG), Electroencephalography (EEG), and Photoplethysmography (PPG), and also depending on various environments like during driving, studying, and working. The stressors, techniques, results, advantages, limitations, and issues for each study are highlighted and expected to provide a path for future research studies. Also, a multimodal stress detection system using a wearable sensor-based deep learning technique has been proposed at the end.

Due to the proliferation of the Internet of Things (IoT), the IoT devices are becoming utilized at the edge network at a much higher rate. Conventionally, the IoT devices lack the computation resources required for carrying out ultra-edge analytics. In this paper, we go beyond the typical edge analytics paradigm, which is mostly limited to user-smartphones, and investigate how to embed intelligence into the ultra-edge IoT sensors. To conceptualize the smart IoT sensors with enhanced intelligence, we select the arrhythmia detection task employing Electrocardiogram (ECG) trace as one of the mobile health (mHealth) cases. The existing approaches are not feasible for ultra-edge IoT sensors due to the extensive noise-filtering and manual feature extraction phase. Hence, in this paper, to facilitate the analytics, we propose a Deep Learning-based Lightweight Arrhythmia Classification (DL-LAC) method, which employs only single-lead ECG trace and does not require noise-filtering and manual feature extraction steps. As the proposed technique, we design a one-dimensional Convolutional Neural Network (CNN) architecture. Complying with the ANSI/AAMI EC57:1998 standard, four heartbeat types are taken into consideration as class labels. The efficiency and the generalization ability of the proposed model are evaluated, employing four different datasets from PhysioNet. The experimental results demonstrate that the proposed DL method outperforms traditional methods such as the Delay Differential Equation (DDE)-based optimization, K-Nearest Neighbor (KNN), and Random Forest (RF). The proposed DL-LAC illustrates encouraging performance in terms of time and memory requirement when the trained model is transferred to virtualized microcontrollers connected to IoT sensors.

The application of Artificial Intelligence (AI) to financial investment is a research area that has attracted extensive research attention since the 1990s, when there was an accelerated technological development and popularization of the personal computer. Since then, countless approaches have been proposed to deal with the problem of price prediction in the stock market. This paper presents a systematic review of the literature on Artificial Intelligence applied to investments in the stock market based on a sample of 2326 papers from the Scopus website between 1995 and 2019. These papers were divided into four categories: portfolio optimization, stock market prediction using AI, financial sentiment analysis, and combinations involving two or more approaches. For each category, the initial introductory research to its state-of-the-art applications are described. In addition, an overview of the review leads to the conclusion that this research area is gaining continuous attention and the literature is becoming increasingly specific and thorough.

Aiming at the difference in density between the light guide point distribution of the light guide plate (LGP) images, the different size, shape and brightness of LGP defects, and the limitation of a small number of defect samples, our paper proposes a two-stage multiscale residual attention network based on “segmentation + decision” for LGP defect detection. In the first stage, a segmentation subnet is constructed to achieve accurate location of multiscale defects by using the U-shaped structure and the designed Multiscale Residual Attention Unit (MRAU). In the second stage, a decision subnet is constructed to achieve accurate decision whether the LGP image is defective under the guidance of features extracted by segmentation subnet. The design of the network enables the model to be trained using a small number of samples, which is a significant requirement for practical applications. Finally, a lot of experimental studies were conducted on two datasets, including compared with related deep learning methods, proving the effectiveness of residual attention mechanism and proving the effectiveness of multiscale mechanism. Experimental results show that the detection method in this paper has advantages in both detection accuracy and universality under the limitation of a small number of samples. In particular, the detection indicator F1-Score respectively reached 99.85% and 95.46% on the two datasets. Combined with an image acquisition platform, a surface defect detection system is implemented to realize the real-time production quality inspection on the production lines.