Search

Many methods achieve the visual object tracking task with deep learning technologies. As the deep features of different levels contain various semantic information and functions, this paper presents a multi-hierarchy feature aggregation approach to tackle the specific issues in the tracking task, which consists of two aspects. On one hand, this paper integrates the features captured by the offline and online classifiers at the score level, which constructs complementary roles of these classifiers to enhance the stability of classification. Besides, the proposed offline classifier is continuously optimized with different levels of features to reinforce classification constraints. On the other hand, we design a butterfly attention module to promote the capacity of multi-hierarchy feature aggregation in the regression network, which aims to fuse and strengthen the multi-scale features by attending to their spatial information. It can capture more spatial contexts by utilizing the self-attention mechanism during the fusion procedure, and preserve the hierarchy of the features during the strengthening process. Extensive experiments on four public datasets, i.e., VOT2018, OTB100, NFS and LaSOT datasets, demonstrate the effectiveness of the proposed methods.

Context and rationale Intrusion Detection, the ability to detect malware and other attacks, is a crucial aspect to ensure cybersecurity. So is the ability to identify this myriad of attacks. Objective Artificial Neural Networks (as well as other machine learning bio-inspired approaches) are an established and proven method of accurate classification. ANNs are extremely versatile – a wide range of setups can achieve significantly different classification results. The main objective and contribution of this paper is the evaluation of the way the hyperparameters can influence the final classification result. Method and results In this paper, a wide range of ANN setups is put to comparison. We have performed our experiments on two benchmark datasets, namely NSL-KDD and CICIDS2017. Conclusions The most effective arrangement achieves the multi-class classification accuracy of 99.909% on an established benchmark dataset.

Single-object tracking is regarded as a challenging task in computer vision, especially in complex spatio-temporal contexts. The changes in the environment and object deformation make it difficult to track. In the last 10 years, the application of correlation filters and deep learning enhances the performance of trackers to a large extent. This paper summarizes single-object tracking algorithms based on correlation filters and deep learning. Firstly, we explain the definition of single-object tracking and analyze the components of general object tracking algorithms. Secondly, the single-object tracking algorithms proposed in the past decade are summarized according to different categories. Finally, this paper summarizes the achievements and problems of existing algorithms by analyzing experimental results and discusses the development trends.

The problem of pursuing the data items of a large database whose distances to a query item are the least is known as Similarity Search (Nearest Neighbor Search) problem. There exist various algorithms to address this problem. Some of the well known algorithms are i) exact algorithms ii) approximation algorithms and iii) randomized algorithms. This paper has made study only on exact and approximation algorithms because randomized algorithm produces approximate results with some probability. Recently, there are several approximation algorithms are proposed by the researchers because this type of algorithms minimizes the problem of Curse of Dimensionality. This paper mainly has two major sections. In first section, various methods under exact and approximation algorithms are discussed with regard to storage, preprocessing and query time. In the second section, efficient algorithms for similarity search suitable for certain physiological characteristics based biometric systems are considered. Biometric system has five main steps viz acquisition of Image, pre-processing, extraction of features, matching and making final decision. In this paper, indexing algorithms for similarity search suitable for iris trait based on different features are discussed in detail. Since the nature of features are distinct and different in biometric traits, there does not exist a universal (one unique) solution which can apply to all traits of biometric systems. Various performance measures like Penetration Rate and Hit Rate are used to determine the correct recognition rate with top best match (rank-1 accuracy).

Machine Learning has emerged in the last years as the main solution to many of nowadays’ data-based decision problems. However, while new and more powerful algorithms and the increasing availability of computational resources contributed to a widespread use of Machine Learning, significant challenges still remain. Two of the most significant nowadays are the need to explain a model’s predictions, and the significant costs of training and re-training models, especially with large datasets or in streaming scenarios. In this paper we address both issues by proposing an approach we deem predictive and user-centric. It is predictive in the sense that it estimates the benefit of re-training a model with new data, and it is user-centric in the sense that it implements an explainable interface that produces interpretable explanations that accompany predictions. The former allows to reduce necessary resources (e.g. time, costs) spent on retraining models when no improvements are expected, while the latter allows for human users to have additional information to support decision-making. We validate the proposed approach with a group of public datasets and present a real application scenario.

The prognostics and health management (PHM) aims to provide a comprehensive solution for equipment health care. Classifying the operation mode of excavator, one of the tasks in the PHM, is important to evaluate the remaining useful lifetime. Several studies have been conducted to classify the operations with either video or sensor data, but they have several limitations to use only one type of data. A model trained with sensor data cannot classify the similar operations such as “digging” and “ditch digging”, whereas a model with video data is vulnerable to surrounding condition like weather. In this paper, to overcome these shortcomings, we propose a deep neural network ensemble called FusionNet that classifies the operations of excavator. Two models are trained with sensor data and video frames respectively, where the feature extractors are transferred to the FusionNet. The proposed network ensemble performs a flexible and well-optimized classification by automatically calculating weights according to the extracted feature vectors and combining them. To verify the proposed model, several experiments are conducted with the real-world data. The proposed model achieves the accuracy of 99.17% which outperforms the conventional methods. We also confirm that the proposed model can address the shortcomings of using only one type of data and maximize the benefits through the automatic weighting of extracted features.

Recently, the Extreme Learning Machine (ELM) algorithm has been applied to various fields due to its rapidity and significant generalization performance. Traditionally, deep learning (DL) and wavelet neural networks (WNN) methods reach a high classification accuracy in machine learning applications. As a result, a new structure based on WNN, deep architecture and ELM is proposed in this paper. The proposed method is based on Extreme Learning Machine Auto-Encoder with DL structure and a composite wavelet activation function used in the hidden nodes. To evaluate the performance of our approach, we used standard benchmark data-sets, namely COIL-20, Pima Indian Diabetes (PID), MNIST and EMNIST. Experimental results show that our method offers satisfactory results and performance compared to other approaches.

Automatic International Classification of Diseases (ICD) coding is an important task in the future of artificial intelligence healthcare. In recent years, a lot of traditional machine learning-based methods have been proposed, and they achieved good results on this task. However, these traditional machine learning-based methods for automatic ICD coding only focus on the semantic features of clinical notes and ignore the feature extraction of ICD titles that are the descriptions of ICD codes. In this paper, we propose a knowledge attention-based deep learning framework called KAICD for automatic ICD coding. KAICD makes full use of the clinic notes and the ICD titles. The semantic features of clinic notes are extracted by a multi-scale convolutional neural network. For ICD titles, we use attention-based Bidirectional Gated Recurrent Unit (Bi-GRU) to build a knowledge database, which can offer additional information. Depending on input clinic notes, we can use the attention mechanism to obtain different knowledge vectors from the knowledge database where some ICD titles are more relevant to the input clinic notes. Last, we concatenate the knowledge vectors and the semantic features of clinic notes, and use them for the final prediction. KAICD is tested on a public dataset Medical Information Mart for Intensive Care III (MIMIC III); it achieves micro-precision of 0.502, micro-recall of 0.428, and micro-f1 of 0.462, which outperforms other competing methods. Furthermore, the results of the ablation study show that the knowledge database of ICD titles learned by the attention-based Bi-GRU enhances the feature expression and improves the prediction performance.

Fraud detection is essential for e-commerce platforms to maintain a fair business environment. Many existing works propose manually designed methods such as label propagation and dense block mining rules on built user-item graphs to detect fraud behaviours, but they are always heuristic and thus have limited performance. Other learning-based methods can either handle only the fraud detection problem well in the transductive scenario when there is only structural information or require rich content features to obtain a good inductive ability. Considering that content features are not always available in practice and there are usually many fraudulent behaviours that belong to newly emerging users and items, how to learn effective inductive rules with structures only is still underexplored. In this paper, we propose a subgraph-based method named SubGNN for collective fraud detection. In SubGNN, first, we extract the subgraphs around the given edges (user behaviours) to be tested. Then, we remove nodes’ global IDs so that SubGNN is entity-independent. Finally, by learning knowledge reasoning rules on extracted heterogeneous subgraphs using our proposed relational graph isomorphism network (R-GIN), a powerful graph neural network (GNN) model, SubGNN can achieve precise fraud detection. Experiments are conducted on publicly available Amazon and Yelp datasets and a newly collected Taobao dataset. The results clearly show the advantages and prospects of our method. When using SubGNN to detect fraudulent transactions on Taobao, the precision is higher than 0.99 and more than 90 % of fraud samples are recalled.

In this paper, we develop a novel Joint Model Compression (referred to as JMC) method by combining structured pruning and dense knowledge distillation techniques to significantly compress original large language model into a deep compressed shallow network. In particular, a new Direct Importance-aware Structured Pruning (referred as DISP) approach is proposed to structurally prune the redundant structures in the Transformer networks directly based on the corresponding parameter matrices in the model. Besides, a Dense Knowledge Distillation (referred to as DKD) method is developed with a many-to-one layer mapping strategy to leverage more comprehensive layer-wise linguistic knowledge for the distillation. Further, the proposed structured pruning and dense knowledge distillation are integrated together to perform the joint compression, which enables us to achieve a significant compression without sacrificing model accuracy. The extensive experimental results across four NLP tasks on seven datasets demonstrate its effectiveness and superiority to the baselines, while maintaining similar performance to original large model with further remarkable benefits for inference-time speedup and memory efficiency.

The outbreak and rapid spread of coronavirus disease 2019 (COVID-19) has had a huge impact on the lives and safety of people around the world. Chest CT is considered an effective tool for the diagnosis and follow-up of COVID-19. For faster examination, automatic COVID-19 diagnostic techniques using deep learning on CT images have received increasing attention. However, the number and category of existing datasets for COVID-19 diagnosis that can be used for training are limited, and the number of initial COVID-19 samples is much smaller than the normal’s, which leads to the problem of class imbalance. It makes the classification algorithms difficult to learn the discriminative boundaries since the data of some classes are rich while others are scarce. Therefore, training robust deep neural networks with imbalanced data is a fundamental challenging but important task in the diagnosis of COVID-19. In this paper, we create a challenging clinical dataset (named COVID19-Diag) with category diversity and propose a novel imbalanced data classification method using deep supervised learning with a self-adaptive auxiliary loss (DSN-SAAL) for COVID-19 diagnosis. The loss function considers both the effects of data overlap between CT slices and possible noisy labels in clinical datasets on a multi-scale, deep supervised network framework by integrating the effective number of samples and a weighting regularization item. The learning process jointly and automatically optimizes all parameters over the deep supervised network, making our model generally applicable to a wide range of datasets. Extensive experiments are conducted on COVID19-Diag and three public COVID-19 diagnosis datasets. The results show that our DSN-SAAL outperforms the state-of-the-art methods and is effective for the diagnosis of COVID-19 in varying degrees of data imbalance.

Multi-modal abstractive summarization for videos is an emerging task, aiming to integrate multi-modal and multi-source inputs (video, audio transcript) into a compressed textual summary. Although recent multi-encoder-decoder models on this task have shown promising performance, they did not explicitly model interactions of multi-source inputs. While some strategies like co-attention are utilized for modeling this interaction, considering ultra-long sequences and additional decoder in this task, the coupling of multi-modal data from multi-encoders and decoder needs complicated structure and additional parameters. In this paper, we propose a concise Decoder-only Multi-modal Transformer (D-MmT) based on the above observations. Specifically, we cut the encoder structure, and introduce an in-out shared multi-modal decoder to make the multi-source and target fully interact and couple in the shared feature space, reducing the model parameter redundancy. Also, we design a concise cascaded cross-modal interaction (CXMI) module in the multi-modal decoder that generates joint fusion representations and spontaneously establishes a fine-grained intra- and inter- association between multi-modalities. In addition, to make full use of the ultra-long sequence information, we introduce a joint in-out loss to make the input transcript also participate in backpropagation to enhance the contextual feature representation. The experimental results on the How2 dataset show that the proposed model outperforms the current state-of-the-art approach with fewer model parameters. Further analysis and visualization show the effectiveness of our proposed framework.

Aspect level sentiment classification aims to extract fine-grained sentiment expressed towards specific aspects from a sentence. The key to this task lies in connecting aspects and their respective sentiment contexts. Existing methods measure the dependency weights between aspects and context words via either the semantic similarity between words captured by attention mechanism or the structural proximity between words in syntactic structures. However, methods in both groups fail to fully exploit explicit syntactic dependency, which we argue should be critical to identify sentiment contexts. In this paper, we propose a novel syntactic-dependency-based attention network (SDATT) to incorporate explicit syntactic dependency for aspect level sentiment classification. SDATT first models the dependency path between each word and the aspect to characterize aspect-oriented syntactic representation of each word. The generated syntactic representations are later fed into the attention layer to help infer the dependency weights for sentiment prediction. Experimental results on five benchmark datasets show the superior performance of the proposed model over state-of-the-art baselines.

The ever-increasing complexity of robots causes failures of them as a side effect. Successful detection of anomalies in robotic systems is a key issue in order to improve their maintenance and consequently reducing economic costs and downtime. Going one step further in the detection of anomalies in robots, different mechanisms to deal with data irregularities are proposed and validated in present paper in order to increase detection rates. More precisely, strategies to overcome missing values and class imbalance are considered as complementary tools to get better one-class classification results. The effect of such strategies is evaluated through cross-validation when applying a standard supervised learning model, the Support Vector Machine. Experiments are run on an up-to-date and public dataset that contains some examples of different software anomalies that the middleware of the robot under analysis may experience.

Recently, many scenarios, such as affective disorders treatment, have sparked rising needs for establishment of personalized emotion recognition (PER) models. Unfortunately, the data sparsity issue violates the basic i.i.d. assumption of supervised learning (i.e., training data and test data are independently and identically distributed). In this paper, we present a semi-supervised joint domain adaption (SSJDA) solution, aiming to inject the hidden domain knowledge from ample labeled data of multiple source individuals into the target subject’s customized model. Specifically, we put forward a novel Progressive Low-Rank Subspace Alignment (PLRSA) approach, which unifies a semi-supervised instance-transfer paradigm and an unsupervised mapping-transfer learning paradigm in a single optimization framework. We leverage the boosting-based TrAdaBoost algorithm and the Transfer Component Analysis (TCA) algorithm for the implementation of instance reweighting and feature matching, respectively. Then we introduce the l 2 , 1 - norm to pass feedback and make the joint learning feasible. The central idea is to progressively minimize the cross-domain distribution discrepancies to finally construct the optimal domain-invariant features. We systematically compare the PLRSA method with five state-of-the-art techniques using two public EEG datasets (DEAP and SEED). Both many-to-one and one-to-one evaluations are performed. The experimental results have confirmed the efficacy of the proposed method.

Multi-view clustering, which utilizes the ample information provided by multiple sources to obtain better performance, has attracted much attention. However, existing clustering algorithms either have no ability to offer confidence for each assignment or suffer from the disturbance of outliers. To address these problems, in this paper, we propose a novel multi-view fuzzy clustering method via transferring softmin to fuzzy models. To obtain fuzzy assignments, we utilize the softmin with temperature and further develop an efficient algorithm to solve the non-convex problem approximately. We also show another explanation for the algorithm from the aspect of the prior distribution of various views. Besides, we design a scalable robust loss function, which interpolates between l2-norm and the squared l2-norm, to enhance the robustness to outliers. Extensive experiments show the superiority of our model under different clustering metrics.

Interpretability is a significant aspect of the distributed word representation learning model. Although the most advanced pretrained models have achieved the best results till date, the interpretability of a pretrained model is difficult to explain clearly. For this reason, based on the interpretability of distributed word embeddings, this paper presents a method of learning word representation using joint context. At present, the existing distributed word representation models for learning word representations usually focus on either neighbor or syntactic context. We argue that it is necessary to simultaneously model both contexts. In particular, the point mutual information obtained by combining the two types of contexts can efficiently express the correlation between the words. We propose two alternative distribution models for learning word representations by employing the neighbor and syntactic contexts via a simple and effective joint learning framework. Furthermore, the proposed models are trained on a public corpus, and the learned representations are evaluated in word analogy, word similarity, and sentence classification tasks. The experimental results demonstrate the potential of the proposed method.

Relation extraction has been an active research interest in the field of Natural Language Processing (NLP). The past works primarily focused on a corpus of formal text which is inherently non-dialogic. Recently, the dialogue-based relation extraction task, which detects relations among speaker-aware entities scattering in dialogues, has been gradually arousing people’s attention. Some sequence-based neural methods have been carried out to obtain the relevant information. However, identifying cross-sentence relations remains unsolved, especially in the context of a specific-domain dialogue system. In this paper, we propose a Relational Attention Enhanced Graph Convolutional Network (RAEGCN), which constructs the whole dialogue as a semantic interactive graph by emphasizing the speaker-related information and leveraging various inter-sentence dependencies. A dense connectivity mechanism is also introduced to empower the multi-hop relational reasoning across sentences, which can capture both local and non-local features simultaneously. Experiments show the significant superiority and robustness of our model on a real-world dataset DialogRE, as compared with previous approaches.

Spoken language understanding (SLU) plays an indispensable role in the dialogue system. The traditional intention detection task is regarded as a classification problem where utterances are associated with pre-defined intents. However, the various expressions of user's intents and constantly emerging novel intents make the annotating time-consuming and labor-intensive, building massive obstacles for extending the model to new tasks. Identifying unexpected user intention and achieving the user's desire goal is a challenging task. Therefore, we conduct zero-shot intention detection based on a transformation-based learning manner. In this paper, we propose an intent-enhanced attentive capsule network (IE-BertCapsNet) further guides the aggregation process of the capsule network and generalizable useful features that can be adapted to emerging intentions. Coupling with the large margin cosine loss function, the proposed model can identify discriminative features by forcing the whole network to minimize inter-class distance and minimize intra-class distance. Finally, we leverage the IE-BertCapsNet’s feature extraction ability and knowledge transferring capability to conduct zero-shot intent detection and generalized zero-shot intent detection. Extensive experiments on five benchmark task-oriented datasets in four languages demonstrate that the proposed model can achieve competitive performance that can better discriminate known intents and detect unknown intents.

Recently, many methods enlarged the kernel size or fused multi-scale features to capture contextual information for parsing the hard samples, such as heavy occlusions, limb absence, and complex poses. Nevertheless, those required higher computational and cannot explore the effective global context to achieve more precise parsing results. In this paper, we propose an end-to-end network for human parsing, named Context Prior based Semantic-Spatial Graph Network (CP-SSGNet), which achieves higher precision and consistency by encoding the context prior in the graph model. CP-SSGNet consists of three modules, Semantic Constraint Module (SCM), Spatial Perceiving Module (SPM), and Intra-class Attention Module (IAM). SCM captures richer global dependencies by encoding semantic structure context prior in a semantic graph, which can reduce semantic structure errors. SPM learns enhanced local features by encoding the spatial consistency context prior in the spatial graph, which can optimize boundaries and reduce the local consistency errors. IAM utilizes the spatial graph with strong–weak connections for intra-class features aggregation to distinguish the inter-class features clearly. Extensive experiments are conducted on two challenging datasets, PASCAL-Person-Part and LIP, effectively achieving state-of-the-arts performance.

Link prediction aims to predict the missing interactions in evolving networks that may appear in the future. It has practical importance in various real-world applications, ranging from friendship recommendation, knowledge graph completion, target advertising, and protein–protein interaction prediction. Most of the recent efforts focus on the structure of the network while ignoring many other essential factors. In this paper, we present a modified Latent Dirichlet Allocation (LDA), and Hidden Naive Bayesian (HNB) based link prediction technique named PILHNB model for link prediction in dynamic social networks by considering behavioral controlling elements like relationship network structure, nodes’ attributes, location-based information of nodes, nodes’ popularity, users’ interests, and learning the evolution pattern of these factors in the networks. Experimental results on six real-world networks demonstrate our proposed models’ effectiveness and efficiency compared with existing state-of-the-art link prediction techniques.

Knowledge distillation (KD) is widely applied in the training of efficient neural network. A compact model, which is trained to mimic the representation of a cumbersome model for the same task, generally obtains a better performance compared with being trained with the ground truth label. Previous KD-based works mainly focus on two aspects: (1) designing various feature representation for knowledge transfer; (2) introducing different training mechanism such as progressive learning or adversarial learning. In this paper, we revisit the standard KD and observe that training with teacher’s logits might suffer from incorrect and uncertain supervision. To tackle these problems, we propose two novel approaches to deal with incorrect logits and uncertain logits respectively, which are called Logits Adjustment (LA) and Dynamic Temperature Distillation (DTD). To be specific, LA rectifies the incorrect logits according to ground truth label and certain rules. While DTD treats the temperature of KD as a dynamic sample wise parameter rather than a static and global hyper-parameter, which actually notes the uncertainty for each sample’s logits. With iteratively updating the sample wise temperature, the student model could pay more attention on the samples that confuse the teacher model. Experiments on CIFAR-10/100, CINIC-10 and Tiny ImageNet verify that the proposed methods yield encouraging improvement compared with the standard KD. Furthermore, considering the simple implementations, LA and DTD can be easily attached to many KD-based frameworks and bring improvements without extra cost of training time and computing resources.

Influence Maximization (IM) is the problem of finding a small set of highly influential users in the social networks. The influence spreads according to an explicit influence propagation model. IM is an essential component in many applications such as Network Monitoring and Viral Marketing. Most of the present IM solutions neglect the highly dynamic behavior of social networks. It can result in either deprived seed qualities or a prolonged processing time. In this paper, we study the IM problem in a social network that evolves with time and proposes a new Link Prediction based Influential Node Tracking (LPINT) framework. In the proposed model, we apply the conditional temporal Restricted Boltzmann Machine (ctRBM) to predict the upcoming snapshot of the graph by predicting the links that may appear in the network by considering the evolutionary network’s temporal and structural pattern. And then, we apply an efficient IM technique for finding the seed nodes in the predicted snapshot of the network. Finally, we evaluate the spread of influence in the latest snapshot of the graph using predicted seed nodes. Extensive experimentation on four real large-scale datasets confirms that our LPINT model attains better performance in terms of influence coverage and influence spread time for considered networks compared to the baseline techniques.

Detail-and-geometry richness is essential to bas-relief modelling. However, existing image-based and model-based bas-relief modelling techniques commonly suffer from detail monotony or geometry loss. In this paper, we introduce a new bas-relief modelling framework for detail abundance with visual attention based mask generation and geometry preservation, which benefits from our two key contributions. For detail richness, we propose a novel semantic neural network of normal transfer to enrich the texture styles on bas-reliefs. For geometry preservation, we introduce a normal decomposition scheme based on Domain Transfer Recursive Filter (DTRF). Experimental results demonstrate that our approach is advantageous on producing bas-relief modellings with both fine details and geometry preservation.

Most existing methods for video person re-identification apply spatial-temporal global average or attention pooling to aggregate frame-level feature and get video-level feature. The obtained video-level feature models only the first-order statistics of the appearance feature from holistic video, resulting in limited representation capability of the feature network. In this paper, we propose a novel Global Statistic Pooling network (GSPnet) which takes full advantage of the second-order information for enhancing modeling capability. Firstly, a novel global statistic pooling module is proposed to summarize both the first- and second-order statistics across frame-level feature, and then transfer them into a compact and robust video-level feature embedding. Secondly, a statistic-based attention block is incorporated into multiple stages of convolutional networks to fully explore the second-order representations from low- to high-level features. To enhance the representation learning ability and further boost re-identification (re-ID) performance, we also propose a multi-level self-attention distillation training scheme, which squeezes the knowledge learned in the deeper portion of the networks into the shallow ones. Extensive experimental results have demonstrated the effectiveness and superiority of our approach on four popular video person re-ID datasets.

In the diversity of contemporary decision-making tasks, where the data is no longer static and changes over time, data stream processing has become an important issue in the field of pattern recognition. In addition, most of the real problems are not balanced, representing their classes in various improportions. Following paper proposes the Prior Imbalance Compensation method, modifying on-the-fly predictions made by the base classifier, aiming at mapping prior probability in the statistics of assigned classes. It is intended to be a less computationally complex competition for popular algorithms such as smote , solving this problem by oversampling the training set. The proposed method has been tested using computer experiments on the example of a set of various data streams, leading to promising results, suggesting its usefulness in solving this type of problems.

Although recent deep generative models are able to generate high-resolution, diverse natural samples from complex datasets, the generated samples still exist some problems in terms of images structure and detailed texture. In this paper, we propose a novel network architecture–SEDA-GAN that can learn the potential relationship in the dimension of the channel to enhance the generation performance of GAN. The proposed architecture applies Squeeze-and-Excitation(SE) block for feature recalibration to model channel-interdependencies within the GAN feature, and it also incorporates a dual-attention(DA) model with a channel attention mechanism in the GAN framework that can obtain global dependencies between channels. After conducting some comparative experiments on CIFAR and ImageNet datasets by using model BIGGAN as a baseline, our model performance has a certain improvement when evaluating on Fr e chet Inception Distance(FID) and Inception Score(IS) respectively.

Recently, deep convolutional neural networks have demonstrated great effectiveness in action recognition with both RGB and optical flow in the past decade. However, existing studies generally treat all frames and pixels equally, potentially leading to poor robustness of models. In this paper, we propose a novel parameter-free spatial–temporal pooling block (referred to as STP) for action recognition in videos to address this challenge. STP is proposed to learn spatial and temporal weights, which are further used to guide information compression. Different from other temporal pooling layers, STP is more efficient as it discards the non-informative frames in a certain clip. In addition, STP applies a novel loss function that forces the model to learn information from sparse and discriminative frames. Moreover, we introduce a dataset for ferry action classification, named Ferryboat-4, which includes four categories: Inshore, Offshore, Traffic, and Negative. This designed dataset can be used for the identification of ferries with abnormal behaviors, providing the essential information to support the supervision, management, and monitoring of ships. All the videos are acquired via real-world cameras. We perform extensive experiments on publicly available datasets as well as Ferryboat-4 and find that the proposed method outperforms several state-of-the-art methods in action classification. Source code and datasets are available at https://github.com/jiaming-wang/STP .

Person re-identification (re-ID) has made substantial progress in recent years; however, it is still challenging to search for the target person in a short time. Re-ID with deep hashing is a shortcut for that but, limited by the expression of binary code, the performance of the hashing method is not satisfactory. Besides, to further speed up retrieval, researchers tend to reduce the number of feature bits, which will cause more performance degradation. In this paper, we design the attribute-based fast retrieval (AFR), which leverages the attribute prediction of the model trained in a binary classification manner tailor-made for hashing. The attribute information is also used to refine the global feature representation by an attribute-guided attention block (AAB). Then, to fully exploit deep feature to generate the hash codes, we propose a binary code learning method, named self-distilling smooth relaxation (SSR). In this method, a simple yet effective regularization is presented to distill the quantized knowledge in the model itself, thus mitigating the lack of semantic guidance in the traditional non-linear relaxations. We manually label attributes for each person in dataset CUHK03 and evaluate our method on four authoritative public benchmarks (Market-1501, Market-1501+500K, CUHK03, and DukeMTMC-reID). The experimental results indicate that with the SSR and AAB, we surpass all the state-of-the-art hashing methods. And compared with reducing the feature bits, the AFR strategy is more effective to save search time.

Hindsight Experience Replay (HER) is a multi-goal reinforcement learning algorithm for sparse reward functions. The algorithm treats every failure as a success for an alternative (virtual) goal that has been achieved in the episode. Virtual goals are randomly selected, irrespective of which are most instructive for the agent. In this paper, we present two improvements over the existing HER algorithm. First, we prioritize virtual goals from which the agent will learn more valuable information. We call this property the instructiveness of the virtual goal and define it by a heuristic measure, which expresses how well the agent will be able to generalize from that virtual goal to actual goals. Secondly, we reduce existing bias in HER by the removal of misleading samples. To test our algorithms, we built three challenging environments with sparse reward functions. Our empirical results in both environments show vast improvement in the final success rate and sample efficiency when compared to the original HER algorithm. A video showing experimental results is available at https://youtu.be/xjAiwJiSeLc.

Representation learning (RL) has gained increasingly considerable attention in intelligent fault diagnosis due to its great capability of automatically learning useful features. Most existing studies focus on developing various variants about RL through modifying loss function of original versions, whereas it is a challenging task. Using the promising sparse filtering as the basic module, this paper presents a simple and effective RL method called multi-scale and multi-pooling sparse filtering (MSMPSF). Instead of modifying loss function of sparse filtering, we simply introduce two fusion mechanisms into sparse filtering, i.e., multi-scale fusion and multi-pooling fusion. In detail, the former aims to learn different local features from the collected signals under multiple scales. The latter tries to fuse various local features using multiple poolings. With these two mechanisms, MSMPSF is capable of capturing complementary fault information hidden in raw signals of several scales and obtaining richly informative feature representations, hence it can perform better. The proposed method is evaluated through experiments on three datasets about gear and bearing. Extensive comparison results confirm that both of two mechanisms facilitate a significant improvement on diagnosis performance. Furthermore, our method receives very reliable and competitive results in terms of diagnosis accuracy and stability in comparison with existing related works.

The task of text-to-image synthesis is a new challenge in the field of image synthesis. In the earlier research, the task of text-to-image synthesis is mainly to achieve the alignment of words and images by the way of retrieval based on the sentences or keywords. With the development of deep learning, especially the application of deep generative models in image synthesis, image synthesis achieves promising progress. The Generative adversarial networks (GANs) are one of the most significant generative models, and GANs have been successfully applied in computer vision, natural language processing and so on. In this paper, we review and summarize the recent research in GANs-based text-to-image synthesis, and provide a summary of the development of classic and advanced models. The input of the GANs-based text-to-image synthesis is not only the general text description as earlier studies, also includes scene layout and dialog text. The typical structure of each categories is elaborated. The general text-based image synthesis is the most commonly in the text-to-image synthesis, and it is subdivided into three groups based on the improvements of text information utilization, network structure and output control conditions. Through the survey, the detailed and logical overview of the evolution of GANs-based text-to-image synthesis is presented. Finally, the challenged problems and the future development of text-to-image synthesis are discussed.

Emotion recognition in conversation (ERC) is an important research topic in artificial intelligence. Different from the emotion estimation in individual utterances, ERC requires proper handling of human interactions in conversations. Several approaches have been proposed for ERC and achieved promising results. In this paper, we propose a correction model for previous approaches, called “Dialogical Emotion Correction Network (DECN)”. This model aims to automatically correct some errors made by emotion recognition strategies and further improve the recognition performance. Specifically, DECN employs a graphical network to model human interactions in conversations. To further utilize the contextual information, DECN also employs the multi-head attention based bi-directional GRU component. Since DECN is a correction model for ERC, it can be easily integrated with any emotion recognition strategy. Experimental results on the IEMOCAP and MELD datasets verify the effectiveness of our proposed method. DECN can improve the performance of emotion recognition strategies with few parameters and low computational complexity.

In this paper, we propose an attention-guided multi-scale fusion network (named as AMS-Net) for crowd counting in dense scenarios. The overall model is mainly comprised by the density and the attention networks. The density network is able to provide a coarse prediction of the crowd distribution (density map), while the attention network helps to distinguish crowded regions from backgrounds. The output of the attention network serves as a mask of the coarse density map. The number of persons in the scene is finally estimated by applying integration on the refined density map. In order to deal with persons of varied resolutions, we introduce a multi-scale fusion strategy which is built upon dilated convolution. Experiments are carried out on the standard benchmark datasets, covering varied over-crowded scenarios. Experimental results demonstrate the effectiveness of the proposed approach.

Existing methods that use document-level information for event detection ignore the dependencies between sentences and also have shortcomings in modeling the dependencies among words. In this paper, we propose a novel Hierarchical Graph Enhanced Event Detection (HGEED) framework to make full use of syntax and document information for the task of event detection. First, a sentence graph is used to model word-to-word dependencies, enriching the local information of words by incorporating syntactic features. Then, a document graph is built to model sentence-to-sentence dependencies, obtaining global semantic representations for word-level prediction. The experiment results on the widely used ACE 2005 and TAC KBP 2015 corpora show that our model can capture local and global information with dependencies and achieve significant improvements as compared to all baselines.

Recommendation with implicit feedback such as “bought” in e-commerce sites and “like” in online-to-offline services is an important problem because of the abundance of users’ online behaviors. Previous works for modeling users’ implicit feedback mainly focus on designing some predefined similarity, learned similarity, or hybrid similarity between two users (or two items). However, these similarities either can not capture both the local and global relations among users (or items) simultaneously, or they are not easy to be tuned due to the tradeoff parameter in the linear hybridization. Moreover, information from one single group, either a user group or an item group, may not be sufficient for modeling users’ preferences well. To this end, in this paper, we first propose a heterogeneous similarity to capture rich correlations among users and items via a concise integration, and then design a novel recommendation algorithm, i.e., pointwise factored heterogeneous similarity model (Poi-FHSM), in which we fully exploit the proposed similarity in a flexible pointwise preference learning paradigm. Finally, we conduct extensive empirical studies on five real-world datasets, and find that our Poi-FHSM performs better than those based on the predefined, learned and hybrid similarity, as well as the state-of-the-art deep learning-based methods, showcasing the effectiveness of our heterogeneous similarity and recommendation algorithm.

Knowledge Graph (KG), which commonly consists of fruitful connected facts about items, presents an unprecedented opportunity to alleviate the sparsity problem in recommender system. However, existing KG based recommendation methods mainly rely on handcrafted meta-path features or simple triple-level entity embedding, which cannot automatically capture entities’ long-term relational dependencies for the recommendation. Specially, entity embedding learning is not properly designed to combine user-item interaction information with KG context information. In this paper, a two-channel neural interaction method named Knowledge Graph enhanced Neural Collaborative Filtering with Residual Recurrent Network (KGNCF-RRN) is proposed, which leverages both long-term relational dependencies KG context and user-item interaction for recommendation. (1) For the KG context interaction channel, we propose Residual Recurrent Network (RRN) to construct context-based path embedding, which incorporates residual learning into traditional recurrent neural networks (RNNs) to efficiently encode the long-term relational dependencies of KG. The self-attention network is then applied to the path embedding to capture the polysemy of various user interaction behaviours. (2) For the user-item interaction channel, the user and item embeddings are fed into a newly designed two-dimensional interaction map. (3) Finally, above the two-channel neural interaction matrix, we employ a convolutional neural network to learn complex correlations between user and item. Extensive experimental results on three benchmark datasets show that our proposed approach outperforms existing state-of-the-art approaches for knowledge graph based recommendation.

Aspect-based sentiment analysis has obtained great success in recent years. Most of the existing work focuses on determining the sentiment polarity of the given aspect according to the given text, while little attention has been paid to the visual information as well as multimodality content for aspect-based sentiment analysis. Multimodal content is becoming increasingly popular in mainstream online social platforms and can help better extract user sentiments toward a given aspect. There are only few studies focusing on this new task: Multimodal Aspect-based Sentiment Analysis (MASA), which performs aspect-based sentiment analysis by integrating both texts and images. In this paper, we propose a mutimodal interaction model for MASA to learn the relationship among the text, image and aspect via interaction layers and adversarial training. Additionally, we build a new large-scale dataset for this task, named MASAD, which involves seven domains and 57 aspect categories with 38 k image-text pairs. Extensive experiments have been conducted on the proposed dataset to provide several baselines for this task. Though our models obtain significant improvement for this task, empirical results show that MASA is more challenging than textual aspect-based sentiment analysis, which indicates that MASA remains a challenging open problem and requires further efforts.

Story ending generation is a challenging and under-explored task, which aims at generating a coherent, reasonable, and logical story ending given a context. Previous studies mainly focus on utilizing the contextual information and commonsense knowledge to generate story endings. However, there are still some issues must be addressed in the story endings generation processing, such as sentimental consistency and interference from secondary information. In this paper, we propose a Gated Mechanism based Transformer Network (GMTF). The GMTF model utilizes the sentimental trend to make story ending generation more sentimentally consistent with the context. For a given story context, we utilize a sentiment analysis tool VADER to obtain the sentimental trend. Then, the sentimental information and contextual information are input jointly into the transformer network to capture the key clues. Furthermore, the gated mechanism is applied to filter irrelative information and the weights of attention layers for encoder and decoder are shared to make the most of the contextual clues. The experimental results on ROCStories dataset demonstrate that the proposed method achieves 27.03% on BLEU-1, 7.62% on BLEU-2, 1.71 on Grammar, and 1.31 on Logicality, respectively. Specifically, our model outperforms the state-of-the-art model IE+MSA by 0.23%, 0.22%, 1.78%, 5.64%, respectively and the Transformer model by 3.06%, 1.05%, 5.55%, 48.86%, respectively. Both automatic and manual evaluations show that our model can generate more reasonable and appropriate story endings compared with the related well-established approaches.

Zero-shot action recognition (ZSAR) aims to recognize novel actions that have not been seen in the training stage. However, ZSAR always suffers from serious domain shift problem, which causes poor performance. This is because: 1) Videos contain complicated intrinsic structures, including cross-sample visual correlations and cross-category semantic relationships, which make it challenging to generalize domain shift over categories and transfer knowledge across videos. 2) Existing methods do not disentangle unique and shared information underlying unseen videos during embedding. They are always weakly adaptive to novel categories and easily shift unseen videos to irrelevant action prototypes. In this paper, we propose a novel Coupling Adversarial Graph Embedding (CAGE) method for ZSAR, which formulates an effective visual-to-semantic embedding to alleviate the domain shift problem. Our model implements in a transductive setting that assumes accessing to a full set of unseen videos. Firstly, a structured graph is built for expressing both seen and unseen videos, which integrally captures visual and semantic relationships between them. Then, an effective visual-to-semantic embedding is formulated based on graph convolutional network (GCN), which is generalized to disjoint action categories and optimized for label propagation. In addition, a couple of adversarial constraints are proposed to characterize unique information of unseen videos and purify shared information across categories, which further improve the adaptability and discriminability of our model. Experiments on Olympic sports, HMDB51 and UCF101 datasets show that our model achieves impressive performance on ZSAR task.

Urban hotspots spatiotemporal prediction is a long-term but challenging task for urban management and smart city construction. Accurate urban hotspots spatiotemporal prediction can improve urban planning, scheduling and, security capability, reduce resource consumption. Existing deep spatiotemporal prediction methods mainly utilize geographic grid based image, some given network structure or some additional data to capture spatiotemporal dynamic. However, we observed that mining some latent self-semantics from raw data and fusing them with geospatial based grid images can also improve the performance of spatiotemporal predictions. In this paper, we propose Geographic-Semantic Ensemble Neural Network (GSEN), a novel deep learning approach to stack geographical prediction neural network and semantical prediction neutral network. GSEN model integrates the structures of Predictive Recurrent Neural Network (PredRNN), Graph Convolutional Predictive Recurrent Neural Network (GC-PredRNN), and Ensemble Layer to capture spatiotemporal dynamics from different views. And this model can also be correlated with some latent high-level dynamics in the real-world without any external data. We evaluate our proposed model on three different domains real-world datasets and the experimental studies demonstrate the generalization and effectiveness of GSEN in different urban hotspots spatiotemporal prediction tasks.

Image segmentation is an important field of computer vision and has attracted significant research attention in the recent years. In this paper, we provide a survey of regional level set image segmentation models based on the energy functional similarity measure. Our survey begins with an introduction to region-based level set image segmentation and an overview of its general steps. Then the different segmentation models are summarized. We define and survey six categories of regional level set image segmentation models based on energy functional similarity measures. For every category, we present the mainstream approaches from the literature as examples. Experimental analyses are conducted to compare the segmentation performance of various methods, which allow us to draw meaningful conclusions about their mutual advantages and disadvantages. Finally, we conclude this survey by highlighting several promising directions which need to be further explored by the research community in the future.

Autonomous underwater vehicles (AUVs) highly depend on the quality of captured underwater images to perform a variety of tasks. However, compared with everyday images taken in air, underwater images are hazy, with color shift, and in relatively low quality, posing significant challenges to available mature vision algorithms to achieve expected performance. There are, currently, two major lines of approaches to tackle these challenges: the physical image formation model-based and the neural-network-based approaches. In this paper, we propose an integrated approach, where the revised underwater image formation model, i.e., the Akkaynak-Treibitz model, is embedded into the network design for the benefit of combining the advantages of these two approaches. The embedded physical model guides for network learning, and the generative adversarial network (GAN) is adopted for coefficients estimation. We conduct extensive experiments and compare with state-of-the-art approaches quantitatively and qualitatively on nearly all the available underwater datasets, and our method achieves significant improvements.

Spatial–temporal prediction is a fundamental problem for intelligent transportation system (ITS), which is important for traffic management such as vehicle controls and travel plans. Deep learning has achieved success in spatial–temporal prediction with adequate data. However, many cities still suffer from data scarcity due to lack of essential infrastructure and services for data collection. Therefore, spatial–temporal prediction of cities with scarce data can be regarded as a few-shot learning problem. In this paper, we propose a novel transfer learning method to tackle spatial–temporal prediction tasks with only a small collection of data. Our proposed model aims to transfer the knowledge from multiple source cities to the target city by considering the different spatial–temporal distribution similarities between cities. Specifically, our model is designed as a spatial–temporal network based on a first-order meta-learning algorithm Reptile with an attention mechanism. Different from meta-learning algorithms that aim to learn a well-generalized initialization which can be adapted to any new task, our model achieves better performance on the specific target city by considering the different distribution similarities between multiple source cities and the target city. In addition, as it is difficult to learn the long-term spatial–temporal dependency with limited data in the target city, we propose a generation mechanism to learn and transfer long-term temporal features from source cities which have abundant long-period data to the target city. In the experiments, we compare our model with other state-of-the-art methods in real-world traffic prediction task. The experiments demonstrate the effectiveness of the proposed model.

The development of a functional myoelectric control represents a big challenge within the researchers community, due to the complexity of mapping the user’s movement intention onto the control signals. It is continuously gaining attention since it could be useful for building natural, intuitive and tailored human–machine interfaces. In this context, muscle synergies-based approaches are playing an important role since they may be useful to exploit the modular organization of the musculoskeletal system. Muscle synergies-based myo-control schemes have shown promising results when they are trained and validated at the same limb pose. However, dealing with a muscle-to-force mapping variability across multiple limb poses remains an open challenge, thus keeping these techniques unusable in several real application scenarios, e.g. rehabilitation contexts. In this paper, the authors propose a method able to compute the synergies-to-force mapping of a new limb pose by interpolation, with the knowledge of the synergies-to-force mapping related to a limited set of limb poses. The proposed interpolation-based approach has been evaluated on three different kind of mappings: muscle-to-force, “Pose-Shared” synergies-to-force and “Pose-Related” synergies-to-force. The muscle-to-force mapping considers a direct map between muscles and hand force. Both synergies-to-force approaches consider a map between muscle synergies and hand force, but, the “Pose-Shared” mapping assumes that the muscle patterns can be factorized using data coming from different limb poses, whereas the “Pose-Related” one assumes that each pose has its own set of muscle primitives that can be clustered together. The generalization capability of the proposed approach has been evaluated by comparing performances obtained in untrained conditions with the ones obtained in trained upper limb poses. Results showed that synergies-based approach substantially reduce the performance loss when tested on untrained upper-limb’s poses, demonstrating that muscle synergies may be suitable to be shared across different working conditions. Moreover, the feasibility of the proposed approach has been preliminary tested in an online condition, demonstrating that the subject was able to accomplish the force task by controlling a virtual cursor with his muscular activations.

End-to-end aspect-based sentiment analysis (E2E-ABSA) is a sequence labeling task which detects aspect terms and the corresponding sentiment simultaneously. Previous works ignore the useful task-specific knowledge and embed the vital aspect and sentiment attributes implicitly in the intermediate layers. In this paper, we propose a hierarchical multi-task learning framework, which explicitly leverages task-related knowledge via the supervision of intermediate layers. Specifically, aspect term extraction, sentiment lexicon detection, and aspect sentiment detection are designed to encode the aspect boundary and sentiment information. The tasks are in charge of different perspectives and levels of knowledge, which provide multi-fold regulation effects to optimize the main task. Unlike vanilla multi-task learning, all the tasks are integrated into a hierarchical structure to help the higher-level tasks make full use of the lower-level tasks’ information. Experimental results on three datasets demonstrate that the proposed method achieves state-of-the-art results. Further analysis shows that the proposed method achieves better performance than single-task and vanilla multi-task learning methods and yields a more discriminative feature representation.