Search

The capability of Convolutional Neural Networks (CNNs) for sparse representation has significant application to complex tasks like Representation Learning (RL). However, labelled datasets of sufficient size for learning this representation are not easily obtainable. The unsupervised learning capability of Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) provide a promising solution to this issue through their capacity to learn representations for novel data samples and classification tasks. In this research, a texture-based latent space disentanglement technique is proposed to enhance learning of representations for novel data samples. A comparison is performed among different VAEs and GANs with the proposed approach for synthesis of new data samples. Two different VAE architectures are considered, a single layer dense VAE and a convolution based VAE, to compare the effectiveness of different architectures for learning of the representations. The GANs are selected based on the distance metric for disjoint distribution divergence estimation of complex representation learning tasks. The proposed texture-based disentanglement has been shown to provide a significant improvement for disentangling the process of representation learning by conditioning the random noise and synthesising texture rich images of fruit and vegetables.

Multi-view classification with limited sample size and data augmentation is a very common machine learning (ML) problem in medicine. With limited data, a triplet network approach for two-stage representation learning has been proposed. However, effective training and verifying the features from the representation network for their suitability in subsequent classifiers are still unsolved problems. Although typical distance-based metrics for the training capture the overall class separability of the features, the performance according to these metrics does not always lead to an optimal classification. Consequently, an exhaustive tuning with all feature-classifier combinations is required to search for the best end result. To overcome this challenge, we developed a novel nearest-neighbor (NN) validation strategy based on the triplet metric. This strategy is supported by a theoretical foundation to provide the best selection of the features with a lower bound of the highest end performance. The proposed strategy is a transparent approach to identify whether to improve the features or the classifier. This avoids the need for repeated tuning. Our evaluations on real-world medical imaging tasks (i.e., radiation therapy delivery error prediction and sarcoma survival prediction) show that our strategy is superior to other common deep representation learning baselines [i.e., autoencoder (AE) and softmax]. The strategy addresses the issue of feature's interpretability which enables more holistic feature creation such that the medical experts can focus on specifying relevant data as opposed to tedious feature engineering.

Most existing multiview clustering methods are based on the original feature space. However, the feature redundancy and noise in the original feature space limit their clustering performance. Aiming at addressing this problem, some multiview clustering methods learn the latent data representation linearly, while performance may decline if the relation between the latent data representation and the original data is nonlinear. The other methods which nonlinearly learn the latent data representation usually conduct the latent representation learning and clustering separately, resulting in that the latent data representation might be not well adapted to clustering. Furthermore, none of them model the intercluster relation and intracluster correlation of data points, which limits the quality of the learned latent data representation and therefore influences the clustering performance. To solve these problems, this article proposes a novel multiview clustering method via proximity learning in latent representation space, named multiview latent proximity learning (MLPL). For one thing, MLPL learns the latent data representation in a nonlinear manner which takes the intercluster relation and intracluster correlation into consideration simultaneously. For another, through conducting the latent representation learning and consensus proximity learning simultaneously, MLPL learns a consensus proximity matrix with k connected components to output the clustering result directly. Extensive experiments are conducted on seven real-world datasets to demonstrate the effectiveness and superiority of the MLPL method compared with the state-of-the-art multiview clustering methods.

Non-Euclidean property of graph structures has faced interesting challenges when deep learning methods are applied. Graph convolutional networks (GCNs) can be regarded as one of the successful approaches to classification tasks on graph data, although the structure of this approach limits its performance. In this work, a novel representation learning approach is introduced based on spectral convolutions on graph-structured data in a semisupervised learning setting. Our proposed method, COnvOlving cLiques (COOL), is constructed as a neighborhood aggregation approach for learning node representations using established GCN architectures. This approach relies on aggregating local information by finding maximal cliques. Unlike the existing graph neural networks which follow a traditional neighborhood averaging scheme, COOL allows for aggregation of densely connected neighboring nodes of potentially differing locality. This leads to substantial improvements on multiple transductive node classification tasks.

The localization and segmentation of the novel coronavirus disease of 2019 (COVID-19) lesions from computerized tomography (CT) scans are of great significance for developing an efficient computer-aided diagnosis system. Deep learning (DL) has emerged as one of the best choices for developing such a system. However, several challenges limit the efficiency of DL approaches, including data heterogeneity, considerable variety in the shape and size of the lesions, lesion imbalance, and scarce annotation. In this article, a novel multitask regression network for segmenting COVID-19 lesions is proposed to address these challenges. We name the framework MT-nCov-Net. We formulate lesion segmentation as a multitask shape regression problem that enables partaking the poor-, intermediate-, and high-quality features between various tasks. A multiscale feature learning (MFL) module is presented to capture the multiscale semantic information, which helps to efficiently learn small and large lesion features while reducing the semantic gap between different scale representations. In addition, a fine-grained lesion localization (FLL) module is introduced to detect infection lesions using an adaptive dual-attention mechanism. The generated location map and the fused multiscale representations are subsequently passed to the lesion regression (LR) module to segment the infection lesions. MT-nCov-Net enables learning complete lesion properties to accurately segment the COVID-19 lesion by regressing its shape. MT-nCov-Net is experimentally evaluated on two public multisource datasets, and the overall performance validates its superiority over the current cutting-edge approaches and demonstrates its effectiveness in tackling the problems facing the diagnosis of COVID-19.

Self-supervised learning has attracted plenty of recent research interest. However, most works for self-supervision in speech are typically unimodal and there has been limited work that studies the interaction between audio and visual modalities for cross-modal self-supervision. This work (1) investigates visual self-supervision via face reconstruction to guide the learning of audio representations; (2) proposes an audio-only self-supervision approach for speech representation learning; (3) shows that a multi-task combination of the proposed visual and audio self-supervision is beneficial for learning richer features that are more robust in noisy conditions; (4) shows that self-supervised pretraining can outperform fully supervised training and is especially useful to prevent overfitting on smaller sized datasets. We evaluate our learned audio representations for discrete emotion recognition, continuous affect recognition and automatic speech recognition. We outperform existing self-supervised methods for all tested downstream tasks. Our results demonstrate the potential of visual self-supervision for audio feature learning and suggest that joint visual and audio self-supervision leads to more informative audio representations for speech and emotion recognition., Accepted for publication in IEEE Transactions on Affective Computing; v3: Publication-ready version including additional experiments and discussion

We propose an unsupervised cross-view facial expression adaptation network(UCFEAN) to simultaneously generate and recognize cross-view facial expressions in images in an unsupervised manner. The main idea of UCFEAN is to convert the unsupervised domain adaptation between two image spaces with different appearance into semi-supervised learning(SSL) in feature spaces with the same semantic content. The cyclic image generation of cross-view facial expressions based on the generative adversarial network(GAN) is carried out to project unlabeled target images and labelled source images to the corresponding feature spaces with the same semantic content. This helps realize the unsupervised feature learning of the target image. Labels of facial expressions represented in the projected target features can then be learned using the projected source features, because the distributions of the projected features in the two domains are close enough for knowledge transfer by using SSL. Three techniques are developed to train UCFEAN in an effective and stable manner. Extensive experiments are conducted to evaluate the UCFEAN on two multi-view facial expression image databases including RaFD and Multi-PIE. The results show that the proposed method can generate realistic target images of the facial expression and recognize cross-view facial expressions with high precision.

This paper proposes an image fusion framework based on separate representation learning, called IFSepR. We believe that both the co-modal image and the multi-modal image have common and private features based on prior knowledge, exploiting this disentangled representation can help to image fusion, especially to fusion rule design. Inspired by the autoencoder network and contrastive learning, a multi-branch encoder with contrastive constraints is built to learn the common and private features of paired images. In the fusion stage, based on the disentangled features, a general fusion rule is designed to integrate the private features, then combining the fused private features and the common feature are fed into the decoder, reconstructing the fused image. We perform a series of evaluations on three typical image fusion tasks, including multi-focus image fusion, infrared and visible image fusion, medical image fusion. Quantitative and qualitative comparison with five state-of-art image fusion methods demonstrates the advantages of our proposed model.

We focus on unsupervised representation learning for skeleton based action recognition. Existing unsupervised approaches usually learn action representations by motion prediction but they lack the ability to fully learn inherent semantic similarity. In this paper, we propose a novel framework named Prototypical Contrast and Reverse Prediction (PCRP) to address this challenge. Different from plain motion prediction, PCRP performs reverse motion prediction based on encoder-decoder structure to extract more discriminative temporal pattern, and derives action prototypes by clustering to explore the inherent action similarity within the action encoding. Specifically, we regard action prototypes as latent variables and formulate PCRP as an expectation-maximization (EM) task. PCRP iteratively runs (1) E-step as to determine the distribution of action prototypes by clustering action encoding from the encoder while estimating concentration around prototypes, and (2) M-step as optimizing the model by minimizing the proposed ProtoMAE loss, which helps simultaneously pull the action encoding closer to its assigned prototype by contrastive learning and perform reverse motion prediction task. Besides, the sorting can also serve as a temporal task similar as reverse prediction in the proposed framework. Extensive experiments on N-UCLA, NTU 60, and NTU 120 dataset present that PCRP outperforms main stream unsupervised methods and even achieves superior performance over many supervised methods.

In the fields of pattern recognition and data mining, two problems need to be addressed. First, the curse of dimensionality degrades the performance of many practical data processing techniques. Second, due to the existence of noise and outliers, feature extraction on corrupted data can not be effectively achieved. Recently, some representation based methods have produced promising results. However, these methods cannot handle the case in which nonlinear similarity exists and have failed to provide the quantized interpretability for the importance of features. In this paper, we propose a novel low rank and sparse representation method to realize dimensionality reduction and robustly extract latent low dimensional discriminative features. Specifically, we first adopt multiple kernel learning to map the original data into an embedded reproducing kernel Hilbert space (RKHS) and then kernel based similarity discriminative projection is learned to explore the within-class and between-class variability. Notably, this low dimensional feature learning strategy is definitely integrated into the low rank matrix recovery of the kernel matrix. Next, we introduce the regularization of l2;1 norm on error matrix to eliminate noise and on projection matrix to lead the selected features to be more compact and interpretable. The non-convex optimization problem is effectively solved by the alternating direction method of multipliers (ADM-M) methods. Extensive experiments on seven benchmark datasets are conducted to demonstrate the effectiveness of our method.

Multi-Panda Tracking (MPT) is a video-based tracking task for panda individuals, which is conducive to the observation and measurement of distribution and status of pandas. Different from tracking general objects such as pedestrians and vehicles, MPT is extremely challenging due to the indistinguishable appearances and diversified postures of pandas. In this case, existing tracking methods cannot appropriately tackle with the excessive occlusion between different panda individuals, hence suffering from identity switch, missing and inaccurate detections. To address these problems, we propose a simple yet effective MPT framework in the tracking-by-detection paradigm, which is benefited both from a short-term prediction filtering module and a discriminative feature learning network. In particular, the short-term prediction filtering module introduces similarity learning to enhance the temporal consistency among detections, which is capable of supplementing the missing detections and discarding false positive detections. Besides, the discriminative feature learning network leverages a two-branch network to learn both local and global discriminative features, so as to distinguish different panda individuals with a very similar appearance with a subtle difference. To evaluate the proposed method, we annotate a large-scale MPT dataset, named PANDA2021, which is particularly challenging due to the similar appearance and dramatic occlusion between panda individuals. Experiments on PANDA2021 demonstrate that the proposed MPT method significantly outperforms the competing methods. Moreover, experimental results on pedestrian tracking dataset MOT16 further demonstrate that the proposed MPT method achieves comparative performance with competing methods.

Andriod malware poses a serious threat to users privacy, money, equipment and file integrity. A series of data-driven malware detection methods were proposed. However, there exist two key challenges for these methods: (1) how to learn effective feature representation from raw data; (2) how to reduce the dependence on the prior knowledge or human labors in feature learning. Inspired by the success of deep learning methods in the feature representation learning community, we propose a malware detection framework which starts with learning rich-features by a novel unsupervised feature learning algorithm Merged Sparse Auto-Encoder (MSAE). In order to extract more compact and discriminative feature from the rich-features to further boost the malware detection capability, a hybrid deep network learning algorithm Stacked Hybrid Learning MSAE and SDAE (SHLMD) is established by further incorporating a classical deep learning method Stacked Denoising Auto-encoders (SDAE). After that, we feed the feature learned by MSAE and SHLMD respectively to classification algorithms to train a malware detection model. Evaluation results on two real-world datasets demonstrate that SHLMD achieves 94.46% and 90.57% accuracy respectively, which outperforms the classical unsupervised feature representation learning Sparse Auto-encoder (SAE).

Vibration signals are widely utilized in many fields, which can reflect machine health state. Those typical deep learning techniques cannot learn impulsive features from vibration signals due to interference of strong background noise. Supervised learning techniques greatly rely on vast labeled data, which limits implementation of deep learning in industry applications. Hence, a new deep neural network (DNN), sparse representation convolutional autoencoder (SRCAE) is proposed to extract impulsive components of vibration signals for machinery fault diagnosis in an unsupervised manner. A sparse representation (SR) block is proposed to extract impulsive components of vibration signals and transform time-domain signal to a sparse domain by sparse mapping of convolutional graph. The SR block is inserted into deep network to remove noise and learn impulsive features for machinery fault diagnosis. Furthermore, an unsupervised selective feature transmission mechanism (USFTM) is proposed to improve training efficiency and realize feature filtering simultaneously. Finally, the effectiveness of SRCAE is verified on rotary machine fault diagnosis experiments. The testing results show that SRCAE has good noise filtering and impulsive components extraction performance. The recognition accuracy of SRCAE reached 97.16% based on the five-fold cross validation, which demonstrates the outperformance of SRCAE in comparison with state-of-the-art DNNs.

Broad learning system (BLS) is a novel and efficient model, which facilitates representation learning and classification by concatenating feature nodes and enhancement nodes. In spite of the efficient properties, BLS is still suboptimal when facing with imbalance problem. Besides, outliers and noises in imbalanced data remain a challenge for BLS. To address the above issues, in this paper we firstly propose a weighted BLS, which assigns a weight to each training sample, and adopt a general weighting scheme, which augments the weight of samples from the minority class. To further explore the prior distribution of original data, we design a density based weight generation mechanism to guide the specific weight matrix generation and propose the adaptive weighted broad learning system (AWBLS). This mechanism considers the inter-class and intra-class distance simultaneously in the density calculation. Finally, we propose the incremental weighted ensemble broad learning system (IWEB) by utilizing a progressive mechanism to further improve the stability and robustness of AWBLS. Extensive comparative experiments on 30 real-world data sets verfy that IWEB outperforms most of the imbalance ensemble classification methods.

Modern convolutional neural network (CNN)-based object detectors focus on feature configuration during training but often ignore feature optimization during inference. In this article, we propose a new feature optimization approach to enhance features and suppress background noise in both the training and inference stages. We introduce a generic inference-aware feature filtering (IFF) module that can be easily combined with existing detectors, resulting in our iffDetector. Unlike conventional open-loop feature calculation approaches without feedback, the proposed IFF module performs the closed-loop feature optimization by leveraging high-level semantics to enhance the convolutional features. By applying the Fourier transform to analyze our detector, we prove that the IFF module acts as a negative feedback that can theoretically guarantee the stability of the feature learning. IFF can be fused with CNN-based object detectors in a plug-and-play manner with little computational cost overhead. Experiments on the PASCAL VOC and MS COCO datasets demonstrate that our iffDetector consistently outperforms state-of-the-art methods with significant margins.

Recently, the correlation filter (CF) and Siamese network have become the two most popular frameworks in object tracking. Existing CF trackers, however, are limited by feature learning and context usage, making them sensitive to boundary effects. In contrast, Siamese trackers can easily suffer from the interference of semantic distractors. To address the above problems, we propose an end-to-end target-insight correlation network (TICNet) for object tracking, which aims at breaking the above limitations on top of a unified network. TICNet is an asymmetric dual-branch network involving a target-background awareness model (TBAM), a spatial-channel attention network (SCAN), and a distractor-aware filter (DAF) for end-to-end learning. Specifically, TBAM aims to distinguish a target from the background in the pixel level, yielding a target likelihood map based on color statistics to mine distractors for DAF learning. SCAN consists of a basic convolutional network, a channel-attention network, and a spatial-attention network, aiming to generate attentive weights to enhance the representation learning of the tracker. Especially, we formulate a differentiable DAF and employ it as a learnable layer in the network, thus helping suppress distracting regions in the background. During testing, DAF, together with TBAM, yields a response map for the final target estimation. Extensive experiments on seven benchmarks demonstrate that TICNet outperforms the state-of-the-art methods while running at real-time speed.

Fake news detection attracts many researchers' attention due to the negative impacts on the society. Most existing fake news detection approaches mainly focus on semantic analysis of news' contents. However, the detection performance will dramatically decrease when the content of news is short. In this paper, we propose a novel $fake news detection multi-task learning (FDML)$ model based on the following observations: 1) some certain topics have higher percentages of fake news; and 2) some certain news authors have higher intentions to publish fake news. FDML model investigates the impact of topic labels for the fake news and introduce contextual information of news at the same time to boost the detection performance on the short fake news. Specifically, the FDML model consists of representation learning and multi-task learning parts to train the fake news detection task and the news topic classification task, simultaneously. As far as we know, this is the first fake news detection work that integrates the above two tasks. The experiment results show that the FDML model outperforms state-of-the-art methods on real-world fake news dataset.