Search

Recent years have witnessed a tremendous increasing of first-person videos captured by wearable devices. Such videos record information from different perspectives than the traditional third-person view, and thus show a wide range of potential usages. However, techniques for analyzing videos from different views can be fundamentally different, not to mention co-analyzing on both views to explore the shared information. In this paper, we take the challenge of cross-view video co-analysis and deliver a novel learning-based method. At the core of our method is the notion of "joint attention", indicating the shared attention regions that link the corresponding views, and eventually guide the shared representation learning across views. To this end, we propose a multi-branch deep network, which extracts cross-view joint attention and shared representation from static frames with spatial constraints, in a self-supervised and simultaneous manner. In addition, by incorporating the temporal transition model of the joint attention, we obtain spatial-temporal joint attention that can robustly capture the essential information extending through time. Our method outperforms the state-of-the-art on the standard cross-view video matching tasks on public datasets. Furthermore, we demonstrate how the learnt joint information can benefit various applications through a set of qualitative and quantitative experiments.

The task of multi-label image recognition is to predict a set of object labels that present in an image. As objects normally co-occur in an image, it is desirable to model label dependencies to improve recognition performance. To capture and explore such important information, we propose Graph Convolutional Networks based models for multi-label recognition, where directed graphs are constructed over classes and information is propagated between classes to learn inter-dependent class-level representations. Following this idea, we design two particular models that approach multi-label classification from different views. In our first model, the prior knowledge about the class dependencies is integrated into classifier learning. Specifically, we propose Classifier-Learning-GCN to map class-level semantic representations (\eg, word embedding) into classifiers that maintain the inter-class topology. In our second model, we decompose the visual representation of an image into a set of label-aware features and propose Prediction-Learning-GCN to encode such features into inter-dependent image-level prediction scores. Furthermore, we also present an effective correlation matrix construction approach to capture inter-class relationships and consequently guide information propagation among classes. Empirical results on generic multi-label recognition demonstrate that the effectiveness of both two proposed models. Moreover, the proposed methods also show advantages in other multi-label related applications.

This article introduces a new iterative approach to explainable feature learning. During each iteration, new features are generated, first by applying arithmetic operations on the input set of features. These are then evaluated in terms of probability distribution agreements between values of samples belonging to different classes. Finally, a graph-based approach for feature selection is proposed, which allows for selecting high-quality and uncorrelated features to be used in feature generation during the next iteration. As shown by the results, the proposed method improved the accuracy of all tested classifiers, where the best accuracies were achieved using random forest. In addition, the method turned out to be insensitive to both of the input parameters, while superior performances in comparison to the state of the art were demonstrated on nine out of 15 test sets and achieving comparable results in the others. Finally, we demonstrate the explainability of the learned feature representation for knowledge discovery.

Scene parsing, or semantic segmentation, aims at labeling all pixels in an image with the predefined categories of things and stuff. Learning a robust representation for each pixel is crucial for this task. Existing state-of-the-art (SOTA) algorithms employ deep neural networks to learn (discover) the representations needed for parsing from raw data. Nevertheless, these networks discover desired features or representations only from the given image (content), ignoring more generic knowledge contained in the dataset. To overcome this deficiency, we make the first attempt to explore the meaningful supportive knowledge, including general visual concepts (i.e., the generic representations for objects and stuff) and their relations from the whole dataset to enhance the underlying representations of a specific scene for better scene parsing. Specifically, we propose a novel supportive knowledge mining module (SKMM) and a knowledge augmentation operator (KAO), which can be easily plugged into modern scene parsing networks. By taking image-specific content and dataset-level supportive knowledge into full consideration, the resulting model, called knowledge augmented neural network (KANN), can better understand the given scene and provide greater representational power. Experiments are conducted on three challenging scene parsing and semantic segmentation datasets: Cityscapes, Pascal-Context, and ADE20K. The results show that our KANN is effective and achieves better results than all existing SOTA methods.

With the powerful learning ability of deep convolutional networks, deep clustering methods can extract the most discriminative information from individual data and produce more satisfactory clustering results. However, existing deep clustering methods usually ignore the relationship between the data. Fortunately, the graph convolutional network can handle such relationship, opening up a new research direction for deep clustering. In this paper, we propose a cross-attention based deep clustering framework, named Cross-Attention Fusion based Enhanced Graph Convolutional Network (CaEGCN), which contains four main modules: the cross-attention fusion module which innovatively concatenates the Content Auto-encoder module (CAE) relating to the individual data and Graph Convolutional Auto-encoder module (GAE) relating to the relationship between the data in a layer-by-layer manner, and the self-supervised model that highlights the discriminative information for clustering tasks. While the cross-attention fusion module fuses two kinds of heterogeneous representation, the CAE module supplements the content information for the GAE module, which avoids the over-smoothing problem of GCN. In the GAE module, two novel loss functions are proposed that reconstruct the content and relationship between the data, respectively. Finally, the self-supervised module constrains the distributions of the middle layer representations of CAE and GAE to be consistent. Experimental results on different types of datasets prove the superiority and robustness of the proposed CaEGCN.

Achieving robust cross contexts speech emotion recognition (SER) has become a critical next direction of research for wide adoption of SER technology. The core challenge is in the large variability of affective speech that is highly contextualized. Prior works have worked on this as a transfer learning problem that mostly focuses on developing domain adaptation strategy. However, many of the existing speech emotion corpora, even those considered as large scale, are still limited in size resulting in an unsatisfactory transfer result. On the other hand, directly collecting context-specific corpus often results in an even smaller data size leading to an inevitably non-robust accuracy. In order to mitigate this issue, we propose the concept of enhancing the affect-related variability when learning the in-context acoustic latent representation by integrating out-of-context emotion data. Specifically, we utilize adversarial autoencoder network as our backbone with multiple out-of-context emotion labels derived for each in-context samples that serve as an auxiliary constraint in learning the latent representation. We extensively evaluate our framework using three in-context databases with three out-of-context databases. In this work, we demonstrate not only an improved recognition accuracy but also a comprehensive analysis on the effectiveness of this representation learning strategy.

Online users are typically active on multiple social media networks (SMNs), which constitute a multiplex social network. With improvements in cybersecurity awareness, users increasingly choose different usernames and provide different profiles on different SMNs. Thus, it is becoming increasingly challenging to determine whether given accounts on different SMNs belong to the same user; this can be expressed as an interlayer link prediction problem in a multiplex network. To address the challenge of predicting interlayer links, feature or structure information is leveraged. Existing methods that use network embedding techniques to address this problem focus on learning a mapping function to unify all nodes into a common latent representation space for prediction; positional relationships between unmatched nodes and their common matched neighbors (CMNs) are not utilized. Furthermore, the layers are often modeled as unweighted graphs, ignoring the strengths of the relationships between nodes. To address these limitations, we propose a framework based on multiple types of consistency between embedding vectors (MulCEVs). In MulCEV, the traditional embedding-based method is applied to obtain the degree of consistency between the vectors representing the unmatched nodes, and a proposed distance consistency index based on the positions of nodes in each latent space provides additional clues for prediction. By associating these two types of consistency, the effective information in the latent spaces is fully utilized. In addition, MulCEV models the layers as weighted graphs to obtain representation. In this way, the higher the strength of the relationship between nodes, the more similar their embedding vectors in the latent representation space will be. The results of our experiments on several real-world and synthetic datasets demonstrate that the proposed MulCEV framework markedly outperforms current embedding-based methods, especially when the number of training iterations is small.

Long-term visual place recognition (VPR) is challenging as the environment is subject to drastic appearance changes across different temporal resolutions, such as time of the day, month, and season. A wide variety of existing methods address the problem by means of feature disentangling or image style transfer but ignore the structural information that often remains stable even under environmental condition changes. To overcome this limitation, this article presents a novel structure-aware feature disentanglement network (SFDNet) based on knowledge transfer and adversarial learning. Explicitly, probabilistic knowledge transfer (PKT) is employed to transfer knowledge obtained from the Canny edge detector to the structure encoder. An appearance teacher module is then designed to ensure that the learning of appearance encoder does not only rely on metric learning. The generated content features with structural information are used to measure the similarity of images. We finally evaluate the proposed approach and compare it to state-of-the-art place recognition methods using six datasets with extreme environmental changes. Experimental results demonstrate the effectiveness and improvements achieved using the proposed framework. Source code and some trained models will be available at http://www.tianshu.org.cn.

The capability of Convolutional Neural Networks (CNNs) for sparse representation has significant application to complex tasks like Representation Learning (RL). However, labelled datasets of sufficient size for learning this representation are not easily obtainable. The unsupervised learning capability of Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) provide a promising solution to this issue through their capacity to learn representations for novel data samples and classification tasks. In this research, a texture-based latent space disentanglement technique is proposed to enhance learning of representations for novel data samples. A comparison is performed among different VAEs and GANs with the proposed approach for synthesis of new data samples. Two different VAE architectures are considered, a single layer dense VAE and a convolution based VAE, to compare the effectiveness of different architectures for learning of the representations. The GANs are selected based on the distance metric for disjoint distribution divergence estimation of complex representation learning tasks. The proposed texture-based disentanglement has been shown to provide a significant improvement for disentangling the process of representation learning by conditioning the random noise and synthesising texture rich images of fruit and vegetables.

Freezing of gait (FoG) is identified as a sudden and brief episode of movement cessation despite the intention to continue walking. It is one of the most disabling symptoms of Parkinson's disease (PD) and often leads to falls and injuries. Many computer-aided FoG detection methods have been proposed to use data collected from unimodal sources, such as motion sensors, pressure sensors, and video cameras. However, there are limited efforts of multimodal-based methods to maximize the value of all the information collected from different modalities in clinical assessments and improve the FoG detection performance. Therefore, in this study, a novel end-to-end deep architecture, namely graph fusion neural network (GFN), is proposed for multimodal learning-based FoG detection by combining footstep pressure maps and video recordings. GFN constructs multimodal graphs by treating the encoded features of each modality as vertex-level inputs and measures their adjacency patterns to construct complementary FoG representations, thus reducing the representation redundancy among different modalities. In addition, since GFN is devised to process multimodal graphs of arbitrary structures, it is expected to achieve superior performance with inputs containing missing modalities, compared to the alternative unimodal methods. A multimodal FoG dataset was collected, which included clinical assessment videos and footstep pressure sequences of 340 trials from 20 PD patients. Our proposed GFN demonstrates a great promise of multimodal FoG detection with an area under the curve (AUC) of 0.882. To the best of our knowledge, this is one of the first studies to utilize multimodal learning for automated FoG detection, which offers significant opportunities for better patient assessments and clinical trials in the future.

This paper aims to predict a set of apps a user will open on her mobile device in the next time slot. Such an information is essential for many smartphone operations, e.g., app pre-loading and content pre-caching, to improve user experience. However, it is hard to build an explicit model that accurately captures the complex environment context and predicts a set of apps at one time. This paper presents a deep reinforcement learning framework, named as DeepAPP, which learns a model-free predictive neural network from historical app usage data. Meanwhile, an online updating strategy is designed to adapt the predictive network to the time-varying app usage behavior. To transform DeepAPP into a practical deep reinforcement learning system, several challenges are addressed by developing a context representation method for complex contextual environment, a general agent for overcoming data sparsity and a lightweight personalized agent for minimizing the prediction time. Extensive experiments on a large-scale anonymized app usage dataset reveal that DeepAPP provides high accuracy (precision 70.6% and recall of 62.4%) and reduces the prediction time of the state-of-the-art by 6.58 times. A field experiment of 29 participants demonstrates DeepAPP can effectively reduce launch time of apps.

Projection techniques are often used to visualize high-dimensional data, allowing users to better understand the overall structure of multi-dimensional spaces on a 2D screen. Although many such methods exist, comparably little work has been done on generalizable methods of inverse-projection -- the process of mapping the projected points, or more generally, the projection space back to the original high-dimensional space. In this paper we present NNInv, a deep learning technique with the ability to approximate the inverse of any projection or mapping. NNInv learns to reconstruct high-dimensional data from any arbitrary point on a 2D projection space, giving users the ability to interact with the learned high-dimensional representation in a visual analytics system. We provide an analysis of the parameter space of NNInv, and offer guidance in selecting these parameters. We extend validation of the effectiveness of NNInv through a series of quantitative and qualitative analyses. We then demonstrate the method's utility by applying it to three visualization tasks: interactive instance interpolation, classifier agreement, and gradient visualization.

This paper aims to solve two important issues that frequently occur in existing automatic personality analysis systems: 1. Attempting to use very short video segments or even single frames to infer personality traits; 2. Lack of methods to encode person-specific facial dynamics for personality recognition. Hence, we proposes a novel Rank Loss which utilizes the natural temporal evolution of facial actions, rather than personality labels, for self-supervised learning of facial dynamics. Our approach first trains a generic U-net model that can infer general facial dynamics learned from unlabelled face videos. Then, the generic model is frozen, and a set of intermediate filters are incorporated into this architecture. The self-supervised learning is then resumed with only person-specific videos. This way, the learned filters' weights are person-specific, making them a valuable source for modeling person-specific facial dynamics. We then concatenate the weights of the learned filters as a person-specific representation, which can be directly used to predict the personality traits without needing other parts of the network. We evaluate the proposed approach on both self-reported personality and apparent personality datasets. Besides achieving promising results in personality trait estimation from videos, we show that fusion of tasks reaches highest accuracy, and that multi-scale dynamics are more informative than single-scale dynamics.

While we instantaneously recognize a face as attractive, it is much harder to explain what exactly defines personal attraction. This suggests that attraction depends on implicit processing of complex, culturally and individually defined features. Generative adversarial neural networks (GANs), which learn to mimic complex data distributions, can potentially model subjective preferences unconstrained by pre-defined model parameterization. Here, we present generative brain-computer interfaces (GBCI), coupling GANs with brain-computer interfaces. GBCI first presents a selection of images and captures personalized attractiveness reactions toward the images via electroencephalography. These reactions are then used to control a GAN model, finding a representation that matches the features constituting an attractive image for an individual. We conducted an experiment (N=30) to validate GBCI using a face-generating GAN and producing images that are hypothesized to be individually attractive. In double-blind evaluation of the GBCI-produced images against matched controls, we found GBCI yielded highly accurate results. Thus, the use of EEG responses to control a GAN presents a valid tool for interactive information-generation. Furthermore, the GBCI-derived images visually replicated known effects from social neuroscience, suggesting that the individually responsive, generative nature of GBCI provides a powerful, new tool in mapping individual differences and visualizing cognitive-affective processing.

Learning from complementary labels (CLs) is a useful learning paradigm, where the CL specifies the classes that the instance does not belong to, instead of providing the ground truth as in the ordinary supervised learning scenario. In general, although it is less laborious and more efficient to collect CLs compared with ordinary labels, the less informative signal in the complementary supervision is less helpful to learn competent feature representation. Consequently, the final classifier's performance greatly deteriorates. In this article, we leverage generative adversarial networks (GANs) to derive an algorithm GAN-CL to effectively learn from CLs. In addition to the role in original GAN, the discriminator also serves as a normal classifier in GAN-CL, with the objective constructed partly with the complementary information. To further prove the effectiveness of our schema, we study the global optimality of both generator and discriminator for the GAN-CL under mild assumptions. We conduct extensive experiments on benchmark image datasets using deep models, to demonstrate the compelling improvements, compared with state-of-the-art CL learning approaches.

This paper proposes an image fusion framework based on separate representation learning, called IFSepR. We believe that both the co-modal image and the multi-modal image have common and private features based on prior knowledge, exploiting this disentangled representation can help to image fusion, especially to fusion rule design. Inspired by the autoencoder network and contrastive learning, a multi-branch encoder with contrastive constraints is built to learn the common and private features of paired images. In the fusion stage, based on the disentangled features, a general fusion rule is designed to integrate the private features, then combining the fused private features and the common feature are fed into the decoder, reconstructing the fused image. We perform a series of evaluations on three typical image fusion tasks, including multi-focus image fusion, infrared and visible image fusion, medical image fusion. Quantitative and qualitative comparison with five state-of-art image fusion methods demonstrates the advantages of our proposed model.

Currently, with the rapid development of mobile Internet, micro-video has become a prevailing format of user-generated contents (UGCs) on various social media platforms. Several studies have been conducted towards to understanding high-level micro-video semantics, such as venue categorization, memorability analysis, and popularity prediction. However, these approaches supported tasks with only a single output, which exhibited limitations when attempting to use them to resolve tasks with multiple outputs, especially the multi-label micro-video classification. To tackle this problem, in this paper, we propose a dual multi-modal low-rank decomposition (DMLRD) method for multi-label micro-video classification tasks. To learn more comprehensive micro-video representations, we first learn the low-rank-regularized modality-specific and modality-shared components by considering the consistency and the complementarity among modalities simultaneously. Meanwhile, the less descriptive power of each modality aroused by inherent properties can be solved. To obtain unseen label representations, we next construct a sparsity-regularized multi-matrix normal estimation term to jointly encode the latent relationship structures among labels and dimensions. Experiments on a publicly available large-scale micro-video dataset demonstrate the effectiveness of our proposed model over the state-of-art methods.

Detecting salient objects in videos is a very challenging task. Current state-of-the-art methods are dominated by motion based deep neural networks, among which optical flow is often leveraged as motion representation. Though with robust performance, these optical flow-based video salient object detection methods face at least two problems that may hinder their generalization and application. First, computing optical flow as a pre-processing step does not support direct end-to-end learning; second, little attention has been given to the quality of visual features due to high computational cost of spatiotemporal feature encoding. In this paper we propose a novel self-sufficient feature enhancing network (SFENet) for video salient object detection, which leverages optical flow estimation as an auxiliary task while being end-to-end trainable. With a joint training scheme of both salient object detection and optical flow estimation, its multi-task architecture can be totally self-sufficient for achieving good performance without any pre-processing. Furthermore, for improving feature quality, we design four lightweight modules in spatial and temporal domains, including cross-layer fusion, multi-level warping, spatial-channel attention and boundary-aware refinement. The proposed method is evaluated through extensive experiments on five video salient object detection datasets. Experimental results show that our SFENet can be easily trained with fast convergence speed. It significantly outperforms previous methods in terms of various evaluation metrics. Moreover, with optical flow estimation and unsupervised video object segmentation as example applications, our method also yields state-of-the-art results on standard datasets.

The challenging problem in the traffic system is lane detection. This Lane detection which attracts the computer vision community’s attention. For computer vision and machine learning techniques, the Lane detection which acts as the multi-feature detection problem. Many machine learning techniques are used for lane detection. Driver support system is one of the most important features in the modern vehicles to ensure the safety of the driver and decrease the vehicle accidents on road. Road Lane detection is the most challenging task and complex tasks now-a-days. Road localization and relative position between vehicle and roads which also includes with this. But in this journal, we propose a new method. Here, an on- board camera to be used which is looking outwards are presented here in this work. This proposed technique which can be used for all types of roads like painted, unpainted, curved, straight roads etc in different weather conditions. No need for camera calibration and coordination of the transform, may be any changing illumination situation, shadow effects, various road types. No representation for speed limits. This includes that the system acquires the front view using a camera mounted on the vehicles and detect the Lane by applying the code from the Python Programming process. This proposed system does not require any more information about roads. This system which demonstrates a robust performance for Lane detection.

In this paper, we propose a simple yet effective self-supervised method called spatio-temporal contrastive learning (ST-CL for 3D skeleton-based action recognition. ST-CL acquires action-specific features by regarding the spatio-temporal continuity of motion tendency as the supervisory signal. To yield effective representations, ST-CL first designs a novel contrastive proxy task by providing different spatio-temporal observation scenes for the same 3D action and pulling them together in the embedding space. Second, three key components are devised in the action encoding to efficiently extract representations in contrastive tasks: (1 Information Representation introduces the awareness of joint type when analyzing the motion dynamics. (2 Non-local GCN learns a data-driven graph topology structure and promotes a spatial message passing among long-range joints in each frame. (3 Multi-Scale TCN makes larger receptive fields for capturing richer longe-range temporal dynamics amomg adjacent frames. In ST-CL, the effective proxy tasks yield useful representations and efficient action encoding further enhances representation capacity. \revise{As validated on the four large-scale datasets, ST-CL is a strong baseline with high performance and efficiency for the contrastive learning study of the skeleton data. Compared to previous self-supervised methods, the proposed ST-CL achieves significant improvement consistently with a smaller model size and better training efficiency.

In this paper we present a new classification method based on Dictionary Learning (DL). The main contribution consists of a kernel version of incoherent DL, derived from its standard linear counterpart. We also propose an improvement of the AK-SVD algorithm concerning the representation update. Our algorithms are tested on several popular databases of classification problems.

Extensive studies have been conducted on human action recognition, whereas relatively few methods have been proposed for hand action recognition. Although it is very natural and straightforward to apply a human action recognition method to hand action recognition, this approach cannot always lead to state-of-the-art performance. One of the important reasons is that both the between-class difference and the within-class difference in hand actions are much smaller than those in human actions. In this paper, we study first-person hand action recognition from RGB-D sequences. To explore whether pretrained networks substantially influence accuracy, 8 classic pretrained networks and one pretrained network designed by us are introduced for extracting RGB-D features. A Lie group is introduced for hand pose representation. Ablation studies are conducted to compare the discriminative power of the RGB modality, depth modality, pose modality, and their combinations. In our method, a fixed number of frames are randomly sampled to represent an action. This temporal modeling strategy is simple but is proven more effective than both the graph convolutional network (GCN) and the recurrent neural network (RNN), which are widely adopted by conventional methods. Evaluation experiments on two public datasets demonstrate that our method markedly outperforms recent baselines.

The analysis of large-scale multimodal data has become very popular recently. Image captioning, whose goal is to describe the content of image with natural language automatically, is an essential and challenging task in artificial intelligence. Commonly, most existing image caption methods utilize the mixture of Convolutional Neural Network and Recurrent Neural Network framework. These methods either pay attention to global representation at the image level or only focus on the specific concepts, such as regions and objects. To make the most of characteristics about a given image, in this study, we present a novel model named Multilevel Attention Networks and Policy Reinforcement Learning for image caption generation. Specifically, our model is composed of a multilevel attention network module and a policy reinforcement learning module. In the multilevel attention network, the object-attention network aims to capture global and local details about objects, whereas the region-attention network obtains global and local features about regions. After that, a policy reinforcement learning algorithm is adopted to overcome the exposure bias problem in the training phase and solve the loss-evaluation mismatching problem at the caption generation stage. With the attention network and policy algorithm, our model can automatically generate accurate and natural sentences for any particular image. We carry out extensive experiments on the MSCOCO and Flickr30k data sets, demonstrating that our model is superior to other competitive methods.

Realistic speech-driven 3D facial animation is a challenging problem due to the complex relationship between speech and face. In this paper, we propose a deep architecture, called Geometry-guided Dense Perspective Network (GDPnet), to achieve speaker-independent realistic 3D facial animation. The encoder is designed with dense connections to strengthen feature propagation and encourage the re-use of audio features, and the decoder is integrated with an attention mechanism to adaptively recalibrate point-wise feature responses by explicitly modeling interdependencies between different neuron units. We also introduce a non-linear face reconstruction representation as a guidance of latent space to obtain more accurate deformation, which helps solve the geometry-related deformation and is good for generalization across subjects. Huber and HSIC (Hilbert-Schmidt Independence Criterion) constraints are adopted to promote the robustness of our model and to better exploit the non-linear and high-order correlations. Experimental results on the public dataset and real scanned dataset validate the superiority of our proposed GDPnet compared with state-of-the-art model. The code is available for research purposes at http://cic.tju.edu.cn/faculty/likun/projects/GDPnet.

Semantic segmentation is paramount for autonomous vehicles to have a deeper understanding of the surrounding traffic environment and enhance safety. Deep neural networks (DNN) have achieved remarkable performances in semantic segmentation. However, training such a DNN requires a large amount of labelled data at pixel level. In practice, it is a labour-intensive task to manually annotate dense pixel-level labels. To tackle the problem associated with a small amount of labelled data, Deep Domain Adaptation (DDA) methods have recently been developed to examine the use of synthetic driving scenes so as to significantly reduce the manual annotation cost. Despite remarkable advances, these methods unfortunately suffer from the generalisability problem that fails to provide a holistic representation of the mapping from the source image domain to the target image domain. In this paper, we therefore develop a novel ensembled DDA to train models with different up-sampling strategies, discrepancy and segmentation loss functions. The models are, therefore, complementary with each other to achieve better generalisation in the target image domain. Such a design does not only improve the adapted semantic segmentation performance, but also strengthen the model reliability and robustness. Extensive experimental results demonstrate the superiorities of our approach over several state-of-the-art methods.

Depth maps acquired with ToF cameras have a limited accuracy due to the high noise level and to the multi-path interference. Deep networks can be used for refining ToF depth, but their training requires real world acquisitions with ground truth, which is complex and expensive to collect. A possible workaround is to train networks on synthetic data, but the domain shift between the real and synthetic data reduces the performances. In this paper, we propose three approaches to perform unsupervised domain adaptation of a depth denoising network from synthetic to real data. These approaches are respectively acting at the input, at the feature and at the output level of the network. The first approach uses domain translation networks to transform labeled synthetic ToF data into a representation closer to real data, that is then used to train the denoiser. The second approach tries to align the network internal features related to synthetic and real data. The third approach uses an adversarial loss, implemented with a discriminator trained to recognize the ground truth statistic, to train the denoiser on unlabeled real data. Experimental results show that the considered approaches are able to outperform other state-of-the-art techniques and achieve superior denoising performances.

Classification of Alzheimer’s disease (AD) is one of the most challenging issues for neurologists. Manual methods are time consuming and may not be accurate all the time. Since, brain is the most affected region in AD, a proper classification framework using brain images may provide more accurate results. Deep Learning (DL) is a popular representation of machine learning techniques, that emulate the functionalities of a human brain to process information and creates patterns that help in making complex decisions. The ability to absorb information, even from the unstructured and unlabeled data, makes DL one of the first choices by researchers. In this paper, some of the most popular DL models are discussed along with their implementation results for AD classification. All brain Magnetic Resonance (MR) images are acquired from the online data-set, “Alzheimer’s Disease Neuroimaging Initiative (ADNI)”. From the performance comparison amongst all the discussed models, it is observed that the DenseNet-121 model achieves a convincing result with an average performance rate of 88.78%. But one limitation of the DenseNet model is that it uses lots of convolutional operations that make the model computationally slower than many of the discussed models. Depth-wise convolution is a popular way to make a convolutional operation faster and better. Hence, to improve the execution time, we have proposed replacing the convolution layers in the original DenseNet-121 architecture with depth-wise convolution layers. The new architecture also improved the performance of the model with an average rate of 90.22%.

Co-location pattern mining refers to discovering neighboring relationships of spatial features distributed in geographic space. With the rapid growth of spatial datasets, the usefulness of co-location patterns is strongly limited by the large number of discovered patterns containing multiple redundancies. To address this problem, in this article, we propose a novel approach for discovering the super participation index-closed (SPI-closed) co-location patterns which are a newly proposed lossless condensed representation of co-location patterns by considering distributions of the spatial instances. In the proposed approach, first, a linear-time method is designed to generate complete and correct neighboring cliques using extended neighboring relationships. Based on these cliques, a hash structure is then constructed to store the distributions of the co-location patterns in a condensed way. Finally, using this hash structure, the SPI-closed co-location patterns (SCPs) are efficiently discovered even if the prevalence threshold is changed, while similar approaches have to restart their mining processes. To confirm the efficiency of the proposed method, we compared its performance with similar approaches in the literature on multiple real and synthetic spatial datasets. The experiments confirm that our new approach is more efficient, effective, and flexible than similar approaches.

Most NN-RSs focus on accuracy by building representations from the direct user-item interactions (e.g., user-item rating matrix), while ignoring the underlying relatedness between users and items (e.g., users who rate the same ratings for the same items should be embedded into similar representations), which is an ideological disadvantage. On the other hand, ME models directly employ inner products as a default loss function metric that cannot project users and items into a proper latent space, which is a methodological disadvantage. In this paper, we propose a supervised collaborative representation learning model - Magnetic Metric Learning (MML) - to map users and items into a unified latent vector space, enhancing the representation learning for NN-RSs. Firstly, MML utilizes dual triplets to model not only the observed relationships between users and items, but also the underlying relationships between users as well as items to overcome the ideological disadvantage. Specifically, a modified metric-based dual loss function is proposed in MML to gather similar entities and disperse the dissimilar ones. With MML, we can easily compare all the relationships (user to user, item to item, user to item) according to the weighted metric, which overcomes the methodological disadvantage. We conduct extensive experiments on four real-world datasets with large item space. The results demonstrate that MML can learn a proper unified latent space for representations from the user-item matrix with high accuracy and effectiveness, and lead to a performance gain over the state-of-the-art RS models by an average of 17%., Accepted by IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING(TKDE)

In this article, the unsupervised domain adaptation problem, where an approximate inference model is to be learned from a labeled dataset and expected to generalize well on an unlabeled dataset, is considered. Unlike the existing work, we explicitly unveil the importance of the latent variables produced by the feature extractor, that is, encoder, where contains the most representative information about their input samples, for the knowledge transfer. We argue that an estimator of the representation of the two datasets can be used as an agent for knowledge transfer. To be specific, a novel variational inference approach is proposed to approximate a latent distribution from the unlabeled dataset that can be used to accurately predict its input samples. It is demonstrated that the discriminative knowledge of the latent distribution that is learned from the labeled dataset can be progressively transferred to that is learned from the unlabeled dataset by simultaneously optimizing the estimator via the variational inference and our proposed regularization for shifting the mean of the estimator. The experiments on several benchmark datasets demonstrate that the proposed method consistently outperforms state-of-the-art methods for both object classification and digit classification.

The recent success in supervised multi-view stereopsis (MVS) relies on the onerously collected real-world 3D data. While the latest differentiable rendering techniques enable unsupervised MVS, they are restricted to discretized (e.g., point cloud) or implicit geometric representation, suffering from either low integrity for a textureless region or less geometric details for complex scenes. In this paper, we propose SurRF, an unsupervised MVS pipeline by learning Surface Radiance Field, i.e., a radiance field defined on a continuous and explicit 2D surface. Our key insight is that, in a local region, the explicit surface can be gradually deformed from a continuous initialization along view-dependent camera rays by differentiable rendering. That enables us to define the radiance field only on a 2D deformable surface rather than in a dense volume of 3D space, leading to compact representation while maintaining complete shape and realistic texture for large-scale complex scenes. We experimentally demonstrate that the proposed SurRF produces competitive results over the-state-of-the-art on various real-world challenging scenes, without any 3D supervision. Moreover, SurRF shows great potential in owning the joint advantages of mesh (scene manipulation), continuous surface (high geometric resolution), and radiance field (realistic rendering).

The ability to accurately predict human encounters can inspire a variety of promising applications, ranging from epidemiology to data forwarding in opportunistic networks. This work aims at designing a low cost, highly accurate human encounter prediction model based on Wi-Fi datasets. By leveraging the temporal dependency of human mobility, we propose the distributed human encounter prediction (DHEP) model, which uses the Wi-Fi access history and inferred contact information of only the person of interest to estimate future encounters of that person. We implement the proposed DHEP model using a recurrent neural network and a feed-forward neural network. An embedding model that learns the low-dimensional representation of a person's location is proposed to reduce the number of training parameters. The experimental results on two large Wi-Fi datasets show the proposed RNN-based DHEP model outperforms existing models, and achieves 87% to 91% accuracy based on University at Buffalo (UB) traces. We also compare DH P with the centralized human encounter prediction (CHEP) model, which gathers the contact history of all people for predicting future encounters. Despite a slightly lower performance than CHEP, DHEP has a low overhead and can protect data privacy.

Software testing plays a critical role in the development and the assurance of the software quality. However, the quality of the code that is responsible for testing may, itself, be affected by poor design choices, known as test smells. In the literature, test smells may be interpreted differently by developers, which in turn can lead to different detection tools and results. In our work, we have selected the mostly commonly used detection tools and have investigated their overall agreement across different projects and different test smells. The found results were evaluated according to the average level of agreement, where we observed a definite disagreement between the tools. To overcome this gap of misinterpretation, we propose in this paper a multi-label classification approach to detect test smells based on a deep representation of the test code. We conducted our experiments using 4 problem-transformation techniques and 4 ensemble techniques. To evaluate our experimental results, we built a benchmark using a tool-based approach for labelling and made it publicly available. Binary Relevance and RAkEL are found to be the best multi-label techniques that achieve high performance results.

Accurate medical image segmentation of brain tumors is necessary for the diagnosing, monitoring, and treating disease. In recent years, with the gradual emergence of multi-sequence magnetic resonance imaging (MRI), multi-modal MRI diagnosis has played an increasingly important role in the early diagnosis of brain tumors by providing complementary information for a given lesion. Different MRI modalities vary significantly in context, as well as in coarse and fine information. As the manual identification of brain tumors is very complicated, it usually requires the lengthy consultation of multiple experts. The automatic segmentation of brain tumors from MRI images can thus greatly reduce the workload of doctors and buy more time for treating patients. In this paper, we propose a multi-modal brain tumor segmentation framework that adopts the hybrid fusion of modality-specific features using a self-supervised learning strategy. The algorithm is based on a fully convolutional neural network. Firstly, we propose a multi-input architecture that learns independent features from multi-modal data, and can be adapted to different numbers of multi-modal inputs. Compared with single-modal multi-channel networks, our model provides a better feature extractor for segmentation tasks, which learns cross-modal information from multi-modal data. Secondly, we propose a new feature fusion scheme, named hybrid attentional fusion. This scheme enables the network to learn the hybrid representation of multiple features and capture the correlation information between them through an attention mechanism. Unlike popular methods, such as feature map concatenation, this scheme focuses on the complementarity between multi-modal data, which can significantly improve the segmentation results of specific regions. Thirdly, we propose a self-supervised learning strategy for brain tumor segmentation tasks. Our experimental results demonstrate the effectiveness of the proposed model against other state-of-the-art multi-modal medical segmentation methods.

Machine vision based planar grasping detection is challenging due to uncertainty about object shape, pose, size, etc. Previous methods mostly focus on predicting discrete gripper configurations, and may miss some ground-truth grasp postures. In this paper, a pixel-level grasp detection method is proposed, which uses deep neural network to predict pixel-level gripper configurations on RGB images. Firstly, a novel Oriented Arrow Representation model (OAR-model) is introduced to represent the gripper configuration of parallel-jaw and three-fingered gripper, which can partly improve the applicability to different grippers. Then, the Adaptive Grasping Attribute model (AGA-model) is proposed to adaptively represent the grasping attribute of objects, for resolving angle conflicts in training and simplifying pixel-level labeling. Lastly, the Adaptive Feature Fusion and Grasp Aware Network (AFFGA-Net) is proposed to predict pixel-level OAR-models on RGB images. AFFGA-Net improves the robustness in unstructured scenarios by using Hybrid Atrous Spatial Pyramid and Adaptive Decoder connected in sequence. On the public Cornell dataset and actual objects, our structure achieves 99.09\% and 98.0\% grasp detection accuracy respectively. In over 2,400 robotic grasp trials, our structure achieves average success rate of 98.77\% in single-object scenarios and 93.69\% in cluttered scenarios. Moreover, AFFGA-Net completes a grasp detection pipeline within 15 ms.

Nowadays, online spamming has already been a remarkable threat to contents security of Internet of Things (IoT). Due to constant technical progress, online spamming activities have been more and more concealed. This brings much fuzziness to spammer detection scenarios, yielding the issue of fuzzy detection of spammers. Although existing detection techniques for spammers utilized idea of deep learning, they still ignore to release power of label spaces. As real nature about a user may be usually fuzzy, but the label annotated for a user is always certain. To remedy such gap, this paper proposes a label smoothing-based fuzzy detection method for spammers (Fuz-Spam). First of all, deep representation is still utilized to deeply fuse features, which acts as the foundation of neural computing. On this basis, generative adversarial learning is introduced to transform previous label spaces into distributed forms. In addition, two groups of experiments are carried out on two real-world datasets for evaluation. The results demonstrate that the Fuz-Spam improves identification efficiency about 10% to 20% than previous ones, and that the Fuz-Spam is endowed with proper stability.

The user alignment problem that establishes a correspondence between users across networks is a fundamental issue in various social network analyses and applications. Since symbolic representations of users suffer from sparsity and noise when computing their cross-network similarities, the state-of-the-art methods embed users into the low-dimensional representation space, where their features are preserved and establish user correspondence based on the similarities of their low-dimensional embeddings. Many embedding-based methods try to align latent spaces of two networks by learning a mapping function before computing similarities. However, most of them learn the mapping function largely based on the limited labeled aligned user pairs and ignore the distribution discrepancy of user representations from different networks, which may lead to the overfitting problem and affect the performance. To address the above problems, we propose a cycle-consistent adversarial mapping model to establish user correspondence across social networks. The model learns mapping functions across the latent representation spaces, and the representation distribution discrepancy is addressed through the adversarial training between the mapping functions and the discriminators as well as the cycle-consistency training. Besides, the proposed model utilizes both labeled and unlabeled users in the training process, which may alleviate the overfitting problem and reduce the number of labeled users required. Results of extensive experiments demonstrate the effectiveness of the proposed model on user alignment on real social networks.

In this paper, we explore the mask representation in instance segmentation with Point-of-Interest (PoI) features. Differentiating multiple potential instances within a single PoI feature is challenging because learning a high-dimensional mask feature for each instance using vanilla convolution demands a heavy computing burden. To address this challenge, we propose an instance-aware convolution. It decomposes this mask representation learning task into two tractable modules as instance-aware weights and instance-agnostic features. The former is to parametrize convolution for producing mask features corresponding to different instances, improving mask learning efficiency by avoiding employing several independent convolutions. Meanwhile, the latter serves as mask templates in a single point. Together, instance-aware mask features are computed by convolving the template with dynamic weights, used for the mask prediction. Along with instance-aware convolution, we propose PointINS, a simple and practical instance segmentation approach, building upon dense one-stage detectors. Through extensive experiments, we evaluated the effectiveness of our framework built upon RetinaNet and FCOS. PointINS in ResNet101 backbone achieves a 38.3 mask mean average precision (mAP) on COCO dataset, outperforming existing point-based methods by a large margin. It gives a comparable performance to the region-based Mask R-CNN with faster inference., Comment: Accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence

Face hallucination technologies have been widely developed during the past decades, among which the sparse manifold learning (SML)-based approaches have become the popular ones and achieved promising performance. However, these SML methods always failed in handling noisy images due to the least-square regression (LSR) they used for error approximation. To this end, we propose, in this article, a smooth correntropy representation (SCR) model for noisy face hallucination. In SCR, the correntropy regularization and smooth constraint are combined into one unified framework to improve the resolution of noisy face images. Specifically, we introduce the correntropy induced metric (CIM) rather than the LSR to regularize the encoding errors, which admits the proposed method robust to noise with uncertain distributions. Besides, the fused LASSO penalty is added into the feature space to ensure similar training samples holding similar representation coefficients. This encourages the SCR not only robust to noise but also can well exploit the inherent typological structure of patch manifold, resulting in more accurate representations in noise environment. Comparison experiments against several state-of-the-art methods demonstrate the superiority of SCR in super-resolving noisy low-resolution (LR) face images.

In addition to 3D geometry, accurate representation of texture is important when digitizing real objects in virtual worlds. Based on a single consumer RGBD sensor, accurate texture representation for static objects can be realized by fusing multi-frame information; however, extending the process to dynamic objects, which typically have time-varying textures, is difficult. Thus, to address this problem, we propose a compact keyframe-based representation that decouples a dynamic texture into a basic static texture and a set of multiplicative changing maps. With this representation, the proposed method first aligns textures recorded from multiple keyframes with the reconstructed dynamic geometry of the object. Errors in the alignment and geometry are then compensated in an innovative iterative linear optimization framework. With the reconstructed texture, we then employ a scheme to synthesize the dynamic object from arbitrary viewpoints. By considering temporal and local pose similarities jointly, dynamic textures in all keyframes are fused to guarantee high-quality image generation. Experimental results demonstrate that the proposed method handles various dynamic objects, including faces, bodies, cloth, and toys. In addition, qualitative and quantitative comparisons demonstrate that the proposed method outperforms state-of-the-art solutions.

Studying the deep learning-based molecular representation has great significance on predicting molecular property, promoted the development of drug screening and new drug discovery, and improving human well-being for avoiding illnesses. It is essential to learn the characterization of drug for various downstream tasks, such as molecular property prediction. In particular, the 3D structure features of molecules play an important role in biochemical function and activity prediction. The 3D characteristics of molecules largely determine the properties of the drug and the binding characteristics of the target. However, most current methods merely rely on 1D or 2D properties while ignoring the 3D topological structure, thereby degrading the performance of molecular inferring. In this paper, we propose 3DMol-Net to enhance the molecular representation, considering both the topology and rotation invariance (RI) of the 3D molecular structure. Specifically, we construct a molecular graph with soft relations related to the spatial arrangement of the 3D coordinates to learn 3D topology of arbitrary graph structure and employ an adaptive graph convolutional network to predict molecular properties and biochemical activities. Comparing with current graph-based methods, 3DMol-Net demonstrates superior performance in terms of both regression and classification tasks. Further verification of RI and visualization also show better robustness and representation capacity of our model.

When deploying memory-intensive applications to public clouds, one important yet challenging question to answer is how to select a specific instance type whose memory capacity is large enough to prevent out-of-memory errors while the cost is minimized without violating performance requirements. The state-of-the-practice solution is trial and error, causing both performance overhead and additional monetary cost. This paper investigates two memory scaling mechanisms in public clouds: physical memory (good performance and high cost) and virtual memory (degraded performance and no additional cost). In order to analyze the trade-off between performance and cost of the two scaling options, a performance-cost model is developed that is driven by a lightweight analytic prediction approach through a compact representation of the memory footprint. In addition, for those scenarios when the footprint is unavailable, a meta-model-based prediction method is proposed using just-in-time migration mechanisms. The proposed techniques have been extensively evaluated with various benchmarks and real-world applications on Amazon Web Services: the performance-cost model is highly accurate and the proposed just-in-time migration approach reduces the monetary cost by up to 66\%.

Sound event detection (SED) and acoustic scene classification (ASC) are two key tasks related to each other in the field of computational auditory scene analysis. For example, during sound event detection, scene information can be used to exclude sound events that are unlikely to occur in this scene. In other words, scene information can improve the accuracy of sound event detection. However, existing works rarely detect sound events by considering acoustic scene information. Based on the internal relationship between sound events and scene information, this paper proposes a scene-dependent sound event detection (SDSED) approach, which combines scene information and sound event information using multi-task learning. In the proposed approach, we share common feature representation for the two tasks simultaneously. Meanwhile, a temporal attention mechanism is used to extract informative features from sound recordings. We test the proposed approach on Synthetic Sound Scenes dataset. Experimental results show that our proposed approach outperforms the state-of-the-art approaches. Compared with the referenced approach, our approach improves the segment-based F-score by 4.29% and reduces the segment-based error rate by 4.8%.

The approximation learning of fuzzy concepts associated with fuzzy rough sets and three-way decisions is a useful technology for the representation, learning, and transformation of uncertain knowledge. However, how to integrate the advantages of fuzzy rough approximations and three-way approximations has rarely been carefully investigated so far. In this paper, we focus on exploring the connection and interplay of these two methods, and further propose the hierarchical fuzzy rough approximations in the dynamic fuzzy open-world environment. We utilize the temporal-spatial perspectives of three-way decisions to construct multi-granularity structures and implement multi-granularity learning. The temporality of data and the spatiality of model parameters are both considered in such frameworks. Subsequently, we discuss the interpretation and representation of fuzzy three-way regions in fuzzy rough sets with some definitions and properties. The max, double, and aggregated evaluation-based models of three-way approximations are proposed to gradually transform a fuzzy concept to a crisp concept by the time-evolving attributes. Finally, the comparative experimental results between static and dynamic approaches demonstrate the effectiveness of our proposed hierarchical approximation learning models.

Congestion recognition is necessary for vehicle routing, traffic control, and many other applications in intelligent transportation systems. Besides, traffic facilities in the three-dimensional road network, which contains the fundamental spatiotemporal features for congestion recognition, provides multi-source traffic information. To exploit these traffic big data, in this paper, we propose an attention mechanism-based digraph convolution network (ADGCN) enabled framework to tackle the congestion recognition problem. It can be divided into two parts, spatial relevance modeling and temporal relevance modeling. At first, the representation incorporates spatiotemporal traffic information with the three-dimensional urban network, and partially decouples the global network topology to a single-knot digraph. Then a digraph-based convolution network is used to capture high-order spatial features. Finally, to proceed with time-series features, the multi-modal attention mechanism is introduced to catch the long-range temporal dependence and the congestion classifier is defined accordingly. This distinguishes the proposed model from the conventional congestion recognition methods. Comprehensive experiments are conducted based on real traffic data. The results demonstrate the advantages of the proposed framework over the existing spatiotemporal analysis methods.

Travel time prediction of a given trajectory plays an indispensable role in intelligent transportation systems. Although many prior researches have struggled for accurate prediction results, most of them achieve inferior performance due to insufficient feature extraction of travel speed and road network structure from the trajectory data, which confirms the challenges involved in this topic. To overcome those issues, we propose a novel neural Network for Travel Time Prediction based on tensor decomposition and graph embedding, named TTPNet, which can extract travel speed and representation of road network structure effectively from historical trajectories, as well as predict the travel time with better accuracy. Specifically, TTPNet consists of three components: the first module (Travel Speed Features Layer) leverages non-negative tensor decomposition to restore travel speed distributions on different roads in the previous hour, and integrates a CNN-RNN model to extract both long-term and short-term travel speed features of the query trajectory; the second module (Road Network Structure Features Layer) utilizes graph embedding to generate the representation of local and global road network structure; the last module (Deep LSTM Prediction Layer) completes the final predicting task. Empirical results over two real-world large-scale datasets show that our proposed TTPNet model can achieve significantly better performance and remarkable robustness.

The increasing complexity of Intelligent Transportation Systems (ITS), that comprise a wide variety of applications and services, has imposed a necessity for high-performance Modern Hardware Devices (MHDs). The performance challenge has become more noticeable with the integration of Machine Learning (ML) techniques deployed in large-scale settings. ML has effectively supported the field of ITS by providing efficient and optimized solutions to problems that were otherwise tackled using traditional statistical and analytical approaches. Addressing the hardware deployment needs of ITS in the era of ML is a challenging problem that involves temporal, spatial, environmental, and economical factors. This survey reviews the recent literature of ML-driven ITS, in which MHDs were utilized, with a focus on performance indicators. A taxonomy is then synthesized, giving a complete representation of what the current capabilities of the surveyed ITS rely on in terms of ML techniques and technological infrastructure. To alleviate the difficulties faced in the non-trivial task of selecting suitable ML techniques and MHDs for an ITS with a specific complexity level, a performance evaluation framework is proposed. The presented survey sets the basis for developing suitable hardware, facilitating the integration of ML within ITS, and bridging the gap between research and real-world deployments.

With the development of neural architecture search, the performance of deep neural networks has been considerably enhanced with less human expertise. While the existing work mainly focuses on the development of optimizers, the design of encoding scheme is still in its infancy. This article thus proposes a novel encoding scheme for neural architecture search, termed action command encoding (ACEncoding). Inspired by the gene expression process, ACEncoding defines several action commands to indicate the addition and clone of layers, connections, and local modules, where an architecture grows from empty according to multiple action commands. ACEncoding provides a compact and rich search space that can be explored by various optimizers efficiently. Furthermore, a surrogate-assisted performance evaluator is tailored for ACEncoding, termed sequence-to-rank (Seq2Rank). By integrating the Seq2Seq model with RankNet, Seq2Rank embeds the variable-length encoding of ACEncoding into a continuous space, and then predicts the rankings of architectures based on the continuous representation. In the experiments, ACEncoding brings improvement to neural architecture search with existing encoding schemes and Seq2Rank shows better accuracy than existing performance evaluators. The neural architectures obtained by ACEncoding and Seq2Rank have competitive test errors and complexities on image classification tasks, and also show high transferability between different data sets.

Learning to drive in urban areas is an open challenge for autonomous vehicles (AVs), as complex decision making requirements are needed in multi-task co-ordinations environments. In this paper, we propose a hybrid framework with a new perception model involving affordance learning to simplify the surrounding urban scenes for training an AV agent, along with a planned trajectory and the associated driving measurements. Our proposed solution encompasses two main aspects. Firstly, a supervised learning network is used to map the input sensory data into affordance predictions. The predicted affordances provide a low-dimensional representation of surrounding scenes of the AV in the form of key perception indicators, e.g., true or false with respect to a traffic light signal. Secondly, a deep deterministic policy gradient model that maps the perception information into a series of actions is devised. We evaluate the proposed solution using the CARLA driving simulator in an urban town and evaluate the performance in a new, unseen town under different weather conditions. The quantitative and qualitative results indicate that our proposed solution can generalize well to cope with different traffic solutions and environmental conditions. Our proposed solution also outperforms other baseline methods in a comparative study in handling various AV driving tasks with different levels of difficulty. In addition, the model trained with simulated scenes yields promising prediction results when testing on recorded video streams on real-world highway and suburb environments with varying traffic and weather conditions.

Community detection is of great help to understand the structures and functions of complex networks. It has become one of popular research topics in the field of complex networks analysis. Due to the simplicity, flexibility, effectiveness and interpretability, Nonnegative Matrix Factorization (NMF)-based methods have been widely employed for community detection. However, most existing NMF-based community detection methods are linear and their performance is limited when facing networks with diversified structure information. In view of this, we propose a nonlinear NMF-based method named NMFGAAE, which is composed of two main modules: NMF and Graph Attention Auto-Encoder (GAAE). This approach can boost the performance of NMF-based community detection methods by the aid of graph neural networks and deep clustering. More specifically, GAAE introduces an attention mechanism directed by NMF-based community detection to learn the node representations, while NMF can simultaneously factor these representations to uncover the community structure. We design a unified framework to jointly optimize GAAE and NMF modules, which is very beneficial to obtain better community detection results. We conduct extensive experiments on synthetic and real-world networks. The results show that NMFGAAE not only performs better than state-of-the-art NMF-based community detection methods, but also outperforms some network representation based baselines.