Search

Reinforcement learning (RL) has emerged as a promising approach for scheduling semiconductor operations. Yet, it is still challenging to solve large-scale scheduling problems based on an RL method since learning complexity grows fast as the size of shop floor increases. This challenge becomes more apparent when solving the scheduling problems with a diverse number of job types, which leads to the difficulties in exploration and function approximation in RL. This article presents a scheduling method for semiconductor packaging facilities using deep RL in which an agent allocates a job to one of machines in a centralized manner. Specifically, a novel state representation is introduced to effectively accommodate the variations in the number of available machines and the production requirements. Furthermore, we propose a continuous representation of an action to maintain the size of the action space even when the numbers of jobs, machines, and operation types are subject to change. Extensive experiments on large-scale datasets demonstrate that the proposed method mostly outperforms the metaheuristics and rule-based methods, as well as the other RL approaches considered in terms of makespan while requiring much less computation time than the metaheuristics.

Recent years have witnessed a tremendous increasing of first-person videos captured by wearable devices. Such videos record information from different perspectives than the traditional third-person view, and thus show a wide range of potential usages. However, techniques for analyzing videos from different views can be fundamentally different, not to mention co-analyzing on both views to explore the shared information. In this paper, we take the challenge of cross-view video co-analysis and deliver a novel learning-based method. At the core of our method is the notion of "joint attention", indicating the shared attention regions that link the corresponding views, and eventually guide the shared representation learning across views. To this end, we propose a multi-branch deep network, which extracts cross-view joint attention and shared representation from static frames with spatial constraints, in a self-supervised and simultaneous manner. In addition, by incorporating the temporal transition model of the joint attention, we obtain spatial-temporal joint attention that can robustly capture the essential information extending through time. Our method outperforms the state-of-the-art on the standard cross-view video matching tasks on public datasets. Furthermore, we demonstrate how the learnt joint information can benefit various applications through a set of qualitative and quantitative experiments.

The task of multi-label image recognition is to predict a set of object labels that present in an image. As objects normally co-occur in an image, it is desirable to model label dependencies to improve recognition performance. To capture and explore such important information, we propose Graph Convolutional Networks based models for multi-label recognition, where directed graphs are constructed over classes and information is propagated between classes to learn inter-dependent class-level representations. Following this idea, we design two particular models that approach multi-label classification from different views. In our first model, the prior knowledge about the class dependencies is integrated into classifier learning. Specifically, we propose Classifier-Learning-GCN to map class-level semantic representations (\eg, word embedding) into classifiers that maintain the inter-class topology. In our second model, we decompose the visual representation of an image into a set of label-aware features and propose Prediction-Learning-GCN to encode such features into inter-dependent image-level prediction scores. Furthermore, we also present an effective correlation matrix construction approach to capture inter-class relationships and consequently guide information propagation among classes. Empirical results on generic multi-label recognition demonstrate that the effectiveness of both two proposed models. Moreover, the proposed methods also show advantages in other multi-label related applications.

This article introduces a new iterative approach to explainable feature learning. During each iteration, new features are generated, first by applying arithmetic operations on the input set of features. These are then evaluated in terms of probability distribution agreements between values of samples belonging to different classes. Finally, a graph-based approach for feature selection is proposed, which allows for selecting high-quality and uncorrelated features to be used in feature generation during the next iteration. As shown by the results, the proposed method improved the accuracy of all tested classifiers, where the best accuracies were achieved using random forest. In addition, the method turned out to be insensitive to both of the input parameters, while superior performances in comparison to the state of the art were demonstrated on nine out of 15 test sets and achieving comparable results in the others. Finally, we demonstrate the explainability of the learned feature representation for knowledge discovery.

Due to the personalized and diversified expression habits, the evaluation information in multiple criteria decision-making (MCDM) problems, especially those with multiple experts, may appear in heterogeneous forms. Converting heterogeneous expressions to a single representation would lead to information loss. To solve this challenge, this article aims to address the MCDM problems with heterogeneous linguistic expressions by unifying the operations of heterogeneous linguistic representations. To do so, we propose to use expectation values and entropy measures to describe the meaning of linguistic evaluation information. The expectation functions of ten kinds of linguistic representations are respectively defined based on the semantics of linguistic terms. The entropy measures of these representations are developed to reflect the inherent uncertainty of evaluations. Afterward, a computation model for heterogeneous linguistic representations is presented based on their expectation values and entropy. On this basis, an MCDM framework with personalized heterogeneous linguistic information is constructed. A numerical example about selecting the best green logistics for a business-to-customer e-commerce enterprise shows the advantages of the proposed method in modeling personalized linguistic evaluations and retaining uncertain information in the computation process.

Scene parsing, or semantic segmentation, aims at labeling all pixels in an image with the predefined categories of things and stuff. Learning a robust representation for each pixel is crucial for this task. Existing state-of-the-art (SOTA) algorithms employ deep neural networks to learn (discover) the representations needed for parsing from raw data. Nevertheless, these networks discover desired features or representations only from the given image (content), ignoring more generic knowledge contained in the dataset. To overcome this deficiency, we make the first attempt to explore the meaningful supportive knowledge, including general visual concepts (i.e., the generic representations for objects and stuff) and their relations from the whole dataset to enhance the underlying representations of a specific scene for better scene parsing. Specifically, we propose a novel supportive knowledge mining module (SKMM) and a knowledge augmentation operator (KAO), which can be easily plugged into modern scene parsing networks. By taking image-specific content and dataset-level supportive knowledge into full consideration, the resulting model, called knowledge augmented neural network (KANN), can better understand the given scene and provide greater representational power. Experiments are conducted on three challenging scene parsing and semantic segmentation datasets: Cityscapes, Pascal-Context, and ADE20K. The results show that our KANN is effective and achieves better results than all existing SOTA methods.

With the powerful learning ability of deep convolutional networks, deep clustering methods can extract the most discriminative information from individual data and produce more satisfactory clustering results. However, existing deep clustering methods usually ignore the relationship between the data. Fortunately, the graph convolutional network can handle such relationship, opening up a new research direction for deep clustering. In this paper, we propose a cross-attention based deep clustering framework, named Cross-Attention Fusion based Enhanced Graph Convolutional Network (CaEGCN), which contains four main modules: the cross-attention fusion module which innovatively concatenates the Content Auto-encoder module (CAE) relating to the individual data and Graph Convolutional Auto-encoder module (GAE) relating to the relationship between the data in a layer-by-layer manner, and the self-supervised model that highlights the discriminative information for clustering tasks. While the cross-attention fusion module fuses two kinds of heterogeneous representation, the CAE module supplements the content information for the GAE module, which avoids the over-smoothing problem of GCN. In the GAE module, two novel loss functions are proposed that reconstruct the content and relationship between the data, respectively. Finally, the self-supervised module constrains the distributions of the middle layer representations of CAE and GAE to be consistent. Experimental results on different types of datasets prove the superiority and robustness of the proposed CaEGCN.

Achieving robust cross contexts speech emotion recognition (SER) has become a critical next direction of research for wide adoption of SER technology. The core challenge is in the large variability of affective speech that is highly contextualized. Prior works have worked on this as a transfer learning problem that mostly focuses on developing domain adaptation strategy. However, many of the existing speech emotion corpora, even those considered as large scale, are still limited in size resulting in an unsatisfactory transfer result. On the other hand, directly collecting context-specific corpus often results in an even smaller data size leading to an inevitably non-robust accuracy. In order to mitigate this issue, we propose the concept of enhancing the affect-related variability when learning the in-context acoustic latent representation by integrating out-of-context emotion data. Specifically, we utilize adversarial autoencoder network as our backbone with multiple out-of-context emotion labels derived for each in-context samples that serve as an auxiliary constraint in learning the latent representation. We extensively evaluate our framework using three in-context databases with three out-of-context databases. In this work, we demonstrate not only an improved recognition accuracy but also a comprehensive analysis on the effectiveness of this representation learning strategy.

Understanding in-network information diffusion is a fundamental problem in many applications and one of the primary challenges is to predict the information cascade size. Most of the existing models rely either on hypothesized point process (e.g., Poisson and Hawkes processes), or simply predict the information propagation via deep neural networks. However, they fail to simultaneously capture the underlying global and local structures of a cascade and the propagation uncertainty in the diffusion, which may result in unsatisfactory prediction performance. To address these, in this work we propose a novel probabilistic cascade prediction framework: Hierarchical Cascade Normalizing Flows (CasFlow). CasFlow allows a non-linear information diffusion inference and models the information diffusion process by learning the latent representation of both the structural and temporal information. It is a pattern-agnostic model leveraging normalizing flows to learn the node-level and cascade-level latent factors in an unsupervised manner. In addition, CasFlow is capable of capturing both the cascade representation uncertainty and node infection uncertainty, while enabling hierarchical pattern learning of information diffusion. Extensive experiments conducted on real-world datasets demonstrate that CasFlow reduces the prediction error to 21.0% by only observing half an hour of cascades, compared to state-of-the-art approaches, while also enabling model interpretability.

Differentiable ARchiTecture Search (DARTS) uses a continuous relaxation of network representation and dramatically accelerates Neural Architecture Search (NAS) by almost thousands of times in GPU-day. However, the searching process of DARTS is unstable, which suffers severe degradation when training epochs become large, thus limiting its application. In this article, we claim that this degradation issue is caused by the imbalanced norms between different nodes and the highly correlated outputs from various operations. We then propose an improved version of DARTS, namely iDARTS, to deal with the two problems. In the training phase, it introduces node normalization to maintain the norm balance. In the discretization phase, the continuous architecture is approximated based on the similarity between the outputs of the node and the decorrelated operations rather than the values of the architecture parameters. Extensive evaluation is conducted on CIFAR-10 and ImageNet, and the error rates of 2.25% and 24.7% are reported within 0.2 and 1.9 GPU-day for architecture search, respectively, which shows its effectiveness. Additional analysis also reveals that iDARTS has the advantage in robustness and generalization over other DARTS-based counterparts.

Online users are typically active on multiple social media networks (SMNs), which constitute a multiplex social network. With improvements in cybersecurity awareness, users increasingly choose different usernames and provide different profiles on different SMNs. Thus, it is becoming increasingly challenging to determine whether given accounts on different SMNs belong to the same user; this can be expressed as an interlayer link prediction problem in a multiplex network. To address the challenge of predicting interlayer links, feature or structure information is leveraged. Existing methods that use network embedding techniques to address this problem focus on learning a mapping function to unify all nodes into a common latent representation space for prediction; positional relationships between unmatched nodes and their common matched neighbors (CMNs) are not utilized. Furthermore, the layers are often modeled as unweighted graphs, ignoring the strengths of the relationships between nodes. To address these limitations, we propose a framework based on multiple types of consistency between embedding vectors (MulCEVs). In MulCEV, the traditional embedding-based method is applied to obtain the degree of consistency between the vectors representing the unmatched nodes, and a proposed distance consistency index based on the positions of nodes in each latent space provides additional clues for prediction. By associating these two types of consistency, the effective information in the latent spaces is fully utilized. In addition, MulCEV models the layers as weighted graphs to obtain representation. In this way, the higher the strength of the relationship between nodes, the more similar their embedding vectors in the latent representation space will be. The results of our experiments on several real-world and synthetic datasets demonstrate that the proposed MulCEV framework markedly outperforms current embedding-based methods, especially when the number of training iterations is small.

Long-term visual place recognition (VPR) is challenging as the environment is subject to drastic appearance changes across different temporal resolutions, such as time of the day, month, and season. A wide variety of existing methods address the problem by means of feature disentangling or image style transfer but ignore the structural information that often remains stable even under environmental condition changes. To overcome this limitation, this article presents a novel structure-aware feature disentanglement network (SFDNet) based on knowledge transfer and adversarial learning. Explicitly, probabilistic knowledge transfer (PKT) is employed to transfer knowledge obtained from the Canny edge detector to the structure encoder. An appearance teacher module is then designed to ensure that the learning of appearance encoder does not only rely on metric learning. The generated content features with structural information are used to measure the similarity of images. We finally evaluate the proposed approach and compare it to state-of-the-art place recognition methods using six datasets with extreme environmental changes. Experimental results demonstrate the effectiveness and improvements achieved using the proposed framework. Source code and some trained models will be available at http://www.tianshu.org.cn.

The capability of Convolutional Neural Networks (CNNs) for sparse representation has significant application to complex tasks like Representation Learning (RL). However, labelled datasets of sufficient size for learning this representation are not easily obtainable. The unsupervised learning capability of Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) provide a promising solution to this issue through their capacity to learn representations for novel data samples and classification tasks. In this research, a texture-based latent space disentanglement technique is proposed to enhance learning of representations for novel data samples. A comparison is performed among different VAEs and GANs with the proposed approach for synthesis of new data samples. Two different VAE architectures are considered, a single layer dense VAE and a convolution based VAE, to compare the effectiveness of different architectures for learning of the representations. The GANs are selected based on the distance metric for disjoint distribution divergence estimation of complex representation learning tasks. The proposed texture-based disentanglement has been shown to provide a significant improvement for disentangling the process of representation learning by conditioning the random noise and synthesising texture rich images of fruit and vegetables.

3D reconstruction from single-view images is a long-standing research problem. To date, there are various methods based on point clouds and volumetric representations. In spite of success in 3D models generation, it is quite challenge for these approaches to deal with models with complex topology and fine geometric details. Thanks to the recent advance of deep shape representations,learning the structure and detail representation using deep neural networks is a promising direction. In this paper, we propose a novel approach named STD-Net to reconstruct 3D models utilizing mesh representation that is well suited for characterizing complex structures and geometry details. Our method consists of (1) an auto-encoder network for recovering the structure of an object with bounding box representation from a single-view image; (2) a topology-adaptive GCN for updating vertex position for meshes of complex topology; and (3) a unified mesh deformation block that deforms the structural boxes into structure-aware meshes. Evaluation on ShapeNet shows that STD-Net has better performance than state-of-the-art methods in reconstructing complex structures and fine geometric details.

The predicted utility of information stored in working memory (WM) is hypothesized to influence the strategic allocation of WM resources. Prior work has shown that when information is prioritized, it is remembered with greater precision relative to other remembered items. However, these paradigms often complicate interpretation of the effects of predicted utility on item fidelity due to a concurrent memory load. Likewise, no fMRI studies have examined whether the predicted utility of an item modulates fidelity in the neural representation of items during the memory delay without a concurrent load. In the current study, we used fMRI to investigate whether predicted utility influences fidelity of WM representations in the brain. Using a generative model multivoxel analysis approach to estimate the quality of remembered representations across predicted utility conditions, we observed that items with greater predicted utility are maintained in memory with greater fidelity, even when they are the only item being maintained. Further, we found that this pattern follows a parametric relationship where more predicted utility corresponded to greater fidelity. These precision differences could not be accounted for based on a redistribution of resources among already-remembered items. Rather, we interpret these results in terms of a gating mechanism that allows for pre-allocation of resources based on predicted value alone. This evidence supports a theoretical distinction between resource allocation that occurs as a result of load and resource pre-allocation that occurs as a result of predicted utility.

Freezing of gait (FoG) is identified as a sudden and brief episode of movement cessation despite the intention to continue walking. It is one of the most disabling symptoms of Parkinson's disease (PD) and often leads to falls and injuries. Many computer-aided FoG detection methods have been proposed to use data collected from unimodal sources, such as motion sensors, pressure sensors, and video cameras. However, there are limited efforts of multimodal-based methods to maximize the value of all the information collected from different modalities in clinical assessments and improve the FoG detection performance. Therefore, in this study, a novel end-to-end deep architecture, namely graph fusion neural network (GFN), is proposed for multimodal learning-based FoG detection by combining footstep pressure maps and video recordings. GFN constructs multimodal graphs by treating the encoded features of each modality as vertex-level inputs and measures their adjacency patterns to construct complementary FoG representations, thus reducing the representation redundancy among different modalities. In addition, since GFN is devised to process multimodal graphs of arbitrary structures, it is expected to achieve superior performance with inputs containing missing modalities, compared to the alternative unimodal methods. A multimodal FoG dataset was collected, which included clinical assessment videos and footstep pressure sequences of 340 trials from 20 PD patients. Our proposed GFN demonstrates a great promise of multimodal FoG detection with an area under the curve (AUC) of 0.882. To the best of our knowledge, this is one of the first studies to utilize multimodal learning for automated FoG detection, which offers significant opportunities for better patient assessments and clinical trials in the future.

This paper aims to predict a set of apps a user will open on her mobile device in the next time slot. Such an information is essential for many smartphone operations, e.g., app pre-loading and content pre-caching, to improve user experience. However, it is hard to build an explicit model that accurately captures the complex environment context and predicts a set of apps at one time. This paper presents a deep reinforcement learning framework, named as DeepAPP, which learns a model-free predictive neural network from historical app usage data. Meanwhile, an online updating strategy is designed to adapt the predictive network to the time-varying app usage behavior. To transform DeepAPP into a practical deep reinforcement learning system, several challenges are addressed by developing a context representation method for complex contextual environment, a general agent for overcoming data sparsity and a lightweight personalized agent for minimizing the prediction time. Extensive experiments on a large-scale anonymized app usage dataset reveal that DeepAPP provides high accuracy (precision 70.6% and recall of 62.4%) and reduces the prediction time of the state-of-the-art by 6.58 times. A field experiment of 29 participants demonstrates DeepAPP can effectively reduce launch time of apps.

Projection techniques are often used to visualize high-dimensional data, allowing users to better understand the overall structure of multi-dimensional spaces on a 2D screen. Although many such methods exist, comparably little work has been done on generalizable methods of inverse-projection -- the process of mapping the projected points, or more generally, the projection space back to the original high-dimensional space. In this paper we present NNInv, a deep learning technique with the ability to approximate the inverse of any projection or mapping. NNInv learns to reconstruct high-dimensional data from any arbitrary point on a 2D projection space, giving users the ability to interact with the learned high-dimensional representation in a visual analytics system. We provide an analysis of the parameter space of NNInv, and offer guidance in selecting these parameters. We extend validation of the effectiveness of NNInv through a series of quantitative and qualitative analyses. We then demonstrate the method's utility by applying it to three visualization tasks: interactive instance interpolation, classifier agreement, and gradient visualization.

The logarithmic representation of infinitesimal generators is generalized to the cases when the evolution operator is unbounded. The generalized result is applicable to the representation of infinitesimal generators of unbounded evolution operators, where unboundedness of evolution operator is an essential ingredient of nonlinear analysis. In conclusion a general framework for the identification between the infinitesimal generators with evolution operators is established. A mathematical framework for such an identification is indispensable to the rigorous treatment of nonlinear transforms: e.g., transforms appearing in the theory of integrable systems., To appear in Math. Meth. Appl. Sci

This paper aims to solve two important issues that frequently occur in existing automatic personality analysis systems: 1. Attempting to use very short video segments or even single frames to infer personality traits; 2. Lack of methods to encode person-specific facial dynamics for personality recognition. Hence, we proposes a novel Rank Loss which utilizes the natural temporal evolution of facial actions, rather than personality labels, for self-supervised learning of facial dynamics. Our approach first trains a generic U-net model that can infer general facial dynamics learned from unlabelled face videos. Then, the generic model is frozen, and a set of intermediate filters are incorporated into this architecture. The self-supervised learning is then resumed with only person-specific videos. This way, the learned filters' weights are person-specific, making them a valuable source for modeling person-specific facial dynamics. We then concatenate the weights of the learned filters as a person-specific representation, which can be directly used to predict the personality traits without needing other parts of the network. We evaluate the proposed approach on both self-reported personality and apparent personality datasets. Besides achieving promising results in personality trait estimation from videos, we show that fusion of tasks reaches highest accuracy, and that multi-scale dynamics are more informative than single-scale dynamics.

Measuring and improving the efficiency of a sustainable supply chain in an effective way for strategic decision making is important in the business world. In this article, an alternative coherent data envelopment analysis (CoDEA) with a representation of the intramural structure is established for evaluating the efficiency of a sustainable supply chain. The proposed CoDEA model not only maintains the traditional value of data envelopment analysis (DEA) in its “black box” approach, which avoids the intermediate measures among different nodes in the supply chain, but also overcomes some main pitfalls in previously developed DEA models. The usefulness of the proposed model is illustrated in both simple and complex supply chain situations, yielding reasonable efficiency scores as compared with those from the existing methods. Moreover, the dummy decision-making unit (DMU) introduced in CoDEA offers additional benefits in the scenario of a small supply chain. The approach allows the implicit rule that requires three times the ratio between the total measurement factors and total DMUs to be always satisfied, providing a higher discriminatory power of CoDEA. Based on the case studies covering both simple and complex supply chains, the results and sensitivity analysis ascertain that the proposed CoDEA model is a flexible and reasonable alternative with a good efficiency evaluation, contributing toward an efficient and sustainable supply chain management process pertaining to sustainable strategy or policy recommendations.

Passenger-flow anomaly detection and prediction are essential tasks for intelligent operation of the metro system. Accurate passenger-flow representation is the foundation of them. However, spatiotemporal dependencies, complex dynamic changes, and anomalies of passenger-flow data bring great challenges to data representation. Taking advantage of the time-varying characteristics of data, we propose a novel passenger-flow representation model based on low-rank dynamic mode decomposition (DMD), which also integrates the global low-rank nature and sparsity to explore the spatiotemporal consistency of data and depict abrupt data, respectively. The model can detect anomalies and predict short-term passenger flow conveniently and flexibly. For anomaly detection, we further introduce a strong temporal Toeplitz regularization to characterize the temporal periodic change of data, so as to more accurately detect anomalies. We conduct experiments with smart card transaction data from the Beijing metro system to assess the performance of the model in two use cases. In terms of anomaly detection, the experimental results demonstrate that our method can detect anomalies efficiently, especially for time sequence anomalies. As for short-term prediction, our model is superior to other methods in most cases.

While we instantaneously recognize a face as attractive, it is much harder to explain what exactly defines personal attraction. This suggests that attraction depends on implicit processing of complex, culturally and individually defined features. Generative adversarial neural networks (GANs), which learn to mimic complex data distributions, can potentially model subjective preferences unconstrained by pre-defined model parameterization. Here, we present generative brain-computer interfaces (GBCI), coupling GANs with brain-computer interfaces. GBCI first presents a selection of images and captures personalized attractiveness reactions toward the images via electroencephalography. These reactions are then used to control a GAN model, finding a representation that matches the features constituting an attractive image for an individual. We conducted an experiment (N=30) to validate GBCI using a face-generating GAN and producing images that are hypothesized to be individually attractive. In double-blind evaluation of the GBCI-produced images against matched controls, we found GBCI yielded highly accurate results. Thus, the use of EEG responses to control a GAN presents a valid tool for interactive information-generation. Furthermore, the GBCI-derived images visually replicated known effects from social neuroscience, suggesting that the individually responsive, generative nature of GBCI provides a powerful, new tool in mapping individual differences and visualizing cognitive-affective processing.

Learning from complementary labels (CLs) is a useful learning paradigm, where the CL specifies the classes that the instance does not belong to, instead of providing the ground truth as in the ordinary supervised learning scenario. In general, although it is less laborious and more efficient to collect CLs compared with ordinary labels, the less informative signal in the complementary supervision is less helpful to learn competent feature representation. Consequently, the final classifier's performance greatly deteriorates. In this article, we leverage generative adversarial networks (GANs) to derive an algorithm GAN-CL to effectively learn from CLs. In addition to the role in original GAN, the discriminator also serves as a normal classifier in GAN-CL, with the objective constructed partly with the complementary information. To further prove the effectiveness of our schema, we study the global optimality of both generator and discriminator for the GAN-CL under mild assumptions. We conduct extensive experiments on benchmark image datasets using deep models, to demonstrate the compelling improvements, compared with state-of-the-art CL learning approaches.

This paper proposes an image fusion framework based on separate representation learning, called IFSepR. We believe that both the co-modal image and the multi-modal image have common and private features based on prior knowledge, exploiting this disentangled representation can help to image fusion, especially to fusion rule design. Inspired by the autoencoder network and contrastive learning, a multi-branch encoder with contrastive constraints is built to learn the common and private features of paired images. In the fusion stage, based on the disentangled features, a general fusion rule is designed to integrate the private features, then combining the fused private features and the common feature are fed into the decoder, reconstructing the fused image. We perform a series of evaluations on three typical image fusion tasks, including multi-focus image fusion, infrared and visible image fusion, medical image fusion. Quantitative and qualitative comparison with five state-of-art image fusion methods demonstrates the advantages of our proposed model.

Currently, with the rapid development of mobile Internet, micro-video has become a prevailing format of user-generated contents (UGCs) on various social media platforms. Several studies have been conducted towards to understanding high-level micro-video semantics, such as venue categorization, memorability analysis, and popularity prediction. However, these approaches supported tasks with only a single output, which exhibited limitations when attempting to use them to resolve tasks with multiple outputs, especially the multi-label micro-video classification. To tackle this problem, in this paper, we propose a dual multi-modal low-rank decomposition (DMLRD) method for multi-label micro-video classification tasks. To learn more comprehensive micro-video representations, we first learn the low-rank-regularized modality-specific and modality-shared components by considering the consistency and the complementarity among modalities simultaneously. Meanwhile, the less descriptive power of each modality aroused by inherent properties can be solved. To obtain unseen label representations, we next construct a sparsity-regularized multi-matrix normal estimation term to jointly encode the latent relationship structures among labels and dimensions. Experiments on a publicly available large-scale micro-video dataset demonstrate the effectiveness of our proposed model over the state-of-art methods.

In recent years, enhanced computing power and information accessibility as well as algorithmic advances have resulted to amazing outcomes in taxonomy, information generation, deterioration, reinforcement learning. In spite of these accomplishments, the closeness with the availability limitations of chip production together with the growing volume of data sets is motivate more and more scientists investigate opportunities to use the power of quantum computing to increase the speed of traditional machine learning methods. To examine quantum machine learning works and explore views for a mix of classical machine and quantum computing specialists. The chapter gathers the mainstream quantum image representation, basic quantum algorithms, learning theory and discusses the similarities, differences for some of the quantum image representation. Mostly importance will be given on illuminating the basic of quantum with its image representation, algorithms and learning theory, how they contrast and their best traditional partners and why quantum assets are required to give benefits to learning issues. Practical issues are also discussed; such as how classical information can be uploaded into the quantum.

Detecting salient objects in videos is a very challenging task. Current state-of-the-art methods are dominated by motion based deep neural networks, among which optical flow is often leveraged as motion representation. Though with robust performance, these optical flow-based video salient object detection methods face at least two problems that may hinder their generalization and application. First, computing optical flow as a pre-processing step does not support direct end-to-end learning; second, little attention has been given to the quality of visual features due to high computational cost of spatiotemporal feature encoding. In this paper we propose a novel self-sufficient feature enhancing network (SFENet) for video salient object detection, which leverages optical flow estimation as an auxiliary task while being end-to-end trainable. With a joint training scheme of both salient object detection and optical flow estimation, its multi-task architecture can be totally self-sufficient for achieving good performance without any pre-processing. Furthermore, for improving feature quality, we design four lightweight modules in spatial and temporal domains, including cross-layer fusion, multi-level warping, spatial-channel attention and boundary-aware refinement. The proposed method is evaluated through extensive experiments on five video salient object detection datasets. Experimental results show that our SFENet can be easily trained with fast convergence speed. It significantly outperforms previous methods in terms of various evaluation metrics. Moreover, with optical flow estimation and unsupervised video object segmentation as example applications, our method also yields state-of-the-art results on standard datasets.

Physiological studies have shown that there are some differences in speech and facial activities between depressive and healthy individuals. Based on this fact, we propose a novel Spatio-Temporal Attention (STA) network and a Multimodal Attention Feature Fusion (MAFF) strategy to obtain the multimodal representation of depression cues for predicting the individual depression level. Specifically, we firstly divide the speech amplitude spectrum/video into fixed-length segments and input these segments into the STA network, which not only integrates the spatial and temporal information through attention mechanism, but also emphasizes the audio/video frames related to depression detection. The audio/video segment-level feature is obtained from the output of the last full connection layer of the STA network. Secondly, this paper employs the eigen evolution pooling method to summarize the changes of each dimension of the audio/video segment-level features to aggregate them into the audio/video level feature. Thirdly, the multimodal representation with modal complementary information is generated using the MAFF and inputs into the support vector regression predictor for estimating depression severity. Experimental results on the AVEC2013 and AVEC2014 depression databases illustrate the effectiveness of our method.

The challenging problem in the traffic system is lane detection. This Lane detection which attracts the computer vision community’s attention. For computer vision and machine learning techniques, the Lane detection which acts as the multi-feature detection problem. Many machine learning techniques are used for lane detection. Driver support system is one of the most important features in the modern vehicles to ensure the safety of the driver and decrease the vehicle accidents on road. Road Lane detection is the most challenging task and complex tasks now-a-days. Road localization and relative position between vehicle and roads which also includes with this. But in this journal, we propose a new method. Here, an on- board camera to be used which is looking outwards are presented here in this work. This proposed technique which can be used for all types of roads like painted, unpainted, curved, straight roads etc in different weather conditions. No need for camera calibration and coordination of the transform, may be any changing illumination situation, shadow effects, various road types. No representation for speed limits. This includes that the system acquires the front view using a camera mounted on the vehicles and detect the Lane by applying the code from the Python Programming process. This proposed system does not require any more information about roads. This system which demonstrates a robust performance for Lane detection.

In this paper, we propose a simple yet effective self-supervised method called spatio-temporal contrastive learning (ST-CL for 3D skeleton-based action recognition. ST-CL acquires action-specific features by regarding the spatio-temporal continuity of motion tendency as the supervisory signal. To yield effective representations, ST-CL first designs a novel contrastive proxy task by providing different spatio-temporal observation scenes for the same 3D action and pulling them together in the embedding space. Second, three key components are devised in the action encoding to efficiently extract representations in contrastive tasks: (1 Information Representation introduces the awareness of joint type when analyzing the motion dynamics. (2 Non-local GCN learns a data-driven graph topology structure and promotes a spatial message passing among long-range joints in each frame. (3 Multi-Scale TCN makes larger receptive fields for capturing richer longe-range temporal dynamics amomg adjacent frames. In ST-CL, the effective proxy tasks yield useful representations and efficient action encoding further enhances representation capacity. \revise{As validated on the four large-scale datasets, ST-CL is a strong baseline with high performance and efficiency for the contrastive learning study of the skeleton data. Compared to previous self-supervised methods, the proposed ST-CL achieves significant improvement consistently with a smaller model size and better training efficiency.

Neural network quantization has shown to be an effective way for network compression and acceleration. However, existing binary or ternary quantization methods suffer from two major issues. First, low bit-width input/activation quantization easily results in severe prediction accuracy degradation. Second, network training and quantization are always treated as two non-related tasks, leading to accumulated parameter training error and quantization error. In this work, we introduce a novel scheme, named Residual Quantization, to train a neural network with both weights and inputs constrained to low bit-width, e.g., binary or ternary values. On one hand, by recursively performing residual quantization, the resulting binary/ternary network is guaranteed to approximate the full-precision network with much smaller errors. On the other hand, we mathematically re-formulate the network training scheme in an EM-like manner, which iteratively performs network quantization and parameter optimization. During expectation, the low bit-width network is encouraged to approximate the full-precision network. During maximization, the low bit-width network is further tuned to gain better representation capability. Extensive experiments well demonstrate that the proposed quantization scheme outperforms previous low bit-width methods and achieves much closer performance to the full-precision counterpart.

This paper investigates a class of unified stochastic linear-quadratic-Gaussian (LQG) social optima problems involving a large number of weakly-coupled interactive agents under a generalized setting. For each individual agent, the control and state process enters both diffusion and drift terms in its linear dynamics, and the control weight might be indefinite in cost functional. This setup is innovative and has great theoretical and realistic significance as its applications in mathematical finance (e.g., portfolio selection in mean-variation model). Using some fully-coupled variational analysis under the person-by-person optimality principle, and the mean-field approximation method, the decentralized social control is derived by a class of new type consistency condition (CC) system for typical representative agent. Such CC system is some mean-field forward-backward stochastic differential equation (MF-FBSDE) combined with embedding representation. The well-posedness of such forward-backward stochastic differential equation (FBSDE) system is carefully examined. The related social asymptotic optimality is related to the convergence of the average of a series of weakly-coupled backward stochastic differential equation (BSDE). They are verified through some Lyapunov equations.

In this paper we present a new classification method based on Dictionary Learning (DL). The main contribution consists of a kernel version of incoherent DL, derived from its standard linear counterpart. We also propose an improvement of the AK-SVD algorithm concerning the representation update. Our algorithms are tested on several popular databases of classification problems.

In this paper, we explore the challenging 1-to-N map matching problem, which exploits a compact description of map data, to improve the scalability of map matching techniques used by various robot vision tasks. We propose a first method explicitly aimed at fast succinct map matching, which consists only of map-matching subtasks. These tasks include offline map matching attempts to find a compact part-based scene model that effectively explains each map using fewer larger parts. The tasks also include an online map matching attempt to efficiently find correspondence between the part-based maps. Our part-based scene modeling approach is unsupervised and uses common pattern discovery (CPD) between the input and known reference maps. This enables a robot to learn a compact map model without human intervention. We also present a practical implementation that uses the state-of-the-art CPD technique of randomized visual phrases (RVP) with a compact bounding box (BB) based part descriptor, which consists of keypoint and descriptor BBs. The results of our challenging map-matching experiments, which use a publicly available radish dataset, show that the proposed approach achieves successful map matching with significant speedup and a compact description of map data that is tens of times more compact. Although this paper focuses on the standard 2D point-set map and the BB-based part representation, we believe our approach is sufficiently general to be applicable to a broad range of map formats, such as the 3D point cloud map, as well as to general bounding volumes and other compact part representations., Offprint of IROS2013 paper

Extensive studies have been conducted on human action recognition, whereas relatively few methods have been proposed for hand action recognition. Although it is very natural and straightforward to apply a human action recognition method to hand action recognition, this approach cannot always lead to state-of-the-art performance. One of the important reasons is that both the between-class difference and the within-class difference in hand actions are much smaller than those in human actions. In this paper, we study first-person hand action recognition from RGB-D sequences. To explore whether pretrained networks substantially influence accuracy, 8 classic pretrained networks and one pretrained network designed by us are introduced for extracting RGB-D features. A Lie group is introduced for hand pose representation. Ablation studies are conducted to compare the discriminative power of the RGB modality, depth modality, pose modality, and their combinations. In our method, a fixed number of frames are randomly sampled to represent an action. This temporal modeling strategy is simple but is proven more effective than both the graph convolutional network (GCN) and the recurrent neural network (RNN), which are widely adopted by conventional methods. Evaluation experiments on two public datasets demonstrate that our method markedly outperforms recent baselines.

The analysis of large-scale multimodal data has become very popular recently. Image captioning, whose goal is to describe the content of image with natural language automatically, is an essential and challenging task in artificial intelligence. Commonly, most existing image caption methods utilize the mixture of Convolutional Neural Network and Recurrent Neural Network framework. These methods either pay attention to global representation at the image level or only focus on the specific concepts, such as regions and objects. To make the most of characteristics about a given image, in this study, we present a novel model named Multilevel Attention Networks and Policy Reinforcement Learning for image caption generation. Specifically, our model is composed of a multilevel attention network module and a policy reinforcement learning module. In the multilevel attention network, the object-attention network aims to capture global and local details about objects, whereas the region-attention network obtains global and local features about regions. After that, a policy reinforcement learning algorithm is adopted to overcome the exposure bias problem in the training phase and solve the loss-evaluation mismatching problem at the caption generation stage. With the attention network and policy algorithm, our model can automatically generate accurate and natural sentences for any particular image. We carry out extensive experiments on the MSCOCO and Flickr30k data sets, demonstrating that our model is superior to other competitive methods.

Realistic speech-driven 3D facial animation is a challenging problem due to the complex relationship between speech and face. In this paper, we propose a deep architecture, called Geometry-guided Dense Perspective Network (GDPnet), to achieve speaker-independent realistic 3D facial animation. The encoder is designed with dense connections to strengthen feature propagation and encourage the re-use of audio features, and the decoder is integrated with an attention mechanism to adaptively recalibrate point-wise feature responses by explicitly modeling interdependencies between different neuron units. We also introduce a non-linear face reconstruction representation as a guidance of latent space to obtain more accurate deformation, which helps solve the geometry-related deformation and is good for generalization across subjects. Huber and HSIC (Hilbert-Schmidt Independence Criterion) constraints are adopted to promote the robustness of our model and to better exploit the non-linear and high-order correlations. Experimental results on the public dataset and real scanned dataset validate the superiority of our proposed GDPnet compared with state-of-the-art model. The code is available for research purposes at http://cic.tju.edu.cn/faculty/likun/projects/GDPnet.

Semantic segmentation is paramount for autonomous vehicles to have a deeper understanding of the surrounding traffic environment and enhance safety. Deep neural networks (DNN) have achieved remarkable performances in semantic segmentation. However, training such a DNN requires a large amount of labelled data at pixel level. In practice, it is a labour-intensive task to manually annotate dense pixel-level labels. To tackle the problem associated with a small amount of labelled data, Deep Domain Adaptation (DDA) methods have recently been developed to examine the use of synthetic driving scenes so as to significantly reduce the manual annotation cost. Despite remarkable advances, these methods unfortunately suffer from the generalisability problem that fails to provide a holistic representation of the mapping from the source image domain to the target image domain. In this paper, we therefore develop a novel ensembled DDA to train models with different up-sampling strategies, discrepancy and segmentation loss functions. The models are, therefore, complementary with each other to achieve better generalisation in the target image domain. Such a design does not only improve the adapted semantic segmentation performance, but also strengthen the model reliability and robustness. Extensive experimental results demonstrate the superiorities of our approach over several state-of-the-art methods.

Recently, many deep neural networks were designed to process 3D point clouds, but a common drawback is that rotation invariance is not ensured, leading to poor generalization to arbitrary orientations. In this article, we introduce a new low-level purely rotation-invariant representation to replace common 3D Cartesian coordinates as the network inputs. Also, we present a network architecture to embed these representations into features, encoding local relations between points and their neighbors, and the global shape structure. To alleviate inevitable global information loss caused by the rotation-invariant representations, we further introduce a region relation convolution to encode local and non-local information. We evaluate our method on multiple point cloud analysis tasks, including (i) shape classification, (ii) part segmentation, and (iii) shape retrieval. Extensive experimental results show that our method achieves consistent, and also the best performance, on inputs at arbitrary orientations, compared with all the state-of-the-art methods.

Depth maps acquired with ToF cameras have a limited accuracy due to the high noise level and to the multi-path interference. Deep networks can be used for refining ToF depth, but their training requires real world acquisitions with ground truth, which is complex and expensive to collect. A possible workaround is to train networks on synthetic data, but the domain shift between the real and synthetic data reduces the performances. In this paper, we propose three approaches to perform unsupervised domain adaptation of a depth denoising network from synthetic to real data. These approaches are respectively acting at the input, at the feature and at the output level of the network. The first approach uses domain translation networks to transform labeled synthetic ToF data into a representation closer to real data, that is then used to train the denoiser. The second approach tries to align the network internal features related to synthetic and real data. The third approach uses an adversarial loss, implemented with a discriminator trained to recognize the ground truth statistic, to train the denoiser on unlabeled real data. Experimental results show that the considered approaches are able to outperform other state-of-the-art techniques and achieve superior denoising performances.

Classification of Alzheimer’s disease (AD) is one of the most challenging issues for neurologists. Manual methods are time consuming and may not be accurate all the time. Since, brain is the most affected region in AD, a proper classification framework using brain images may provide more accurate results. Deep Learning (DL) is a popular representation of machine learning techniques, that emulate the functionalities of a human brain to process information and creates patterns that help in making complex decisions. The ability to absorb information, even from the unstructured and unlabeled data, makes DL one of the first choices by researchers. In this paper, some of the most popular DL models are discussed along with their implementation results for AD classification. All brain Magnetic Resonance (MR) images are acquired from the online data-set, “Alzheimer’s Disease Neuroimaging Initiative (ADNI)”. From the performance comparison amongst all the discussed models, it is observed that the DenseNet-121 model achieves a convincing result with an average performance rate of 88.78%. But one limitation of the DenseNet model is that it uses lots of convolutional operations that make the model computationally slower than many of the discussed models. Depth-wise convolution is a popular way to make a convolutional operation faster and better. Hence, to improve the execution time, we have proposed replacing the convolution layers in the original DenseNet-121 architecture with depth-wise convolution layers. The new architecture also improved the performance of the model with an average rate of 90.22%.

The data-driven intelligent health state diagnosis strategy has been successfully applied in modern industrial equipments. However, many of existing researches suffer from two major deficiencies: Firstly, the sample independence assumption is widely adopted, so the influences of the relationship between samples on the overall performances are not explored. Secondly, most of the above-mentioned application objects are typical key functional components (such as bearings, gearboxes, etc.), and researches on the intelligent health state diagnosis of planar parallel manipulator are rarely reported. To address these issues, a novel intelligent health state diagnosis method, termed multi-scale graph-guided convolutional network with node attention (MSGCN-NA), is proposed for a 3-PRR (P and R represent prismatic and revolute pairs respectively) planar parallel manipulator. Specifically, the developed MSGCN-NA model mainly contains two parts: a) the one is an unsupervised convolutional auto-encoder, which is employed for the extraction of deep representation features, and then combined with Pearson metric to establish the adjacency matrix. b) the other part is the constructed multi-scale graph convolutional network, in which the node attention mechanism is adopted to achieve cross-scale fusion of different neighborhood information. The effectiveness of the proposed MSGCN-NA method is fully verified based on the simulation and experimental scenarios, the results show that MSGCN-NA can achieve superior diagnosis performances.

Co-location pattern mining refers to discovering neighboring relationships of spatial features distributed in geographic space. With the rapid growth of spatial datasets, the usefulness of co-location patterns is strongly limited by the large number of discovered patterns containing multiple redundancies. To address this problem, in this article, we propose a novel approach for discovering the super participation index-closed (SPI-closed) co-location patterns which are a newly proposed lossless condensed representation of co-location patterns by considering distributions of the spatial instances. In the proposed approach, first, a linear-time method is designed to generate complete and correct neighboring cliques using extended neighboring relationships. Based on these cliques, a hash structure is then constructed to store the distributions of the co-location patterns in a condensed way. Finally, using this hash structure, the SPI-closed co-location patterns (SCPs) are efficiently discovered even if the prevalence threshold is changed, while similar approaches have to restart their mining processes. To confirm the efficiency of the proposed method, we compared its performance with similar approaches in the literature on multiple real and synthetic spatial datasets. The experiments confirm that our new approach is more efficient, effective, and flexible than similar approaches.

Most NN-RSs focus on accuracy by building representations from the direct user-item interactions (e.g., user-item rating matrix), while ignoring the underlying relatedness between users and items (e.g., users who rate the same ratings for the same items should be embedded into similar representations), which is an ideological disadvantage. On the other hand, ME models directly employ inner products as a default loss function metric that cannot project users and items into a proper latent space, which is a methodological disadvantage. In this paper, we propose a supervised collaborative representation learning model - Magnetic Metric Learning (MML) - to map users and items into a unified latent vector space, enhancing the representation learning for NN-RSs. Firstly, MML utilizes dual triplets to model not only the observed relationships between users and items, but also the underlying relationships between users as well as items to overcome the ideological disadvantage. Specifically, a modified metric-based dual loss function is proposed in MML to gather similar entities and disperse the dissimilar ones. With MML, we can easily compare all the relationships (user to user, item to item, user to item) according to the weighted metric, which overcomes the methodological disadvantage. We conduct extensive experiments on four real-world datasets with large item space. The results demonstrate that MML can learn a proper unified latent space for representations from the user-item matrix with high accuracy and effectiveness, and lead to a performance gain over the state-of-the-art RS models by an average of 17%., Accepted by IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING(TKDE)

As a class of context-aware systems, context-aware service recommendation aims to bind high-quality services to users while taking into account their context requirements, including invocation time, location, social profiles, connectivity, and so on. However, current CASR approaches are not scalable with the huge amount of service data (QoS and context information, users reviews and feedbacks). In addition, they lack a rich representation of contextual information as they adopt a simple matrix view. Moreover, current CASR approaches adopt the traditional user-service relation and they do not allow for multi-relational interactions between users and services in different contexts. To offer a scalable and context-sensitive service recommendation with great analysis and learning capabilities, we provide a rich and multi-relational representation of the CASR knowledge, based on the concept of knowledge graph. The constructed context-aware service knowledge graph (C-SKG) is, then, transformed into a low-dimentional vector space to facilitate its processing. For this purpose, we adopt Dilated Recurrent Neural Networks to propose a context-aware knowledge graph embedding, based on the principles of first-order and subgraph-aware proximity. Finally, a recommendation algorithm is defined to deliver the top-rated services according to the target user's context. Experiments have proved the accuracy and scalability of our CASR approach.

In this article, the unsupervised domain adaptation problem, where an approximate inference model is to be learned from a labeled dataset and expected to generalize well on an unlabeled dataset, is considered. Unlike the existing work, we explicitly unveil the importance of the latent variables produced by the feature extractor, that is, encoder, where contains the most representative information about their input samples, for the knowledge transfer. We argue that an estimator of the representation of the two datasets can be used as an agent for knowledge transfer. To be specific, a novel variational inference approach is proposed to approximate a latent distribution from the unlabeled dataset that can be used to accurately predict its input samples. It is demonstrated that the discriminative knowledge of the latent distribution that is learned from the labeled dataset can be progressively transferred to that is learned from the unlabeled dataset by simultaneously optimizing the estimator via the variational inference and our proposed regularization for shifting the mean of the estimator. The experiments on several benchmark datasets demonstrate that the proposed method consistently outperforms state-of-the-art methods for both object classification and digit classification.

The recent success in supervised multi-view stereopsis (MVS) relies on the onerously collected real-world 3D data. While the latest differentiable rendering techniques enable unsupervised MVS, they are restricted to discretized (e.g., point cloud) or implicit geometric representation, suffering from either low integrity for a textureless region or less geometric details for complex scenes. In this paper, we propose SurRF, an unsupervised MVS pipeline by learning Surface Radiance Field, i.e., a radiance field defined on a continuous and explicit 2D surface. Our key insight is that, in a local region, the explicit surface can be gradually deformed from a continuous initialization along view-dependent camera rays by differentiable rendering. That enables us to define the radiance field only on a 2D deformable surface rather than in a dense volume of 3D space, leading to compact representation while maintaining complete shape and realistic texture for large-scale complex scenes. We experimentally demonstrate that the proposed SurRF produces competitive results over the-state-of-the-art on various real-world challenging scenes, without any 3D supervision. Moreover, SurRF shows great potential in owning the joint advantages of mesh (scene manipulation), continuous surface (high geometric resolution), and radiance field (realistic rendering).

Cyber-Physical-Social Systems (CPSS) provide great value to our lives, but they also cause data overload problems. Data-driven personalized recommendation service is one of the most efficient means to solve such problems, which is currently receiving wide attention from research and industrial communities. The most important task of personalized recommender systems is to predict the click-through rate of given items, which is especially true for personalized advertisement recommendation systems. Recently, a number of deep click-through models have been proposed, which obtain low-dimensional dense embedding vectors of features, and then concatenate them together and input into multi-layer perceptron to learn the nonlinear relationship between the features. However, the existing models don't dig deep enough into the user preferences and habits in users behavior history. In this paper, we propose a new model: Self-attention based Deep Neural Network (DeepSA), which addresses this issue by constructing Ad-related graph and training graph embedding vectors to enhance the representation of the advertisement for capturing user interests, and learns the internal correlation between user behaviors via the self-attention mechanism, which better explores interests and preferences hidden in users' historical behaviors. Experiments on two public datasets and an industrial dataset demonstrate the proposed method outperforms the state-of-the-art models

Network embedding (NE) aims to encode the relations of vertices into a low-dimensional space. After NE, we can obtain the learned vectors of vertices that preserve the proximity of network structures for subsequent applications, e.g., vertex classification and link prediction. In existing NE models, they usually exploit the skip-gram with a negative sampling method to optimize their objective functions. Generally, this method learns the vertex representation only from the local connectivity of vertices (i.e., neighbors). However, there is a larger scope of vertex connectivity in real-world scenarios: a vertex may have multifaceted aspects and should belong to overlapping communities. Taking a social network as the overlapping example, a user may subscribe to the channels of politics, economy, and sports simultaneously, but the politics share more common attributes with the economy and less with the sports. In this article, we propose an adversarial learning approach (ACNE) for modeling overlapping communities of vertices. Specifically, we map the association between communities and vertices into an embedding space. Moreover, we take further research on enhancing our ACNE with the following two operations. First, in the initialization stage, we adopt a walking strategy with perception to obtain paths containing more possible boundary vertices to improve overlapping community detection. Then, after representation learning with ACNE, we use soft community assignments from a simple classifier as supervision to update the weights of ACNE. This self-training mechanism referred to as ACNE-ST can help ACNE to achieve better performance. Experimental results demonstrate that the proposed methods, including ACNE and ACNE-ST, can outperform the state-of-the-art models on the subsequent tasks of vertex classification and overlapping community detection.