Search

Paper cooperation network embodies expert topic similarity in an extent, thus, a novel method is proposed for expert topic analysis and extraction by combining paper cooperation network and topic model. In the method, we extract each paper’ author information and construct an expert cooperation network. At the same time, by means of LDA model, a probabilistic topic model is also built to analyze papers’ latent topics. Then, by making full use of the feature that adjacent nodes in the expert cooperation network share similar themes distribution, we makes a constraint on expert topic distribution in Gibbs sampling process of solving the probabilistic topic model. Experimental results on NIPS dataset show that the proposed method can effectively extract expert topics, and the expert paper cooperation network plays a very good supporting role on the extracting task.

Many methods achieve the visual object tracking task with deep learning technologies. As the deep features of different levels contain various semantic information and functions, this paper presents a multi-hierarchy feature aggregation approach to tackle the specific issues in the tracking task, which consists of two aspects. On one hand, this paper integrates the features captured by the offline and online classifiers at the score level, which constructs complementary roles of these classifiers to enhance the stability of classification. Besides, the proposed offline classifier is continuously optimized with different levels of features to reinforce classification constraints. On the other hand, we design a butterfly attention module to promote the capacity of multi-hierarchy feature aggregation in the regression network, which aims to fuse and strengthen the multi-scale features by attending to their spatial information. It can capture more spatial contexts by utilizing the self-attention mechanism during the fusion procedure, and preserve the hierarchy of the features during the strengthening process. Extensive experiments on four public datasets, i.e., VOT2018, OTB100, NFS and LaSOT datasets, demonstrate the effectiveness of the proposed methods.

Context and rationale Intrusion Detection, the ability to detect malware and other attacks, is a crucial aspect to ensure cybersecurity. So is the ability to identify this myriad of attacks. Objective Artificial Neural Networks (as well as other machine learning bio-inspired approaches) are an established and proven method of accurate classification. ANNs are extremely versatile – a wide range of setups can achieve significantly different classification results. The main objective and contribution of this paper is the evaluation of the way the hyperparameters can influence the final classification result. Method and results In this paper, a wide range of ANN setups is put to comparison. We have performed our experiments on two benchmark datasets, namely NSL-KDD and CICIDS2017. Conclusions The most effective arrangement achieves the multi-class classification accuracy of 99.909% on an established benchmark dataset.

Single-object tracking is regarded as a challenging task in computer vision, especially in complex spatio-temporal contexts. The changes in the environment and object deformation make it difficult to track. In the last 10 years, the application of correlation filters and deep learning enhances the performance of trackers to a large extent. This paper summarizes single-object tracking algorithms based on correlation filters and deep learning. Firstly, we explain the definition of single-object tracking and analyze the components of general object tracking algorithms. Secondly, the single-object tracking algorithms proposed in the past decade are summarized according to different categories. Finally, this paper summarizes the achievements and problems of existing algorithms by analyzing experimental results and discusses the development trends.

With the rapid development of deep learning technology and improvement in computing capability, deep learning has been widely used in the field of hyperspectral image (HSI) classification. In general, deep learning models often contain many trainable parameters and require a massive number of labeled samples to achieve optimal performance. However, in regard to HSI classification, a large number of labeled samples is generally difficult to acquire due to the difficulty and time-consuming nature of manual labeling. Therefore, many research works focus on building a deep learning model for HSI classification with few labeled samples. In this article, we concentrate on this topic and provide a systematic review of the relevant literature. Specifically, the contributions of this paper are twofold. First, the research progress of related methods is categorized according to the learning paradigm, including transfer learning, active learning and few-shot learning. Second, a number of experiments with various state-of-the-art approaches has been carried out, and the results are summarized to reveal the potential research directions. More importantly, it is notable that although there is a vast gap between deep learning models (that usually need sufficient labeled samples) and the HSI scenario with few labeled samples, the issues of small-sample sets can be well characterized by fusion of deep learning methods and related techniques, such as transfer learning and a lightweight model. For reproducibility, the source codes of the methods assessed in the paper can be found at https://github.com/ShuGuoJ/HSI-Classification.git.

Road landmark inventory is becoming an important data product for the maintenance of transport infrastructures. Several commercial sensors are available which include synchronized optical cameras that allowto build 360° panoramic images of the surroundings of the vehicle used for road inspection. This paper is devoted to the analysis of such panorama images,specifically the area that contains themost relevant information. Road lane landmark detection is posed as a two class classification problem that may be solved bymachine learningapproaches, such as Random Forest (RF) and ensembles of Extreme Learning Machines (V-ELM). Besides model parameter selection, a central problem is the construction of a labeled training and validation datasetto cope with the highly uncontrolled conditions of image capture. Besides, human labor cost makes image data labeling a very expensive process. This paper proposes an open ended Active Learning (AL) approach involving a human oraclein the loop who provides the data labeling and can trigger the AL process when detection quality is degraded by the change in imaging conditions. The paper reports encouraging results over a collection of sample images selected from an industrial road landmark inventory operation. As an additional contribution, this paper assesses the ability of AL to overcomesome of the issues raised by highly class imbalanced datasets.

Depth estimation is a classic task in computer vision, which is of great significance for many applications such as augmented reality, target tracking and autonomous driving. Traditional monocular depth estimation methods are based on depth cues for depth prediction with strict requirements, e.g. shape-from-focus/ defocus methods require low depth of field on the scenes and images. Recently, a large body of deep learning methods have been proposed and has shown great promise in handling the traditional ill-posed problem. This paper aims to review the state-of-the-art development in deep learning-based monocular depth estimation. We give an overview of published papers between 2014 and 2020 in terms of training manners and task types. We firstly summarize the deep learning models for monocular depth estimation. Secondly, we categorize various deep learning-based methods in monocular depth estimation. Thirdly, we introduce the publicly available dataset and the evaluation metrics. And we also analysis the properties of these methods and compare their performance. Finally, we highlight the challenges in order to inform the future research directions.

While meta learning approaches have achieved remarkable success, obtaining a stable and unbiased meta-learner remains a significant challenge, since the initial model of a meta-learner could be too biased towards existing tasks to adapt to new tasks. In order to avoid a biased meta-learner and improve its generalizability, this paper proposes a generic meta learning method that aims to learn an unbiased meta-learner towards a variety of tasks before its initial model is adapted to unseen tasks. Specifically, this paper presents a meta weight learning method for minimizing the inequality of performance across different training tasks. An end-to-end training approach is introduced for the proposed algorithm that allows for effectively learning weight and initializing the network model. Alternatively, a variety of measurement methods of weight is also designed to test the effectiveness of different weight learning methods on the improvement of model-agnostic meta-learning algorithm. The simulation results show that the proposed meta weight learning method not only outperforms state-of-the-art meta learning algorithms, but also is superior to other manually designed measurement methods of weight on discrete and continuous control problems.

The natural concept ontology structure of clothes has enabled easy management of large quantities of fashion images for online retailers and it is meaningful to study how to automatically recognize fashion images for both commercial promotion and academic research. In this paper, a new hierarchical approach is developed for large-scale fashion recognition. We first embed concept ontology into deep convolutional neural network (CNN) by adopting multiple deep CNN branches to learn node-specific features and classifiers explicitly. Then, we introduce a hierarchical knowledge distillation method to further improve the performance of fashion recognition. Finally, we employ the proposed approach for fashion recommendation. To deal with hierarchical deep learning constrains, we leverage back propagation to simultaneously refine the shared deep CNNs and the diverse CNN branches for relevant node features and classifiers by using our joint objective function. The main advantages of this paper lie in (1) providing an effective way for recognizing rich semantic explanations of fashion images without training large or multiple networks, and (2) saving the storage&time costs by learning personalized features and classifiers for each tree node. The experimental results on both our organized fashion dataset and the public DeepFashion dataset have verified the effectiveness and efficiency of the proposed approach on both hierarchical fashion recognition and within category fine-grained fashion recommendation.

Analyzing the runtime behaviors of Android apps is crucial for malware detection. In this paper, we attempt to learn the behavior level features of an app from function calls. The challenges of this task are twofold. First, the absence of function attributes hinders the understanding of app behaviors. Second, the graphical representation of function calls cannot be directly processed by classical machine learning algorithms. In this paper, we develop two methods to overcome these challenges. Based on function embedding, we first propose the concept of enhanced function call graphs (E-FCGs) to characterize app runtime behaviors. We then develop a Graph Convolutional Network (GCN) based algorithm to obtain vector representations of E-FCGs. Extensive experiments show that the features learned by our method can achieve surprisingly high detection performance on a variety of classifiers (e.g., LR, DT, SVM, KNN, RF, MLP and CNN), significantly outperforming the traditional static features.

In this paper, we propose a method for obtaining sentence-level embeddings. While the problem of securing word-level embeddings is very well studied, we propose a novel method for obtaining sentence-level embeddings. This is obtained by a simple method in the context of solving the paraphrase generation task. If we use a sequential encoder-decoder model for generating paraphrase, we would like the generated paraphrase to be semantically close to the original sentence. One way to ensure this is by adding constraints for true paraphrase embeddings to be close and unrelated paraphrase candidate sentence embeddings to be far. This is ensured by using a sequential pair-wise discriminator that shares weights with the encoder that is trained with a suitable loss function. Our loss function penalizes paraphrase sentence embedding distances from being too large. This loss is used in combination with a sequential encoder-decoder network. We also validated our method by evaluating the obtained embeddings for a sentiment analysis task. The proposed method results in semantic embeddings and outperforms the state-of-the-art on the paraphrase generation and sentiment analysis task on standard datasets. These results are also shown to be statistically significant., Comment: This work is an extension of our COLING-2018 paper arXiv:1806.00807

In the past few decades, numerous studies have demonstrated the effectiveness of diet modification in preventing and controlling different chronic diseases, bringing increased attention to the creation of nutritional recipes. Currently, the development of recipes is primarily based on personal experience rather than nutritional specifications. To this end, this paper first uses kernel canonical correlation analysis to demonstrate the relationship between nutrients in food and recipes. Based on this relationship, a new recipe expression method is put forward, which objectively reflects differing importance of an ingredient in differing recipes. Then, recipes are composed using an auto-encoder in deep learning, and a fusion model of two auto-encoders is proposed for better concocting recipes. This paper uses two machine learning methods, namely, non-negative matrix factorization and two-step regularized least squares, to form recipes. To tackle overfitting and instability in non-negative matrix factorization during the training of recipe model, we introduce the Frobenius norm to redefine the objective function and add non-smooth sparse matrices. Similar food has similar taste, but their nutrients might differ. This paper also considers nutrients as a kernel matrix of the two-step regularized least squares, which can effectively avoid the occurrence of different food combinations with similar taste. Experimental results show that developing recipes based on nutrients in food is feasible and effective in the context of machine learning.

In order to address such negative issues as GAN’s mediocre image quality, high-demand for training samples and computation resources, this paper proposes the collaborative Generative Adversarial Network with Visual perception and memory reasoning (ESA-CGAN). This not only makes use of the vision self-attention mechanism and objects salience model to analyze the global information and detailed features of objects, but also designs cross-correlation self-attention module so as to make a balance between the computation efficiency and computational on the one hand and the statistical efficiency and the ability to simulate remote dependencies on the other hand. Based on convolutional long-term and short-term memory network, this paper optimizes the attention feature map so as to highlight the features of objects themselves and improve their generative abilities. Meanwhile, a cooperative learning mechanism between generators is designed, which combines self-constructed generation model and pre-training generation model to form a generation model group. It not only improves effectively the generation ability of the model, improve the computing efficiency, but also restrains the collapse of the model from another angle. In fact, the model proposed here has completed numerical experiments on multiple common standard datasets and self-configuring datasets, and has made comparisons with several mainstream generation antagonistic network models in terms of the performance of image data augmentation. The experimental results demonstrate that the model has excellent simulation ability to enable itself to effectively realize the augmentation of data, thus making it highly applicable in the future.

Deep learning technologies have been rapidly developing recently. They have shown excellent performances in many fields. However, deep learning networks have weak adaptability to small sample sizes. Usually, deep learning networks require tens of thousands of samples to avoid overfitting. In contrast, the kernel method can deal with small sample sizes. In this paper, we use only dozens of samples to train a successful classifier. Deep multiple kernel learning (DMKL) has emerged in recent years. It combines the ideas of deep learning and multiple kernel learning. Unfortunately, the DMKL network with a fixed structure cannot adapt to data of different dimensions and sizes. Therefore, in this paper, we propose a novel depth-width-scaling multiple kernel learning (DWS-MKL) algorithm. It has the ability to adjust the architecture according to the input data. We optimize the estimation of leave-one-out error by using the span bound instead of the dual objective function. Finally, we choose a large number of data sets from the UCI benchmark data sets for classification tasks. We also evaluate DWS-MKL on the MNIST data set. The experimental results show that different frameworks have different performances on the same dataset. Our DWS-MKL algorithm obtains better classification results than the state-of-the-art kernel learning algorithms, as determined by a thorough comparison. The encouraging results demonstrate that our method has the potential to improve generalization performance of the model.

Person reidentification (ReID) refers to the task of verifying the identity of a pedestrian observed from nonoverlapping views in a surveillance camera network. It has recently been validated that reranking can achieve remarkable performance improvements in person ReID systems. However, current reranking approaches either require feedback from users or suffer from burdensome computational costs. In this paper, we propose to exploit a density-adaptive smooth kernel technique to achieve efficient and effective reranking. Specifically, we adopt a smooth kernel function to formulate the neighbor relationships among data samples with a density-adaptive parameter. Based on this new formulation, we present two simple yet effective reranking methods, termed \emph{inverse} density-adaptive kernel based reranking (inv-DAKR) and \emph{bidirectional} density-adaptive kernel based reranking (bi-DAKR), in which the local density information in the vicinity of each gallery sample is elegantly exploited. Moreover, we extend the proposed inv-DAKR and bi-DAKR methods to incorporate the available extra probe samples and demonstrate that when and why these extra probe samples are able to improve the local neighborhood and thus further refine the ranking results. Extensive experiments are conducted on six benchmark datasets, including: PRID450s, VIPeR, CUHK03, GRID, Market-1501 and Mars. The experimental results demonstrate that our proposals are effective and efficient., 39 pages, 18 figures and 12 tables. This paper is an extended version of our preliminary work on ICPR 2018

Manual annotation of video segmentation datasets requires an immense amount of human effort, thus, reduction of human annotation costs is an active topic of research. While many papers deal with the propagation of masks through frames of a video, only a few results attempt to optimize annotation task selection. In this paper we present a deep learning based solution to the latter problem and train it using Reinforcement Learning. Our approach utilizes a modified version of the Dueling Deep Q-Network sharing weight parameters across the temporal axis of the video. This technique enables the trained agent to select annotation tasks from the whole video. We evaluate our annotation task selection method by means of a hierarchical supervoxel segmentation based mask propagation algorithm.

CAPTCHA, Completely Automated Public Turing test to tell Computers and Humans Apart, is widely used as a security mechanism to classify human and computer. This security mechanism is based on the Turing Test, which has been conceived to ensure network security. Usability is another fundamental issue, which can avoid human users proceeding tedious and time-consuming operation. CAPTCHA design should consider security and usability simultaneously. This paper provides a review on the development of CAPTCHA technologies for human and computer classification, along with their applications and instantiations. Different from previous CAPTCHA survey, this review discusses the CAPTCHA mechanism from usability and security aspects, therefore attacking (anti-classification) and defending (classification) technologies towards current CAPTCHA are both reviewed. Besides, recent emerging CAPTCHA and the attacking techniques are also introduced in this paper, such as game CAPTCHA, deep learning-based attacking, and etc.

This paper introduces a new procedure to train Minimal Learning Machines (MLM) for regression tasks. Besides that, we propose a new prediction process in MLM. A well-known drawback concerning the (original) MLM model formulation is the lack of sparseness.The most recent efforts on this problem strongly rely on the selection of reference points before training and prediction steps in MLM, all based on some supposition regarding the data. In the opposite direction, here, we explore another formulation of MLM in which we do not rely on any assumption regarding the data for prior selection. Instead, our proposal, named Lightweight Minimal Learning Machine (LW-MLM), builds a regularized system that imposes sparseness. We thrive in such a sparse criterion, not by selection but instead using a piece of weighted information into the model. We validate the contributions of this paper through four types of experiments to evaluate different aspects of our proposal: the prediction error performance, the goodness-of-fit of estimated vs. measured values, the norm values which are related to the sparsity, and finally, the prediction error in high dimensional settings. Based on the results, we show that LW-MLM is a valid alternative since achieved similar or higher accuracy rates against other variants being all seen as statistically equivalent.

Intelligent transportation is an indispensable part of the smart city and the primary development direction of the future transportation systems. Vehicle detection and recognition, which is one of the most important aspects of intelligent transportation, plays a very important role in various areas of our daily life; one such important area is criminal investigation. In the fine-grained vehicle type detection and recognition, several difficult issues such as problems in data acquisition and tagging, dramatic variance in the data of different vehicle types, and challenges in identifying vehicles of the same brand with highly similar appearances remain unsolved. For the problems of data acquisition and tagging, this paper presents a strategy for automatic data acquisition and tagging based on object detection that can label the vehicle images efficiently while rapidly acquiring all types of fine-grained models. Considering the problem of data imbalance in the training process, this paper proposes a Faster-RCNN based data equalizing strategy (Faster-BRCNN), thereby improving the performance of object detection. In view of the severe information attenuation caused by the feature information transfer obstruct between layers in the traditional deep learning network, the lack of mutual dependency of these features, and the inability of the network to focus on the important region and characteristics, we propose an intensive dense attention network (DA-Net). Through its intensive connection and attention unit, we enhance the model's detection ability. The proposed method achieves mAP of 94.5% and 95.8% in the Stanford Cars and FZU Cars datasets, respectively, thereby verifying its effectiveness.

This paper proposes a Neurocomputing application that reorders the Web results obtained from different Web Search Engines emulating the way our brain takes decisions. The proposed application is based on the Random Neural Network with Deep Learning Clusters that evaluates and adapts Web result relevance by associating independently each Deep Learning Cluster to a specific Web Search Engine. In addition, this paper presents a Deep Learning Cluster to perform as a Management Cluster that decides the final result relevance based on the inputs from each independent Deep Learning cluster. The performance of the proposed Management Cluster is evaluated when included as an additional layer to the Deep Learning Clusters. On average; the proposed Deep Learning cluster structure improves Smart Search performance.

Facial expressions play an important role during communications, allowing information regarding the emotional state of an individual to be conveyed and inferred. Research suggests that automatic facial expression recognition is a promising avenue of enquiry in mental healthcare, as facial expressions can also reflect an individual's mental state. In order to develop user-friendly, low-cost and effective facial expression analysis systems for mental health care, this paper presents a novel deep convolution network based emotion analysis framework to support mental state detection and diagnosis. The proposed system is able to process facial images and interpret the temporal evolution of emotions through a new solution in which deep features are extracted from the Fully Connected Layer 6 of the AlexNet, with a standard Linear Discriminant Analysis Classifier exploited to obtain the final classification outcome. It is tested against 5 benchmarking databases, including JAFFE, KDEF,CK+, and databases with the images obtained ‘in the wild’ such as FER2013 and AffectNet. Compared with the other state-of-the-art methods, we observe that our method has overall higher accuracy of facial expression recognition. Additionally, when compared to the state-of-the-art deep learning algorithms such as Vgg16, GoogleNet, ResNet and AlexNet, the proposed method demonstrated better efficiency and has less device requirements. The experiments presented in this paper demonstrate that the proposed method outperforms the other methods in terms of accuracy and efficiency which suggests it could act as a smart, low-cost, user-friendly cognitive aid to detect, monitor, and diagnose the mental health of a patient through automatic facial expression analysis.

This paper investigates network representation learning which involves network structures and labels. Most methods proposed so far try to utilize different kinds of network related data available in just one perfect model to learn the set of perfect embeddings, and then evaluate its performance comparing with other methods for downstream applications, such as node classification or link prediction. In this paper, we introduce ensemble learning to the study of network representation learning. In conventional scenario, ensemble methods train multiple individual learners and combine all their results about the same input to produce an improved one. Inspired by this, we try to train multiple individual models using network related data, and aggregate all the learned embeddings as the final network representations, to expect a performance boost for downstream applications. In order to learn good and diverse individual models, bootstrap sampling is used and different individual model structures are designed. Experimental results show that the idea of aggregating network embeddings really works, and can outperform existing excellent methods under specific experimental setups.

Aspect-Based Sentiment Analysis (ABSA) is a fine-grained sentiment analysis task, which aims to predict sentiment polarities of given aspects or target terms in text. ABSA contains two subtasks: Aspect-Category Sentiment Analysis (ACSA) and Aspect-Term Sentiment Analysis (ATSA). Aspect embeddings have been extensively used for representing aspect-categories on ACSA task. Based on our observations, existing aspect embeddings cannot properly represent the relation between aspect-categories and aspect-terms. To address this limitation, this paper presents a learning method which trains aspect embeddings according to the relation between aspect-categories and aspect-terms. According to the cosine measure metric we proposed in this paper, the limitation is successfully alleviated in the aspect embeddings which are trained by our method. The trained aspect embeddings can be used as initialization in existing models to solve ACSA task. We conduct experiments on SemEval datasets for ACSA task, and the results indicate that our pre-trained aspect embeddings are capable of improving the performance of sentiment analysis.

The Convolutional Neural Networks (CNNs), in domains like computer vision, mostly reduced the need for handcrafted features due to its ability to learn the problem-specific features from the raw input data. However, the selection of dataset-specific CNN architecture, which mostly performed by either experience or expertise is a time-consuming and error-prone process. To automate the process of learning a CNN architecture, this paper attempts at finding the relationship between Fully Connected (FC) layers with some of the characteristics of the datasets. The CNN architectures, and recently datasets also, are categorized as deep, shallow, wide, etc. This paper tries to formalize these terms along with answering the following questions. (i) What is the impact of deeper/shallow architectures on the performance of the CNN w.r.t. FC layers?, (ii) How the deeper/wider datasets influence the performance of CNN w.r.t. FC layers?, and (iii) Which kind of architecture (deeper/ shallower) is better suitable for which kind of (deeper/ wider) datasets. To address these findings, we have performed experiments with three CNN architectures having different depths. The experiments are conducted by varying the number of FC layers. We used four widely used datasets including CIFAR-10, CIFAR-100, Tiny ImageNet, and CRCHistoPhenotypes to justify our findings in the context of the image classification problem. The source code of this research is available at https://github.com/shabbeersh/Impact-of-FC-layers., This paper is accepted for publication in Neurocomputing Journal

Large amounts of training data in machine learning can keep the accuracy high to a certain extent, but the time costs are high because of the exorbitant amount of data and their dimensionality. Therefore, how to simultaneously select the most useful training data set and extract the main features of the samples, especially for image data, are essential problems that urgently need to be solved in the field of large-scale machine learning. Herein, a training sample selection method that is based on the fuzzy c-means clustering algorithm (FCM) is proposed for the problems. It first utilises condensed nearest neighbour (CNN) to make a preliminary selection of training samples. Then, it utilises the FCM to get the centres of the selected data, and, finally, it effectively condenses the sample using a compression parameter. Meanwhile, considering the critical influence of the sample features on the classification model, this paper selects the hierarchical extreme learning machine (H-ELM) model to better solve the classification task. Based on this, the paper presents the FCM-CNN-H-ELM framework for data classification, which combines FCM-Based CNN and H-ELM. The results of the experiments show that the proposed training sample selection method and classification framework can guarantee consistent, even higher, prediction results with a small number of training samples, and significantly reduce the training time.

Virtualization in the data center network is used to overcome the resistance of Internet ossification. Node mapping in virtual network embedding (VNE) process is more important than link mapping because each node mapping has a lot of choices with high probability. Historical data always have valuable information which displays how to be mapped virtual networks with special features. Previous works use physical network historical information, and they do not pay attention to the characteristics of virtual networks during the embedding process. To address this problem, this paper proposes SoGVNE (Self-organized Green Virtual Network Embedding) which uses previous successful virtual node mapping information to train a self-organizing map (SOM) neural network. Then, it will respond to future requests quickly and with fewer resources. SoGVNE uses virtual networks historical data and is able to embed more than one virtual network concurrently. Using supervised SOM makes the SoGVNE aware of resources with renewable energy and uses them to manage energy consumption. This paper evaluates SoGVNE with reinforcement-learning, Monte-Carlo and Presto methods. The results indicate the high efficiency of the proposed method in learning the node mapping process and achieving high profit and acceptance ratio with low cost.

In online-learning, the data is incrementally received and the distributions from which it is drawn may keep changing over time. This phenomenon is widely known as concept drift. Such changes may affect the generalization of a learned model to future data. This problem may be exacerbated by the form of the drift itself changing over time. Quantitative measures to describe and analyze the concept drift have been proposed in previous work. A description composed from these measures is called a concept drift map. We believe that these maps could be useful for guiding how much knowledge in the old model should be forgotten. Therefore, this paper presents an adaptive online learning model that uses a concept drift map to regulate the forgetting factor of an extreme learning machine. Specifically, when a batch of new instances are labeled, the distribution of each class on each attribute is firstly estimated, and then it is compared with the distribution estimated in the previous batch to calculate the magnitude of concept drift, which is further used to regulate the forgetting factor and to update the learning model. Therefore, the novelty of this paper lies in that a quantitative distance metric between two distributions constructed on continuous attribute space is presented to construct concept drift map which can be further associated with the forgetting factor to make the learning model adapt the concept drift. Experimental results on several benchmark stream data sets show the proposed model is generally superior to several previous algorithms when classifying a variety of data streams subject to drift, indicating its effectiveness and feasibility.

The Multi-Layered Echo-State Network (ML-ESN) is a recently developed, highly powerful type of recurrent neural network. It has succeeded in dealing with several non-linear benchmark problems. On account of its rich dynamics, ML-ESN is exploited in this paper, for the first time, as a recurrent Autoencoder (ML-ESNAE) to extract new features from original data representations. Further, the challenging and crucial task of optimally determining the ML-ESNAE architecture and training parameters is addressed, in order to extract more efficient features from the data. Traditionally, in a ML-ESN, the number of parameters (hidden neurons, sparsity rates, weights) are randomly chosen and manually altered to achieve a minimum learning error. On one hand, this random setting may not guarantee best generalization results. On the other, it can increase the network’s complexity. In this paper, a novel bi-level evolutionary optimization approach is thus proposed for the ML-ESNAE, to deal with these challenges. The first level offers Pareto multi-objective architecture optimization, providing maximum learning accuracy while maintaining a reduced complexity target. Next, every Pareto optimal solution obtained from the first level undergoes a mono-objective weights optimization at the second level. Particle Swarm Optimization (PSO) is used as an evolutionary tool for both levels 1 and 2. An empirical study shows that the evolved ML-ESNAE produces a noticeable improvement in extracting new, more expressive data features from original ones. A number of application case studies, using a range of benchmark datasets, show that the extracted features produce excellent results in terms of classification accuracy. The effectiveness of the evolved ML-ESNAE is demonstrated for both noisy and noise-free data. In conclusion, the evolutionary ML-ESNAE is proposed as a new benchmark for the evolutionary AI and machine learning research community.

Currently, knowledge discovery in databases is an essential first step when identifying valid, novel and useful patterns for decision making. There are many real-world scenarios, such as bankruptcy prediction, option pricing or medical diagnosis, where the classification models to be learned need to fulfill restrictions of monotonicity (i.e. the target class label should not decrease when input attributes values increase). For instance, it is rational to assume that a higher debt ratio of a company should never result in a lower level of bankruptcy risk. Consequently, there is a growing interest from the data mining research community concerning monotonic predictive models. This paper aims to present an overview of the literature in the field, analyzing existing techniques and proposing a taxonomy of the algorithms based on the type of model generated. For each method, we review the quality metrics considered in the evaluation and the different data sets and monotonic problems used in the analysis. In this way, this paper serves as an overview of monotonic classification research in specialized literature and can be used as a functional guide for the field.

Twin Support Vector Machines (TWSVMs) have emerged as an efficient alternative to Support Vector Machines (SVM) for learning from imbalanced datasets. The TWSVM learns two non-parallel classifying hyperplanes by solving a couple of smaller sized problems. However, it is unsuitable for large datasets, as it involves matrix operations. In this paper, we discuss a Twin Neural Network (Twin NN) architecture for learning from large unbalanced datasets. The objective functions of the networks in the Twin NN are designed to realize the idea of the Twin SVM with non-parallel decision boudaries for the respective classes, while also being able to reduce model complexity. The Twin NN optimizes the feature map, allowing for better discrimination between classes. The paper also discusses an extension of the Twin NN for multiclass datasets. This architecture trains as many neural networks as the number of classes, and has the additional advantage that it does not have any hyper-parameter which requires tuning. Results presented in the paper demonstrate that the Twin NN generalizes well and scales well on large unbalanced datasets.

Feature selection is of great importance for two possible scenarios: (1) prediction, i.e., improving (or minimally degrading) the predictions of a target variable while discarding redundant or uninformative features and (2) discovery, i.e., identifying features that are truly dependent on the target and may be genuine causes to be determined in experimental verifications (for example for the task of drug target discovery in genomics). In both cases, if variables have a large number of missing values, imputing them may lead to false positives; features that are not associated with the target become dependent as a result of imputation. In the first scenario, this may not harm prediction, but in the second one, it will erroneously select irrelevant features. In this paper, we study the risk/benefit trade-off of missing value imputation in the context of feature selection, using causal graphs to characterize when structural bias arises. Our aim is also to investigate situations in which imputing missing values may be beneficial to reduce false negatives, a situation that might arise when there is a dependency between feature and target, but the dependency is below the significance level when only complete cases are considered. However, the benefits of reducing false negatives must be balanced against the increased number of false positives. In the case of binary target variable and continuous features, the t-test is often used for univariate feature selection. In this paper, we also introduce a de-biased version of the t-test allowing us to reap the benefits of imputation, while not incurring the penalty of increasing the number of false positives.

In recent years, person re-identification (ReID) has received much attention since it is a fundamental task in intelligent surveillance systems and has widespread application prospects in numerous fields. Given an image of a pedestrian captured from one camera, the task is to identify this pedestrian from the gallery set captured by other multiple cameras. It is a challenging issue since the appearance of a pedestrian may suffer great changes across different cameras. The task has been greatly boosted by deep learning technology. There are mainly six types of deep learning-based methods designed for this issue, i.e. identification deep model, verification deep model, distance metric-based deep model, part-based deep model, video-based deep model and data augmentation-based deep model. In this paper, we first give a comprehensive review of current six types of deep learning methods. Second, we present the detailed descriptions of existing person ReID datasets. Then, some state-of-the-art performances of methods over recent years on several representative ReID datasets are summarized. Finally, we conclude this paper and discuss the future directions of the person ReID.

In this paper, we present a multi-view transfer learning model named Multi-view Transfer Discriminative Model (MTDM) for both image and text classification tasks. Transfer learning, which aims to learn a robust classifier for the target domain using data from a different distribution, has been proved to be effective in many real-world applications. However, most of the existing transfer learning methods map across domain data into a high-dimension space which the distance between domains is closed. This strategy always fails in the multi-view scenario. On the contrary, the multi-view learning methods are also difficult to extend in the transfer learning settings. One of our goals in this paper is to develop a model which can perform better in both multi-view and transfer learning settings. On the one hand, the problem of multi-view is implemented by the paradigm of learning using privileged information (LUPI), which could guarantee the principle of complementary and consensus. On the other hand, the model adequately utilizes the source domain data to build a robust classifier for the target domain. We evaluate our model on both image and text classification tasks and show the effectiveness compared with other baseline approaches.

Traditional recommendation methods based on matrix factorization techniques have yielded immense success because of their good scalability. However, they still face the problem of data sparsity, which may lead to a reduction in recommendation performance. As it is hard to learn good latent features in the sparse user-item rating matrix. In recent years, deep learning is very appealing in learning effective representations. Its non-linear characteristics just remedy the shortcomings of matrix factorization. In this paper, a novel method deep variational matrix factorization recommendation (DVMF) is proposed for large scale sparse dataset. DVMF is based on latent factors to predict the ratings. The latent features of the users and items are respectively obtained through a deep nonlinear structure. Based on the latent factors and combined with matrix factorization method, the paper presents algorithm optimization method of DVMF. The experiments on three real-world datasets from different domains show that DVMF is able to provide higher accuracy than recommendation algorithms based on matrix factorization or deep learning individually on large scale sparse dataset.

In the era of big data, all types of data with increasing samples and high-dimensional attributes are demonstrating their important roles in various fields, such as data mining, pattern recognition and machine learning, etc. Meanwhile, machine learning algorithms are being effectively applied in large-scale data processing. This paper mainly reviews the classical dimensionality reduction and sample selection methods based on machine learning algorithms for large-scale data processing. Firstly, the paper provides a brief overview to the classical sample selection and dimensionality reduction methods. Then, it pays attention to the applications of those methods and their combinations with the classical machine learning methods, such as clustering, random forest, fuzzy set, and heuristic algorithms, particularly deep leaning methods. Furthermore, the paper primarily introduces the application frameworks that combine sample selection and dimensionality reduction in the context of two aspects: sequential and simultaneous, which almost all get the ideal results in the processing of the large-scale training data contrasting to the original models. Lastly, we further conclude that sample selection and dimensionality reduction methods are essential and effective for the modern large-scale data processing. In the future work, the machine learning algorithms, especially the deep learning methods, will play a more important role in the processing of large-scale data.

Unpredictable failures and unscheduled maintenance of physical systems increases production resources, produces more harmful waste for the environment, and increases system life cycle costs. Efficient remaining useful life (RUL) estimation can alleviate such an issue. The RUL is predicted by making use of the data collected from several types of sensors that continuously record different indicators about a working asset, such as vibration intensity or exerted pressure. This type of continuous monitoring data is sequential in time, as it is collected at a certain rate from the sensors during the asset's work. Long Short-Term Memory (LSTM) neural network models have been demonstrated to be efficient throughout the literature when dealing with sequential data because of their ability to retain a lot of information over time about previous states of the system. This paper proposes using a new LSTM architecture for predicting the RUL when given short sequences of monitored observations with random initial wear. By using LSTM, this paper proposes a new objective function that is suitable for the RUL estimation problem, as well as a new target generation approach for training LSTM networks, which requires making lesser assumptions about the actual degradation of the system.

In recent years, convolutional neural networks (CNNs) have shown great performance in various fields such as image classification, pattern recognition, and multi-media compression. Two of the feature properties, local connectivity and weight sharing, can reduce the number of parameters and increase processing speed during training and inference. However, as the dimension of data becomes higher and the CNN architecture becomes more complicated, the end-to-end approach or the combined manner of CNN is computationally intensive, which becomes limitation to CNN’s further implementation. Therefore, it is necessary and urgent to implement CNN in a faster way. In this paper, we first summarize the acceleration methods that contribute to but not limited to CNN by reviewing a broad variety of research papers. We propose a taxonomy in terms of three levels, i.e. structure level, algorithm level, and implementation level, for acceleration methods. We also analyze the acceleration methods in terms of CNN architecture compression, algorithm optimization, and hardware-based improvement. At last, we give a discussion on different perspectives of these acceleration and optimization methods within each level. The discussion shows that the methods in each level still have large exploration space. By incorporating such a wide range of disciplines, we expect to provide a comprehensive reference for researchers who are interested in CNN acceleration.

Twin extreme learning machine (TELM) is an efficient and effective method for pattern classification, based on widely known extreme learning machine (ELM). However, TELM is mainly used to deal with supervised learning problems. In this paper, we extend TELM to handle semi-supervised learning problems and propose a novel Laplacian twin extreme learning machine (LapTELM), which simultaneously trains two related and paired semi-supervised ELMs with two nonparallel separating planes for the final classification. The proposed method exploits the geometry structure property of the unlabeled samples and incorporates it as a manifold regularization term. This allows LapTELM to reap the benefits of fully exploring the plentiful unlabeled samples while retaining the learning ability and efficiency of TELM. Moreover, the paper shows that semi-supervised and supervised TELM can form an unified learning framework. Compared with several mainstream semi-supervised learning methods, the experimental results on the synthetic and several real-world data sets verify the effectiveness and efficiency of LapTELM.

Short term electric load forecasting, as an important tool in the electricity market, plays a critical role in the management of electric systems. Proposing an accuracy and optimization method is not only a challenging task but also an indispensable part of the energy system. More and more accurate forecasting methods are needed by different people in different areas. This paper proposes a novel short-term electric load forecasting method EMD-Mixed-ELM which based on empirical mode decomposition (EMD) and extreme learning machine (ELM). EMD-Mixed-ELM first uses the empirical mode decomposition to decompose the load series for capturing the complicated features of the electric load and de-noising the data. Considering that the performance of extreme learning machine (ELM) is greatly influenced by the choice of kernel, the mixed kernel method is proposed for ELM. The mixed kernel combines the RBF kernel and the UKF kernel. The forecasting results of the EMD-Mixed-ELM are proved to be better than all the other three methods (RBF-ELM, UKF-ELM and Mixed-ELM) and other existing methods (MFES, ESPLSSVM and Combined method). To verify the forecasting ability of the EMD-Mixed-ELM, half-hourly electric load data from the state of New South Wales, Victoria and Queensland in Australia are used in this paper as a case study. The experimental results clearly indicate that for this three datasets, the forecasting accuracy of the proposed method is superior to other methods.

This paper focuses on the task of targeted sentiment classification over dynamic memory networks. To remedy the flaw of the previous related work, only doing well in simple sentences, the new dynamic memory networks, in which the input module, question module and memory module are optimally designed, are established to model the task into question answering system for the first time. Specifically, in the input module, position embedding is considered to eliminate the influence of sentiment-irrelevant words that close to the target, and skip connection can help to build rich representations over the sentences. In the question module, for dealing with many cases where the target is composed of multiple words, the commonly used average target vector is replaced by the designed target sentiment question, which is encoded by a GRU. In the memory module, two improvements are performed. On the one hand, weight bias, caused by using original soft attention to extract memory information at each attention step, is avoided by both exploring attention and inner attention based GRU networks. On the other hand, memory updating block is capable to distill sentiment-relevant features from memory information, which is extracted by multiple attention block using multiple attention mechanism. Finally, experimental results demonstrate the effectiveness of the proposed models for the task of targeted sentiment classification among three different datasets. Additionally, the best model of this paper achieves state-of-the-art performance on different types of data.

A small dataset often makes it difficult to build a reliable learning model, and thus some researchers have proposed virtual sample generation (VSG) methods to add artificial samples into small datasets to extend the data size. However, for some datasets the assumption of the distribution of data in the VSG methods may be vague, and when data only has a few attributes, such approaches may not work effectively. Other researchers thus proposed attribute extension methods to generate attributes to convert data into a higher dimensional space. Unfortunately, the resulting dataset may become a sparse dataset with many null or zero values in extended attributes, and then a large quantity of such attributes will reduce the representativeness of instances for the learning model. Therefore, based on fuzzy theories, this paper proposes a novel sample attribute extending (SEA) method to extend a suitable quantity of attributes to improve small dataset learning. In order to verify the validity of the SEA method, using SVR and BPNN, this paper adopts two real cases and two public datasets to conduct the learning of the predictive model, and uses the paired t-test to statistically examine the significance of improvement. The experimental results show that the proposed SEA method can effectively improve the learning accuracy of small datasets.

Neural Networks and Support Vector Machines (SVMs) are two of the most popular and efficient supervised classification models. However, in the context of large datasets many complexity issues arise due to high memory requirements and high computational cost. In the context of the application of Data Mining algorithms, data reduction techniques attempt to reduce the size of training datasets in terms of the number of instances by selecting some of the existing instances or by generating new training instances. The idea is to speed up the application of the data mining algorithm with minimum or no sacrifice in performance. Data reduction techniques have been extensively used in the context of k-Nearest Neighbor classification, a lazy classifier that works by directly using a training dataset rather than building a model. This paper explores the application of data reduction techniques as a preprocessing step before the training step of Neural Networks and SVMs. Furthermore, the paper proposes a new data reduction technique that is based on k-median clustering algorithm. Our experimental results illustrate that, in the case of SVMs, data reduction techniques can effectively reduce the dataset size incurring small performance degradation. In the case of Neural Networks, the performance loss is somewhat greater, for the same data reduction rate, but both SVM and Neural Network models outperform the k-NN approach that is typically used in Data Mining applications.

An important feature characteristic of the data streams in many of today's big data applications is the intrinsic uncertainty, which could happen for both item occurrence and attribute value. While this has already posed great challenges for fundamental data mining tasks such as classification, things are made even more complicated by the fact that completely-labeled examples are usually unavailable in such settings, leaving researchers the only option to learn classifiers on partially-labeled examples on uncertain data streams. Furthermore, there will be concept drift on evolving data streams. To address these challenges, this paper therefore focuses on the study of learning from positive and unlabeled examples (PU) on uncertain data streams. To the best of our knowledge, this paper is the first work to address the uncertainty issue in both item occurrence (occurrence level) and attribute value (attribute level) for the problem of PU learning over streaming data. Firstly, we propose an algorithm to classify positive and unlabeled examples with both uncertainties (PUU). The algorithm extracts reliable positive and negative examples by clustering-based method, and then trains the classifier with Weighted Extreme Learning Machine (Weighted ELM). Secondly, we propose an algorithm of PU learning over uncertain data streams (PUUS). It adopts ensemble model and trains base classification by PUU. In order to detect concept drift, PUUS uses cluster set similarities between the current data block and history data block. We propose different update strategies for different concept drift to adapt to evolving uncertain data streams. Experimental results show that PUUS can effectively classify uncertain data streams with just positive and unlabeled data, while achieving in the meantime good performance in detecting and handling concept drifts.

An instance is often represented from different aspects (views or modalities), which leads to high-dimensional features and even multiple labels. In this paper, we focus on the feature selection problem in multi-label classification, for which a trivial solution is handling the labels dividedly. Obviously, such a scheme may not work well by leaving the label relationship out of consideration. Recently, several research works conduct feature selection directly under a multi-label framework by implicitly or explicitly modeling label relationship. However, these works assume that all labels share the same feature subset or subspace, which is not reasonable enough for some scenarios since different labels tend to convey different semantics. To address this problem, we develop a novel approach in this paper to select label-dependent features for multi-label classification. Specifically, we (1) formulate a convex model based on a more general and practical assumption that different labels convey different semantics with specific features; (2) design an alternating optimization algorithm based on Nesterov's method and L1-ball projection for efficiently finding the optimal solution, which can realize multi-label classification, feature selection, and label relationship estimation simultaneously. Finally, experiments on publicly available datasets show that the proposed algorithm achieves better performance than several related methods.

Personal recommendation technology is becoming a useful and popular solution to solve the problem of information overload with the popularity of big data cloud services. But most recommendation algorithms pay too much attention to the similarity to focus on the social trust between users. So this paper focus on the research of hybrid Recommendation algorithm for big data based on the optimization combining with the similarity and trust in sociology. In this paper, we introduced some user trust models including trust path model and loop trust model, and then we took these models into the calculation of mixed weighting. The experiment results show that the recommendation algorithm considering the trust models has the higher accuracy than the traditional recommendation algorithm, and we have a 2% increase in both MEA (Mean Absolute Error) and RMSE (Root Mean Square Error).

The Internet of Things (IoT) refers to an infrastructure that integrates things over standard wired/wireless networks and allows them to exchange information with each other. The IoT is a very complex heterogeneous network, enabling seamless integration of these things is a huge challenge. A publish/subscribe method of integration can be formulated to solve the problems of interconnecting billions of heterogeneous things. In our work, an IoT framework that uses an abstraction layer that decouples an application from the service calls and network interfaces is required to send and receive messages on a particular thing. This paper provides definitions and classifications for heterogeneous data/events/services according to the properties of the things in order to integrate them into a framework for description. Based on these definitions and classifications, heterogeneous data/events/services in the IoT were integrated via topic description through the Data Distribution Service (DDS) middleware standard for real-time publish/subscribe. This paper also concludes with general remarks and a discussion of future work.

This paper presents a method for determining how close a neural network (and its design) is to the systems it is intended to represent and model. The output of this method is a normalized numerical value (an index) which can be compared to other indices. It can also be used to compare one artificial neural network with another. The objective is to develop a numerical index of quality that would make it possible to compare a variety of methods and computational techniques. A brief revisit to the concept from which this paper originated is presented in section 2 (where authors summarize the Statistical Index of Quality), and the outline of the Neural Network Index of Quality is shown in subsection 3.2. Subsection 3.3 illustrates sample calculations and comparisons with other similar methods for neural network evaluation. Finally, section 4 addresses certain implications and outlines future research in pursuit of automated quality analysis of computational techniques. As proposed, the method is effective to compare different computational techniques adequately, with results close to ‘1’ for well-designed networks and results close to ‘0’ for neural networks with possible design flaws. The proposed method is designed for fixed-size, supervised artifical neural networks.

In this paper, a self-organizing map (SOM) neural network is used to visualize corrective actions of failure modes and effects analysis (FMEA). SOM is a popular unsupervised neural network model that aims to produce a low-dimensional map (typically a two-dimensional map) for visualizing high-dimensional data. With regards to FMEA, it is a popular methodology to identify potential failure modes for a product or a process, to assess the risk associated with those failure modes, also, to identify and carry out corrective actions to address the most serious concerns. Despite the popularity of FMEA in a wide range of industries, two well-known shortcomings are the complexity of the FMEA worksheet and its intricacy of use. To the best of our knowledge, the use of computation techniques for solving the aforementioned shortcomings is limited. The use of SOM in FMEA is new. In this paper, corrective actions in FMEA are described in their severity, occurrence and detect scores. SOM is then used as a visualization aid for FMEA users to see the relationship among corrective actions via a map. Color information from the SOM map is then included to the FMEA worksheet for better visualization. In addition, a Risk Priority Number Interval is used to allow corrective actions to be evaluated and ordered in groups. Such approach provides a quick and easily understandable framework to elucidate important information from a complex FMEA worksheet; therefore facilitating the decision-making tasks by FMEA users. The significance of this study is two-fold, viz., the use of SOM as an effective neural network learning paradigm to facilitate FMEA implementations, and the use of a computational visualization approach to tackle the two well-known shortcomings of FMEA.

The paper investigates application of several methods of feature selection to identification of the most important genes in autism disorder. The study is based on the expression microarray of genes. The applied methods analyze the importance of genes on the basis of different principles of selection. The most important step is to fuse the results of these selections into common set of genes, which are the best associated with autism. These genes may be treated as the biomarkers of this disorder and used in early prediction of autism. The paper proposes and compares three different methods of such fusion: purity of the clusterization space, application of genetic algorithm and random forest in the role of integrator. The numerical experiments are concerned with the identification of the most important biomarkers and their application in autism recognition. They show the applied fusion strategy of many independent selection methods leads to the significant improvement of the autism recognition rate.

The performance of many machine learning algorithms depends crucially on the hyperparameter settings, especially in Deep Learning. Manually tuning the hyperparameters is laborious and time consuming. To address this issue, Bayesian optimization (BO) methods and their extensions have been proposed to optimize the hyperparameters automatically. However, they still suffer from highly computational expense when applying to deep generative models (DGMs) due to their strategy of the black-box function optimization. This paper provides a new hyperparameter optimization procedure at the pre-training phase of the DGMs, where we avoid combining all layers as one black-box function by taking advantage of the layer-by-layer learning strategy. Following this procedure, we are able to optimize multiple hyperparameters in an adaptive way by using Gaussian process. In contrast to the traditional BO methods, which mainly focus on the supervised models, the pre-training procedure is unsupervised where there is no validation error can be used. To alleviate this problem, this paper proposes a new holdout loss, the free energy gap, which takes into account both factors of the model fitting and over-fitting. The empirical evaluations demonstrate that our method not only speeds up the process of hyperparameter optimization, but also improves the performances of DGMs significantly in both the supervised and unsupervised learning tasks.