Search

A self-training algorithm is a classical semi-supervised learning algorithm that uses a small number of labeled samples and a large number of unlabeled samples to train a classifier. However, the existing self-training algorithms consider only the geometric distance between data while ignoring the data distribution when calculating the similarity between samples. In addition, misclassified samples can severely affect the performance of a self-training algorithm. To address the above two problems, this paper proposes a self-training algorithm based on data editing with mass-based dissimilarity (STDEMB). First, the mass matrix with the mass-based dissimilarity is obtained, and then the mass-based local density of each sample is determined based on its k nearest neighbors. Inspired by density peak clustering (DPC), this study designs a prototype tree based on the prototype concept. In addition, an efficient two-stage data editing algorithm is developed to edit misclassified samples and efficiently select high-confidence samples during the self-training process. The proposed STDEMB algorithm is verified by experiments using accuracy and F-score as evaluation metrics. The experimental results on 18 benchmark datasets demonstrate the effectiveness of the proposed STDEMB algorithm. • Mass-based dissimilarity is used to o account for the effects of data distribution. • A prototype tree was designed to efficiently identify high confidence samples. • A two-stage data editing algorithm was developed for editing misclassified samples. • Numerous experiments were conducted to prove the performance of the proposed STDEMB. [ABSTRACT FROM AUTHOR]

Twin support vector machine (TSVM) is a practical machine learning algorithm, whereas traditional TSVM can be limited for data with outliers or noises. To address this problem, we propose a novel TSVM with the symmetric LINEX loss function (SLTSVM) for robust classification. There are several advantages of our method: (1) The performance of the proposed SLTSVM for data with outliers or noise can be improved by using the symmetric LINEX loss function. (2) The introduction of regularization term can effectively improve the generalization ability of our model. (3) An efficient iterative algorithm is developed to solve the optimization problems of our SLTSVM. (4) The convergence and time complexity of the iterative algorithm are analyzed in detail. Furthermore, our model does not involve loss function parameter, which makes our method more competitive. Experimental results on synthetic, benchmark and image datasets with label noises and feature noises demonstrate that our proposed method slightly outperforms other state-of-the-art methods on most datasets. • This paper proposes SLTSVM by introducing symmetric LIENX loss function. • An interpretable method is proposed for selecting the loss parameter. • The convergence of the designed solution algorithm are further analyzed. • Experiments show that our method is efficient for data with outliers or noises. [ABSTRACT FROM AUTHOR]

Large neural networks usually perform well for executing machine learning tasks. However, models that achieve state-of-the-art performance involve arbitrarily large number of parameters and therefore their training is very expensive. It is thus desired to implement methods with small per-iteration costs, fast convergence rates, and reduced tuning. This paper proposes a multivariate adaptive gradient descent method that meets the above attributes. The proposed method updates every element of the model parameters separately in a computationally efficient manner using an adaptive vector-form learning rate, resulting in low per-iteration cost. The adaptive learning rate computes the absolute difference of current and previous model parameters over the difference in subgradients of current and previous state estimates. In the deterministic setting, we show that the cost function value converges at a linear rate for smooth and strongly convex cost functions. Whereas in both the deterministic and stochastic setting, we show that the gradient converges in expectation at the order of O (1 / k) for a non-convex cost function with Lipschitz continuous gradient. In addition, we show that after T iterates, the cost function of the last iterate scales as O (log (T) / T) for non-smooth strongly convex cost functions. Effectiveness of the proposed method is validated on convex functions, smooth non-convex function, non-smooth convex function, and four image classification data sets, whilst showing that its execution requires hardly any tuning unlike existing popular optimizers that entail relatively large tuning efforts. Our empirical results show that our proposed algorithm provides the best overall performance when comparing it to tuned state-of-the-art optimizers. [ABSTRACT FROM AUTHOR]

The concept of randomized neural networks (RNNs), such as the random vector functional link network (RVFL) and extreme learning machine (ELM), is a widely accepted and efficient network method for constructing single-hidden layer feedforward networks (SLFNs). Due to its exceptional approximation capabilities, RNN is being extensively used in various fields. While the RNN concept has shown great promise, its performance can be unpredictable in imperfect conditions, such as weight noises and outliers. Thus, there is a need to develop more reliable and robust RNN algorithms. To address this issue, this paper proposes a new objective function that addresses the combined effect of weight noise and training data outliers for RVFL networks. Based on the half-quadratic optimization method, we then propose a novel algorithm, named noise-aware RNN (NARNN), to optimize the proposed objective function. The convergence of the NARNN is also theoretically validated. We also discuss the way to use the NARNN for ensemble deep RVFL (edRVFL) networks. Finally, we present an extension of the NARNN to concurrently address weight noise, stuck-at-fault, and outliers. The experimental results demonstrate that the proposed algorithm outperforms a number of state-of-the-art robust RNN algorithms. [ABSTRACT FROM AUTHOR]

In this paper, a complementary learning scheme for experience transference of unknown continuous-time linear systems is proposed. The algorithm is inspired in the complementary learning properties that exhibit the hippocampus and neocortex learning systems via the striatum. The hippocampus is modelled as pattern-separated data of a human optimized controller. The neocortex is modelled as a Q-reinforcement learning algorithm which improves the hippocampus control policy. The complementary learning (striatum) is designed as an inverse reinforcement learning algorithm which relates the hippocampus and neocortex learning models to seek and transfer the weights of the hidden expert's utility function. Convergence of the proposed approach is analysed using Lyapunov recursions. Simulations are given to verify the proposed approach. [ABSTRACT FROM AUTHOR]

We use a recently developed synchronous Spiking Neural Network (SNN) model to study the problem of learning hierarchically-structured concepts. We introduce an abstract data model that describes simple hierarchical concepts. We define a feed-forward layered SNN model, with learning modeled using Oja's local learning rule, a well known biologically-plausible rule for adjusting synapse weights. We define what it means for such a network to recognize hierarchical concepts; our notion of recognition is robust, in that it tolerates a bounded amount of noise. Then, we present a learning algorithm by which a layered network may learn to recognize hierarchical concepts according to our robust definition. We analyze correctness and performance rigorously; the amount of time required to learn each concept, after learning all of the sub-concepts, is approximately O 1 η k ℓmax log (k) + 1 ɛ + b log (k) , where k is the number of sub-concepts per concept, ℓmax is the maximum hierarchical depth, η is the learning rate, ɛ describes the amount of uncertainty allowed in robust recognition, and b describes the amount of weight decrease for "irrelevant" edges. An interesting feature of this algorithm is that it allows the network to learn sub-concepts in a highly interleaved manner. This algorithm assumes that the concepts are presented in a noise-free way; we also extend these results to accommodate noise in the learning process. Finally, we give a simple lower bound saying that, in order to recognize concepts with hierarchical depth two with noise-tolerance, a neural network should have at least two layers. The results in this paper represent first steps in the theoretical study of hierarchical concepts using SNNs. The cases studied here are basic, but they suggest many directions for extensions to more elaborate and realistic cases. [ABSTRACT FROM AUTHOR]

Find the global optimal solution of the model is one promising research topic in computational intelligent community. Dependent on analogies to natural processes, the evolutionary swarm intelligent algorithms are widely used for solving global optimization problems which directed by the fitness values. In this paper, we propose one efficient fractional global learning machine (Fragmachine) which includes two stages (descending and ascending) to determine the optimal search path. The neural network model is used to approach the given fitness value. Specifically, for the descending stage, the integer gradient of the network output with respect the current location is employed to find the next descending point, while for the ascending stage, the fractional gradient is implemented to climb and escape from the local optimal point. We further propose one adaptive learning rate during training which relies on both the current gradient (integer or fractional) information and the fitness value. Finally, a series of numerical experiments verify the effectiveness of the proposed algorithm, Fragmachine. [ABSTRACT FROM AUTHOR]

In recent years transfer learning has attracted much attention due to its ability to adapt a well-trained model from one domain to another. Fine-tuning is one of the most widely-used methods which exploit a small set of labeled data in the target domain for adapting the network. Including a few methods using the labeled data in the source domain, most transfer learning methods require labeled datasets, and it restricts the use of transfer learning to new domains. In this paper, we propose a fully unsupervised self-tuning algorithm for learning visual features in different domains. The proposed method updates a pre-trained model by minimizing the triplet loss function using only unlabeled data in the target domain. First, we propose the relevance measure for unlabeled data by the bagged clustering method. Then triplets of the anchor, positive, and negative data points are sampled based on the ranking violations of the relevance scores and the Euclidean distances in the embedded feature space. This fully unsupervised self-tuning algorithm improves the performance of the network significantly. We extensively evaluate the proposed algorithm using various metrics, including classification accuracy, feature analysis, and clustering quality, on five benchmark datasets in different domains. Besides, we demonstrate that applying the self-tuning method on the fine-tuned network help achieve better results. • Fully-unsupervised feature learning algorithm for neural networks. • Relevance score from the bagged clustering in random feature subspaces. • Ranking inconsistency of the relevance score and Euclidean distance for Triplet sampling. • Self-tuning neural networks from the Triplet samples. • Extensive Experiments to evaluate the self-tuning algorithm. [ABSTRACT FROM AUTHOR]

Abstract Multi-label learning is the problem where each instance is associated with multiple labels simultaneously. Binary Relevance (BR) is a representative algorithm for multi-label learning. However, it may suffer the class-imbalance issue especially when the label space is large. Besides, it ignores the label correlations, which is of importance to improve the performance. Moreover, labels might have positive and negative correlations in real applications, but existing methods seldom exploit the negative label correlations. In this paper, we propose a novel Cost-sensitive multi-label learning model with Positive and Negative Label pairwise correlations (CPNL), which extends BR to tackle the above issues. The kernel extension of the linear model is also provided to explore complex input–output relationships. Moreover, we adopt two accelerated gradient methods (AGM) to efficiently solve the linear and kernel models. Experimental results show that our approach CPNL achieves a competitive performance to some state-of-the-art approaches for multi-label learning. Highlights • We propose a novel cost-sensitive multi-label learning model with positive and negative label pairwise correlations. • We introduce a new regularizer to exploit the negative label pairwise correlations. • Kernel extension of the linear model is provided to explore nonlinear input–output relationships. • Two accelerated gradient methods (AGM) are adopted to efficiently solve the linear and kernel model. • Extensive experiments have verified the effectiveness of our proposed method. [ABSTRACT FROM AUTHOR]

It is well known that the support vector machine (SVM) is an effective learning algorithm. The alternating direction method of multipliers (ADMM) algorithm has emerged as a powerful technique for solving distributed optimisation models. This paper proposes a distributed SVM algorithm in a master–slave mode (MS-DSVM), which integrates a distributed SVM and ADMM acting in a master–slave configuration where the master node and slave nodes are connected, meaning the results can be broadcasted. The distributed SVM is regarded as a regularised optimisation problem and modelled as a series of convex optimisation sub-problems that are solved by ADMM. Additionally, the over-relaxation technique is utilised to accelerate the convergence rate of the proposed MS-DSVM. Our theoretical analysis demonstrates that the proposed MS-DSVM has linear convergence, meaning it possesses the fastest convergence rate among existing standard distributed ADMM algorithms. Numerical examples demonstrate that the convergence and accuracy of the proposed MS-DSVM are superior to those of existing methods under the ADMM framework. [ABSTRACT FROM AUTHOR]

For a domain adaptation learning problem, how to minimize the distribution mismatch between different domains is one of key challenges. In real applications, it is reasonable to obtain an optimal latent space for both domains so as to reduce the domain distribution discrepancy as much as possible. We therefore propose in this paper a Robust Latent Regression (RLR) framework with discriminative regularization by uncovering a compact and more informative latent space as well as leveraging the source domain knowledge, which learns a discriminative representation of domain data by considering the recognition task in the procedure of domain adaptation learning. On the one hand, to leverage the prior information in the source domain, RLR incorporates both the source and target classification loss functions as parts of its objective function, and simultaneously trains these two classifiers by encoding the common components of the classifier models as a low-rank regularization term, thus exploiting the discriminative information shared by different domains. On the other hand, to guarantee that the latent space is more compact and discriminative, the intrinsic geometric structure of data, and the local and global structural consistencies over labels are exploited simultaneously and incorporated into RLR. Lastly, to make our algorithm robust to the outliers and noise, we additionally introduce the l 2 , 1 -norm into the loss function. To solve the proposed problem, an effective iterative algorithm is proposed. Extensive experiments are conducted on several visual datasets and the results show that the proposed approach achieves outstanding performance for almost all learning tasks compared with several representative algorithms. [ABSTRACT FROM AUTHOR]

Graph-based algorithms have been successfully applied in machine learning and data mining tasks. A simple but, widely used, approach to build graphs from vector-based data is to consider each data instance as a vertex and connecting pairs of it using a similarity measure. Although this abstraction presents some advantages, such as arbitrary shape representation of the original data, it is still tied to some drawbacks, for example, it is dependent on the choice of a pre-defined distance metric and is biased by the local information among data instances. Aiming at exploring alternative ways to build graphs from data, this paper proposes an algorithm for constructing a new type of graph, called Attribute-based Decision Graph–AbDG . Given a vector-based data set, an AbDG is built by partitioning each data attribute range into disjoint intervals and representing each interval as a vertex. The edges are then established between vertices from different attributes according to a pre-defined pattern. Classification is performed through a matching process among the attribute values of the new instance and AbDG. Moreover, AbDG provides an inner mechanism to handle missing attribute values, which contributes for expanding its applicability. Results of classification tasks have shown that AbDG is a competitive approach when compared to well-known multiclass algorithms. The main contribution of the proposed framework is the combination of the advantages of attribute-based and graph-based techniques to perform robust pattern matching data classification, while permitting the analysis the input data considering only a subset of its attributes. [ABSTRACT FROM AUTHOR]