Search

Support vector machine (SVM) is a particularly powerful and flexible supervised learning model that analyzes data for both classification and regression, whose usual algorithm complexity scales polynomially with the dimension of data space and the number of data points. To tackle the big data challenge, a quantum SVM algorithm was proposed, which is claimed to achieve exponential speedup for least squares SVM (LS-SVM). Here, inspired by the quantum SVM algorithm, we present a quantum-inspired classical algorithm for LS-SVM. In our approach, an improved fast sampling technique, namely indirect sampling, is proposed for sampling the kernel matrix and classifying. We first consider the LS-SVM with a linear kernel, and then discuss the generalization of our method to nonlinear kernels. Theoretical analysis shows our algorithm can make classification with arbitrary success probability in logarithmic runtime of both the dimension of data space and the number of data points for low rank, low condition number, and high dimensional data matrix, matching the runtime of the quantum SVM. [ABSTRACT FROM AUTHOR]

Twin support vector machine (TWSVM), which constructs two nonparallel classifying hyperplanes, is widely applied to various fields. However, TWSVM solves two quadratic programming problems (QPPs) separately such that the final classifiers lack consistency and enough prediction accuracy. Moreover, by reason of only considering the 1-norm penalty for slack variables, TWSVM is not well defined in the geometrical view. In this article, we propose a novel elastic net nonparallel hyperplane support vector machine (ENNHSVM), which adopts elastic net penalty for slack variables and constructs two nonparallel separating hyperplanes simultaneously. We further discuss the properties of ENNHSVM theoretically and derive the violation tolerance upper bound to better demonstrate the relative violations of training samples in the same class. In particular, we design a safe screening rule for ENNHSVM to speed up the calculations. We finally compare the performance of ENNHSVM on both synthetic datasets and benchmark datasets with the Lagrangian SVM, the twin parametric-margin SVM, the elastic net SVM, the TWSVM, and the nonparallel hyperplane SVM. [ABSTRACT FROM AUTHOR]

One-shot semantic segmentation poses the challenging task of segmenting object regions from unseen categories with only one annotated example as guidance. Thus, how to effectively construct robust feature representations from the guidance image is crucial to the success of one-shot semantic segmentation. To this end, we propose in this article a simple, yet effective approach named rich embedding features (REFs). Given a reference image accompanied with its annotated mask, our REF constructs rich embedding features of the support object from three perspectives: 1) global embedding to capture the general characteristics; 2) peak embedding to capture the most discriminative information; 3) adaptive embedding to capture the internal long-range dependencies. By combining these informative features, we can easily harvest sufficient and rich guidance even from a single reference image. In addition to REF, we further propose a simple depth-priority context module to obtain useful contextual cues from the query image. This successfully raises the performance of one-shot semantic segmentation to a new level. We conduct experiments on pattern analysis, statical modeling and computational learning (Pascal) visual object classes (VOC) 2012 and common object in context (COCO) to demonstrate the effectiveness of our approach. [ABSTRACT FROM AUTHOR]

The brain–computer interface (BCI) P300 speller analyzes the P300 signals from the brain to achieve direct communication between humans and machines, which can assist patients with severe disabilities to control external machines or robots to complete expected tasks. Therefore, the classification method of P300 signals plays an important role in the development of BCI systems and technologies. In this article, a novel ensemble support vector recurrent neural network (E-SVRNN) framework is proposed and developed to acquire more accurate and efficient electroencephalogram (EEG) signal classification results. First, we construct a support vector machine (SVM) to formulate EEG signals recognizing model. Second, the SVM formulation is transformed into a standard convex quadratic programming (QP) problem. Third, the convex QP problem is solved by combining a varying parameter recurrent neural network (VPRNN) with a penalty function. Experimental results on BCI competition II and BCI competition III datasets demonstrate that the proposed E-SVRNN framework can achieve accuracy rates as high as 100% and 99%, respectively. In addition, the results of comparison experiments verify that the proposed E-SVRNN possesses the best recognition accuracy and information transfer rate (ITR) compared with most of the state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Learning with label proportions (LLP) deals with the problem that the training data are provided as bags, where the label proportions of training bags rather than the labels of individual training instances are accessible. Existing LLP studies assume that the label proportions of all training bags are accessible. However, in many applications, it is time-consuming to mark all training bags with label proportions, which leads to the problem of learning with both marked and unmarked bags, namely, semisupervised LLP (SLLP). In this work, we propose semisupervised proportional support vector machine (SS- $\propto $ SVM), which extends the proportional SVM ($\propto $ SVM) model to its semisupervised version. To the best of our knowledge, SS- $\propto $ SVM is the first attempt to cope with the SLLP problem. Two realizations, alter-SS- $\propto $ SVM and conv-SS- $\propto $ SVM, which are based on alternating optimization and convex relaxation, respectively, are developed to solve the proposed SS- $\propto $ SVM model. Moreover, we design a cutting plane (CP) method to optimize conv-SS- $\propto $ SVM with a guaranteed convergence rate and present a fast accelerated proximal gradient method to solve the multiple kernel learning subproblem in conv-SS- $\propto $ SVM efficiently. Empirical experiments not only justify the superiority of SS- $\propto $ SVM over its supervised counterpart in classification accuracy but also demonstrate the high competitive computational efficiency of the CP optimization of conv-SS- $\propto $ SVM. [ABSTRACT FROM AUTHOR]

Multi-instance learning (MIL) has garnered considerable attention in recent years due to its favorable performance in various scenarios. Nonetheless, most previous studies have implicitly expressed the correlation between instances and bags. Moreover, the importance of negative instances has been largely overlooked. Hence, we seek to present an explicit and intuitively understandable method that can compensate for these deficiencies. In this article, we creatively introduce a metric-based multiconcept MIL approach based on two aspects. First, the triplet-based bag embedding method identifies instance categories and builds attention weights for every instance explicitly. Accordingly, bag embedding is accomplished under the limitation of weak supervision. Second, the developed instance correlation metric approach in the superbag considers the multiconcept issue to boost the model generalization performance. We have designed a rich variety of experiments to demonstrate the performance of our algorithm. The artificial data experiment reveals the interpretability of the proposed network. The results of the comparison experiment confirm that our method shows favorable performance in multiple tasks. Finally, we illustrate the motivation of the presented method by the ablation experiments. [ABSTRACT FROM AUTHOR]

In classification, the use of 0–1 loss is preferable since the minimizer of 0–1 risk leads to the Bayes optimal classifier. However, due to the nonconvexity of 0–1 loss, this optimization problem is NP-hard. Therefore, many convex surrogate loss functions have been adopted. Previous works have shown that if a Bayes-risk consistent loss function is used as a surrogate, the minimizer of the empirical surrogate risk can achieve the Bayes optimal classifier as the sample size tends to infinity. Nevertheless, the comparison of convergence rates of minimizers of different empirical surrogate risks to the Bayes optimal classifier has rarely been studied. Which characterization of the surrogate loss determines its convergence rate to the Bayes optimal classifier? Can we modify the loss function to achieve a faster convergence rate? In this article, we study the convergence rates of empirical surrogate minimizers to the Bayes optimal classifier. Specifically, we introduce the notions of consistency intensity and conductivity to characterize a surrogate loss function and exploit this notion to obtain the rate of convergence from an empirical surrogate risk minimizer to the Bayes optimal classifier, enabling fair comparisons of the excess risks of different surrogate risk minimizers. The main result of this article has practical implications including: 1) showing that hinge loss (SVM) is superior to logistic loss (Logistic regression) and exponential loss (Adaboost) in the sense that its empirical minimizer converges faster to the Bayes optimal classifier and 2) guiding the design of new loss functions to speed up the convergence rate to the Bayes optimal classifier with a data-dependent loss correction method inspired by our theorems. [ABSTRACT FROM AUTHOR]

In this article, anomaly detection is considered for hyperspectral imagery in the Gaussian background with an unknown covariance matrix. The anomaly to be detected occupies multiple pixels with an unknown pattern. Two adaptive detectors are proposed based on the generalized likelihood ratio test design procedure and ad hoc modification of it. Surprisingly, it turns out that the two proposed detectors are equivalent. Analytical expressions are derived for the probability of false alarm of the proposed detector, which exhibits a constant false alarm rate against the noise covariance matrix. Numerical examples using simulated data reveal how some system parameters (e.g., the background data size and pixel number) affect the performance of the proposed detector. Experiments are conducted on five real hyperspectral data sets, demonstrating that the proposed detector achieves better detection performance than its counterparts. [ABSTRACT FROM AUTHOR]

This work proposes a decision tree (DT)-based method for initializing a dendritic neuron model (DNM). Neural networks become larger and larger, thus consuming more and more computing resources. This calls for a strong need to prune neurons that do not contribute much to their network’s output. Pruning those with low contribution may lead to a loss of accuracy of DNM. Our proposed method is novel because 1) it can reduce the number of dendrites in DNM while improving training efficiency without affecting accuracy and 2) it can select proper initialization weight and threshold of neurons. The Adam algorithm is used to train DNM after its initialization with our proposed DT-based method. To verify its effectiveness, we apply it to seven benchmark datasets. The results show that decision-tree-initialized DNM is significantly better than the original DNM, k-nearest neighbor, support vector machine, back-propagation neural network, and DT classification methods. It exhibits the lowest model complexity and highest training speed without losing any accuracy. The interactions among attributes can also be observed in its dendritic neurons. [ABSTRACT FROM AUTHOR]

Feature selection aims to select strongly relevant features and discard the rest. Recently, embedded feature selection methods, which incorporate feature weights learning into the training process of a classifier, have attracted much attention. However, traditional embedded methods merely focus on the combinatorial optimality of all selected features. They sometimes select the weakly relevant features with satisfactory combination abilities and leave out some strongly relevant features, thereby degrading the generalization performance. To address this issue, we propose a novel embedded framework for feature selection, termed feature selection boosted by unselected features (FSBUF). Specifically, we introduce an extra classifier for unselected features into the traditional embedded model and jointly learn the feature weights to maximize the classification loss of unselected features. As a result, the extra classifier recycles the unselected strongly relevant features to replace the weakly relevant features in the selected feature subset. Our final objective can be formulated as a minimax optimization problem, and we design an effective gradient-based algorithm to solve it. Furthermore, we theoretically prove that the proposed FSBUF is able to improve the generalization ability of traditional embedded feature selection methods. Extensive experiments on synthetic and real-world data sets exhibit the comprehensibility and superior performance of FSBUF. [ABSTRACT FROM AUTHOR]

Categorizing aerial photographs with varied weather/lighting conditions and sophisticated geomorphic factors is a key module in autonomous navigation, environmental evaluation, and so on. Previous image recognizers cannot fulfill this task due to three challenges: 1) localizing visually/semantically salient regions within each aerial photograph in a weakly annotated context due to the unaffordable human resources required for pixel-level annotation; 2) aerial photographs are generally with multiple informative attributes (e.g., clarity and reflectivity), and we have to encode them for better aerial photograph modeling; and 3) designing a cross-domain knowledge transferal module to enhance aerial photograph perception since multiresolution aerial photographs are taken asynchronistically and are mutually complementary. To handle the above problems, we propose to optimize aerial photograph’s feature learning by leveraging the low-resolution spatial composition to enhance the deep learning of perceptual features with a high resolution. More specifically, we first extract many BING-based object patches (Cheng et al., 2014) from each aerial photograph. A weakly supervised ranking algorithm selects a few semantically salient ones by seamlessly incorporating multiple aerial photograph attributes. Toward an interpretable aerial photograph recognizer indicative to human visual perception, we construct a gaze shifting path (GSP) by linking the top-ranking object patches and, subsequently, derive the deep GSP feature. Finally, a cross-domain multilabel SVM is formulated to categorize each aerial photograph. It leverages the global feature from low-resolution counterparts to optimize the deep GSP feature from a high-resolution aerial photograph. Comparative results on our compiled million-scale aerial photograph set have demonstrated the competitiveness of our approach. Besides, the eye-tracking experiment has shown that our ranking-based GSPs are over 92% consistent with the real human gaze shifting sequences. [ABSTRACT FROM AUTHOR]

Image ordinal estimation is to predict the ordinal label of a given image, which can be categorized as an ordinal regression (OR) problem. Recent methods formulate an OR problem as a series of binary classification problems. Such methods cannot ensure that the global ordinal relationship is preserved since the relationships among different binary classifiers are neglected. We propose a novel OR approach, termed convolutional OR forest (CORF), for image ordinal estimation, which can integrate OR and differentiable decision trees with a convolutional neural network for obtaining precise and stable global ordinal relationships. The advantages of the proposed CORF are twofold. First, instead of learning a series of binary classifiers independently, the proposed method aims at learning an ordinal distribution for OR by optimizing those binary classifiers simultaneously. Second, the differentiable decision trees in the proposed CORF can be trained together with the ordinal distribution in an end-to-end manner. The effectiveness of the proposed CORF is verified on two image ordinal estimation tasks, i.e., facial age estimation and image esthetic assessment, showing significant improvements and better stability over the state-of-the-art OR methods. [ABSTRACT FROM AUTHOR]

Margin is an important concept in machine learning; theoretical analyses further reveal that the distribution of margin plays a more critical role than the minimum margin in generalization power. Recently, several approaches have achieved performance breakthroughs by optimizing the margin distribution, but their computational cost, which is usually higher than before, still hinders them to be widely applied. In this article, we propose margin distribution analysis (MDA), which optimizes the margin distribution more simply by maximizing the margin mean and minimizing the margin variance simultaneously. MDA is efficient and resistive to class-imbalance naturally, since its objective distinguishes the margin means of different classes and can be broken up into two linear equations. In practice, it can also cooperate with other frameworks such as reweight-minimization when facing complex circumstances with noise and outliers. Empirical studies validate the superiority of MDA in real-world data sets, and demonstrate that simple approaches can also perform competitively by optimizing margin distribution. [ABSTRACT FROM AUTHOR]

Learning classifiers with imbalanced data can be strongly biased toward the majority class. To address this issue, several methods have been proposed using generative adversarial networks (GANs). Existing GAN-based methods, however, do not effectively utilize the relationship between a classifier and a generator. This article proposes a novel three-player structure consisting of a discriminator, a generator, and a classifier, along with decision boundary regularization. Our method is distinctive in which the generator is trained in cooperation with the classifier to provide minority samples that gradually expand the minority decision region, improving performance for imbalanced data classification. The proposed method outperforms the existing methods on real data sets as well as synthetic imbalanced data sets. [ABSTRACT FROM AUTHOR]

Accurate electroencephalogram (EEG) pattern decoding for specific mental tasks is one of the key steps for the development of brain–computer interface (BCI), which is quite challenging due to the considerably low signal-to-noise ratio of EEG collected at the brain scalp. Machine learning provides a promising technique to optimize EEG patterns toward better decoding accuracy. However, existing algorithms do not effectively explore the underlying data structure capturing the true EEG sample distribution and, hence, can only yield a suboptimal decoding accuracy. To uncover the intrinsic distribution structure of EEG data, we propose a clustering-based multitask feature learning algorithm for improved EEG pattern decoding. Specifically, we perform affinity propagation-based clustering to explore the subclasses (i.e., clusters) in each of the original classes and then assign each subclass a unique label based on a one-versus-all encoding strategy. With the encoded label matrix, we devise a novel multitask learning algorithm by exploiting the subclass relationship to jointly optimize the EEG pattern features from the uncovered subclasses. We then train a linear support vector machine with the optimized features for EEG pattern decoding. Extensive experimental studies are conducted on three EEG data sets to validate the effectiveness of our algorithm in comparison with other state-of-the-art approaches. The improved experimental results demonstrate the outstanding superiority of our algorithm, suggesting its prominent performance for EEG pattern decoding in BCI applications. [ABSTRACT FROM AUTHOR]

Distance metric learning (DML) aims to learn a distance metric to process the data distribution. However, most of the existing methods are $k$ NN DML methods and employ the $k$ NN model to classify the test instances. The drawback of $k$ NN DML is that all training instances need to be accessed and stored to classify the test instances, and the classification performance is influenced by the setting of the nearest neighbor number $k$. To solve these problems, there are several DML methods that employ the SVM model to classify the test instances. However, all of them are nonconvex and the convex support vector DML method has not been explicitly proposed. In this article, we propose a convex model for support vector DML (CSV-DML), which is capable of replacing the $k$ NN model of DML with the SVM model. To make CSV-DML can use the most kernel functions of the existing SVM methods, a nonlinear mapping is used to map the original instances into a feature space. Since the explicit form of nonlinear mapped instances is unknown, the original instances are further transformed into the kernel form, which can be calculated explicitly. CSV-DML is constructed to work directly on the kernel-transformed instances. Specifically, we learn a specific Mahalanobis distance metric from the kernel-transformed training instances and train a DML-based separating hyperplane based on it. An iterated approach is formulated to optimize CSV-DML, which is based on generalized block coordinate descent and can converge to the global optimum. In CSV-DML, since the dimension of kernel-transformed instances is only related to the number of original training instances, we develop a novel parameter reduction scheme for reducing the feature dimension. Extensive experiments show that the proposed CSV-DML method outperforms the previous methods. [ABSTRACT FROM AUTHOR]

Truncated Newton (TN) methods have been a useful technique for large-scale optimization. Instead of obtaining the full Newton direction, a truncated method approximately solves the Newton equation with an inner conjugate gradient (CG) procedure (TNCG for the whole method). These methods have been employed to efficiently solve linear classification problems. However, even in this deeply studied field, various theoretical and numerical aspects were not completely explored. The first contribution of this work is to comprehensively study the global and local convergence when TNCG is applied to linear classification. Because of the lack of twice differentiability under some losses, many past works cannot be applied here. We prove various missing pieces of theory from scratch and clarify many proper references. The second contribution is to study the termination of the CG method. For the first time when TNCG is applied to linear classification, we show that the inner stopping condition strongly affects the convergence speed. We propose using a quadratic stopping criterion to achieve both robustness and efficiency. The third contribution is that of combining the study on inner stopping criteria with that of preconditioning. We discuss how convergence theory is affected by preconditioning and finally propose an effective preconditioned TNCG. [ABSTRACT FROM AUTHOR]

Mitigating label noise is a crucial problem in classification. Noise filtering is an effective method of dealing with label noise which does not need to estimate the noise rate or rely on any loss function. However, most filtering methods focus mainly on binary classification, leaving the more difficult counterpart problem of multiclass classification relatively unexplored. To remedy this deficit, we present a definition for label noise in a multiclass setting and propose a general framework for a novel label noise filtering learning method for multiclass classification. Two examples of noise filtering methods for multiclass classification, multiclass complete random forest (mCRF) and multiclass relative density, are derived from their binary counterparts using our proposed framework. In addition, to optimize the NI_threshold hyperparameter in mCRF, we propose two new optimization methods: a new voting cross-validation method and an adaptive method that employs a 2-means clustering algorithm. Furthermore, we incorporate SMOTE into our label noise filtering learning framework to handle the ubiquitous problem of imbalanced data in multiclass classification. We report experiments on both synthetic data sets and UCI benchmarks to demonstrate our proposed methods are highly robust to label noise in comparison with state-of-the-art baselines. All code and data results are available at https://github.com/syxiaa/Multiclass-Label-Noise-Filtering-Learning. [ABSTRACT FROM AUTHOR]

The two issues on dynamically generated hierarchical neural networks such as the sort of basic neurons and how to compose a layer are considered in this article. On the first issue, a variant version of the least-square support vector regression (SVR) is chosen as a basic neuron. Support vector machine (SVM) is a representative classifier which usually shows good classification performance. Along with the SVMs, SVR was introduced to deal with the regression problem. Especially, least-square SVR has the advantages of high learning speed due to the substitution of the inequality constraints by the equality constraint in the formulation of the optimization problem. Based on the least-square SVR, the multiple least-square (MLS) SVR, which is a type of a linear combination of least-square SVRs with fuzzy clustering, is proposed to improve the modeling performance. In addition, a hierarchical neural network, where the MLS SVR is utilized as the generic node instead of the conventional polynomial, is developed. The key issues of hierarchical neural networks, which are generated dynamically layer by layer, are discussed on how to retain the diversity of the nodes located at the same layer according to the increase of the layer. In order to maintain the diversity of the nodes, various selection methods such as truncation selection and roulette wheel selection (RWS) to choose the nodes among candidate nodes are proposed. In addition, in order to reduce the computational overhead to determine all candidates which exhibit all compositions of the input variables, a new implementation method is proposed. From the viewpoint of the diversity of the selected nodes and the computational aspects, it is shown that the proposed method is preferred over the conventional design methodology. [ABSTRACT FROM AUTHOR]

Kernel recursive least squares (KRLS) is a widely used online machine learning algorithm for time series predictions. In this article, we present the mixed-precision KRLS, producing equivalent prediction accuracy to double-precision KRLS with a higher training throughput and a lower memory footprint. The mixed-precision KRLS applies single-precision arithmetic to the computation components being not only numerically resilient but also computationally intensive. Our mixed-precision KRLS demonstrates the 1.32, 1.15, 1.29, 1.09, and $1.08\times $ training throughput improvements using 24.95%, 24.74%, 24.89%, 24.48%, and 24.20% less memory footprint without losing any prediction accuracy compared to double-precision KRLS for a 3-D nonlinear regression, a Lorenz chaotic time series, a Mackey-Glass chaotic time series, a sunspot number time series, and a sea surface temperature time series, respectively. [ABSTRACT FROM AUTHOR]

In this article, we discuss two novel sparse versions of the classical nearest-centroid classifier. The proposed sparse classifiers are based on $\ell _{1}$ and $\ell _{2}$ distance criteria, respectively, and perform simultaneous feature selection and classification, by detecting the features that are most relevant for the classification purpose. We formally prove that the training of the proposed sparse models, with both distance criteria, can be performed exactly (i.e., the globally optimal set of features is selected) at a linear computational cost. Especially, the proposed sparse classifiers are trained in $O(mn)+O(m\log k)$ operations, where $n$ is the number of samples, $m$ is the total number of features, and $k \leq m$ is the number of features to be retained in the classifier. Furthermore, the complexity of testing and classifying a new sample is simply $O(k)$ for both methods. The proposed models can be employed either as stand-alone sparse classifiers or fast feature-selection techniques for prefiltering the features to be later fed to other types of classifiers (e.g., SVMs). The experimental results show that the proposed methods are competitive in accuracy with state-of-the-art feature selection and classification techniques while having a substantially lower computational cost. [ABSTRACT FROM AUTHOR]

Kernel-based extreme learning machine (KELM), as a natural extension of ELM to kernel learning, has achieved outstanding performance in addressing various regression and classification problems. Compared with the basic ELM, KELM has a better generalization ability owing to no needs of the number of hidden nodes given beforehand and random projection mechanism. Since KELM is derived under the minimum mean square error (MMSE) criterion for the Gaussian assumption of noise, its performance may deteriorate under the non-Gaussian cases, seriously. To improve the robustness of KELM, this article proposes a mixture correntropy-based KELM (MC-KELM), which adopts the recently proposed maximum mixture correntropy criterion as the optimization criterion, instead of using the MMSE criterion. In addition, an online sequential version of MC-KELM (MCOS-KELM) is developed to deal with the case that the data arrive sequentially (one-by-one or chunk-by-chunk). Experimental results on regression and classification data sets are reported to validate the performance superiorities of the new methods. [ABSTRACT FROM AUTHOR]

Sign language is the main type of communication of the deaf community. However, most people do not know this language, which causes communication problems for many people. Many technological solutions have been proposed to overcome this issue. Some use the concept of wearable devices, but gesture recognition in a video sequence is a cheaper and less intrusive solution. In this work, we address the problem of gesture recognition in video. To do so, we employ a two-step method with feature space mapping and classification. First, the body parts of each subject in a video are segmented through a deep neural network architecture. Then, we use Gait Energy Image to encode the motion of the body parts in a compact feature space. Small datasets are usually a problem in this type of application, leading to sparse representation in the feature space. To contour this problem, we evaluate SMOTE as a data augmentation technique in the feature space and classical dimensionality reduction techniques. We evaluate our method on three challenging Brazilian sign language (Libras) datasets, CEFET/RJ-Libras, MINDS-Libras, and LIBRAS-UFOP, achieving global accuracies of 85.40±3.13%, 84.66±1.78%, and 64.91±3.79%, respectively, with singular value decomposition and support vector machine. [ABSTRACT FROM AUTHOR]

Suitability of shallow (one-hidden-layer) networks with translation-invariant kernel units for function approximation and classification tasks is investigated. It is shown that a critical property influencing the capabilities of kernel networks is how the Fourier transforms of kernels converge to zero. The Fourier transforms of kernels suitable for multivariable approximation can have negative values but must be almost everywhere nonzero. In contrast, the Fourier transforms of kernels suitable for maximal margin classification must be everywhere nonnegative but can have large sets where they are equal to zero (e.g., they can be compactly supported). The behavior of the Fourier transforms of multivariable kernels is analyzed using the Hankel transform. The general results are illustrated by examples of both univariable and multivariable kernels (such as Gaussian, Laplace, rectangle, sinc, and cut power kernels). [ABSTRACT FROM AUTHOR]

In this article, an improved nonparallel support vector machine (INPSVM) is proposed for pattern classification. INPSVM inherits almost all advantages of nonparallel support vector machine (NPSVM), i.e., the kernel trick can be directly applied for the nonlinear case and the matrix inversion is avoided. These are completely different from the twin support vector machine (TSVM). Moreover, the INPSVM classifier has some incomparable advantages over TSVM and NPSVM. First, it can effectively eliminate the negative effect of noise, especially feature noise around the decision boundary. Second, the novel classifier has higher classification accuracy for both linear and nonlinear data sets compared with the other algorithms. Finally, a large number of experiments show that INPSVM is superior to other algorithms in efficiency, accuracy, and robustness. [ABSTRACT FROM AUTHOR]

Fractional calculus and entropy are two essential mathematical tools, and their conceptions support a productive interplay in the study of system dynamics and machine learning. In this article, we modify the fractional entropy and propose the cumulative permuted fractional entropy (CPFE). A theoretical analysis is provided to prove that CPFE not only meets the basic properties of the Shannon entropy but also has unique characteristics of its own. We apply it to typical discrete distributions, simulated data, and real-world data to prove its efficiency in the application. This article demonstrates that CPFE can measure the complexity and uncertainty of complex systems so that it can perform reliable and accurate classification. Finally, we introduce CPFE to support vector machines (SVMs) and get CPFE-SVM. The CPFE can be used to process data to make the irregular data linearly separable. Compared with the other five state-of-the-art algorithms, CPFE-SVM has significantly higher accuracy and less computational burden. Therefore, the CPFE-SVM is especially suitable for the classification of irregular large-scale data sets. Also, it is insensitive to noise. Implications of the results and future research directions are also presented. [ABSTRACT FROM AUTHOR]

This paper presents a novel hardware architecture of the Tensorial Support Vector Machine (TSVM) based on Shallow Neural Networks (NN) for the Single Value Decomposition (SVD) computation. The proposed NN achieves a comparable Mean Squared Error and Cosine Similarity to the widely used one-sided Jacobi algorithm. When implemented on an FPGA, the NN offers $324\times $ faster computations than the one-sided Jacobi with reductions up to 58% and 67% in terms of hardware resources and power consumption respectively. When validated on a touch modality classification problem, the NN-based TSVM implementation has achieved a real-time operation while consuming about 88% less energy per classification than the Jacobi-based TSVM with an accuracy loss of at most 3%. Such results offer the ability to deploy intelligence on resource-limited platform for energy-constrained applications. [ABSTRACT FROM AUTHOR]

The aim of unsupervised domain adaptation is to leverage the knowledge in a labeled (source) domain to improve a model’s learning performance with an unlabeled (target) domain—the basic strategy being to mitigate the effects of discrepancies between the two distributions. Most existing algorithms can only handle unsupervised closed set domain adaptation (UCSDA), i.e., where the source and target domains are assumed to share the same label set. In this article, we target a more challenging but realistic setting: unsupervised open set domain adaptation (UOSDA), where the target domain has unknown classes that are not found in the source domain. This is the first study to provide learning bound for open set domain adaptation, which we do by theoretically investigating the risk of the target classifier on unknown classes. The proposed learning bound has a special term, namely, open set difference, which reflects the risk of the target classifier on unknown classes. Furthermore, we present a novel and theoretically guided unsupervised algorithm for open set domain adaptation, called distribution alignment with open difference (DAOD), which is based on regularizing this open set difference bound. The experiments on several benchmark data sets show the superior performance of the proposed UOSDA method compared with the state-of-the-art methods in the literature. [ABSTRACT FROM AUTHOR]

In the light of the general question posed in the title, we write down a very simple randomized learning algorithm, based on boosting, that can be seen as a nonstationary Markov random process. Surprisingly, the decision hyperplanes resulting from this algorithm converge in probability to the exact hard-margin solutions of support vector machines (SVMs). This fact is curious because the hard-margin hyperplane is not a statistical solution, but a purely geometric one—driven by margin maximization and strictly dependent on particular locations of some data points that are placed in the contact region of two classes, namely the support vectors. The proposed algorithm detects support vectors probabilistically, without being aware of their geometric definition. We give proofs of the main convergence theorem and several auxiliary lemmas. The analysis sheds new light on the relation between boosting and SVMs and also on the nature of SVM solutions since they can now be regarded equivalently as limits of certain random trajectories. In the experimental part, correctness of the proposed algorithm is verified against known SVM solvers: libsvm, liblinear, and also against optimization packages: cvxopt (Python) and Wolfram Mathematica. [ABSTRACT FROM AUTHOR]

In this article, we propose a general model for plane-based clustering. The general model reveals the relationship between cluster assignment and cluster updating during clustering implementation, and it contains many existing plane-based clustering methods, e.g., k-plane clustering, proximal plane clustering, twin support vector clustering, and their extensions. Under this general model, one may obtain an appropriate clustering method for a specific purpose. The general model is a procedure corresponding to an optimization problem, which minimizes the total loss of the samples. Thereinto, the loss of a sample derives from both within-cluster and between-cluster information. We discuss the theoretical termination conditions and prove that the general model terminates in a finite number of steps at a local or weak local solution. Furthermore, we propose a distribution loss function that fluctuates with the input data and introduce it into the general model to obtain a plane-based clustering method (DPC). DPC can capture the data distribution precisely because of its statistical characteristics, and its termination that finitely terminates at a weak local solution is given immediately based on the general model. The experimental results show that our DPC outperforms the state-of-the-art plane-based clustering methods on many synthetic and benchmark data sets. [ABSTRACT FROM AUTHOR]

Ordinal regression (OR) is one of the most important machine learning tasks. The kernel method is a major technique to achieve nonlinear OR. However, traditional kernel OR solvers are inefficient due to increased complexity introduced by multiple ordinal thresholds as well as the cost of kernel computation. Doubly stochastic gradient (DSG) is a very efficient and scalable kernel learning algorithm that combines random feature approximation with stochastic functional optimization. However, the theory and algorithm of DSG can only support optimization tasks within the unique reproducing kernel Hilbert space (RKHS), which is not suitable for OR problems where the multiple ordinal thresholds usually lead to multiple RKHSs. To address this problem, we construct a kernel whose RKHS can contain the decision function with multiple thresholds. Based on this new kernel, we further propose a novel DSG-like algorithm, DSGOR. In each iteration of DSGOR, we update the decision functional as well as the function bias with appropriately set learning rates for each. Our theoretic analysis shows that DSGOR can achieve O(1/t) convergence rate, which is as good as DSG, even though dealing with a much harder problem. Extensive experimental results demonstrate that our algorithm is much more efficient than traditional kernel OR solvers, especially on large-scale problems. [ABSTRACT FROM AUTHOR]

Tensors are increasingly encountered in prediction problems. We extend previous results for high-dimensional least-squares convex tensor regression to classification problems with a hinge loss and establish its asymptotic statistical properties. Based on a general convex decomposable penalty, the rate depends on both the intrinsic dimension and the Rademacher complexity of the class of linear functions of tensor predictors. [ABSTRACT FROM AUTHOR]

Due to the effectiveness of anomaly/outlier detection, one-class algorithms have been extensively studied in the past. The representatives include the shallow-structure methods and deep networks, such as the one-class support vector machine (OC-SVM), one-class extreme learning machine (OC-ELM), deep support vector data description (Deep SVDD), and multilayer OC-ELM (ML-OCELM/MK-OCELM). However, existing algorithms are generally built on the minimum mean-square-error (mse) criterion, which is robust to the Gaussian noises but less effective in dealing with large outliers. To alleviate this deficiency, a robust maximum correntropy criterion (MCC)-based OC-ELM (MC-OCELM) is first proposed and then further extended to a hierarchical network to enhance its capability in characterizing complex and large data (named HC-OCELM). The gradient derivation combining with a fixed-point iterative updation scheme is adopted for the output weight optimization. Experiments on many benchmark data sets are conducted for effectiveness validation. Comparisons to many state-of-the-art approaches are provided for the superiority demonstration. [ABSTRACT FROM AUTHOR]

We face a binary multiple instance learning (MIL) problem, whose objective is to discriminate between two kinds of point sets: positive and negative. In the MIL terminology, such sets are called bags, and the points inside each bag are called instances. Considering the case with two classes of instances (positive and negative) and inspired by a well-established instance-space support vector machine (SVM) model, we propose to extend to MIL classification the proximal SVM (PSVM) technique that has revealed very effective for supervised learning, especially in terms of computational time. In particular, our approach is based on a new instance-space model that exploits the benefits coming from both SVM (better accuracy) and PSVM (computational efficiency) paradigms. Starting from the standard MIL assumption, such a model is aimed at generating a hyperplane placed in the middle between two parallel hyperplanes: the first one is a proximal hyperplane that clusters the instances of the positive bags, while the second one constitutes a supporting hyperplane for the instances of the negative bags. Numerical results are presented on a set of MIL test data sets drawn from the literature. [ABSTRACT FROM AUTHOR]

Emotions composed of cognizant logical reactions toward various situations. Such mental responses stem from physiological, cognitive, and behavioral changes. Electroencephalogram (EEG) signals provide a noninvasive and nonradioactive solution for emotion identification. Accurate and automatic classification of emotions can boost the development of human–computer interface. This article proposes automatic extraction and classification of features through the use of different convolutional neural networks (CNNs). At first, the proposed method converts the filtered EEG signals into an image using a time–frequency representation. Smoothed pseudo-Wigner–Ville distribution is used to transform time-domain EEG signals into images. These images are fed to pretrained AlexNet, ResNet50, and VGG16 along with configurable CNN. The performance of four CNNs is evaluated by measuring the accuracy, precision, Mathew’s correlation coefficient, F1-score, and false-positive rate. The results obtained by evaluating four CNNs show that configurable CNN requires very less learning parameters with better accuracy. Accuracy scores of 90.98%, 91.91%, 92.71%, and 93.01% obtained by AlexNet, ResNet50, VGG16, and configurable CNN show that the proposed method is best among other existing methods. [ABSTRACT FROM AUTHOR]

With the rapid development and globalization of the semiconductor industry, hardware security has emerged as a critical concern. New attacking and tampering methods are continuously challenge current hardware protection methods. Combating these powerful attacks is of great importance in securing hardware devices. This paper proposes a novel framework to protect Physical Unclonable Function (PUF) against modeling attacks, denominated as Noisy PUF (NoPUF). NoPUF exploits structural unpredictability to improve overall security. We present several PUF architectures under the proposed framework that could reconfigure a conventional reliable PUF to a noisy PUF. The reconfigured PUF becomes inherently unreliable and hence achieves a higher resistance against modeling attacks. Moreover, since only a small portion of the Challenge-Response Pairs (CRPs) are required for authentication, the designer can use the information obtained from the initial reliable PUF configuration to find CRPs, which are still reliable in the noisy PUF configuration for authentication. Exploiting such information asymmetry between designer and attacker is the nexus of the proposed NoPUF design methodology. Experimental results show that we can achieve a maximum attacker and designer accuracy difference of 44.79% for a 64-stage NoPUF candidate architecture while ensuring high reliability for selected challenges. [ABSTRACT FROM AUTHOR]

Recently, there have been many studies attempting to take advantage of advancements in Artificial Intelligence (AI) in Analog and Mixed-Signal (AMS) circuit design. Automated circuit sizing optimization and improving the accuracy of performance models are the two predominant uses of AI in AMS circuit design. This paper first introduces and explains the basic concepts in AI especially the ones that are more suitable to this application. Next, it surveys some recent studies of various AI techniques for AMS circuit design. Then, it discusses the main approaches as well as the pros and cons of each method. Finally, it gives meaningful insights about the current challenges and open issues, as well as recommends approaches for specific applications. [ABSTRACT FROM AUTHOR]

Prediction accuracy and model explainability are the two most important objectives when developing machine learning algorithms to solve real-world problems. Neural networks are known to possess good prediction performance but suffer from a lack of model interpretability. In this article, we propose to enhance the explainability of neural networks through the following architecture constraints: 1) sparse additive subnetworks; 2) projection pursuit with orthogonality constraint; and 3) smooth function approximation. It leads to an enhanced explainable neural network (ExNN) with a superior balance between prediction performance and model interpretability. We derive sufficient identifiability conditions for the proposed ExNN model. The multiple parameters are simultaneously estimated by a modified minibatch gradient descent method based on the backpropagation algorithm for calculating the derivatives and the Cayley transform for preserving the projection orthogonality. Through simulation study under six different scenarios, we compare the proposed method to several benchmarks, including least absolute shrinkage and selection operator, support vector machine, random forest, extreme learning machine, and multilayer perceptron. It is shown that the proposed ExNN model keeps the flexibility of pursuing high prediction accuracy while attaining improved interpretability. Finally, a real data example is employed as a showcase application. [ABSTRACT FROM AUTHOR]

In this article, we introduce a discriminative ridge regression approach to supervised classification. It estimates a representation model while accounting for discriminativeness between classes, thereby enabling accurate derivation of categorical information. This new type of regression model extends the existing models, such as ridge, lasso, and group lasso, by explicitly incorporating discriminative information. As a special case, we focus on a quadratic model that admits a closed-form analytical solution. The corresponding classifier is called the discriminative ridge machine (DRM). Three iterative algorithms are further established for the DRM to enhance the efficiency and scalability for real applications. Our approach and the algorithms are applicable to general types of data including images, high-dimensional data, and imbalanced data. We compare the DRM with current state-of-the-art classifiers. Our extensive experimental results show the superior performance of the DRM and confirm the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

Automated social text annotation is the task of suggesting a set of tags for shared documents on social media platforms. The automated annotation process can reduce users’ cognitive overhead in tagging and improve tag management for better search, browsing, and recommendation of documents. It can be formulated as a multilabel classification problem. We propose a novel deep learning-based method for this problem and design an attention-based neural network with semantic-based regularization, which can mimic users’ reading and annotation behavior to formulate better document representation, leveraging the semantic relations among labels. The network separately models the title and the content of each document and injects an explicit, title-guided attention mechanism into each sentence. To exploit the correlation among labels, we propose two semantic-based loss regularizers, i.e., similarity and subsumption, which enforce the output of the network to conform to label semantics. The model with the semantic-based loss regularizers is referred to as the joint multilabel attention network (JMAN). We conducted a comprehensive evaluation study and compared JMAN to the state-of-the-art baseline models, using four large, real-world social media data sets. In terms of F1, JMAN significantly outperformed bidirectional gated recurrent unit (Bi-GRU) relatively by around 12.8%–78.6% and the hierarchical attention network (HAN) by around 3.9%–23.8%. The JMAN model demonstrates advantages in convergence and training speed. Further improvement of performance was observed against latent Dirichlet allocation (LDA) and support vector machine (SVM). When applying the semantic-based loss regularizers, the performance of HAN and Bi-GRU in terms of F1 was also boosted. It is also found that dynamic update of the label semantic matrices (JMANd) has the potential to further improve the performance of JMAN but at the cost of substantial memory and warrants further study. [ABSTRACT FROM AUTHOR]

Cost-sensitive learning methods guaranteeing privacy are becoming crucial nowadays in many applications where increasing use of sensitive personal information is observed. However, there has no optimal learning scheme developed in the literature to learn cost-sensitive classifiers under constraint of enforcing differential privacy. Our approach is to first develop a unified framework for existing cost-sensitive learning methods by incorporating the weight constant and weight functions into the classical regularized empirical risk minimization framework. Then, we propose two privacy-preserving algorithms with output perturbation and objective perturbation methods, respectively, to be integrated with the cost-sensitive learning framework. We showcase how this general framework can be used analytically by deriving the privacy-preserving cost-sensitive extensions of logistic regression and support vector machine. Experimental evidence on both synthetic and real data sets verifies that the proposed algorithms can reduce the misclassification cost effectively while satisfying the privacy requirement. A theoretical investigation is also conducted, revealing a very interesting analytic relation, i.e., that the choice of the weight constant and weight functions does not only influence the Fisher-consistent property (population minimizer of expected risk with a specific loss function leads to the Bayes optimal decision rule) but also interacts with privacy-preserving levels to affect the performance of classifiers significantly. [ABSTRACT FROM AUTHOR]

Nontechnical losses (NTLs) are estimated to be considerable and increasing every year. Recently, high-resolution measurements from globally laid smart meters have brought deeper insights on users’ consumption patterns that can be exploited potentially by NTL detection. However, consumption-pattern-based NTL detection is now facing two major challenges: the inefficiency of harnessing high dimensionality and the severe lack of fraudulent samples. To overcome them, an NTL detection model based on deep learning and anomaly detection is proposed in this article, namely bidirectional Wasserstein GAN and support vector data description-based NTL detector (BSBND). Motivated by the powerful ability of generative adversarial networks (GANs) to learn deep representation from high-dimensional distributions of data, in the BSBND, we utilized a BiWGAN for feature extraction from high-dimensional raw consumption records, and a one-class classifier trained only on benign samples—SVDD—is adopted to map features into judgments. Moreover, a novel alternate coordinating algorithm is proposed to optimize the cooperation between the upstream BiWGAN and the downstream SVDD, and also, an interpreting algorithm is proposed to visualize the basis of each fraudulent judgment. Case studies have demonstrated the superiority of the BSBND over the state of the arts, the powerful feature extraction ability of BiWGAN, and also the effectiveness of the proposed coordinating and interpreting algorithms. [ABSTRACT FROM AUTHOR]

We address the problem of classifying trajectories generated by dynamical systems. We consider the model-based approach, which is the classic approach in control theory, and (data-driven) support vector machines, a popular method in the area of machine learning. The analysis points out connections between the two approaches and their relative merits. Examples are given to substantiate the analysis. [ABSTRACT FROM AUTHOR]

This manuscript presents an ultra-low power acoustic feature extractor for always-on voice activity detection (VAD). It extracts voice features by a 10-band passive switched-capacitor (SC) bandpass filter (BPF) bank and digitizes the features using a passive SC envelope-to-digital converter at the low feature rate. The SC feature extractor minimizes the impact of process-voltage-temperature (PVT) variation at the circuit level, and is thereby free from costly chip-wise training or calibration while at the same time being capable of achieving a classification accuracy matching the state of the arts. Experimental results from a VAD feature extractor prototype fabricated in a 0.18- $\mu \text{m}$ CMOS validate the effectiveness of the proposed techniques. It achieves an averaged 90%/86% speech/non-speech hit rates at 10 dB signal-to-noise ratio for all tested chips based on a universal classifier trained with data from one chip. The feature extractor is mostly passive and thus a low power consumption of 270 nW is achieved. In addition, the proposed feature extractor is frequency-scalable, which allows power-efficient multi-purpose acoustic system implementation. [ABSTRACT FROM AUTHOR]

Traditional energy-based learning models associate a single energy metric to each configuration of variables involved in the underlying optimization process. Such models associate the lowest energy state with the optimal configuration of variables under consideration and are thus inherently dissipative. In this article, we propose an energy-efficient learning framework that exploits structural and functional similarities between a machine-learning network and a general electrical network satisfying Tellegen’s theorem. In contrast to the standard energy-based models, the proposed formulation associates two energy components, namely, active and reactive energy with the network. The formulation ensures that the network’s active power is dissipated only during the process of learning, whereas the reactive power is maintained to be zero at all times. As a result, in steady state, the learned parameters are stored and self-sustained by electrical resonance determined by the network’s nodal inductances and capacitances. Based on this approach, this article introduces three novel concepts: 1) a learning framework where the network’s active-power dissipation is used as a regularization for a learning objective function that is subjected to zero total reactive-power constraint; 2) a dynamical system based on complex-domain, continuous-time growth transforms that optimizes the learning objective function and drives the network toward electrical resonance under steady-state operation; and 3) an annealing procedure that controls the tradeoff between active-power dissipation and the speed of convergence. As a representative example, we show how the proposed framework can be used for designing resonant support vector machines (SVMs), where the support vectors correspond to an $LC$ network with self-sustained oscillations. We also show that this resonant network dissipates less active power compared with its non-resonant counterpart. [ABSTRACT FROM AUTHOR]

This brief presents a geometrical approach for obtaining large margin classifiers. The method aims at exploring the geometrical properties of the data set from the structure of a Gabriel graph, which represents pattern relations according to a given distance metric, such as the Euclidean distance. Once the graph is generated, geometrical support vectors (SVs) (analogous to support vector machines (SVMs) SVs) are obtained in order to yield the final large margin solution from a Gaussian mixture model. Experiments with 20 data sets have shown that the solutions obtained with the proposed method are statistically equivalent to those obtained with SVMs. However, the present method does not require optimization and can also be extended to large data sets using the cascade SVM concept. [ABSTRACT FROM AUTHOR]

In linear support vector regression (SVR), the regularization and error sensitivity parameters are used to avoid overfitting the training data. A proper selection of parameters is very essential for obtaining a good model, but the search process may be complicated and time-consuming. In an earlier work by Chu et al. (2015), an effective parameter-selection procedure by using warm-start techniques to solve a sequence of optimization problems has been proposed for linear classification. We extend their techniques to linear SVR, but address some new and challenging issues. In particular, linear classification involves only the regularization parameter, but linear SVR has an extra error sensitivity parameter. We investigate the effective range of each parameter and the sequence in checking the two parameters. Based on this work, an effective tool for the selection of parameters for linear SVR has been available for public use. [ABSTRACT FROM AUTHOR]

Oblique random forests (ObRFs) have attracted increasing attention recently. Their popularity is mainly driven by learning oblique hyperplanes instead of expensively searching for axis-aligned hyperplanes in the standard random forest. However, most existing methods are trained in an off-line mode, which assumes that the training data are given as a batch. Efficient dual-incremental learning (DIL) strategies for ObRF have rarely been explored when new inputs from the existing classes or unseen classes come. The goal of this article is to provide an ObRF with DIL capacity to perform classification on-the-fly. First, we propose a batch multiclass ObRF (ObRF-BM) algorithm by using a broad learning system and a multi-to-binary method to obtain an optimal oblique hyperplane in a higher dimensional space and then separate the samples into two supervised clusters at each node, which provides the basis for the following incremental learning strategy. Then, the DIL strategy for ObRF-BM, termed ObRF-DIL, is developed by analytically updating the parameters of all nodes on the classification route of the increment of input samples and the increment of input classes so that the ObRF-BM model can be effectively updated without laborious retraining from scratch. Experimental results using several public data sets demonstrate the superiority of the proposed approach in comparison with several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In this article, the problem of automatically learning multiple parameters for kernel collaborative representation classification (KCRC) is considered. We investigate the KCRC and measure its generalization error via leave-one-out cross-validation (LOO-CV). By taking advantage of the specific properties of KCRC, a closed-form expression is derived for the outputs of LOO-CV. Then, a simple classification rule that provides probabilistic outputs is adopted, and thereby, an effective loss function that is an explicit function with respect to the parameters is proposed as the generalization error. The gradients of the loss function are calculated, and the parameters are learned by minimizing the loss function using a gradient-based optimization algorithm. Furthermore, the proposed approach makes it possible to solve the multiple kernel/feature learning problems of KCRC effectively. Experiment results on six data sets taken from different scenes demonstrate the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

This article presents a novel representation of artificial neural networks (ANNs) that is based on a projection of weights into a new spherical space defined by a radius $r$ and a vector of angles $\Theta $. This spherical representation of ANNs further simplifies the multiobjective learning problem, which is usually treated as a constrained optimization problem that requires great computational effort to maintain the constraints. With the proposed spherical representation, the constrained optimization problem becomes unconstrained, which simplifies the formulation and computational effort required. In addition, it also allows the use of any nonlinear optimization method for the multiobjective learning of ANNs. Results presented in this article show that the proposed spherical representation of weights yields more accurate estimates of the Pareto set than the classical multiobjective approach. Regarding the final solution selected from the Pareto set, our approach was effective and outperformed some state-of-the-art methods on several data sets. [ABSTRACT FROM AUTHOR]