Search

Ensemble machine learning methods are frequently used for classification problems and it is known that they may boost the prediction accuracy. Support Vector Machines are widely used as base classifiers during the construction of different types of ensembles. In this study, we have constructed a weighted functional margin classifier ensemble on panel financial ratios to discriminate between solid and unhealthy banks for Turkish commercial bank case. We proposed a novel ensemble generation method enhanced by a pruning strategy to increase the prediction performance and developed a novel aggregation approach for ensemble learning by using the idea of weighted sums. The prediction performances are compared with a panel logistic regression which is considered a benchmark method for panel data. The results show that the proposed ensemble method is more successful than the straight SVM and the classical generalized linear model approach.

Objective: Classifier transfers usually come with dataset shifts. To overcome them, online strategies have to be applied. For practical applications, limitations in the computational resources for the adaptation of batch learning algorithms, like the SVM, have to be considered. Approach: We review and compare several strategies for online learning with SVMs. We focus on data selection strategies which limit the size of the stored training data [...] Main Results: For different data shifts, different criteria are appropriate. For the synthetic data, adding all samples to the pool of considered samples performs often significantly worse than other criteria. Especially, adding only misclassified samples performed astoundingly well. Here, balancing criteria were very important when the other criteria were not well chosen. For the transfer setups, the results show that the best strategy depends on the intensity of the drift during the transfer. Adding all and removing the oldest samples results in the best performance, whereas for smaller drifts, it can be sufficient to only add potential new support vectors of the SVM which reduces processing resources. Significance: For BCIs based on EEG models, trained on data from a calibration session, a previous recording session, or even from a recording session with one or several other subjects, are used. This transfer of the learned model usually decreases the performance and can therefore benefit from online learning which adapts the classifier like the established SVM. We show that by using the right combination of data selection criteria, it is possible to adapt the classifier and largely increase the performance. Furthermore, in some cases it is possible to speed up the processing and save computational by updating with a subset of special samples and keeping a small subset of samples for training the classifier.

The most successful modeling approach to automatic speech recognition (ASR) is to use a set of hidden Markov models (HMMs) as the acoustic models for subword or whole-word speech units and to use the statistical N-gram model as language model for words and/or word classes in sentences. All the model parameters, including HMMs and N-gram models, are estimated from a large amount of training data according to certain criterion. It has been shown that success of this kind of data-driven modeling approach highly depends on the goodness of estimated models. As for HMM-based acoustic models, the dominant estimation method is the Baum-Welch algorithm which is based on the maximum likelihood (ML) criterion. As an alternative to the ML estimation, discriminative training (DT) has also been extensively studied for HMMs in ASR. It has been demonstrated that various DT techniques, such as maximum mutual information (MMI), minimum classification error (MCE) and minimum phone error (MPE), can significantly improve speech recognition performance over the conventional maximum likelihood (ML) estimation. More recently, we have proposed the large margin estimation (LME) of HMMs for speech recognition (Li et al., 2005; Liu et al., 2005a; Li & Jiang, 2005; Jiang et al., 2006), where Gaussian mixture HMMs are estimated based on the principle of maximizing the minimum margin. From the theoretical results in machine learning (Vapnik, 1998), a large margin classifier implies a good generalization power and generally yields much lower generalization errors in new test data, as shown in support vector machine and boosting method. As in Li et al., 2005 and Li & Jiang, 2005, estimation of large margin CDHMMs turns out to be a constrained minimax optimization problem. In the past few years, several optimization methods have been proposed to solve this problem, such as iterative localized optimization in Li et al., 2005, constrained joint optimization method in Li & Jiang, 2005 and Jiang et al., 2006, and semi-definite programming (SDP) method in Li & Jiang, 2006a and Li & Jiang 2006b. In this paper, we present a general Approximation-optiMization (AM) approach to solve the LME problem of Gaussian mixture HMMs in ASR. Similar to the EM algorithm, each iteration of the AM method consists of two distinct steps: namely A-step and M-step. In A-step, the original LME problem is approximated by a simple convex optimization problem in a close proximity of initial model parameters. In M-step, the approximate convex optimization problem is solved by using efficient convex optimization algorithms.

In this paper, we propose a maximum margin classifier that deals with uncertainty in data input. More specifically, we reformulate the SVM framework such that each training example can be modeled by a multi-dimensional Gaussian distribution described by its mean vector and its covariance matrix -- the latter modeling the uncertainty. We address the classification problem and define a cost function that is the expected value of the classical SVM cost when data samples are drawn from the multi-dimensional Gaussian distributions that form the set of the training examples. Our formulation approximates the classical SVM formulation when the training examples are isotropic Gaussians with variance tending to zero. We arrive at a convex optimization problem, which we solve efficiently in the primal form using a stochastic gradient descent approach. The resulting classifier, which we name SVM with Gaussian Sample Uncertainty (SVM-GSU), is tested on synthetic data and five publicly available and popular datasets; namely, the MNIST, WDBC, DEAP, TV News Channel Commercial Detection, and TRECVID MED datasets. Experimental results verify the effectiveness of the proposed method., Comment: IEEE Transactions on Pattern Analysis and Machine Intelligence. (c) 2017 IEEE. DOI: 10.1109/TPAMI.2017.2772235 Author's accepted version. The final publication is available at http://ieeexplore.ieee.org/document/8103808/

Standard support vector machine (SVM) achieves good generalization by maximizing margin and the leading optimization problem can be solved by quadratic programming (QP). Geometrically, such margin description benefits from closed-formed Euclidian distance formula between the support vectors to the decision plane (point-to-plane) based on L2 norm. However, for non-L2 norm learning machines, such as L1- or infinite-norm, due to their non-differentiability, it is difficult to obtain close-formed point-to-plane distance and thus rarely seen large margin classifiers based on other norms in literatures. In this paper, we proposed an infinite-norm large margin classifier, termed as InfLMC. Firstly, for any given points and a plane, the foresaid close-formed distance and projection formula, based on infinite nom, are mathematically described, and then, similar to L2-SVM, infinite norm margin can be directly derived. Thus, the proposed InfLMC is constructed by maximizing margin and simultaneously minimizing experience error. Furthermore, the leading optimization problem can be solved by a linear programming problem (LP) rather than QP in standard SVM. Finally, the experimental results on some artificial and UCI datasets show its performance in test correctness and running time-consume.

Adversarial robustness has become an important research topic given empirical demonstrations on the lack of robustness of deep neural networks. Unfortunately, recent theoretical results suggest that adversarial training induces a strict tradeoff between classification accuracy and adversarial robustness. In this paper, we propose and then study a new regularization for any margin classifier or deep neural network. We motivate this regularization by a novel generalization bound that shows a tradeoff in classifier accuracy between maximizing its margin and average margin. We thus call our approach an average margin (AM) regularization, and it consists of a linear term added to the objective. We theoretically show that for certain distributions AM regularization can both improve classifier accuracy and robustness to adversarial attacks. We conclude by using both synthetic and real data to empirically show that AM regularization can strictly improve both accuracy and robustness for support vector machine's (SVM's), relative to unregularized classifiers and adversarially trained classifiers.

Cross entropy with softmax is the standard loss function for classification in neural networks. However, this function can suffer from limitations on discriminative power, lack of generalization, and propensity to overfitting. In order to address these limitations, several approaches propose to enforce a margin on the top of the neural network specifically at the softmax function. In this work, we present a novel formulation that aims to produce generalization and noise label robustness not only by imposing a margin at the top of the neural network, but also by using the entire structure of the mini-batch data. Based on the distance used for SVM to obtain maximal margin, we propose a broader distance definition called 1-to-N distance and an approximated probability function as the basis for our proposed loss function. We perform empirical experimentation on MNIST, CIFAR-10, and ImageNet32 datasets to demonstrate that our loss function has better generalization and noise label robustness properties than the traditional cross entropy method, showing improvements in the following tasks: generalization robustness, robustness in noise label data, and robustness against adversarial examples attacks.

This brief presents an energy-efficient electrocardiogram processor for arrhythmia detection with a weak-strong hybrid classifier that includes a weak linear classifier (WLC) and a strong support vector machine (SVM) classifier. WLC can only identify the beats with distinct characteristics by performing simple threshold comparisons based on beat interval feature and a novel morphology feature named QRS area ratio. The beats that are unclassified by WLC will activate the more powerful but energy-guzzling SVM classifier. Principal component analysis (PCA) is applied for feature dimension reduction to lower the complexity of SVM classifier and a sparse matrix computing architecture is exploited to reduce the computation burden of PCA. Implemented in SMIC 40LL CMOS process, the processor has a total area of 0.12 mm2. It achieves 1.98-uW power consumption in WLC mode and 3.76-uW in SVM mode under 1.1-V voltage supply and 10-KHz operating frequency, with energy dissipation of 6.8/30.3 nJ per beat classification for the two modes, respectively. The overall accuracy for MIT-BIH arrhythmia database is 98.2% with energy reduction of 41.7% compared to a single SVM classifier.

Classification is an important task in various areas. In many real-world applications, class imbalance and overlapping problems have been reported as major issues in the application of traditional classification algorithms. An imbalance problem occurs when training data contain considerably more representatives of one class than of other classes. Class overlap occurs when a region in the data space contains a similar number of data for each class. When a class overlap occurs in imbalanced data sets, classification becomes even more complicated. Although various approaches have been proposed to deal separately with class imbalance and overlapping problems, only a few studies have attempted to address both problems simultaneously. In this paper, we propose an overlap-sensitive margin (OSM) classifier based on a modified fuzzy support vector machine and k-nearest neighbor algorithm to address imbalanced and overlapping data sets. The main idea of the proposed OSM classifier is to separate the data space into soft- and hard-overlap regions using the modified fuzzy support vector machine algorithm. The separated spaces are then classified using the decision boundaries of the support vector machine and 1-nearest neighbor algorithms. Furthermore, by separating a data set into soft- and hard-overlap regions, one can determine which part of the data is to be examined more closely for classification in real-world situations. Experiments using synthetic and real-world data sets demonstrated that the proposed OSM classifier outperformed existing methods for imbalanced and overlapping situations.

Purpose The most complex organ in the human body is the brain. The unrestrained growth of cells in the brain is called a brain tumor. The cause of a brain tumor is still unknown and the survival rate is lower than other types of cancers. Hence, early detection is very important for proper treatment. Methods In this study, an efficient computer-aided diagnosis (CAD) system is presented for brain image classification by analyzing MRI of the brain. At first, the MRI brain images of normal and abnormal categories are modeled by using the statistical features of dual tree m-band wavelet transform (DTMBWT). A maximum margin classifier, support vector machine (SVM) is then used for the classification and validated with k-fold approach. Results Results show that the system provides promising results on a repository of molecular brain neoplasia data (REMBRANDT) with 97.5% accuracy using 4th level statistical features of DTMBWT. Conclusion Viewing the experimental results, we conclude that the system gives a satisfactory performance for the brain image classification.

In this study, the algorithmic implementation of multi-category nonparallel hyperplane support vector machines is described. First, a least square version of Nonparallel Hyperplane Support Vector Machine (NHSVM) is developed for binary classification problems. Solution of the primal problem corresponding to the proposed NHSVM reduces to a system of linear equations as opposed to a quadratic programming problem in NHSVM. This formulation results in a much simpler and faster approach for constructing a nonparallel hyperplane binary classifier, termed as Least Squares Nonparallel Hyperplane Support Vector Machine (LSNHSVM). Further, LSNHSVM is generalized to solve multi- category classification problems. This multi-class classifier is the named as Multicategory Least Squares Nonparallel Hyperplane Support Vector Machine (MLSNHSVM). Unlike most of the previous methods that usually cast a multi-category classification problem into a series of multiple independent binary classification problem, MLSNHSVM constructs a direct multi-category classifier by solving a system of linear equations. The proposed MLSNHSVM is in close accordance with the principle of solving multi-category problems directly . Experimental results demonstrate that MLSNHSVM has significantly higher classification accuracy as compared to other multi-class classifiers and is considerably efficient than multi-class SVM in terms of computational time.

In this paper, we discuss the distributed design for binary classification based on the nonlinear support vector machine in a time-varying multi-agent network when the training data sets are distributedly located and unavailable to all agents. In particular, the aim is to find a global large margin classifier and then enable each agent to classify any new input data into one of the two labels in the binary classification without sharing its all local data with other agents. We formulate the support vector machine problem into a distributed optimization problem in approximation and employ a distributed algorithm in a time-varying network to solve it. Our algorithm is a stochastic one with the high convergence rate and the low communication cost. With the jointly-connected connectivity condition, we analyze the consensus rate and the convergence rate of the given algorithm. Then some experimental results on various classification training data sets are also provided to illustrate the effectiveness of the given algorithm.

This paper proposes a novel twin-hyperspheres support vector machine (THSVM) classifier for binary classification, called the automatic variable-weighted THSVM (VTHSVM) classifier. By solving a single optimization problem, this classifier not only finds a pair of hyperspheres for classification, but also automatically constructs a weight vector for each class in order to describe the dissimilarity of different classes. This VTHSVM is extended to the kernel case by the fact that a kernel can be written as a sum of one’s evaluated on each variable separately. The main advantage of this method is that it allows the use of adaptive distance, which is suitable to find an as compact as possible hypersphere for each class. Experiments with synthetic and benchmark datasets indicate VTHSVM obtains better performance than some other classifiers.

We developed unconstrained large margin distribution machines (ULDMs) for pattern classification.The ULDM maximizes the margin mean and minimizes the margin variance.The ULDM can be trained by solving a set of linear equations.We clarify the characteristics of ULDMs.Performance of the ULDM is compared with that of the L1 SVM, LS SVM, and the LDM. Large margin distribution machines (LDMs) maximize the margin mean and minimize the margin variance, and show good generalization performance compared to support vector machines (SVMs). But because two additional hyperparameters are necessary, model selection needs more time. In this paper we propose unconstrained large margin distribution machines (ULDMs). In the ULDM, the objective function is the sum of the margin mean (a linear term), the margin variance (a quadratic term), and the weighted regularization term (a quadratic term), and constraints are not included. Therefore, the solution is expressed by a set of linear equations with one hyperparameter for the regularization term. Theoretical analysis proves that the decision boundary between two classes passes through the mean of all mapped training data if the numbers of training data of both classes are the same. The case where the numbers are different is analyzed for a one-dimensional input and how the decision boundary is determined is clarified. Using benchmark data sets, we show that the generalization performance of ULDMs is comparable to or better than that of SVMs.

Support vector machine (SVM) is a supervised machine learning method which can be used for both classification and regression models. In this paper, we introduce a new model of SVM and support vector regression which any of training samples containing inputs and outputs are considered the random variables with known or unknown probability functions. In this new models, we need the mathematical expectation for any of training samples but when these are unknown we apply nonparametric statistical methods. Also constraints occurrence have probability function which helps obtain maximum margin and achieve robustness. We obtain the optimal separating hyperplane and the optimal hyperplane regression by solving the quadratic optimization problems. Finally the proposed methods are illustrated by several experiments.

This paper proposes a two-step method to construct a nonlinear classifier consisting of multiple local linear classifiers interpolated with a basis function. In the first step, a geometry-based approach is first introduced to detect local linear partitions and build local linear classifiers. A coarse nonlinear classifier can then be constructed by interpolating the local linear classifiers. In the second step, a support vector machine (SVM) formulation is used to further implicitly optimize the linear parameters of the nonlinear classifier. In this way, the nonlinear classifier is constructed in exactly the same way as a standard SVM, using a special data-dependent quasi-linear kernel composed of the information of the local linear partitions. Numerical experiments on several real-world datasets demonstrate the effectiveness of the proposed classifier and show that, in cases where traditional nonlinear SVMs run into overfitting problems, the proposed classifier is effective in improving the classification performance. © 2017 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

Minimax probability machine (MPM) is an interesting discriminative classifier based on generative prior knowledge. It can directly estimate the probabilistic accuracy bound by minimizing the maximum probability of misclassification. The structural information of data is an effective way to represent prior knowledge, and has been found to be vital for designing classifiers in real-world problems. However, MPM only considers the prior probability distribution of each class with a given mean and covariance matrix, which does not efficiently exploit the structural information of data. In this paper, we use two finite mixture models to capture the structural information of the data from binary classification. For each subdistribution in a finite mixture model, only its mean and covariance matrix are assumed to be known. Based on the finite mixture models, we propose a structural MPM (SMPM). SMPM can be solved effectively by a sequence of the second-order cone programming problems. Moreover, we extend a linear model of SMPM to a nonlinear model by exploiting kernelization techniques. We also show that the SMPM can be interpreted as a large margin classifier and can be transformed to support vector machine and maxi-min margin machine under certain special conditions. Experimental results on both synthetic and real-world data sets demonstrate the effectiveness of SMPM.

Support Vector Regression (SVR) solves regression problems based on the concept of Support Vector Machine (SVM). In this paper, we introduce a novel model of SVR in which any training samples containing inputs and outputs are considered the random variables with known or unknown distribution functions. Constraints occurrence have a probability density function which helps to obtain maximum margin and achieve robustness. The optimal hyperplane regression can be obtained by solving a quadratic optimization problem. The proposed method is illustrated by several experiments including artificial data sets and real-world benchmark data sets.

One-class classification problems have been widely encountered in the fields that the negative class patterns are difficult to be collected, and the one-class support vector machine is one of the popular algorithms for solving them. However, one-class support vector machine is a vector-based learning algorithm, and it cannot work directly when the input pattern is a tensor. This paper proposes a tensor-based maximum margin classifier for one-class classification problems, and develops a One-Class Support Higher Order Tensor Machine (HO-OCSTM) which can separate most of the target patterns from the origin with the maximum margin in the higher order tensor space. HO-OCSTM directly employs the higher order tensors as the input patterns, and it is more proper for small sample study. Moreover, the direct use of tensor representation has the advantage of retaining the structural information of data, which helps improve the generalization ability of the proposed algorithm. We implement HO-OCSTM by the alternating projection method and solve a convex quadratic programming similar to the standard one-class support vector machine algorithm at each iteration. The experimental results have shown the high recognition accuracy of the proposed method.

A hybrid classifier obtained by hybridizing Support Vector Machines (SVM) and Artificial Neural Network (ANN) classifiers is presented here for diagnosis of gear faults. The distinctive features obtained from vibration signals of a running gearbox, which was operated in normal and fault-induced conditions, were used to feed the SVM-ANN hybrid classifier. Time-domain vibration signals were divided in segments. Features such as peaks in time domain and in spectrum, central moments, and standard deviations were obtained from signal segments. Based on the experimental results, it was shown that SVM-ANN hybrid classifier can successfully identify gear condition and that the hybrid SVM-ANN classifier performs much better than standard versions of ANNs and SVM. The effectiveness of the hybrid classifier under noise was also investigated. It was shown that if vibration signals are preprocessed by Discrete Wavelet Transform (DWT), efficacy of the SVM-ANN hybrid is significantly enhanced.

This paper proposed a new framework of one-class classification based on continuous cognition.The framework is implemented with LSSVM and the corresponding classifier is called OC-LSSVM.Several simulation and real datasets are used to test the performance of OC-LSSVM.OC-LSSVM shows state-of-the-art performance compared to established methods. One-class classification is a basic problem in machine learning. Unlike the existing typical one-class classifiers designed from the angle of probability or geometric, this paper attempts to study this problem from the bionics point of view. Using the continuous cognition characteristic as the starting point, we propose a new framework of one-class classifier, named multiple regression model (OC-MR), which can be seen as a natural extension of multiple regression for one-class classification problem. This paper applies least squares support vector machine (LSSVM) as an example to show themodeling process of the proposed method and the corresponding one-class classifier is named one-class least squares support vector machine (OC-LSSVM). Various simulation and real-life datasets are used to test the performance of the proposed OC-LSSVM. The existing popular one-class classification methods including Parzen kernel density estimation, support vector data description and Gaussian mixture model are also applied in order to achieve a comprehensive comparison. The results show that OC-LSSVM has achieved the best performance in most of the simulation and real-life datasets due to its good robustness, which highlights the efficacy of OC-LSSVM.

There is an intrinsic adversarial nature in the security domain such as spam filtering and malware detection systems that attempt to mislead the detection system. This adversarial nature makes security applications different from the classical machine learning problems; for instance, an adversary (attacker) might change the distribution of test data and violate the data stationarity, a common assumption in machine learning techniques. Since machine learning methods are not inherently adversary-aware, a classifier designer should investigate the robustness of a learning system under attack. In this respect, recent studies have modeled the identified attacks against machine learning-based detection systems. Based on this, a classifier designer can evaluate the performance of a learning system leveraging the modeled attacks. Prior research explored a gradient-based approach in order to devise an attack against a classifier with differentiable discriminant function like SVM. However, there are several powerful classifiers with non-differentiable decision boundary such as Random Forest, which are commonly used in different security domain and applications. In this paper, we present a novel approach to model an attack against classifiers with non-differentiable decision boundary. In the experimentation, we first present an example that visually shows the effect of a successful attack on the MNIST handwritten digits classification task. Then we conduct experiments for two well-known applications in the security domain: spam filtering and malware detection in PDF files. The experimental results demonstrate that the proposed attack successfully evades Random Forest classifier and effectively degrades the classifier’s performance.

We consider the problem of how to design and implement communication-efficient versions of parallel kernel support vector machines, a widely used classifier in statistical machine learning, for distributed memory clusters and supercomputers. The main computational bottleneck is the training phase, in which a statistical model is built from an input data set. Prior to our study, the parallel isoefficiency of a state-of-the-art implementation scaled as $W=\Omega (P^3)$ , where $W$ is the problem size and $P$ the number of processors; this scaling is worse than even a one-dimensional block row dense matrix vector multiplication, which has $W=\Omega (P^2)$ . This study considers a series of algorithmic refinements, leading ultimately to a Communication-Avoiding SVM method that improves the isoefficiency to nearly $W=\Omega (P)$ . We evaluate these methods on 96 to 1,536 processors, and show average speedups of $3-16\times$ ( $7\times$ on average) over Dis-SMO, and a 95 percent weak-scaling efficiency on six real-world datasets, with only modest losses in overall classification accuracy. The source code can be downloaded at [1] .

Malaria, being a life-threatening disease caused by parasites, demands its rapid and accurate diagnosis. In this paper, we develop a computer-assisted malaria-infected life-cycle stages classification based on a hybrid classifier using thin blood smear images. The major issues are: feature extraction, feature selection and classification of erythrocytes infected with different life-cycle stages of malaria. Feature set (134 dimensional features) has been defined by the combination of the proposed features along with the existing features. Features such as prediction error, co-occurrence of linear binary pattern, chrominance channel histogram, R–G color channel difference histogram and Gabor features are the newly proposed features in our system. In the feature selection, a two-stage algorithm utilizing the filter method to rank the feature, along with the incremental feature selection technique, has been analyzed. Moreover, the performance of all the individual classifiers (Naive Bayes, support vector machine, k-nearest neighbors and artificial neural network) is evaluated. Finally, the three individual classifiers are combined to develop a hybrid classifier using different classifier combining techniques. From the experimental results, it may be concluded that hybrid classifier formed by the combination of SVM, k-NN and ANN with majority voting technique provides satisfactory results compared to other individual classifiers as well as other hybrid model. An accuracy of 96.54 ± 0.73% has been achieved on the collected clinical database. The results show an improvement in accuracy (11.62, 6.7, 3.39 and 2.39%) as compared to the state-of-the-art individual classifiers, i.e., Naive Bayes, SVM, k-NN and ANN, respectively.

The traditional data uncertainty (DU) classifier fails to encode the importance of each sample for solving the minimum problem. Moreover, it considers only linear information for classification. To overcome these, we propose four classifiers for action recognition. They are called regularized DU (RDU) classifier, RDU coefficient (RDUC) classifier, kernel RDU (KRDU) classifier, and kernel RDUC (KRDUC) classifier, respectively. Extensive experiments on four benchmark action databases demonstrate that the proposed four classifiers achieve better recognition rates than the traditional DU classifier and several state-of-the-art methods. Moreover, the computation costs of the KRDU and KRDUC classifiers are much less than that of the DU classifier.

Sample mean is generally the simplest feasible description of samples.Margin distribution can characterize the generalization performance.Double Distribution Support Vector Machine (DDSVM) maximizes the margin distribution of two classes sample means.DDSVM can obtain efficient and more balanced classification performance. This paper studies the role of the sample mean in binary classifier based on the margin theory. Support Vector Machine (SVM) with maximized minimum margin is widely used in pattern recognition, but it sometimes induces the weak margin distribution which is negative for the generalization performance. Therefore Double Distribution Support Vector Machine (DDSVM) is proposed to obtain strong generalization performance by maximizing the margin distribution of two classes sample means and the minimum margin. The sample mean is usually a good description of samples, and DDSVM can increase the margin distribution and improve the generalization performance. DDSVM is a general learning approach, and its superiority is verified both theoretically and experimentally.

Support vector machine initially developed to perform binary classification. This paper presents a multi-class support vector machine classifier and ordinal regression to classify the type of bone mineral density. This paper compares the performance of four multi-class approaches, one-against-all, one-against-one, Weston and Watkins, and Crammer and Singer. Results from our real life data conclude that Crammer and Singer may be better approach depending on training error and the percentage of correctly classified test data. Also, we fined that the training error become more less when the regulization parameter and kernel parameter become large.

The support vector machine (SVM) is a state-of-the-art method in supervised classification. In this paper the Cluster Support Vector Machine (CLSVM) methodology is proposed with the aim to increase the sparsity of the SVM classifier in the presence of categorical features, leading to a gain in interpretability. The CLSVM methodology clusters categories and builds the SVM classifier in the clustered feature space. Four strategies for building the CLSVM classifier are presented based on solving: the SVM formulation in the original feature space, a quadratically constrained quadratic programming formulation, and a mixed integer quadratic programming formulation as well as its continuous relaxation. The computational study illustrates the performance of the CLSVM classifier using two clusters. In the tested datasets our methodology achieves comparable accuracy to that of the SVM in the original feature space, with a dramatic increase in sparsity.

This paper proposes a new differential particle swarm optimization (DPSO) method for obtaining optimum support vector machine (SVM) parameters used for electrical fault diagnosis in radial distribution systems. Further, a multiple-stage DPSO-SVM classifier is developed to enhance classification accuracy in the fault diagnosis. Also, time-domain reflectometry (TDR) method with pseudo-random binary sequence (PRBS) excitation is utilized for generating the dataset required for validating this proposed approach. According to the characteristic of echo responses found in different types of faults, 12 features are extracted as input vectors for purposes of classification. The proposed fault diagnosis approach is tested on a typical radial distribution system to classify ten types of short-circuit faults accurately. Further, to demonstrate the superiority of the proposed DPSO algorithm, comparative studies of fault diagnosis are performed using SVM having parameters selected using cross-validation, GA and PSO. The overall classification accuracy obtained for fault diagnosis is 98.5%, which shows the effectiveness of the proposed approach.

Semi-supervised learning is an attractive method in classification problems when insufficient training information is available. In this investigation, a new semi-supervised classifier is proposed based on the concept of maximum vector-angular margin, (called S$^3$MAMC), the main goal of which is to find an optimal vector $c$ as close as possible to the center of the dataset consisting of both labeled samples and unlabeled samples. This makes S$^3$MAMC better generalization with smaller VC (Vapnik-Chervonenkis) dimension. However, S$^3$MAMC formulation is a non-convex model and therefore it is difficult to solve. Following that we present two optimization algorithms, mixed integer quadratic program (MIQP) and DC (difference of convex functions) program algorithms, to solve the S$^3$MAMC. Compared with the supervised learning methods, numerical experiments on real and synthetic databases demonstrate that the S$^3$MAMC can improve generalization when the labelled samples are relatively few. In addition, the S$^3$MAMC has competitive experiment results in generalization compared to the traditional semi-supervised classification methods.

We propose a method for improving the classification performance of a classifier, termed the classifier graph, by embedding it in a graph of classifiers. Our graph-based method has the advantage of enabling delicate classification from different levels of interpretation and abstraction. For the problem that thresholds corresponding to different classifiers are correlated and thus have mutual effects on the final performance, we provide a generalization of the receiver operator characteristic curve that properly tunes them jointly to obtain optimal performance. This method is successfully applied to the detection of noise artifacts (glitches) in the gravitational-wave data. We thus obtain an improvement up to 10% on the classification performance compared with that of a single classifier. The methods of this paper provide an effective way to improve the classification performance with multiple classifiers.

Many pattern recognition problems involve characterizing samples with continuous labels instead of discrete categories. While regression models are suitable for these learning tasks, these labels are often discretized into binary classes to formulate the problem as a conventional classification task (e.g., classes with low versus high values). This methodology brings intrinsic limitations on the classification performance. The continuous labels are typically normally-distributed, with many samples close to the boundary threshold, resulting in poor classification rates. Previous studies only use the discretized labels to train binary classifiers, neglecting the original, continuous labels. This study demonstrates that, even in binary classification problems, exploiting the original labels before splitting the classes can lead to better classification performance. This work proposes an optimal classifier based on the Bayesian maximum a posterior (MAP) criterion for these problems, which effectively utilizes the real-valued labels. We derive the theoretical average performance of this classifier, which can be considered as the expected upper bound performance for the task. Experimental evaluations on synthetic and real data sets show the improvement achieved by the proposed classifier, in contrast to conventional classifiers trained with binary labels. These evaluations clearly demonstrate the optimality of the proposed classifier, and the precision of the expected upper bound obtained by our derivation.

Motivation Protein fold classification is a critical step in protein structure prediction. There are two possible ways to classify protein folds. One is through template-based fold assignment and the other is ab-initio prediction using machine learning algorithms. Combination of both solutions to improve the prediction accuracy was never explored before. Results We developed two algorithms, HH-fold and SVM-fold for protein fold classification. HH-fold is a template-based fold assignment algorithm using the HHsearch program. SVM-fold is a support vector machine-based ab-initio classification algorithm, in which a comprehensive set of features are extracted from three complementary sequence profiles. These two algorithms are then combined, resulting to the ensemble approach TA-fold. We performed a comprehensive assessment for the proposed methods by comparing with ab-initio methods and template-based threading methods on six benchmark datasets. An accuracy of 0.799 was achieved by TA-fold on the DD dataset that consists of proteins from 27 folds. This represents improvement of 5.4–11.7% over ab-initio methods. After updating this dataset to include more proteins in the same folds, the accuracy increased to 0.971. In addition, TA-fold achieved >0.9 accuracy on a large dataset consisting of 6451 proteins from 184 folds. Experiments on the LE dataset show that TA-fold consistently outperforms other threading methods at the family, superfamily and fold levels. The success of TA-fold is attributed to the combination of template-based fold assignment and ab-initio classification using features from complementary sequence profiles that contain rich evolution information. Availability and Implementation http://yanglab.nankai.edu.cn/TA-fold/ Supplementary information Supplementary data are available at Bioinformatics online.

Recently, a class of classifiers, called relative margin machine, has been developed. Relative margin machine has shown significant improvements over the large margin counterparts on real-world problems. In binary classification, the most widely used loss function is the hinge loss, which results in the hinge loss relative margin machine. The hinge loss relative margin machine is sensitive to outliers. In this article, we proposed to change maximizing the shortest distance used in relative margin machine into maximizing the quantile distance, the pinball loss which is related to quantiles was used in classification. The proposed method is less sensitive to noise, especially the feature noise around the decision boundary. Meanwhile, the computational complexity of the proposed method is similar to that of the relative margin machine.

In statistical process monitoring, data mining algorithms are applied for control chart pattern recognition (CCPR) not only to detect but also to identify abnormal patterns associated with assignable causes. Recently, the use of support vector machines (SVMs) has achieved remarkable results in statistical process control applications due to its excellent generalization performance. Although there is a lot of research that highlights the superiority of support vector machine over other algorithms used in systems based on data mining control, there are no studies that analyze the design elements of a SVM classifier. Consequently, a comprehensive review, classification and an analysis of information found in the literature are presented in this paper regarding to the SVM classifier. The aim of this research is to provide researchers a starting point to potentiate the performance of the SVM classifier for assuring the best possible classification and improving the detection efficiency. Sixty-one research articles from 2001 to 2015 are critically analyzed based on their methodology following a classification scheme derived from the support vector classification framework. The analysis showed that the feature extraction and selection play a crucial role on the performance of classifiers. Studies revealed that using extracted features or a combination of raw data and feature extraction achieves better classification performances than using only raw data as an input. On the other hand, there is an ample gap to extend the research on internal structure configurations under various types of kernels, multi-classification, and ensemble approaches. Finally, this paper evidences that the application of nature inspired algorithms for kernel parameter selection in auto-correlated SVM-based process monitoring systems remains unexplored.

This study discusses the optimal design of an automatic inspection system for processing light-emitting diode (LED) chips. Based on support vector machine (SVM) with optimal theory, the classifications of micro-defects in light area and electrode area on the chip surface, and develop a robust classification module will be analyzed. In order to design the SVM-based defect classification system effectively, the multiple quality characteristics parameter design. The Taguchi method is used to improve the classifier design, and meanwhile, PCA is used for analysis of multiple quality characteristics on influence of characteristics on multi-class intelligent classifier, to regularly select effective features, and reduce classification data. Aim to reduce the classification data and dimensions, and with features containing higher score of principal component as decision tree support vector machine classification module training basis, the optimal multi-class support vector machine model was established for subdivision of micro-defects of electrode area and light area. The comparison of traditional binary structure support vector machine and neural network classifier was conducted. The overall recognition rate of the inspection system herein was more than 96%, and the classification speed for 500 micro-defects was only 3 s. It is clear that we have effectively established an inspection process, which is highly effective even under disturbance. The process can realize the subdivision of micro-defects, and with quick classification, high accuracy, and high stability. It is applicable to precise LED detection and can be used for accurate inspection of LED of mass production effectively to replace visual inspection, economizing on labor cost.

Support vector machines (SVMs) play a very dominant role in data classification because of their good generalization performance. However, they suffer from the high computational complexity in the classification stage when there are a considerable number of support vectors (SVs). It is desirable to design efficient algorithms in the classification stage to deal with datasets obtained from real-time pattern recognition systems. To this end, we propose a novel classifier called HMLSVMs (Hierarchical Mixing Linear Support Vector Machines), which has a hierarchical structure with a mixing linear SVM classifier at each node. It predicts the label of a sample using only a few hyperplanes. We also give a generalization error bound for the class of locally linear SVMs (LLSVMs) based on the Rademacher theory, which ensures that overfitting can be effectively avoided. Experimental evaluations show that the proposed classifier achieves a high efficiency in the classification stage, while the classification performance approaches that of kernel SVMs. HighlightsH-MLSVMs does not require a large memory to store kernel values in the training process, because H-MLSVMs is composed of LSVMs, and there are very efficient algorithms for training LSVMs 48,49.The hierarchical structure makes H-MLSVMs predict the labels of new arrived samples via a few of LSVMs, and it is much faster than nonlinear SVMs classifiers.We quantify the generalization error bound for the class of LLSVMs based on the Rademacher complexity, and the stop criterion based on minimizing this bound ensures that H-MLSVMs can effectively avoid overfitting and have a good classification performance.

Extreme Learning Machine provides very competitive performance to other related classical predictive models for solving problems such as regression, clustering, and classification. An ELM possesses the advantage of faster computational time in both training and testing. However, one of the main challenges of an ELM is the selection of the optimal number of hidden nodes. This paper presents a new approach to node selection of an ELM based on a 1-norm support vector machine (SVM). In this method, the targets of SVM yi ∈{+1, –1} are derived using the mean or median of ELM training errors as a threshold for separating the training data, which are projected to SVM dimensions. We present an integrated architecture that exploits the sparseness in solution of SVM to prune out the inactive hidden nodes in ELM. Several experiments are conducted on real-world benchmark datasets, and the results attained attest to the efficiency of the proposed method.

Purpose The purpose of this paper is to tune support vector machine (SVM) classifier using grey wolf optimizer (GWO). Design/methodology/approach The schema of the work aims at extracting the features from the collected data followed by a SVM classifier and metaheuristic optimization to tune the classifier parameters. Findings The optimal tuning of classifier parameters lowers errors due to manual elucidation and decreases the risk in human perceptions and repeated visual dignosis. Originality/value A novel, GWO based tuning algorithm is used for SVM classifier, which is implemented in classifying the complex and nonlinear biomedical signals like intracranial electroencephalogram.

Display Omitted Detection of perfumes by using machine learning algorithms.Fuzzy clustering c-mean: type-1 (FCM).Support vector machines (SVM).Soft computing techniques.Electronic (or artificial) nose. Identification of more than three perfumes is very difficult for the human nose. It is also a problem to recognize patterns of perfume odor with an electronic nose that has multiple sensors. For this reason, a new hybrid classifier has been presented to identify type of perfume from a closely similar data set of 20 different odors of perfumes. The structure of this hybrid technique is the combination of unsupervised fuzzy clustering c-mean (FCM) and supervised support vector machine (SVM). On the other hand this proposed soft computing technique was compared with the other well-known learning algorithms. The results show that the proposed hybrid algorithm's accuracy is 97.5% better than the others.

The present training method of support vector machine has large computational cost. Integrating SVM's geometrical interpretation and structural risk minimization, a new fast support vector machine: Centroid Normal Direct Support Vector Machine (CNDSVM) was proposed. The normal vector of optimal separating hyperplane was determined by the connective direction of centroids. The optimal separating hyperplane was directly obtained through the distributing of the samples normal projection. The experiments on artificial data sets and benchmarks data sets show that the method can get almost the same classify result as SVM and LSSVM with very low computational cost.

This paper presents the Quaternion Support Vector Machines for classification as a generalization of the real- and complex- valued Support Vector Machines. In this framework we handle the design of kernels involving the Clifford or quaternion product. The QSVM allows to change the metric involved in the quaternion product. The application section shows experiments in pattern recognition and colour image processing.

We propose a lightweight sparsity-based algorithm, namely, the basic thresholding classifier (BTC), for hyperspectral image (HSI) classification. BTC is a pixelwise classifier which uses only the spectral features of a given test pixel. It performs the classification using a predetermined dictionary consisting of labeled training pixels. It then produces the class label and residual vector of the test pixel. Since incorporating spatial and spectral information in HSI classification is quite an effective way of improving classification accuracy, we extend our proposal to a three-step spatial–spectral framework. First, every pixel of a given HSI is classified using BTC. The resulting residual vectors form a cube which could be interpreted as a stack of images representing residual maps. Second, each residual map is filtered using an averaging filter. Finally, the class label of each test pixel is determined based on minimal residual. Numerical results on public data sets show that our proposal outperforms well-known support vector machine-based techniques and sparsity-based greedy approaches like simultaneous orthogonal matching pursuit in terms of both classification accuracy and computational cost.

Aim of this study: Bone age assessment and X-ray assessment on hand radiographs is an expensive and time consuming process in radiology. This research work presents a powerful Graphical User Interface (GUI) for IRMA that gives exceptional medical image retrieval results. Methods: This study gives complete details regarding the execution of a novel Discriminative Dictionary Learning (DDL) for the purpose of matching query and input medical image samples. In order to further boost the quality of the medical images, Principal Component Analysis (PCA) based noise level estimation scheme is introduced. This helps to eliminate noises included in the medical images. Furthermore, to enhance the performance, a new relevance score scheme via Kernel Support Vector Machine (KSVM) is formulated, which efficiently make use of medical images features in order to discriminate the images accurately. This system initiated a new discriminative name called ‘pairwise similarity measurement’ for the discriminativeness of pairs of query and input medical database images and subsequently integrate it with the classification error for discriminativeness in classifier production to generate a unified objective function. Results: Experimentation results assessed the proposed DDL in IRMA framework on a numerous medical images like BAA images and X-Ray skull images and results confirm that DDL performs far better than the state-of the-art approaches significantly in terms of precision, recall, sensitivity, specificity and accuracy. Conclusion: The resulting DDL has a considerable impact on medical CBIR applications.

This paper proposes a novel L1-norm loss based twin support vector machine (L1LTSVM) classifier for binary recognition. In this L1LTSVM, each optimization problem simultaneously minimizes the L1-norm based losses for the two classes of points, which results in a different dual problem compared with twin support vector machine (TWSVM). Compared with TWSVM, the main advantages of this L1LTSVM classifier are: first, the dual problems of L1LTSVM do not need to inverse the kernel matrices during the learning process, indicating L1LTSVM not only has a partly sparse decision function, but also can be solved efficiently by some SVM-type learning algorithms, and then is suitable for large scale problems. Second, this L1LTSVM has more perfect and practical geometric interpretation. Experimental results on several synthetic as well as benchmark datasets indicate the significant advantage of L1LTSVM in the generalization performance.

Background Electroencephalogram (EEG) and electromyogram (EMG) recordings are often used in rodents to study sleep architecture and sleep-associated neural activity. These recordings must be scored to designate what sleep/wake state the animal is in at each time point. Manual sleep-scoring is very time-consuming, so machine-learning classifier algorithms have been used to automate scoring. New method Instead of using single classifiers, we implement a multiple classifier system. The multiple classifier is built from six base classifiers: decision tree, k-nearest neighbors, naive Bayes, support vector machine, neural net, and linear discriminant analysis. Decision tree and k-nearest neighbors were improved into ensemble classifiers by using bagging and random subspace. Confidence scores from each classifier were combined to determine the final classification. Ambiguous epochs can be rejected and left for a human to classify. Results Support vector machine was the most accurate base classifier, and had error rate of 0.054. The multiple classifier system reduced the error rate to 0.049, which was not significantly different from a second human scorer. When 10% of epochs were rejected, the remaining epochs’ error rate dropped to 0.018. Comparison with existing method(s) Compared with the most accurate single classifier (support vector machine), the multiple classifier reduced errors by 9.4%. The multiple classifier surpassed the accuracy of a second human scorer after rejecting only 2% of epochs. Conclusions Multiple classifier systems are an effective way to increase automated sleep scoring accuracy. Improvements in autoscoring will allow sleep researchers to increase sample sizes and recording lengths, opening new experimental possibilities.

In this paper, we mainly focus on two issues (1) SVM is very sensitive to noise. (2) The solution of SVM does not take into consideration of the intrinsic structure and the discriminant information of the data. To address these two problems, we first propose an integration model to integrate both the local manifold structure and the local discriminant information into l1 graph embedding. Then we add the integration model into the objection function of υ-support vector machine. Therefore, a discriminant sparse neighborhood preserving embedding υ-support vector machine (υ-DSNPESVM) method is proposed. The theoretical analysis demonstrates that υ-DSNPESVM is a reasonable maximum margin classifier and can obtain a very lower generalization error upper bound by minimizing the integration model and the upper bound of margin error. Moreover, in the nonlinear case, we construct the kernel sparse representation-based l1 graph for υ-DSNPESVM, which is more conducive to improve the classification accuracy than l1 graph constructed in the original space. Experimental results on real datasets show the effectiveness of the proposed υ-DSNPESVM method.

Black-box classifiers, such as artificial neural network and support vector machine, are a popular classifier because of its remarkable performance. They are applied in various fields such as inductive inferences, classifications, or regressions. However, by its characteristics, they cannot provide appropriate explanations how the classification results are derived. Therefore, there are plenty of actively discussed researches about interpreting trained black-box classifiers. In this paper, we propose a method to make a fuzzy logic-based classifier using extracted rules from the artificial neural network and support vector machine in order to interpret internal structures. As an object of classification, an anomalous propagation echo is selected which occurs frequently in radar data and becomes the problem in a precipitation estimation process. After applying a clustering method, learning dataset is generated from clusters. Using the learning dataset, artificial neural network and support vector machine are implemented. After that, decision trees for each classifier are generated. And they are used to implement simplified fuzzy logic-based classifiers by rule extraction and input selection. Finally, we can verify and compare performances. With actual occurrence cased of the anomalous propagation echo, we can determine the inner structures of the black-box classifiers.

This study proposes a new method for content-based image retrieval by finding an optimal classifier. The optimal classifier is achieved by a new active learning support vector machine (SVM) which combines the model selection with the active learning. The unlabelled samples close to the boundary of the SVM classifier are selected based on the feature similarity for the active learning, and the adaptive regularisation is used to select the optimal model. The combination of model selection with active learning accelerates the convergence of the classifier. The new method can improve the image retrieval accuracy and reduce the time consumption. The experimental results show that the proposed method has a better performance with fewer samples and less time consumption for image retrieval.

This paper attempts to combine Laplacian Twin Support Vector Machine (Lap-TWSVM) and Decision Tree Twin Support Vector Machine classifier to obtain an effective tree based classifier for semi-supervised multi-category classification. This classifier is termed as Tree Based Multi-Category Laplacian Twin Support Vector Machine (TB-Lap-TWSVM) classifier. TB-Lap-TWSVM is an improved One Against All (OAA) partition based tree classifier which takes advantage of both Lap-TWSVM and Decision Tree methodology. This paper concentrates on the application of TB-Lap-TWSVM to Content Based Image Retrieval problem. Further, extensive experiments on color images are carried out on different databases to establish the efficacy of the proposed model vis a vis TB-Lap-SVM and OAA-Lap-TWSVM.