Search

Optimizing the antecedent part of neurofuzzy system is an active research topic, for which different approaches have been developed. However, current approaches typically suffer from high computational complexity or lack of ability to extract knowledge from a given set of training data. In this paper, we introduce a novel incremental training algorithm for the class of neurofuzzy systems that are structured based on local linear classifiers. Linear discriminant analysis is utilized to transform the data into a space in which linear discriminancy of training samples is maximized. The neurofuzzy classifier is then built in the transformed space, starting from the simplest form (a global linear classifier). If the overall performance of the classifier was not satisfactory, it would be iteratively refined by incorporating additional local classifiers. In addition, rule consequent parameters are optimized using a local least square approach. Our refinement strategy is motivated by LOLIMOT, which is a greedy partition algorithm for structure training and has been successfully applied in a number of identification problems. The proposed classifier is compared to several benchmark classifiers on a number of well-known datasets. The results prove the efficacy of the proposed classifier in achieving high performance while incurring low computational effort.

Ensemble machine learning methods are frequently used for classification problems and it is known that they may boost the prediction accuracy. Support Vector Machines are widely used as base classifiers during the construction of different types of ensembles. In this study, we have constructed a weighted functional margin classifier ensemble on panel financial ratios to discriminate between solid and unhealthy banks for Turkish commercial bank case. We proposed a novel ensemble generation method enhanced by a pruning strategy to increase the prediction performance and developed a novel aggregation approach for ensemble learning by using the idea of weighted sums. The prediction performances are compared with a panel logistic regression which is considered a benchmark method for panel data. The results show that the proposed ensemble method is more successful than the straight SVM and the classical generalized linear model approach.

Classification is an essential task in the field of machine learning, where finding a maximum margin classifier is one of its central problems. In this work, an evolutionary algorithm is constructed, relying on the convexity properties of the version space, to evolve a population of perceptron classifiers in order to find a solution that approximates the maximum margin. Unlike other methods whose solutions explore the problem’s dual formulation, usually requiring the solution of a linear constraint quadratic programming problem, the proposed method requires only the evaluation of the margin values. Hyperspherical coordinates are used to guarantee feasibility when generating new individuals and for the population to be uniformly distributed through the search space. To control the number of generations, we developed a stop criteria based on a lower bound function which asymptotically approximates the margin curves providing a stop margin that satisfies a $$\beta $$ -approximation of the optimal margin. Experiments were performed on artificial and real datasets, and the obtained results indicate the potential to adopt the proposed algorithm for solving practical problems.

It is well known that there is an increasing interest in edge computing to reduce the distance between cloud and end devices, especially for machine learning (ML) methods. However, when related to latency-sensitive applications, little work can be found in ML literature on suitable embedded systems implementations. This article presents new ways to implement the decision rule of a large margin classifier based on Gabriel graphs as well as an efficient implementation of this on an embedded system. The proposed approach uses the nearest neighbor method as the decision rule, and the implementation starts from an RTL pipeline architecture developed for binary large margin classifiers and proposes the integration in a hardware/software co-design. Results showed that the proposed approach was statistically similar to the classifier and had a speedup factor of up to eight times compared to the classifier executed in software, with performance suitable for ML latency-sensitive applications.

Objective: Classifier transfers usually come with dataset shifts. To overcome them, online strategies have to be applied. For practical applications, limitations in the computational resources for the adaptation of batch learning algorithms, like the SVM, have to be considered. Approach: We review and compare several strategies for online learning with SVMs. We focus on data selection strategies which limit the size of the stored training data [...] Main Results: For different data shifts, different criteria are appropriate. For the synthetic data, adding all samples to the pool of considered samples performs often significantly worse than other criteria. Especially, adding only misclassified samples performed astoundingly well. Here, balancing criteria were very important when the other criteria were not well chosen. For the transfer setups, the results show that the best strategy depends on the intensity of the drift during the transfer. Adding all and removing the oldest samples results in the best performance, whereas for smaller drifts, it can be sufficient to only add potential new support vectors of the SVM which reduces processing resources. Significance: For BCIs based on EEG models, trained on data from a calibration session, a previous recording session, or even from a recording session with one or several other subjects, are used. This transfer of the learned model usually decreases the performance and can therefore benefit from online learning which adapts the classifier like the established SVM. We show that by using the right combination of data selection criteria, it is possible to adapt the classifier and largely increase the performance. Furthermore, in some cases it is possible to speed up the processing and save computational by updating with a subset of special samples and keeping a small subset of samples for training the classifier.

Nowadays, datasets are always dynamic and patterns in them are changing. Instances with different labels are intertwined and often linearly inseparable, which bring new challenges to traditional learning algorithms. This paper proposes adaptive hyper-sphere (AdaHS), an adaptive incremental classifier, and its kernelized version: Nys-AdaHS. The classifier incorporates competitive training with a border zone. With adaptive hidden layer and tunable radii of hyper-spheres, AdaHS has strong capability of local learning like instance-based algorithms, but free from slow searching speed and excessive memory consumption. The experiments showed that AdaHS is robust, adaptive, and highly accurate. It is especially suitable for dynamic data in which patterns are changing, decision borders are complicated, and instances with the same label can be spherically clustered.

Based on the constrained non-negative matrix factor algorithm, the article presents a new approach to facial recognition recognition. Our proposed method incorporated two tasks in an automatic expression analysis system: facial feature extraction and classification into expressions. To obtain local and geometric structure information in the data as much as possible, we amalgamate max-margin relegation into the constrained NMF optimization, resulting in a multiplicative updating algorithm is additionally proposed for solving optimization quandary. Experimental results on JAFFE dataset demonstrate that the effectiveness of the proposed method with improved performances over the conventional dimension reduction methods.

The most successful modeling approach to automatic speech recognition (ASR) is to use a set of hidden Markov models (HMMs) as the acoustic models for subword or whole-word speech units and to use the statistical N-gram model as language model for words and/or word classes in sentences. All the model parameters, including HMMs and N-gram models, are estimated from a large amount of training data according to certain criterion. It has been shown that success of this kind of data-driven modeling approach highly depends on the goodness of estimated models. As for HMM-based acoustic models, the dominant estimation method is the Baum-Welch algorithm which is based on the maximum likelihood (ML) criterion. As an alternative to the ML estimation, discriminative training (DT) has also been extensively studied for HMMs in ASR. It has been demonstrated that various DT techniques, such as maximum mutual information (MMI), minimum classification error (MCE) and minimum phone error (MPE), can significantly improve speech recognition performance over the conventional maximum likelihood (ML) estimation. More recently, we have proposed the large margin estimation (LME) of HMMs for speech recognition (Li et al., 2005; Liu et al., 2005a; Li & Jiang, 2005; Jiang et al., 2006), where Gaussian mixture HMMs are estimated based on the principle of maximizing the minimum margin. From the theoretical results in machine learning (Vapnik, 1998), a large margin classifier implies a good generalization power and generally yields much lower generalization errors in new test data, as shown in support vector machine and boosting method. As in Li et al., 2005 and Li & Jiang, 2005, estimation of large margin CDHMMs turns out to be a constrained minimax optimization problem. In the past few years, several optimization methods have been proposed to solve this problem, such as iterative localized optimization in Li et al., 2005, constrained joint optimization method in Li & Jiang, 2005 and Jiang et al., 2006, and semi-definite programming (SDP) method in Li & Jiang, 2006a and Li & Jiang 2006b. In this paper, we present a general Approximation-optiMization (AM) approach to solve the LME problem of Gaussian mixture HMMs in ASR. Similar to the EM algorithm, each iteration of the AM method consists of two distinct steps: namely A-step and M-step. In A-step, the original LME problem is approximated by a simple convex optimization problem in a close proximity of initial model parameters. In M-step, the approximate convex optimization problem is solved by using efficient convex optimization algorithms.

Distance weighted discrimination (DWD) is an interesting large margin classifier that has been shown to enjoy nice properties and empirical successes. The original DWD only handles binary c...

In this paper, we propose a maximum margin classifier that deals with uncertainty in data input. More specifically, we reformulate the SVM framework such that each training example can be modeled by a multi-dimensional Gaussian distribution described by its mean vector and its covariance matrix -- the latter modeling the uncertainty. We address the classification problem and define a cost function that is the expected value of the classical SVM cost when data samples are drawn from the multi-dimensional Gaussian distributions that form the set of the training examples. Our formulation approximates the classical SVM formulation when the training examples are isotropic Gaussians with variance tending to zero. We arrive at a convex optimization problem, which we solve efficiently in the primal form using a stochastic gradient descent approach. The resulting classifier, which we name SVM with Gaussian Sample Uncertainty (SVM-GSU), is tested on synthetic data and five publicly available and popular datasets; namely, the MNIST, WDBC, DEAP, TV News Channel Commercial Detection, and TRECVID MED datasets. Experimental results verify the effectiveness of the proposed method., Comment: IEEE Transactions on Pattern Analysis and Machine Intelligence. (c) 2017 IEEE. DOI: 10.1109/TPAMI.2017.2772235 Author's accepted version. The final publication is available at http://ieeexplore.ieee.org/document/8103808/

Standard support vector machine (SVM) achieves good generalization by maximizing margin and the leading optimization problem can be solved by quadratic programming (QP). Geometrically, such margin description benefits from closed-formed Euclidian distance formula between the support vectors to the decision plane (point-to-plane) based on L2 norm. However, for non-L2 norm learning machines, such as L1- or infinite-norm, due to their non-differentiability, it is difficult to obtain close-formed point-to-plane distance and thus rarely seen large margin classifiers based on other norms in literatures. In this paper, we proposed an infinite-norm large margin classifier, termed as InfLMC. Firstly, for any given points and a plane, the foresaid close-formed distance and projection formula, based on infinite nom, are mathematically described, and then, similar to L2-SVM, infinite norm margin can be directly derived. Thus, the proposed InfLMC is constructed by maximizing margin and simultaneously minimizing experience error. Furthermore, the leading optimization problem can be solved by a linear programming problem (LP) rather than QP in standard SVM. Finally, the experimental results on some artificial and UCI datasets show its performance in test correctness and running time-consume.

We study the implicit bias of gradient descent methods in solving a binary classification problem over a linearly separable dataset. The classifier is described by a nonlinear ReLU model and the objective function adopts the exponential loss function. We first characterize the landscape of the loss function and show that there can exist spurious asymptotic local minima besides asymptotic global minima. We then show that gradient descent (GD) can converge to either a global or a local max-margin direction, or may diverge from the desired max-margin direction in a general context. For stochastic gradient descent (SGD), we show that it converges in expectation to either the global or the local max-margin direction if SGD converges. We further explore the implicit bias of these algorithms in learning a multi-neuron network under certain stationary conditions, and show that the learned classifier maximizes the margins of each sample pattern partition under the ReLU activation.

Adversarial robustness has become an important research topic given empirical demonstrations on the lack of robustness of deep neural networks. Unfortunately, recent theoretical results suggest that adversarial training induces a strict tradeoff between classification accuracy and adversarial robustness. In this paper, we propose and then study a new regularization for any margin classifier or deep neural network. We motivate this regularization by a novel generalization bound that shows a tradeoff in classifier accuracy between maximizing its margin and average margin. We thus call our approach an average margin (AM) regularization, and it consists of a linear term added to the objective. We theoretically show that for certain distributions AM regularization can both improve classifier accuracy and robustness to adversarial attacks. We conclude by using both synthetic and real data to empirically show that AM regularization can strictly improve both accuracy and robustness for support vector machine's (SVM's), relative to unregularized classifiers and adversarially trained classifiers.

Cross entropy with softmax is the standard loss function for classification in neural networks. However, this function can suffer from limitations on discriminative power, lack of generalization, and propensity to overfitting. In order to address these limitations, several approaches propose to enforce a margin on the top of the neural network specifically at the softmax function. In this work, we present a novel formulation that aims to produce generalization and noise label robustness not only by imposing a margin at the top of the neural network, but also by using the entire structure of the mini-batch data. Based on the distance used for SVM to obtain maximal margin, we propose a broader distance definition called 1-to-N distance and an approximated probability function as the basis for our proposed loss function. We perform empirical experimentation on MNIST, CIFAR-10, and ImageNet32 datasets to demonstrate that our loss function has better generalization and noise label robustness properties than the traditional cross entropy method, showing improvements in the following tasks: generalization robustness, robustness in noise label data, and robustness against adversarial examples attacks.

This brief presents an energy-efficient electrocardiogram processor for arrhythmia detection with a weak-strong hybrid classifier that includes a weak linear classifier (WLC) and a strong support vector machine (SVM) classifier. WLC can only identify the beats with distinct characteristics by performing simple threshold comparisons based on beat interval feature and a novel morphology feature named QRS area ratio. The beats that are unclassified by WLC will activate the more powerful but energy-guzzling SVM classifier. Principal component analysis (PCA) is applied for feature dimension reduction to lower the complexity of SVM classifier and a sparse matrix computing architecture is exploited to reduce the computation burden of PCA. Implemented in SMIC 40LL CMOS process, the processor has a total area of 0.12 mm2. It achieves 1.98-uW power consumption in WLC mode and 3.76-uW in SVM mode under 1.1-V voltage supply and 10-KHz operating frequency, with energy dissipation of 6.8/30.3 nJ per beat classification for the two modes, respectively. The overall accuracy for MIT-BIH arrhythmia database is 98.2% with energy reduction of 41.7% compared to a single SVM classifier.

Asymmetric boosting, while acknowledged to be important to imbalanced classification problems like face detection, is often based on the trial-and-error methodology to obtain the best boosted classifier, rather than on principled methods. This thesis solves a number of issues related to asymmetric boosting and the use of asymmetric boosting in face detection. It shows how a proper understanding and use of asymmetric boosting leads to improvement in the learning time, the learning capacity, the detection speed and the detection accuracy of a face detector. First, an integrated framework for both online learning and asymmetric learning of a boosted classifier is presented. In addition, the proposed method adaptively balances the skewness of the weight distribution of the two classes presented to the weak classifiers, allowing them to be trained more equally. An additional constraint on propagating the weights of the data points is introduced, allowing the online learning to converge faster. When compared with the Online Boosting algorithm recently applied to object detection problems, a 0-10% increase in accuracy and 5-30% gain in learning speed were observed. Second, training a face detector using boosting and Haar-like features often requires weeks of computation on a single CPU machine. The bottleneck is in the training of a weak classifier, currently in the order of minutes. Traditional techniques for training a weak classifier usually run in time O(NT log N), with N examples (approximately 10,000), and T Haar-like features (approximately 40,000). A method to train a weak classifier in time O(N+T) is presented, by using only the statistics of the weighted input data. Experimental results reveal a significantly reduced training time of a face detector from weeks to just a few hours. In particular, this method trades off a minimal increase in training time for a very large increase in the set of Haar-like features explored, enjoying a significant gain in accuracy. Third, a generalized framework for representing a boosted classifier with multiple exit nodes is introduced. A method for training such a classifier is also proposed, which combines the recent idea of propagating scores across boosted classifiers and the use of asymmetric goals. A means for determining the ideal asymmetric goal is provided, which is theoretically justified under a conservative bound on the operating point target in the receiver-operator characteristic (ROC) curve, and is empirically near-optimal under the exact bound. Moreover, the method automatically minimizes the number of weak classifiers, avoiding the need to retrain a boosted classifier multiple times as in conventional methods. Experimental results show a significant reduction in the training time and the number of weak classifiers, as well as an improvement in accuracy. Fourth, a set of bounds on the generalization ability of a boosted classifier trained with an asymmetric goal is proposed, as current generalization bounds are not designed for asymmetric errors. The proposed bounds show that, unlike traditional boosting methods where there is no difference between a margin of a positive example and that of a negative example, the penalties applied to the margins are different for different classes. DOCTOR OF PHILOSOPHY (SCE)

Classification is an important task in various areas. In many real-world applications, class imbalance and overlapping problems have been reported as major issues in the application of traditional classification algorithms. An imbalance problem occurs when training data contain considerably more representatives of one class than of other classes. Class overlap occurs when a region in the data space contains a similar number of data for each class. When a class overlap occurs in imbalanced data sets, classification becomes even more complicated. Although various approaches have been proposed to deal separately with class imbalance and overlapping problems, only a few studies have attempted to address both problems simultaneously. In this paper, we propose an overlap-sensitive margin (OSM) classifier based on a modified fuzzy support vector machine and k-nearest neighbor algorithm to address imbalanced and overlapping data sets. The main idea of the proposed OSM classifier is to separate the data space into soft- and hard-overlap regions using the modified fuzzy support vector machine algorithm. The separated spaces are then classified using the decision boundaries of the support vector machine and 1-nearest neighbor algorithms. Furthermore, by separating a data set into soft- and hard-overlap regions, one can determine which part of the data is to be examined more closely for classification in real-world situations. Experiments using synthetic and real-world data sets demonstrated that the proposed OSM classifier outperformed existing methods for imbalanced and overlapping situations.

Purpose The most complex organ in the human body is the brain. The unrestrained growth of cells in the brain is called a brain tumor. The cause of a brain tumor is still unknown and the survival rate is lower than other types of cancers. Hence, early detection is very important for proper treatment. Methods In this study, an efficient computer-aided diagnosis (CAD) system is presented for brain image classification by analyzing MRI of the brain. At first, the MRI brain images of normal and abnormal categories are modeled by using the statistical features of dual tree m-band wavelet transform (DTMBWT). A maximum margin classifier, support vector machine (SVM) is then used for the classification and validated with k-fold approach. Results Results show that the system provides promising results on a repository of molecular brain neoplasia data (REMBRANDT) with 97.5% accuracy using 4th level statistical features of DTMBWT. Conclusion Viewing the experimental results, we conclude that the system gives a satisfactory performance for the brain image classification.

This paper describes a framework for Dynamic Classifier Selection (DCS) whose novelty resides in its use of features that address the difficulty posed by the classification problem in terms of orienting both pool generation and classifier selection. The classification difficulty is described by meta-features estimated from problem data using complexity measures. Firstly, these features are used to drive the classifier pool generation expecting a better coverage of the problem space, and then, a dynamic classifier selection based on similar features estimates the ability of the classifiers to deal with the test instance. The rationale here is to dynamically select a classifier trained on a subproblem (training subset) having a similar level of difficulty as that observed in the neighborhood of the test instance defined in a validation set. A robust experimental protocol based on 30 datasets, and considering 20 replications, has confirmed that a better understanding of the classification problem difficulty may positively impact the performance of a DCS. For the pool generation method, it was observed that in 126 of 180 experiments (70.0%) adopting the proposed pool generator allowed an improvement of the accuracy of the evaluated DCS methods. In addition, the main results from the proposed framework, in which pool generation and classifier selection are both based on problem difficulty features, are very promising. In 165 of 180 experiments (91.6%), it was also observed that the proposed DCS framework based on the problem difficulty achieved a better classification accuracy when compared to 6 well known DCS methods in the literature.

In this study, the algorithmic implementation of multi-category nonparallel hyperplane support vector machines is described. First, a least square version of Nonparallel Hyperplane Support Vector Machine (NHSVM) is developed for binary classification problems. Solution of the primal problem corresponding to the proposed NHSVM reduces to a system of linear equations as opposed to a quadratic programming problem in NHSVM. This formulation results in a much simpler and faster approach for constructing a nonparallel hyperplane binary classifier, termed as Least Squares Nonparallel Hyperplane Support Vector Machine (LSNHSVM). Further, LSNHSVM is generalized to solve multi- category classification problems. This multi-class classifier is the named as Multicategory Least Squares Nonparallel Hyperplane Support Vector Machine (MLSNHSVM). Unlike most of the previous methods that usually cast a multi-category classification problem into a series of multiple independent binary classification problem, MLSNHSVM constructs a direct multi-category classifier by solving a system of linear equations. The proposed MLSNHSVM is in close accordance with the principle of solving multi-category problems directly . Experimental results demonstrate that MLSNHSVM has significantly higher classification accuracy as compared to other multi-class classifiers and is considerably efficient than multi-class SVM in terms of computational time.

Multi-label classification associates an unseen instance with multiple relevant labels. In recent years, a variety of methods have been proposed to handle the multi-label problems. Classifier chains is one of the most popular multi-label methods because of its efficiency and simplicity. In this paper, we consider to optimize classifier chains from the viewpoint of conditional likelihood maximization. In the proposed unified framework, classifier chains can be optimized in either or both of two aspects: label correlation modeling and multi-label feature selection. In this paper we show that previous classifier chains algorithms are specified in the unified framework. In addition, previous information theoretic multi-label feature selection algorithms are specified with different assumptions on the feature and label spaces. Based on these analyses, we propose a novel multi-label method, k-dependence classifier chains with label-specific features, and demonstrate the effectiveness of the method.

In this paper, we discuss the distributed design for binary classification based on the nonlinear support vector machine in a time-varying multi-agent network when the training data sets are distributedly located and unavailable to all agents. In particular, the aim is to find a global large margin classifier and then enable each agent to classify any new input data into one of the two labels in the binary classification without sharing its all local data with other agents. We formulate the support vector machine problem into a distributed optimization problem in approximation and employ a distributed algorithm in a time-varying network to solve it. Our algorithm is a stochastic one with the high convergence rate and the low communication cost. With the jointly-connected connectivity condition, we analyze the consensus rate and the convergence rate of the given algorithm. Then some experimental results on various classification training data sets are also provided to illustrate the effectiveness of the given algorithm.

This paper proposes a novel twin-hyperspheres support vector machine (THSVM) classifier for binary classification, called the automatic variable-weighted THSVM (VTHSVM) classifier. By solving a single optimization problem, this classifier not only finds a pair of hyperspheres for classification, but also automatically constructs a weight vector for each class in order to describe the dissimilarity of different classes. This VTHSVM is extended to the kernel case by the fact that a kernel can be written as a sum of one’s evaluated on each variable separately. The main advantage of this method is that it allows the use of adaptive distance, which is suitable to find an as compact as possible hypersphere for each class. Experiments with synthetic and benchmark datasets indicate VTHSVM obtains better performance than some other classifiers.

We developed unconstrained large margin distribution machines (ULDMs) for pattern classification.The ULDM maximizes the margin mean and minimizes the margin variance.The ULDM can be trained by solving a set of linear equations.We clarify the characteristics of ULDMs.Performance of the ULDM is compared with that of the L1 SVM, LS SVM, and the LDM. Large margin distribution machines (LDMs) maximize the margin mean and minimize the margin variance, and show good generalization performance compared to support vector machines (SVMs). But because two additional hyperparameters are necessary, model selection needs more time. In this paper we propose unconstrained large margin distribution machines (ULDMs). In the ULDM, the objective function is the sum of the margin mean (a linear term), the margin variance (a quadratic term), and the weighted regularization term (a quadratic term), and constraints are not included. Therefore, the solution is expressed by a set of linear equations with one hyperparameter for the regularization term. Theoretical analysis proves that the decision boundary between two classes passes through the mean of all mapped training data if the numbers of training data of both classes are the same. The case where the numbers are different is analyzed for a one-dimensional input and how the decision boundary is determined is clarified. Using benchmark data sets, we show that the generalization performance of ULDMs is comparable to or better than that of SVMs.

Interpretability of classification systems, which refers to the ability of these systems to express their behavior in an understandable way, has recently gained more attention and it is considered as an important requirement especially for knowledge-based systems. The main objective of this study is to improve the ability of a well-known fuzzy classifier proposed in Ishibuchi and Nojima (2007) to maximize the accuracy while preserve its interpretability. To achieve the above-mentioned objective, we propose two variants of the original fuzzy classifier. In the first variant classifier, the same components of the original classifier were used except NSGA-II which was replaced by an enhanced version called Controlled Elitism NSGA-II. This replacement aims at improving the ability of the first variant classifier to find non-dominated solutions with better interpretability-accuracy trade-off. In the second variant classifier, we further improve the first variant classifier by enhancing the selection method of the antecedent conditions of the rules generated in the initial population of genetic algorithm. Unlike the method applied in the original classifier and the first variant classifier, which uses a random selection of the antecedent conditions, we proposed a feature-based selection method to favor the antecedent conditions associated with the most relevant features. The results show that the two variant classifiers find more non-dominated fuzzy rule-based systems with better generalization ability than the original method which suggests that Controlled Elitism NSGA-II algorithm is more efficient than NSGA-II. In addition, feature-based selection method applied in the second variant classifier allowed this method to successfully obtain high-quality solutions as it has consistently achieved the best error rates for all the data sets compared to the original method and the first variant classifier.

This paper presents an approach to clustering that extends the variance-based Learning Classifier System (XCS-VR). In real world problems, the ability to combine similar rules is crucial in the knowledge discovery and data mining field. Conventionally, XCS-VR is able to acquire generalized rules, but it cannot further acquire more generalized rules from these rules. The proposed approach (called XCS-VRc) accomplishes this by integrating similar generalized rules. To validate the proposed approach, we designed a bench-mark problem to examine whether XCS-VRc can cluster both the generalized and more generalized features in the input data. The proposed XCS-VRc proved to be more efficient than XCS and the conventional XCS-VR.

In this paper, we propose a method to improve ECS-DMR which enables appropriate output for imbalanced data sets. In order to control generalization of LCS in imbalanced data set, we propose a method of applying imbalance ratio of data set to a sigmoid function, and then, appropriately update the matching range. In comparison with our previous work (ECS-DMR), the proposed method can control the generalization of the appropriate matching range automatically to extract the exemplars that cover the given problem space, wchich consists of imbalanced data set. From the experimental results, it is suggested that the proposed method provides stable performance to imbalanced data set. The effect of the proposed method using the sigmoid function considering the data balance is shown.

Support vector machine (SVM) is a supervised machine learning method which can be used for both classification and regression models. In this paper, we introduce a new model of SVM and support vector regression which any of training samples containing inputs and outputs are considered the random variables with known or unknown probability functions. In this new models, we need the mathematical expectation for any of training samples but when these are unknown we apply nonparametric statistical methods. Also constraints occurrence have probability function which helps obtain maximum margin and achieve robustness. We obtain the optimal separating hyperplane and the optimal hyperplane regression by solving the quadratic optimization problems. Finally the proposed methods are illustrated by several experiments.

We propose a novel randomization-based approach for classifier ensemble construction.It samples the parameters of the base classifiers from a pre-defined distribution.As an example we derive the parameter distribution of some linear bagged classifiers.We then simulate bagging by using the derived distribution. Randomization-based techniques for classifier ensemble construction, like Bagging and Random Forests, are well known and widely used. They consist of independently training the ensemble members on random perturbations of the training data or random changes of the learning algorithm. We argue that randomization techniques can be defined also by directly manipulating the parameters of the base classifier, i.e., by sampling their values from a given probability distribution. A classifier ensemble can thus be built without manipulating the training data or the learning algorithm, and then running the learning algorithm to obtain the individual classifiers. The key issue is to define a suitable parameter distribution for a given base classifier. This also allows one to re-implement existing randomization techniques by sampling the classifier parameters from the distribution implicitly defined by such techniques, if it is known or can be approximated, instead of explicitly manipulating the training data and running the learning algorithm. In this work we provide a first investigation of our approach, starting from an existing randomization technique (Bagging): we analytically approximate the parameter distribution for three well-known classifiers (nearest-mean, linear and quadratic discriminant), and empirically show that it generates ensembles very similar to Bagging. We also give a first example of the definition of a novel randomization technique based on our approach.

Bagging and Boosting are two main ensemble approaches consolidating the decisions of several hypotheses. The diversity of the ensemble members is considered to be a significant element to obtain generalization error. Here, an inventive method called EBAGTS (ensemble-based artificially generated training samples) is proposed to generate ensembles. It manipulates training examples in three ways in order to build various hypotheses straightforwardly: drawing a sub-sample from training set, reducing/raising error-prone training instances, and reducing/raising local instances around error-prone regions. The proposed method is a straightforward, generic framework utilizing any base classifier as its ensemble members to assemble a powerfully built combinational classifier. Decision-tree classifier and multilayer perceptron classifier as some basic classifiers have been employed in the experiments to indicate the proposed method accomplish higher predictive accuracy compared to meta-learning algorithms li...

Increase of huge amount of data in every application demands an incremental learning technique for data analysis. One of such data analysis task in dynamic environment is to design an incremental classifier for decision making and consequently updating the knowledge base of the overall system. Classifier construction depicts extraction of interesting patterns from the large repository of data and predicts the future trends based on the existing patterns. The time complexity of the classification system increases gradually and the system becomes inefficient while it is learned repeatedly for adding new group of data with the existing one in a certain interval of time. Without learning the same classifier for the whole data, if the knowledge of old data extracted by the classifier is used together with the new group of data to design the updated classifier, called incremental classifier, then time complexity reduces drastically. In the paper, the concepts of Particle Swarm Optimization technique and Association Rule Mining are used to design an incremental rule based classification system. The incremental classifier is suitable to apply on rice disease dataset for disease prediction as the characteristics of rice diseases change in time due to change of climate, biological, and geographical factors. The proposed method has been applied on both simulated rice disease dataset and benchmark datasets and the classification accuracy is measured and compared with various state of the art classification algorithms. The method is also evaluated based on some statistical measures and statistical test is done to establish its significance and effectiveness.

Generally the performance of a pedestrian detector will decrease rapidly, when it is trained on a fixed training set but applied to specific scenes. The reason is that in the training set only a few samples are useful for the specific scenes while other samples may disturb the accurate detections. Traditional methods solve this problem by transfer learning which suffer the problem of keeping source samples or artificially labeling a few samples in the detection phase. In this paper, we propose a new method to bypass these defects by predicting pedestrian classifier for each sample in the detection phase. A classifier regression model is trained in the source domain in which each sample has a proprietary classifier. In the detection phase, a pedestrian classifier is predicted for each candidate window in an image. Thus, for the samples in the target domain, the pedestrian classifiers are different. Our main contributions are: (1) a new adaptive detector without keeping source samples or labeling a few new target samples; (2) a new dimensionality reduction method for classifier vector which simultaneously ensures the performance of both reconstruction and classification; (3) a two-stage regression neural model which can handle the high-dimensional regression problem effectively. Experiments prove that our method can achieve the state-of-the-art results on two pedestrian datasets.

This paper proposes a two-step method to construct a nonlinear classifier consisting of multiple local linear classifiers interpolated with a basis function. In the first step, a geometry-based approach is first introduced to detect local linear partitions and build local linear classifiers. A coarse nonlinear classifier can then be constructed by interpolating the local linear classifiers. In the second step, a support vector machine (SVM) formulation is used to further implicitly optimize the linear parameters of the nonlinear classifier. In this way, the nonlinear classifier is constructed in exactly the same way as a standard SVM, using a special data-dependent quasi-linear kernel composed of the information of the local linear partitions. Numerical experiments on several real-world datasets demonstrate the effectiveness of the proposed classifier and show that, in cases where traditional nonlinear SVMs run into overfitting problems, the proposed classifier is effective in improving the classification performance. © 2017 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

Minimax probability machine (MPM) is an interesting discriminative classifier based on generative prior knowledge. It can directly estimate the probabilistic accuracy bound by minimizing the maximum probability of misclassification. The structural information of data is an effective way to represent prior knowledge, and has been found to be vital for designing classifiers in real-world problems. However, MPM only considers the prior probability distribution of each class with a given mean and covariance matrix, which does not efficiently exploit the structural information of data. In this paper, we use two finite mixture models to capture the structural information of the data from binary classification. For each subdistribution in a finite mixture model, only its mean and covariance matrix are assumed to be known. Based on the finite mixture models, we propose a structural MPM (SMPM). SMPM can be solved effectively by a sequence of the second-order cone programming problems. Moreover, we extend a linear model of SMPM to a nonlinear model by exploiting kernelization techniques. We also show that the SMPM can be interpreted as a large margin classifier and can be transformed to support vector machine and maxi-min margin machine under certain special conditions. Experimental results on both synthetic and real-world data sets demonstrate the effectiveness of SMPM.

Support Vector Regression (SVR) solves regression problems based on the concept of Support Vector Machine (SVM). In this paper, we introduce a novel model of SVR in which any training samples containing inputs and outputs are considered the random variables with known or unknown distribution functions. Constraints occurrence have a probability density function which helps to obtain maximum margin and achieve robustness. The optimal hyperplane regression can be obtained by solving a quadratic optimization problem. The proposed method is illustrated by several experiments including artificial data sets and real-world benchmark data sets.

In order to solve the problem of indoor place recognition for indoor service robot, a novel algorithm, clustering of features and images (CFI), is proposed in this work. Different from traditional indoor place recognition methods which are based on kernels or bag of features, with large margin classifier, CFI proposed in this work is based on feature matching, image similarity and clustering of features and images. It establishes independent local feature clusters by feature cloud registration to represent each room, and defines image distance to describe the similarity between images or feature clusters, which determines the label of query images. Besides, it improves recognition speed by image scaling, with state inertia and hidden Markov model constraining the transition of the state to kill unreasonable wrong recognitions and achieves remarkable precision and speed. A series of experiments are conducted to test the algorithm based on standard databases, and it achieves recognition rate up to 97% and speed is over 30 fps, which is much superior to traditional methods. Its impressive precision and speed demonstrate the great discriminative power in the face of complicated environment.

The wrong clusters number or poor starting points of each cluster have negative influence on the classification accuracy in the hybrid classifier based credit scoring system. The paper represents a new hybrid classifier based on fuzzy-rough instance selection, which have the same ability as clustering algorithms, but it can eliminate isolated and inconsistent instances without the need of determining clusters number and starting points of each cluster. The unrepresentative instances that cause conflicts with other instances are completely determined by the fuzzy-rough positive region which is only related to intrinsic data structure of datasets. By removing unrepresentative instances, both the training data quality and classifier training time can be improved. To prevent eliminating more instances than strictly necessary, the k-nearest neighbor algorithm is adopted to check the eliminated instances, and the instance whose predicted class is the same with predefined class is added back. SVM classifier with three different kernel functions are applied to the reduced dataset. The experimental results show that the proposed hybrid classifier has better classification accuracy on two real world datasets.

One-class classification problems have been widely encountered in the fields that the negative class patterns are difficult to be collected, and the one-class support vector machine is one of the popular algorithms for solving them. However, one-class support vector machine is a vector-based learning algorithm, and it cannot work directly when the input pattern is a tensor. This paper proposes a tensor-based maximum margin classifier for one-class classification problems, and develops a One-Class Support Higher Order Tensor Machine (HO-OCSTM) which can separate most of the target patterns from the origin with the maximum margin in the higher order tensor space. HO-OCSTM directly employs the higher order tensors as the input patterns, and it is more proper for small sample study. Moreover, the direct use of tensor representation has the advantage of retaining the structural information of data, which helps improve the generalization ability of the proposed algorithm. We implement HO-OCSTM by the alternating projection method and solve a convex quadratic programming similar to the standard one-class support vector machine algorithm at each iteration. The experimental results have shown the high recognition accuracy of the proposed method.

A hybrid classifier obtained by hybridizing Support Vector Machines (SVM) and Artificial Neural Network (ANN) classifiers is presented here for diagnosis of gear faults. The distinctive features obtained from vibration signals of a running gearbox, which was operated in normal and fault-induced conditions, were used to feed the SVM-ANN hybrid classifier. Time-domain vibration signals were divided in segments. Features such as peaks in time domain and in spectrum, central moments, and standard deviations were obtained from signal segments. Based on the experimental results, it was shown that SVM-ANN hybrid classifier can successfully identify gear condition and that the hybrid SVM-ANN classifier performs much better than standard versions of ANNs and SVM. The effectiveness of the hybrid classifier under noise was also investigated. It was shown that if vibration signals are preprocessed by Discrete Wavelet Transform (DWT), efficacy of the SVM-ANN hybrid is significantly enhanced.

This paper proposed a new framework of one-class classification based on continuous cognition.The framework is implemented with LSSVM and the corresponding classifier is called OC-LSSVM.Several simulation and real datasets are used to test the performance of OC-LSSVM.OC-LSSVM shows state-of-the-art performance compared to established methods. One-class classification is a basic problem in machine learning. Unlike the existing typical one-class classifiers designed from the angle of probability or geometric, this paper attempts to study this problem from the bionics point of view. Using the continuous cognition characteristic as the starting point, we propose a new framework of one-class classifier, named multiple regression model (OC-MR), which can be seen as a natural extension of multiple regression for one-class classification problem. This paper applies least squares support vector machine (LSSVM) as an example to show themodeling process of the proposed method and the corresponding one-class classifier is named one-class least squares support vector machine (OC-LSSVM). Various simulation and real-life datasets are used to test the performance of the proposed OC-LSSVM. The existing popular one-class classification methods including Parzen kernel density estimation, support vector data description and Gaussian mixture model are also applied in order to achieve a comprehensive comparison. The results show that OC-LSSVM has achieved the best performance in most of the simulation and real-life datasets due to its good robustness, which highlights the efficacy of OC-LSSVM.

There is an intrinsic adversarial nature in the security domain such as spam filtering and malware detection systems that attempt to mislead the detection system. This adversarial nature makes security applications different from the classical machine learning problems; for instance, an adversary (attacker) might change the distribution of test data and violate the data stationarity, a common assumption in machine learning techniques. Since machine learning methods are not inherently adversary-aware, a classifier designer should investigate the robustness of a learning system under attack. In this respect, recent studies have modeled the identified attacks against machine learning-based detection systems. Based on this, a classifier designer can evaluate the performance of a learning system leveraging the modeled attacks. Prior research explored a gradient-based approach in order to devise an attack against a classifier with differentiable discriminant function like SVM. However, there are several powerful classifiers with non-differentiable decision boundary such as Random Forest, which are commonly used in different security domain and applications. In this paper, we present a novel approach to model an attack against classifiers with non-differentiable decision boundary. In the experimentation, we first present an example that visually shows the effect of a successful attack on the MNIST handwritten digits classification task. Then we conduct experiments for two well-known applications in the security domain: spam filtering and malware detection in PDF files. The experimental results demonstrate that the proposed attack successfully evades Random Forest classifier and effectively degrades the classifier’s performance.

Objective: To provide automatically analyzing and detecting human activities to provide better support in healthcare sector, security purpose etc. Method: We have used UT Kinect-Action 3D dataset containing position of 20 body joint captured by Kinect sensor. We selected two set of joints J1 and J2; after that we have formed some rules for activity classification then we have applied SVM classifier, KNN classifier using Euclidean distance and KNN classifier using minkowski distance for activity classification. Findings: When we have used joint set J1 we got 97.8% accuracy with SVM classifier, 98.8% accuracy with KNN classifier using Euclidean distance, and 98.9% accuracy with KNN classifier using minkowski distance and for joint set J2 we got 97.7% accuracy with SVM classifier, 98.6% accuracy with KNN classifier using Euclidean distance, and 98.7% accuracy with KNN classifier using minkowski distance. Application/Improvement: we have classified four activities hand waving, standing, sitting and picking. In future more activities can also be included in this study. IOT along with this activity recognition method can be used to reduce overheads.