Search

Classifier chains are among the most successful methods in multi-label classification due to their simplicity and promising performance. However the standard versions of classifier chains described in the literature do not usually perform feature selection. In this paper we propose an algorithm CCnet which is a combination of classifier chains and elastic-net regularization. An important advantage of the CCnet is that selection of the relevant features in an integral element of the learning process. We show the stability of our algorithm and analyse the generalization error bound. The difference between generalization error and empirical error is bounded by a term which scales as n − 1 / 2 , where n is a size of a training data. It follows from experiments that the proposed algorithm outperforms the standard versions of classifier chains as well as other state-of-the-art methods. We also show that the feature selection is stable with respect to the order of fitting the models in the chain.

Uncertain objects, where each feature is represented by multiple observations or a given or fitted probability density function, arise in applications such as sensor networks, moving object databases and medical and biological databases. We propose a methodology to classify uncertain objects based on a new probabilistic distance measure between an uncertain object and a group of uncertain objects. This object-to-group probabilistic distance measure is unique in that it accounts separately for the correlations among the features within each class and within each object. We compare the proposed object-to-group classifier to two existing classifiers, namely, the K-Nearest Neighbor classifier on object means (certain-KNN) and the uncertain-naive Bayes classifier. In addition, we compare the object-to-group classifier to an uncertain K-Nearest Neighbor classifier (uncertain-KNN), also proposed here, that uses existing probabilistic distance measures for object-to-object distances. We illustrate the advantages of the proposed classifiers with both simulated and real data.

One of the crucial problems of the classifier ensemble is the so-called combination rule which is responsible for establishing a single decision from the pool of predictors. The final decision is made on the basis of the outputs of individual classifiers. At the same time, some of the individuals do not contribute much to the collective decision and may be discarded. This paper discusses how to design an effective combination rule, based on support functions returned by individual classifiers. We express our interest in aggregation methods which do not require training, because in many real-life problems we do not have an abundance of training objects or we are working under time constraints. Additionally, we show how to use proposed operators for simultaneous classifier combination and ensemble pruning. Our proposed schemes have embedded classifier selection step, which is based on weight thresholding. The experimental analysis carried out on the set of benchmark datasets and backed up with a statistical analysis, proved the usefulness of the proposed method, especially when the number of class labels is high.

In recent years, classifier ensembles have received increasing attention in the machine learning and pattern recognition communities. However, constructing classifier ensembles for one-class classification problems has still remained as a challenging research topic. To pursue this line of research, we need to address issues on how to generate a set of diverse one-class classifiers that are individually accurate and how to combine the outputs of them in an effective way. In this paper, we present BeeOWA, a novel approach to construct highly accurate one-class classifier ensembles. It uses a novel binary artificial bee colony algorithm, called BeePruner, to prune an initial one-class classifier ensemble and find a near-optimal sub-ensemble of base classifiers in a reasonable computational time. To evaluate the fitness of an ensemble solution, BeePruner uses two different measures: an exponential consistency measure and a non-pairwise diversity measure based on the Kappa inter-rater agreement. After one-class classifier pruning, BeeOWA uses a novel exponential induced OWA (ordered weighted averaging) operator, called EIOWA, to combine the outputs of base classifiers in the sub-ensemble. The results of experiments carried out on a number of benchmark datasets show that BeeOWA can outperform several state-of-the-art approaches, both in terms of classification performance and statistical significance.

This paper proposes a novel approach to select the individual classifiers to take part in a Multiple-Classifier System. Individual classifier selection is a key step in the development of multi-classifiers. Several works have shown the benefits of fusing complementary classifiers. Nevertheless, the selection of the base classifiers to be used is still an open question, and different approaches have been proposed in the literature. This work is based on the selection of the appropriate single classifiers by means of an evolutionary algorithm. Different base classifiers, which have been chosen from different classifier families, are used as candidates in order to obtain variability in the classifications given. Experimental results carried out with 20 databases from the UCI Repository show how adequate the proposed approach is; Stacked Generalization multi-classifier has been selected to perform the experimental comparisons.

Facial expression is one of the most expressive ways to display human emotions. Facial expression analysis (FEA) has been broadly studied in the past decades. In our daily life, few of the facial expressions are exactly one of the predefined affective states but are blends of several basic expressions. Even though the concept of ‘blended emotions’ has been proposed years ago, most researchers did not deal with FEA as a multiple outputs problem yet. In this paper, multi-label learning algorithm for FEA is proposed to solve this problem. Firstly, to depict facial expressions more effectively, we model FEA as a multi-label problem, which depicts all facial expressions with multiple continuous values and labels of predefined affective states. Secondly, in order to model FEA jointly with multiple outputs, multi-label Group Lasso regularized maximum margin classifier (GLMM) and Group Lasso regularized regression (GLR) algorithms are proposed which can analyze all facial expressions at one time instead of modeling as a binary learning problem. Thirdly, to improve the effectiveness of our proposed model used in video sequences, GLR is further extended to be a Total Variation and Group Lasso based regression model (GLTV) which adds a prior term (Total Variation term) in the original model. JAFFE dataset and the extended Cohn Kanade (CK+) dataset have been used to verify the superior performance of our approaches with common used criterions in multi-label classification and regression realms.

While classical classification methods such as support vector machine and its extensions (SVMs) obtain strong generalization capability by maximizing the separation margin between binary classes, they are usually sensitive to affine (or scaling) transformation of data. The optimal solutions of SVMs may be misled by the spread of data and preferentially separate classes along large margin directions. To this end, in this paper, we propose a novel minimum class spread constrained support vector machine (MCSSVM) for pattern classification problems by simultaneously considering the maximization of inter-class separation margin and the minimization of within-class spread of data, which corresponds to a distribution-dependent regularization on the classification function by constraining the within-class spread of input data. The basic idea of MCSSVM is to find a class distribution constrained hyperplane such that one class (or normal class) can be enclosed in a minimum q-spread tube, while the other class (or abnormal class) is farthest from this tube. MCSSVM can simultaneously achieve both maximum inter-class margin and minimum within-class spread so as to enhance the generalization capability of the proposed classifier. Moreover, the proposed method only requires simple extensions to existing maximum margin formulations such as SVMs and still preserves their computational efficiency. Generalization bound for MCSSVM is theoretically derived and the validity of MCSSVM is examined by classification of toy and real-world classification problems, which demonstrate the superiority of our method in comparison to other related state-of-the-art algorithms.

The VC dimension measures the complexity of a learning machine, and a low VC dimension leads to good generalization. While SVMs produce state-of-the-art learning performance, it is well known that the VC dimension of a SVM can be unbounded; despite good results in practice, there is no guarantee of good generalization. In this paper, we show how to learn a hyperplane classifier by minimizing an exact, or ? bound on its VC dimension. The proposed approach, termed as the Minimal Complexity Machine (MCM), involves solving a simple linear programming problem. Experimental results show, that on a number of benchmark datasets, the proposed approach learns classifiers with error rates much less than conventional SVMs, while often using fewer support vectors. On many benchmark datasets, the number of support vectors is less than one-tenth the number used by SVMs, indicating that the MCM does indeed learn simpler representations. HighlightsWe learn a hyperplane classifier by minimizing an exact bound on its VC dimension.A fractional programming problem is formulated and reduced to a LP problem.Linear and kernel versions of the approach are explored.The approach, called the Minimal Complexity Machine, generalizes better than SVMs.On numerous benchmark datasets, the MCM uses far fewer support vectors than SVMs.

Learning a classifier when only knowing the features and marginal distribution of class labels in each of the data groups is both theoretically interesting and practically useful. Specifically, we consider the case in which the ratio of the number of data instances to the number of classes is large. We prove sample complexity upper bound in this setting, which is inspired by an analysis of existing algorithms. We further formulate the problem in a density estimation framework to learn a generative classifier. We also develop a practical RBM-based algorithm which shows promising performance on benchmark datasets.

We consider the classifier ensemble problem in this paper. Due to its superior performance to individual classifiers, class ensemble has been intensively studied in the literature. Generally speaking, there are two prevalent research directions on this, i.e., to diversely generate classifier components, and to sparsely combine multiple classifiers. While most current approaches are emphasized on either sparsity or diversity only, we investigate the classifier ensemble by learning both sparsity and diversity simultaneously. We manage to formulate the classifier ensemble problem with the sparsity or/and diversity learning in a general framework. In particular, the classifier ensemble with sparsity and diversity can be represented as a mathematical optimization problem. We then propose a heuristic algorithm, capable of obtaining ensemble classifiers with consideration of both sparsity and diversity. We exploit the genetic algorithm, and optimize sparsity and diversity for classifier selection and combination heuristically and iteratively. As one major contribution, we introduce the concept of the diversity contribution ability so as to select proper classifier components and evolve classifier weights eventually. Finally, we compare our proposed novel method with other conventional classifier ensemble methods such as Bagging, least squares combination, sparsity learning, and AdaBoost, extensively on UCI benchmark data sets and the Pascal Large Scale Learning Challenge 2008 webspam data. The experimental results confirm that our approach leads to better performance in many aspects.

This paper proposes a new Weighted Margin Sparse Embedded (WMSE) classifier for brake cylinder detection, which is a big challenge in Trouble of Freight car Detection System (TFDS) of China. The major contributions of this paper are in three folds. (1) The proposed method is a combination of the Sparse Embedded (SE) and the Weighted Margin LearningS (WML) models, which are iteratively performed toward optimal classifier ensemble. The final classifier in cascades takes advantages of VC-dimension minimization and weighted margin learning, which provides a new investigation into the literature of classifier design. (2) Convergence of the WMSE classifier is theoretically proven, which is a desirable characteristic for object detection due to existence of large-scale training datasets in real applications. (3)To evaluate the performance of the proposed method, we establish and distribute the challenging BeiHang Brake Cylinder (BH-BC) Database containing over 2000 annotated brake cylinder images with various appearances and almost indistinguishable backgrounds. Comparative experimental results on the BH-BC database show that our approach can get a much higher detection performance than the state-of-the-art classifiers (Support Vector Machine and Adaboost).

Among the various traditional approaches of pattern recognition, the statistical approach has been most intensively studied and used in practice. This paper presents a new classifier called MLAC for multiclass classification based on the learning automata. The proposed classifier using a soft decision method could find the optimal hyperplanes in solution space and separate available classes from each other well. We have tested the MLAC classifier on some multiclass datasets including IRIS, WINE and GLASS. The results show a significant improvement in comparison with the previous learning automata based classifiers as it has more accuracy and lower running time. Also, in order to evaluate performance of the proposed MLAC classifier, it has been compared with conventional classifiers such as K-Nearest Neighbor, Multilayer Perceptron, Genetic classifier and Particle Swarm classifier on these datasets in terms of accuracy. The obtained results show that the proposed MLAC classifier not only improves the classification's accuracy, but also reduces time complexity.

This paper investigates the design of a linear-in-the-parameters (LITP) regression classifier for two-class problems. Most existing algorithms generally learn a classifier (model) from the available training data based on some stopping criterions, such as the Akaike's final prediction error (FPE). The drawback here is that the classifier obtained is then not directly obtained based on its generalization capability. The main objective of this paper is to improve the sparsity and generalization capability of a classifier, while reducing the computational expense in producing it. This is achieved by proposing an automatic two-stage locally regularized classifier construction (TSLRCC) method using the extreme learning machine (ELM). In this new algorithm, the nonlinear parameters in each term, such as the width of the Gaussian function and the power of a polynomial term, are firstly determined by the ELM. An initial classifier is then generated by the direct evaluation of these candidates models according to the leave-one-out (LOO) misclassification rate in the first stage. The significance of each selected regressor term is also checked and insignificant ones are replaced in the second stage. To reduce the computational complexity, a proper regression context is defined which allows fast implementation of the proposed method. Simulation results confirm the effectiveness of the proposed technique.

How to combine the outputs from base classifiers is a key issue in ensemble learning. This paper presents a dynamic classifier ensemble method termed as DCE-CC. It dynamically selects a subset of classifiers for test samples according to classification confidence. The weights of base classifiers are learned by optimization of margin distribution on the training set, and the ordered aggregation technique is exploited to estimate the size of an appropriate subset. We examine the proposed fusion method on some benchmark classification tasks, where the stable nearest-neighbor rule and the unstable C4.5 decision tree algorithm are used for generating base classifiers, respectively. Compared with some other multiple classifier fusion algorithms, the experimental results show the effectiveness of our approach. Then we explain the experimental results from the view point of margin distribution.

Threshold-based classifier is a simple yet powerful pattern classification tool, which has been frequently used in applications of object detection and recognition. A threshold-based classifier is usually associated with a unique one-dimensional feature. A properly selected threshold and a binary sign corresponding to the feature govern the classifier. However, the learning process is usually done in a batch manner. The batch algorithms are not suitable for sequentially incoming data because of the limitation of storage and prohibitive computation cost. To deal with sequentially incoming data, this paper proposes an incremental algorithm for incrementally learning the threshold-based classifiers. The proposed method can not only incrementally model the features but also estimate the threshold and training error in a close form. The effectiveness of the proposed algorithm is evaluated in the applications of gender recognition, face detection, and human detection.

An ensemble kernel classifier is proposed in this paper by integrating a kernel principal component analysis (KPCA) with a support vector machine (SVM) as well as an immune clonal selection algorithm (ICSA). The KPCA approach is used to extract features, whereas the SVM technique is employed to deal with classification, and the ICSA is applied to optimize the parameters of the proposed scheme. The proposed ensemble classifier can automatically select the kernel type and optimize its parameter sets, in order to produce various SVM classifiers with different kernels. Regardless of whether the data is linear or nonlinear, an optimum classification result can be obtained. In order to demonstrate the effectiveness of the classifier, it is applied to discriminate the primary open-angle glaucoma (POAG) using a standard classification dataset. Experimental results reveal that the proposed ensemble classifier is accurate and more effective when compared to other approaches in the literature. It is envisaged that ensemble kernel classifier could hold a high potential in classification of pattern recognition problems.

In this paper, we consider the learning rates of support vector machines (SVMs) classifier for density level detection (DLD) problem. Using an established classification framework, we get error decomposition which consists of regularization error and sample error. Based on the decomposition, we obtain learning rates of SVMs classifier for DLD under some assumptions.

In this paper, we investigate the decision making ability of a fully complex-valued radial basis function (FC-RBF) network in solving real-valued classification problems. The FC-RBF classifier is a single hidden layer fully complex-valued neural network with a nonlinear input layer, a nonlinear hidden layer, and a linear output layer. The neurons in the input layer of the classifier employ the phase encoded transformation to map the input features from the Real domain to the Complex domain. The neurons in the hidden layer employ a fully complex-valued Gaussian-like activation function of the type of hyperbolic secant (sech). The classification ability of the classifier is first studied analytically and it is shown that the decision boundaries of the FC-RBF classifier are orthogonal to each other. Then, the performance of the FC-RBF classifier is studied experimentally using a set of real-valued benchmark problems and also a real-world problem. The study clearly indicates the superior classification ability of the FC-RBF classifier.

There are two standard approaches to the classification task: generative, which use training data to estimate a probability model for each class, and discriminative, which try to construct flexible decision boundaries between the classes. An ideal classifier should combine these two approaches. In this paper a classifier combining the well-known support vector machine (SVM) classifier with regularized discriminant analysis (RDA) classifier is presented. The hybrid classifier is used for protein structure prediction which is one of the most important goals pursued by bioinformatics. The obtained results are promising, the hybrid classifier achieves better result than the SVM or RDA classifiers alone. The proposed method achieves higher recognition ratio than other methods described in the literature.

Most existing online algorithms in support vector machines (SVM) can only grow support vectors. This paper proposes an online error tolerance based support vector machine (ET-SVM) which not only grows but also prunes support vectors. Similar to least square support vector machines (LS-SVM), ET-SVM converts the original quadratic program (QP) in standard SVM into a group of easily solved linear equations. Different from LS-SVM, ET-SVM remains support vectors sparse and realizes a compact structure. Thus, ET-SVM can significantly reduce computational time while ensuring satisfactory learning accuracy. Simulation results verify the effectiveness of the newly proposed algorithm.

Support vector machine (SVM) is one of the state-of-the-art tools for linear and non-linear pattern classification. One of the design objectives of an SVM classifier is reducing the number of support vectors without compromising the classification accuracy. For this purpose, a novel technique referred to as diminishing learning (DL) technique is proposed in this paper for a multiclass SVM classifier. In this technique, a sequential classifier is proposed wherein the classes which require stringent boundaries are tested one by one and once the tests for these classes fail, the stringency of the classifier is increasingly relaxed. An automated procedure is also proposed to obtain the optimum classification order for SVM-DL classifier in order to improve the recognition accuracy. The proposed technique is applied for SVM based isolated digit recognition system and is studied using speaker dependent and multispeaker dependent TI46 database of isolated digits. Both LPC and MFCC are used for feature extraction. The features extracted are mapped using self-organized feature maps (SOFM) for dimensionality reduction and the mapped features are used by SVM classifier to evaluate the recognition accuracy using various kernels. The performance of the system using the proposed SVM-DL classifier is compared with those using other techniques: one-against-all (OAA), half-against-half (HAH) and directed acyclic graph (DAG). SVM-DL classifier results in 1-2% increase in recognition accuracy compared to HAH classifier for some of the kernels with both LPC and MFCC feature inputs. For MFCC feature inputs, both HAH and SVM-DL classifiers have 100% recognition accuracy for some of the kernels. The total number of support vectors required is the least for HAH classifier followed by the SVM-DL classifier. The proposed diminishing learning technique is applicable for a number of pattern recognition applications.

The accuracy attained in the mapping of underwater areas is limited by the effect of variations in the water column, which degrade the signal received by the orbital sensor, creating interclasses confusion that introduce errors into the final result of the classification process. In this article we will describe a hybrid classifier ensembles; the classification is done by progressive refining in three stages. At the end of this process, a combining unit links the various partial classifications generated and achieve the accuracy level desired. At the end, the result obtained by the ensemble is compared to the results achieved by the application of multi-class voting scheme methods based on support vector machine: One-Against-the-Rest and One-Against-One. Classification accuracy showed the viability and the potential of using the proposed ensemble to classify images.

Geometric methods provide an intuitive and theoretically solid viewpoint for the solution of many optimization problems in the fields of pattern recognition and machine learning. The support vector machine (SVM) classification is a typical optimization task that has achieved excellent generalization performance in a wide variety of applications. In this paper, the notion of ''scaled convex hull'' (SCH) is presented, through which the nonseparable SVM classifications can be approximately transformed to separable ones: by a suitable selection of the reduction factor, the initially overlapping SCHs (each is generated by the training patterns of each class) can be reduced to become separable, then the maximal margin classifier between them can be trained, which is an approximation of the standard nonseparable SVM. As a practical application of the SCH framework, the popular Gilbert's algorithm has been generalized to approximately solve general (linear and nonlinear, separable and nonseparable) SVM classification problems both accurately and efficiently. The experiments show that the proposed method may achieve better performance than the state-of-the-art methods, an improved sequential minimal optimization and Gilbert's algorithm based on the reduced convex hull (RCH), in terms of the number of kernel evaluations and the execution time.

Imbalanced dataset learning is an important practical issue in machine learning, even in support vector machines (SVMs). In this study, a well known reference model for solving the problem proposed by Veropoulos et al., is first studied. From the aspect of loss function, the reference cost sensitive prototype is identified as a penalty-regularized model. Intuitively, the loss function can change not only the penalty but also the margin to recover the biased decision boundary. This study focuses mainly on the effect from the margin and then extends the model to a more general modification. As proposed in the prototype, the modification first adopts an inversed proportional regularized penalty to re-weight the imbalanced classes. In addition to the penalty regularization, the modification then employs a margin compensation to lead the margin to be lopsided, which enables the decision boundary drift. Two regularization factors, the penalty and margin, are hence suggested for achieving an unbiased classification. The margin compensation, associating with the penalty regularization, is here utilized to calibrate and refine the biased decision boundary to further reduce the bias. With the area under the receiver operating characteristic curve (AuROC) for examining the performance, the modification shows relative higher scores than the reference model, even though the optimal performance is achieved by the reference model. Some useful characteristics found empirically are also included, which may be convenient for the future applications. All the theoretical descriptions and experimental validations show the proposed model's potential to compete for highly unbiased accuracy in a complex imbalanced dataset.

We extend the conditional geometric score (CGS) classifier of Gotoh and Takeda for binary linear classification to a nonlinear one, which we call the @b-support vector classifier (SVC), and investigate the equivalence between the @b-SVC and the (extended) @n-SVC. The CGS classifier has recently been found to be equivalent to the extended @n-SVC of Perez-Cruz et al. and, especially in the convex case, equivalent to the @n-SVC of Scholkopf et al. The CGS problem is to minimize a risk measure known as the conditional value-at-risk (@b-CVaR). In this paper, we discuss theoretical aspects, mainly generalization performance, of the @b-SVC. The formula of a generalization error bound includes the @b-CVaR or a related quantity. It implies that the minimum @b-CVaR leads to a small generalization error bound of the @b-SVC. The viewpoint of CVaR minimization is useful to ensure the validity of not only the @b-SVC but also the (extended) @n-SVC.

Stability and bias-variance analysis are two powerful tools to better understand learning algorithms. We use these tools to analyze a class of support vector machines (SVMs) that try to reduce classifier complexity. The motivation for doing this is to compare the original and modified SVMs on two behavioral dimensions (a) stability and (b) learning behavior. Our preliminary experimental results show that (i) the class of algorithms which reduce classifier complexity by reducing the number of support vectors (SVs) are potentially unstable and (ii) the learning behavior is quite similar to the original SVMs.

Since the loss function is so important in statistical learning, this paper proposes the concept of adding heavier penalties to the heterogeneous examples of a dataset to achieve a stricter convex loss function for optimization. The concept was realized by changing the class labels of support vector machines (SVM) into greater real values. Using the magnified real-valued class labels to convey the additional penalties, an elementary stage-wise classifier was developed to achieve a high training accuracy. In this article, the original theory and induced corresponding properties of the stage-wise classifier are presented for further applications. Two types of re-weighting rules were devised in the connection of consecutive stages to produce the heavier penalties. Compared to a qualified underlying prototype, the empirical results showed that the classification complexity of the proposed classifier was increased accordingly as the accuracy of the classifier was improved due to various additional penalties. Although the stricter penalties might cause an undesirable over-fitting, the flexible re-weighting strategy is still beneficial for some application.

In this paper, a novel classifier, called adaptive local hyperplane, is proposed for pattern classification. The experimental results on 11 real data sets demonstrate that the proposed classifier outperforms, on average, all the other seven benchmarking classifiers. In particular, it is the best classifier in 10 out of 11 data sets, and it is the close second best for just one data set.

Decisions taken by support vector machines (SVM) are hard to interpret from a human perspective. We take advantage of a compact SVM solution previously developed, known as growing support vector classifier (GSVC), to provide interpretation to SVM decisions in terms of input space segmentation in Voronoi sections (determined by the prototypes extracted during the GSVC training method) plus rules built as a linear combination of input variables. We show by means of experiments on public domain datasets that the resulting interpretable machines have high fidelity, and an accuracy comparable to the SVM.

A classifier system is a machine learning system that learns syntactically simple string rules (called classifiers) through a genetic algorithm to guide its performance in an arbitrary environment. In a classifier system, the bucket brigade algorithm is used to solve the problem of credit assignment, which is a critical problem in the field of reinforcement learning. In this paper, we propose a new approach to fuzzy classifier systems and a neuro-fuzzy system referred to as ACSNFIS to implement the proposed fuzzy classifier system. The proposed system is tested by the balancing problem of a cart pole and the back-driving problem of a truck to demonstrate its performance.

This paper presents a two-level stacked generalization scheme composed of three generalizers of support vector machines (SVMs) for image classification. They are color, texture, and high-level concept SVMs. The focus of this paper is to investigate two training strategies based on two-fold cross-validation and non-cross-validation for the proposed classification scheme by evaluating their classification performances, margin of the hyperplane and numbers of support vectors of SVMs. The results show that the non-cross-validation training method performs better, having higher correct classification rates, larger margin of the hyperplane, and smaller numbers of support vectors.

An implicit Lagrangian [Math. Programming Ser. B 62 (1993) 277] formulation of a support vector machine classifier that led to a highly effective iterative scheme [J. Machine Learn. Res. 1 (2001) 161] is solved here by a finite Newton method. The proposed method, which is extremely fast and terminates in 6 or 7 iterations, can handle classification problems in very high dimensional spaces, e.g. over 28,000, in a few seconds on a 400 MHz Pentium II machine. The method can also handle problems with large datasets and requires no specialized software other than a commonly available solver for a system of linear equations. Finite termination of the proposed method is established in this work.

This paper proposes a modified version of support vector machines, called C-ascending support vector machine, to model non-stationary financial time series. The C-ascending support vector machines are obtained by a simple modification of the regularized risk function in support vector machines, whereby the recent e-insensitive errors are penalized more heavily than the distant e-insensitive errors. This procedure is based on the prior knowledge that in the non-stationary financial time series the dependency between input variables and output variable gradually changes over the time, specifically, the recent past data could provide more important information than the distant past data. In the experiment, C-ascending support vector machines are tested using three real futures collected from the Chicago Mercantile Market. It is shown that the C-ascending support vector machines with the actually ordered sample data consistently forecast better than the standard support vector machines, with the worst performance when the reversely ordered sample data are used. Furthermore, the C-ascending support vector machines use fewer support vectors than those of the standard support vector machines, resulting in a sparser representation of solution.