Search

Extreme learning machine (ELM) has shown to be a suitable algorithm for classification problems. Several ensemble meta-algorithms have been developed in order to generalize the results of ELM models. Ensemble approaches introduced in the ELM literature mainly come from boosting and bagging frameworks. The generalization of these methods relies on data sampling procedures, under the assumption that training data are heterogeneously enough to set up diverse base learners. The proposed ELM ensemble model overcomes this strong assumption by using the negative correlation learning (NCL) framework. An alternative diversity metric based on the orthogonality of the outputs is proposed. The error function formulation allows us to develop an analytical solution to the parameters of the ELM base learners, which significantly reduce the computational burden of the standard NCL ensemble method. The proposed ensemble method has been validated by an experimental study with a variety of benchmark datasets, comparing it with the existing ensemble methods in ELM. Finally, the proposed method statistically outperforms the comparison ensemble methods in accuracy, also reporting a competitive computational burden (specially if compared to the baseline NCL-inspired method).

Extreme Learning Machine (ELM) has proven to be an efficient and speedy algorithm for classification. In order to generalize the results of standard ELM, several ensemble meta-algorithms have been implemented. On this manuscript, we propose a hierarchical ensemble methodology that promotes diversity among the elements of an ensemble, explicitly through the loss function in the single-hidden-layer feedforward network version of ELM. The diversity term in the loss function is justified using the concept of regularization from the Negative Correlation Learning framework. Statistical tests show that our proposal is competitive in both performance and diversity measures against bagging and boosting ensemble methodologies.

This work analyses the complementarity and contrast between two metrics commonly used for evaluating the quality of a binary classifier: the correct classification rate or accuracy, C, and the F1 metric, which is very popular when dealing with imbalanced datasets. Based on this analysis, a set of constraints relating C and F1 are defined as a function of the ratio of positive patterns in the dataset. We evaluate the possibility of using a multi-objective evolutionary algorithm guided by this pair of metrics to optimise binary classification models. To check the validity of the constraints, we perform an empirical analysis considering 26 benchmark datasets obtained from the UCI repository and an interesting liver transplant dataset. The results show that the relation is fulfilled and that the use of the algorithm for simultaneously optimising the pair (C,F1) leads to a generally balanced accuracy for both classes. The experiments also reveal that, in some cases, better results are obtained by using the majority class as the positive class instead of using the minority one, which is the most common approach with imbalanced datasets.

Ensembles are a widely implemented approach in the machine learning community and their success is traditionally attributed to the diversity within the ensemble. Most of these approaches foster diversity in the ensemble by data sampling or by modifying the structure of the constituent models. Despite this, there is a family of ensemble models in which diversity is explicitly promoted in the error function of the individuals. The negative correlation learning (NCL) ensemble framework is probably the most well-known algorithm within this group of methods. This article analyzes NCL and reveals that the framework actually minimizes the combination of errors of the individuals of the ensemble instead of minimizing the residuals of the final ensemble. We propose a novel ensemble framework, named global negative correlation learning (GNCL), which focuses on the optimization of the global ensemble instead of the individual fitness of its components. An analytical solution for the parameters of base regressors based on the NCL framework and the global error function proposed is also provided under the assumption of fixed basis functions (although the general framework could also be instantiated for neural networks with nonfixed basis functions). The proposed ensemble framework is evaluated by extensive experiments with regression and classification data sets. Comparisons with other state-of-the-art ensemble methods confirm that GNCL yields the best overall performance.

New diversification strategies, along with other naive strategies as 1/N portfolios, have been proposed in the literature as a method for overcoming concentration limitations of the mean–variance model. However, it is not clear whether these strategies outperform the classical mean–variance model in all scenarios. Motivated by these points, this manuscript contributes an experimental study in which 11 diversification and mean–variance strategies are compiled and compared with a complete repository of 10 portfolio time series problems with three different estimation windows (composing a total of 30 datasets) and then evaluated using four performance metrics. Additionally, a novel purely data-driven method for determining the optimal value of the hyper-parameter associated with each approach is also proposed. Unlike results previously found in the literature, the empirical results obtained in this study show that equally weighed models obtain the worst ranking in all evaluation metrics except for the stability index, which is hypothetically due to the hyper-parameter optimization raising the transaction cost debate.

Accuracy has been used traditionally to evaluate the performance of classifiers. However, it is well known that accuracy is not able to capture all the different factors that characterize the performance of a multiclass classifier. In this manuscript, accuracy is studied and analyzed as a weighted average of the classification rate of each class. This perspective allows us to propose the dispersion of the classification rate of each class as its complementary measure. In this sense, a graphical performance metric, which is defined in a two dimensional space composed by accuracy and dispersion, is proposed to evaluate the performance of classifiers. We show that the combined values of accuracy and dispersion must fall within a clearly bounded two dimensional region, different for each problem. The nature of this region depends only on the a priori probability of each class, and not on the classifier used. Thus, the performance of multiclassifiers is represented in a two dimensional space where the models can be compared in a more fair manner, providing greater awareness of the strategies that are more accurate when trying to improve the performance of a classifier. Furthermore we experimentally analyze the behavior of seven different performance metrics based on the computation of the confusion matrix values in several scenarios, identifying clusters and relationships between measures. As shown in the experimentation, the graphical metric proposed is specially suitable in challenging, highly imbalanced and with a high number of classes datasets. The approach proposed is a novel point of view to address the evaluation of multiclassifiers and it is an alternative to other evaluation measures used in machine learning.

In this paper, the neural network version of Extreme Learning Machine (ELM) is used as a base learner for an ensemble meta-algorithm which promotes diversity explicitly in the ELM loss function. The cost function proposed encourages orthogonality (scalar product) in the parameter space. Other ensemble-based meta-algorithms from AdaBoost family are used for comparison purposes. Both accuracy and diversity presented in our proposal are competitive, thus reinforcing the idea of introducing diversity explicitly.

Artificial neural networks (ANNs) have traditionally been seen as black-box models, because, although they are able to find ``hidden'' relations between inputs and outputs with a high approximation capacity, their structure seldom provides any insights on the structure of the functions being approximated. Several research papers have tried to debunk the black-box nature of ANNs, since it limits the potential use of ANNs in many research areas. This paper is framed in this context and proposes a methodology to determine the individual and collective effects of the input variables on the outputs for classification problems based on the ANOVA-functional decomposition. The method is applied after the training phase of the ANN and allows researchers to rank the input variables according to their importance in the variance of the ANN output. The computation of the sensitivity indices for product unit neural networks is straightforward as those indices can be calculated analytically by evaluating the integrals in the ANOVA decomposition. Unfortunately, the sensitivity indices associated with ANNs based on sigmoidal basis functions or radial basis functions cannot be calculated analytically. In this paper, the indices for those kinds of ANNs are proposed to be estimated by the (quasi-) Monte Carlo method.

Learning from label proportions is the term used for the learning paradigm where the training data is provided in groups (or “bags”), and only the label proportion for each bag is known. The objective is to learn a model to predict the class labels of individual instances. This paradigm presents very different applications, specially concerning anonymous data. Two different iterative strategies are proposed to deal with this type of problems, both based on optimising the class membership of the instances using the estimated pattern distribution per bag and the label proportions. Discriminant analysis is reformulated to deal with non-crisp class memberships. A thorough set of experiments is conducted to test: (1) the performance gap between these approaches and the fully supervised setting, (2) the potential advantages of optimising class memberships by our proposals, and (3) the influence of factors such as the bag size and the number of classes of the problem in the performance.

The recently coined term “learning from label proportions” refers to a new learning paradigm where training data is given by groups (also denoted as “bags”), and the only known information is the label proportion of each bag. The aim is then to construct a classification model to predict the class label of an individual instance, which differentiates this paradigm from the one of multi-instance learning. This learning setting presents very different applications in political science, marketing, healthcare and, in general, all fields in relation with anonymous data. In this paper, two new strategies are proposed to tackle this kind of problems. Both proposals are based on the optimisation of pattern class memberships using the data distribution in each bag and the known label proportions. To do so, linear discriminant analysis has been reformulated to work with non-crisp class memberships. The experimental part of this paper sets different objetives: 1) study the difference in performance, comparing our proposals and the fully supervised setting, 2) analyse the potential benefits of refining class memberships by the proposed approaches, and 3) test the influence of other factors in the performance, such as the number of classes or the bag size. The results of these experiments are promising, but further research should be encouraged for studying more complex data configurations.

Ordinal classication considers those classication problems where the labels of the variable to predict follow a given order. Naturally, labelled data is scarce or di_cult to obtain in this type of problems because, in many cases, ordinal labels are given by an user or expert (e.g. in recommendation systems). Firstly, this paper develops a new strategy for ordinal classi_cation where both labelled and unlabelled data are used in the model construction step (a scheme which is referred to as semi-supervised learning). More specically, the ordinal version of kernel discriminant learning is extended for this setting considering the neighbourhood information of unlabelled data, which is proposed to be computed in the feature space induced by the kernel function. Secondly, a new method for semi-supervised kernel learning is devised in the context of ordinal classi_cation, which is combined with our developed classi_cation strategy to optimise the kernel parameters. The experiments conducted compare 6 different approaches for semi-supervised learning in the context of ordinal classication in a battery of 30 datasets, showing 1) the good synergy of the ordinal version of discriminant analysis and the use of unlabelled data and 2) the advantage of computing distances in the feature space induced by the kernel function.

The machine learning community has traditionally used correct classification rates or accuracy (C) values to measure classifier performance and has generally avoided presenting classification levels of each class in the results, especially for problems with more than two classes. C values alone are insufficient because they cannot capture the myriad of contributing factors that differentiate the performance of two different classifiers. Receiver Operating Characteristic (ROC) analysis is an alternative to solve these difficulties, but it can only be used for two-class problems. For this reason, this paper proposes a new approach for analysing classifiers based on two measures: C and sensitivity (S) (i.e., the minimum of accuracies obtained for each class). These measures are optimised through a two-stage evolutionary process. It was conducted by applying two sequential fitness functions in the evolutionary process, including entropy (E) for the first stage and a new fitness function, area (A), for the second stage. By using these fitness functions, the C level was optimised in the first stage, and the S value of the classifier was generally improved without significantly reducing C in the second stage. This two-stage approach improved S values in the generalisation set (whereas an evolutionary algorithm (EA) based only on the S measure obtains worse S levels) and obtained both high C values and good classification levels for each class. The methodology was applied to solve 16 benchmark classification problems and two complex real-world problems in analytical chemistry and predictive microbiology. It obtained promising results when compared to other competitive multi-class classification algorithms and a multi-objective alternative based on E and S.

This paper proposes a radial basis function neural network (RBFNN), called the q -Gaussian RBFNN, that reproduces different radial basis functions (RBFs) by means of a real parameter q . The architecture, weights and node topology are learnt through a hybrid algorithm (HA). In order to test the overall performance, an experimental study with sixteen data sets taken from the UCI repository is presented. The q -Gaussian RBFNN was compared to RBFNNs with Gaussian, Cauchy and inverse multiquadratic RBFs in the hidden layer and to other probabilistic classifiers, including different RBFNN design methods, support vector machines (SVMs), a sparse classifier (sparse multinomial logistic regression, SMLR) and a non-sparse classifier (regularized multinomial logistic regression, RMLR). The results show that the q -Gaussian model can be considered very competitive with the other classification methods.

We propose a multilogistic regression model based on the combination of linear and product-unit models, where the product-unit nonlinear functions are constructed with the product of the inputs raised to arbitrary powers. The estimation of the coefficients of the model is carried out in two phases. First, the number of product-unit basis functions and the exponents'' vector are determined by means of an evolutionary neural network algorithm. Afterwards, a standard maximum likelihood optimization method determines the rest of the coefficients in the new space given by the initial variables and the product-unit basis functions previously estimated. We compare the performance of our approach with the logistic regression built on the initial variables and several learning classification techniques. The statistical test carried out on twelve benchmark datasets shows that the proposed model is competitive in terms of the accuracy of the classifier.

The Extreme Learning Machine classifier is used to perform the perturbative method known as Sensitivity Analysis. The method returns a measure of class sensitivity per attribute. The results show a strong consistency for classifiers with different random input weights. In order to present the results obtained in an intuitive way, two forms of representation are proposed and contrasted against each other. The relevance of both attributes and classes is discussed. Class stability and the ease with which a pattern can be correctly classified are inferred from the results. The method can be used with any classifier that can be replicated with different random seeds.

This paper presents a novel method for generally adapting ordinal classification models. We essentially rely on the assumption that the ordinal structure of the set of class labels is also reflected in the topology of the instance space. Under this assumption, this paper proposes an algorithm in two phases that takes advantage of the ordinal structure of the dataset and tries to translate this ordinal structure in the total ordered real line and then to rank the patterns of the dataset. The first phase makes a projection of the ordinal structure of the feature space. Next, an evolutionary algorithm tunes the first projection working with the misclassified patterns near the border of their right class. The results obtained in seven ordinal datasets are competitive in comparison with state-of-the-art algorithms in ordinal regression, but with much less computational time in datasets with many patterns.

Ordinal classification problems are an active research area in the machine learning community. Many previous works adapted state-of-art nominal classifiers to improve ordinal classification so that the method can take advantage of the ordinal structure of the dataset. However, these method improvements often rely upon a complex mathematical basis and they usually are attached to the training algorithm and model. This paper presents a novel method for generally adapting classification and regression models, such as artificial neural networks or support vector machines. The ordinal classification problem is reformulated as a regression problem by the reconstruction of a numerical variable which represents the different ordered class labels. Despite the simplicity and generality of the method, results are competitive in comparison with very specific methods for ordinal regression.

We present an extra measurement for classifiers, responding to the need to evaluate them with more than accuracy alone. This measure should be able to express, at least to some degree, the extent to which all classes are taken into account in a classification problem. In this communication we propose sensitivity dispersion (being as it is, the associated statistical dispersion measurement of accuracy), as the appropriate measure to have a more complete evaluation of the quality of classifiers. We use the Evolutionary Extreme Learning Machine algorithm, with a specific fitness function to optimize both measures simultaneously, and we compare it with other classifiers.

This paper proposes a Radial Basis Function Neural Network (RBFNN) which reproduces different Radial Basis Functions (RBFs) by means a real parameter q, named q-Gaussian RBFNN The architecture, weights and node topology are learnt through a Hybrid Algorithm (HA) with the iRprop+ algorithm as the local improvement procedure In order to test its overall performance, an experimental study with eleven datasets, taken from the UCI repository is presented The RBFNN with the q-Gaussian is compared to RBFNN with Gaussian, Cauchy and Inverse Multiquadratic RBFs.

Multi-class pattern recognition has a wide range of applications including handwritten digit recognition (Chiang, 1998), speech tagging and recognition (Athanaselis, Bakamidis, Dologlou, Cowie, Douglas-Cowie & Cox, 2005), bioinformatics (Mahony, Benos, Smith & Golden, 2006) and text categorization (Massey, 2003). This chapter presents a comprehensive and competitive study in multi-class neural learning which combines different elements, such as multilogistic regression, neural networks and evolutionary algorithms. The Logistic Regression model (LR) has been widely used in statistics for many years and has recently been the object of extensive study in the machine learning community. Although logistic regression is a simple and useful procedure, it poses problems when is applied to a real-problem of classification, where frequently we cannot make the stringent assumption of additive and purely linear effects of the covariates. A technique to overcome these difficulties is to augment/replace the input vector with new variables, basis functions, which are transformations of the input variables, and then to use linear models in this new space of derived input features. Methods like sigmoidal feed-forward neural networks (Bishop, 1995), generalized additive models (Hastie & Tibshirani, 1990), and PolyMARS (Kooperberg, Bose & Stone, 1997), which is a hybrid of Multivariate Adaptive Regression Splines (MARS) (Friedman, 1991) specifically designed to handle classification problems, can all be seen as different nonlinear basis function models. The major drawback of these approaches is stating the typology and the optimal number of the corresponding basis functions. Logistic regression models are usually fit by maximum likelihood, where the Newton-Raphson algorithm is the traditional way to estimate the maximum likelihood a-posteriori parameters. Typically, the algorithm converges, since the log-likelihood is concave. It is important to point out that the computation of the Newton-Raphson algorithm becomes prohibitive when the number of variables is large. Product Unit Neural Networks, PUNN, introduced by Durbin and Rumelhart (Durbin & Rumelhart, 1989), are an alternative to standard sigmoidal neural networks and are based on multiplicative nodes instead of additive ones.

This paper proposes a hybrid neural network model using a possible combination of different transfer projection functions (sigmoidal unit, SU, product unit, PU) and kernel functions (radial basis function, RBF) in the hidden layer of a feed-forward neural network. An evolutionary algorithm is adapted to this model and applied for learning the architecture, weights and node typology. Three different combined basis function models are proposed with all the different pairs that can be obtained with SU, PU and RBF nodes: product-sigmoidal unit (PSU) neural networks, product-radial basis function (PRBF) neural networks, and sigmoidal-radial basis function (SRBF) neural networks; and these are compared to the corresponding pure models: product unit neural network (PUNN), multilayer perceptron (MLP) and the RBF neural network. The proposals are tested using ten benchmark classification problems from well known machine learning problems. Combined functions using projection and kernel functions are found to be better than pure basis functions for the task of classification in several datasets.

The paper describes a methodology for constructing a possible combination of different basis functions (sigmoidal and product) for the hidden layer of a feed forward neural network, where the architecture, weights and node typology are learned based on evolutionary programming. This methodology is tested using simulated Gaussian data set classification problems with different linear correlations between input variables and different variances. It was found that combined basis functions are the more accurate for classification than pure sigmoidal or product-unit models. Combined basis functions present competitive results which are obtained using linear discriminant analysis, the best classification methodology for Gaussian data sets.