Search

A large number of measures have been developed for evaluating the performance of classification rules. Some of these have been developed to meet the practical requirements of specific applications, but many others-which here we call 'classification accuracy' criteria-represent different ways of balancing the different kinds of misclassification which may be made. This paper reviews classification accuracy criteria. However, the literature is now so large and diverse that a comprehensive list, covering all the measures and their variants, would probably be impossible. Instead, this paper embeds such measures in general framework, spanning the possibilities, and draws attention to relationships between them. Important points to note are, firstly, that different performance measures, by definition, measure different aspects of performance; secondly, that one should therefore carefully choose a measure to match the objectives of one's study; and, thirdly, that empirical comparisons between instruments measuring different aspects are of limited value. [ABSTRACT FROM AUTHOR]

Credit scoring methods for predicting creditworthiness have proven very effective in consumer finance. In light of the present financial crisis, such methods will become even more important. One of the outstanding issues in credit risk classification is population drift. This term refers to changes occurring in the population due to unexpected changes in economic conditions and other factors. In this paper, we propose a novel methodology for the classification of credit applications that has the potential to adapt to population drift as it occurs. This provides the opportunity to update the credit risk classifier as new labelled data arrives. Assorted experimental results suggest that the proposed method has the potential to yield significant performance improvement over standard approaches, without sacrificing the classifier's descriptive capabilities. [ABSTRACT FROM AUTHOR]

Star-galaxy classification is one of the most fundamental data-processing tasks in survey astronomy and a critical starting point for the scientific exploitation of survey data. Star-galaxy classification for bright sources can be done with almost complete reliability, but for the numerous sources close to a survey's detection limit each image encodes only limited morphological information about the source. In this regime, from which many of the new scientific discoveries are likely to come, it is vital to utilize all the available information about a source, both from multiple measurements and from prior knowledge about the star and galaxy populations. This also makes it clear that it is more useful and realistic to provide classification probabilities than decisive classifications. All these desiderata can be met by adopting a Bayesian approach to star-galaxy classification, and we develop a very general formalism for doing so. An immediate implication of applying Bayes's theorem to this problem is that it is formally impossible to combine morphological measurements in different bands without using colour information as well; however, we develop several approximations that disregard colour information as much as possible. The resultant scheme is applied to data from the UKIRT Infrared Deep Sky Survey (UKIDSS) and tested by comparing the results to deep Sloan Digital Sky Survey (SDSS) Stripe 82 measurements of the same sources. The Bayesian classification probabilities obtained from the UKIDSS data agree well with the deep SDSS classifications both overall (a mismatch rate of 0.022 compared to 0.044 for the UKIDSS pipeline classifier) and close to the UKIDSS detection limit (a mismatch rate of 0.068 compared to 0.075 for the UKIDSS pipeline classifier). The Bayesian formalism developed here can be applied to improve the reliability of any star-galaxy classification schemes based on the measured values of morphology statistics alone. [ABSTRACT FROM AUTHOR]

The performance results of a wide range of different classifiers applied to proteomic mass spectra data, in a blind comparative assessment organised by Bart Mertens, are reviewed. The different approaches are summarised, issues of how to evaluate and compare the predictions are described, and the results of the different methods are examined. Although the different methods perform differently, their rank ordering varies according to how one measures performance, so that one cannot draw unequivocal conclusions about which is 'best.' Instead, it is clear that what matters is not the method by itself, but the interaction of method and user - the degree of sophistication of the user with a method. Nevertheless, such competitions do serve the useful role of setting (constantly improving) baselines against which new researchers can pit their wits and methods, as well as providing standards against which new methods should be assessed. [ABSTRACT FROM AUTHOR]

This paper presents the top 10 data mining algorithms identified by the IEEE International Conference on Data Mining (ICDM) in December 2006: C4.5, k-Means, SVM, Apriori, EM, PageRank, AdaBoost, kNN, Naive Bayes, and CART. These top 10 algorithms are among the most influential data mining algorithms in the research community. With each algorithm, we provide a description of the algorithm, discuss the impact of the algorithm, and review current and further research on the algorithm. These 10 algorithms cover classification, clustering, statistical learning, association analysis, and link mining, which are all among the most important topics in data mining research and development. [ABSTRACT FROM AUTHOR]

The article comments on classifier technology in statistical science. Substantial advances have taken place in recent years. Developments including bootstrap and other resampling approaches have led to significant advances in classification and other statistical models. Substantial advantages offered by certain methods, like ensemble methods and support vector machines, over the earlier methods in enough situations are to be regarded major advances.

Abstract: Many studies have been made to compare the many different methods of supervised classification which have been developed. While conducting a large meta-analysis of such studies, we spotted some anomalous results relating to the Naive Bayes method. This paper describes our detailed investigation into these anomalies. We conclude that a very large comparative study probably mislabelled another method as Naive Bayes, and that the Statlog project used the right method, but possibly incorrectly reported its provenance. Such mistakes, while not too harmful in themselves, can become seriously misleading if blindly propagated by citations which do not examine the source material in detail. [Copyright &y& Elsevier]

In recent decades many highly sophisticated methods have been developed for supervised classification. These developments involve complex models requiring complicated iterative parameter estimation schemes, and can achieve unprecedented performance in terms of misclassification rate. However, in focusing efforts on the single performance criterion of misclassification rate, researchers have abstracted the problem beyond the bounds of practical usefulness, to the extent that the supposed performance improvements are irrelevant in comparison with other factors influencing performance. Examples of such factors are given. An illustration is provided of a new method which, for the particular problem of credit scoring, improves a relevant measure of classification performance while maintaining interpretability. Copyright © 2005 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Supervised classification problems in which the class sizes are very different are common. In such cases, nearest neighbour classifiers exhibit a non-monotonic relationship between the number of nearest neighbours and misclassification rate of each of the two classes separately. [Copyright &y& Elsevier]

We describe a strategy for assigning objects into classes based on characteristics which have not (yet) been measured—for example, assigning people into classes according to how they are predicted to behave in the future. The standard approach would take a training set on which predictor variables, that is variables which can be measured in the present, and behavioural variables, that is the ones to be used to define the classes, had both been measured. The space of behavioural variables would then be partitioned, perhaps using cluster analysis, and a model would be built to predict cluster membership from the predictor variables. New cases, for which only the predictor variables could be measured, would then be assigned to clusters in the space of the unobserved behavioural variables using the predictive model. However, when determining the behavioural classes by this approach no account is taken of future predictive accuracy. We describe a method which combines class homogeneity with predictive performance, yielding a natural single criterion which can be optimized. We illustrate on two real data sets, one arising from a travel survey, and one from a retail banking application. [ABSTRACT FROM PUBLISHER]

The consumer credit market is experiencing unprecedented change, increased competition, and new challenges. To cope with these developments, increasingly sophisticated mathematical and statistical tools are being used. Such tools are used to identify good and bad risks, to monitor customer performance, to characterize different behaviour patterns, and in a wide variety of other ways, at both individual and portfolio level. Examples of such applications and of the modern statistical tools developed to model them are given, including statistical staples such as logistic regression and naive Bayes, but also including more recent developments such as neural networks and recursive partitioning models. A key aspect is the development of methods for assessing performance of the models, and this is examined in detail. [ABSTRACT FROM PUBLISHER]

In this note we use examples from the literature to illustrate some poor practices in assessing the performance of supervised classification rules, and we suggest guidelines for better methodology . We also describe a new assessment criterion that is suitable for the needs of many practical problems. [ABSTRACT FROM AUTHOR]

Abstract: Classifying streaming data requires the development of methods which are computationally efficient and able to cope with changes in the underlying distribution of the stream, a phenomenon known in the literature as concept drift. We propose a new method for detecting concept drift which uses an exponentially weighted moving average (EWMA) chart to monitor the misclassification rate of an streaming classifier. Our approach is modular and can hence be run in parallel with any underlying classifier to provide an additional layer of concept drift detection. Moreover our method is computationally efficient with overhead O(1) and works in a fully online manner with no need to store data points in memory. Unlike many existing approaches to concept drift detection, our method allows the rate of false positive detections to be controlled and kept constant over time. [Copyright &y& Elsevier]