1. Combining cross-validation and confidence to measure fitness
- Author
-
Tony Martinez and D.R. Wilson
- Subjects
Set (abstract data type) ,Artificial neural network ,Computer science ,business.industry ,Generalization ,Artificial intelligence ,business ,Machine learning ,computer.software_genre ,Measure (mathematics) ,computer ,Cross-validation ,Task (project management) - Abstract
Neural network and machine learning algorithms often have parameters that must be tuned for good performance on a particular task. Leave-one-out cross-validation (LCV) accuracy is often used to measure the fitness of a set of parameter values. However, small changes in parameters often have no effect on LCV accuracy. Many learning algorithms can measure the confidence of a classification decision, but often confidence alone is an inappropriate measure of fitness. This paper proposes a combined measure of cross-validation and confidence (CVC) for obtaining a continuous measure of fitness for sets of parameters in learning algorithms. This paper also proposes the refined instance-based learning algorithm which illustrates the use of CVC in automated parameter tuning. Using CVC provides significant improvement in generalization accuracy on a collection of 31 classification tasks when compared to using LCV.
- Published
- 2003