Search

This report is the proceedings from the NIPS'93 postconference workshop entitled 'Pulling It All Together: Methods for Combining Neural Networks.' The goal of the workshop was to examine theoretical aspects and heuristic methods for combining existing neural network algorithms to generate systems which have improved performance properties. This report includes an outline and summary of the workshop as well as the slides used in each participant's presentation. Averaging, Ensemble methods, Committees.

An algorithm for generating tree structured neural networks using a soft-competitive recursive partitioning rule is described. It is demonstrated that the algorithm grows robust, honest estimators. Preliminary results on a 10 class, 240 dimensional OCR classification task are presented which show that the tree out-performs backpropagation. Arguments are made which suggest why this should be the case. The connection of the soft-competitive splitting rule to the twoing rule is described.... Soft-competition, CART, Neural networks

This paper presents a general theoretical framework for ensemble methods of constructing significantly improved regression estimates. Given a population of regression estimators, the authors construct a hybrid estimator that is as good or better in the mean square error sense than any estimator in the population. They argue that the ensemble method presented has several properties: (1) it efficiently uses all the networks of a population -- none of the networks need to be discarded; (2) it efficiently uses all of the available data for training without over-fitting; (3) it inherently performs regularization by smoothing in functional space, which helps to avoid over-fitting; (4) it utilizes local minima to construct improved estimates whereas other neural network algorithms are hindered by local minima; (5) it is ideally suited for parallel computation; (6) it leads to a very useful and natural measure of the number of distinct estimators in a population; and (7) the optimal parameters of the ensemble estimator are given in closed form. Experimental results show that the ensemble method dramatically improves neural network performance on difficult real-world optical character recognition tasks., Sponsored in part by the Office of Naval Research (ONR) and the National Science Foundation (NSF). Pub. in Neural Networks for Speech and Image Processing, 1992.

A data-driven algorithm for partitioning many-class classification problems is presented. The algorithm generates tree-structured hybrid networks with controller nets at tree branches and local expert nets at the leaves. The controller nets recursively partition the feature space according to a novel misclassification minimization rule designed to create groupings of the classes which simplify the classification task. Each local expert is trained only on a subset of the training data corresponding to one of the partitions. The advantage to this approach is that the classification task that each local expert performs is greatly simplified. This simplification helps to avoid the curse of dimensionality and scaling problems by allowing the local expert nets to focus their search for structure in a small portion of the input space.... Cart, Recursive partitioning, Hybrid networks, Misclassification matrix.