Model selection in Neural Networks: Some difficulties

To link to full-text access for this article, visit this link: http://dx.doi.org/10.1016/j.ejor.2004.05.026 Byline: B. Curry, P.H. Morgan Keywords: Neural Networks; Network weights; Hidden layers; Backpropagation; Polytope Abstract: This paper considers two related issues regarding feedforward Neural Networks (NNs). The first involves the question of whether the network weights corresponding to the best fitting network are unique. Our empirical tests suggest an answer in the negative, whether using standard Backpropagation algorithm or our preferred direct (non-gradient-based) search procedure. We also offer a theoretical analysis which suggests that there will almost inevitably be functional relationships between network weights. The second issue concerns the use of standard statistical approaches to testing the significance of weights or groups of weights. Treating feedforward NNs as an interesting way to carry out nonlinear regression suggests that statistical tests should be employed. According to our results, however, statistical tests can in practice be indeterminate. It is rather difficult to choose either the number of hidden layers or the number of nodes on this basis. Author Affiliation: Cardiff Business School, Cardiff University, Aberconway Building, Colum Drive, Cardiff CF10 3EU, UK Article History: Received 26 February 2003; Accepted 25 May 2004