Search

We present a systematic method for incorporating prior knowledge (hints) into the learning-from-examples paradigm. The hints are represented in a canonical form that is compatible with descent techniques for learning. All the hints are fed to the learning process in the form of examples, and examples of the function are treated on equal footing with the rest of the hints. During learning, examples from different hints are selected for processing according to a fixed or adaptive schedule. Fixed schedules specify the relative emphasis of each hint, and adaptive schedules are based on how well each hint has been learned so far. We discuss adaptive minimization which is based on estimates of the overall learning error.

This chapter contains sections titled: The Problem, The Generalized Delta Rule, Simulation Results, Some Further Generalizations, Conclusion

A new descent algorithm based on direction searches in the osculating plane as opposed to the negative gradient is described. The major practical difference between the implementation to this approach and quasi-Newton algorithms is the use of the Hessian in place of the inverse Hessian. Thus, the descent algorithm is most suitable for problems for which it is practical to recalculate the Hessian a number of times during the solution process. A number of comparative results with popular quasi-Newton algorithms are provided. Extensive discussion of the implementation details is included along with documentation of a convenient computer program CRATER for unconstrained optimization analysis. (Author)

This paper describes NOA, a package of Fortran subroutines for minimizing a locally Lipschitz continuous function subject to locally Lipschitzian inequality and equality constraints, general linear constraints and simple upper and lower bounds. The package implements several descent methods that accumulate subgradients of the problem functions and use quadratic programming for search direction finding. We discuss some choices made in the implementation and indicate their potential merits and drawbacks.

The authors demonstrate that the computational capabilities of Kohonen's algorithm provide an unified approach to such diverse fields as sensory mappings, combinatorial optimization, and learning in motor control. For a discrete probability distribution of the training inputs, the formation of the mapping can be described as a probabilistic descent in a potential. In view of their wide applicability, the principles of the algorithm might also be inherent to the maturation of biological brains and could help to achieve a better understanding of these processes from a more unified point of view. >

We give some ideas which lead to descent methods for minimizing nondifferentiable functions. Such methods have been published in several papers and they all involve the same concept, namely the e — subdifferential.

The use of the model of the rooted tree in many phases of linguistics needs no introductory comment. Its most anciently familiar form is the family tree in which so-called lines of descent are represented by edges and paths, languages (given or reconstructed) by vertices, subancestor languages by nodes or vertices of local degree greater than two, the overall ancestor by the root, and the passage of time by direction away from the root.47 In the history of scholarship the setting up of family trees may have drawn inspiration from analogous uses of tree models in other fields of knowledge, notably from the evolutionary tree with its ‘lines of descent’ and from the manuscript stemma used by students of written and printed texts to show the dependency (by copying, etc.) of one witness from another. For reasons of historical accident, as well as for intrinsic ones, it is the last-mentioned influence in particular which has been at work, and which has turned out to be fruitful.

We present an approach to visualizing gradient descent methods and discuss its application in the context of deep neural network (DNN) training. The result is a novel type of training error curve (a) that allows for an exploration of each individual gradient descent iteration at line search level; (b) that reflects how a DNN’s training error varies along each of the descent directions considered; (c) that is consistent with the traditional training error versus training iteration view commonly used to monitor a DNN’s training progress. We show how these three levels of detail can be easily realized as the three stages of Shneiderman’s Visual Information Seeking Mantra. This suggests the design and development of a new interactive visualization tool for the exploration of DNN learning processes. We present an example that showcases a conceivable interactive workflow when working with such a tool. Moreover, we give a first impression of a possible DNN hyperparameter analysis.

In this paper we review recent theoretical approaches for analysing the dynamics of on-line learning in multilayer neural networks using methods adopted from statistical physics. The analysis is based on monitoring a set of macroscopic variables from which the generalisation error can be calculated. A closed set of dynamical equations for the macroscopic variables is derived analytically and solved numerically. The theoretical framework is then employed for defining optimal learning parameters and for analysing the incorporation of second order information into the learning process using natural gradient descent and matrix-momentum based methods. We will also briefly explain an extension of the original framework for analysing the case where training examples are sampled with repetition.