Search

It has been reported that training deep neural networks is more difficult than training shallow neural networks. Hinton et al. proposed deep belief networks with a learning algorithm that trains one layer at a time. A much better generalization can be achieved when pre‐training each layer with an unsupervised learning algorithm. Since then, deep neural networks have been extensively studied. On the other hand, it has been revealed that singular points affect the training dynamics of the learning models such as neural networks and cause a standstill of training. Naturally, training deep neural networks suffer singular points. As described in this paper, we present a deep neural network model that has fewer singular points than the usual one. First, we demonstrate that some singular points in the deep real‐valued neural network, which is equivalent to a deep complex‐valued neural network, have been resolved as its inherent property. Such deep neural networks are less likely to become trapped in local minima or plateaus caused by critical points. Results of experiments on the two spirals problem, which has an extreme nonlinearity, support our theory. Copyright © 2017 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

A complex-valued Hopfield neural network (CHNN) is a model of a multistate Hopfield neural network, and has been applied to the storage of multilevel data. Weak noise tolerance, however, is a disadvantage of CHNNs. Symmetric CHNNs (SCHNNs), modified CHNNs, improve the noise tolerance of CHNNs. In the present work, we study the global and local minima of SCHNNs with one training pattern. In CHNNs, the global minima are the training and rotated patterns, and there are no local minima. In SCHNNs, it has been hard to determine all the global and local minima. It is thought that the global minima are the training and reversed patterns, and that there are no local minima in most cases. In the present work, however, we find many local minima, and show that they are very weak attractors, which reduce noise tolerance very little. In addition, we determine all the global minima of SCHNNs. [ABSTRACT FROM AUTHOR]

Abstract: Most of local minima caused by the hierarchical structure can be resolved by extending the real-valued neural network to complex numbers. It was proved in 2000 that a critical point of the real-valued neural network with hidden neurons always gives many critical points of the real-valued neural network with hidden neurons. These critical points consist of many lines in the parameter space which could be local minima or saddle points. Local minima cause plateaus which have a strong negative influence on learning. However, most of the critical points of complex-valued neural network are saddle points unlike those of the real-valued neural network. This is a prominent property of the complex-valued neural network. [Copyright &y& Elsevier]