Search

It has been reported that training deep neural networks is more difficult than training shallow neural networks. Hinton et al. proposed deep belief networks with a learning algorithm that trains one layer at a time. A much better generalization can be achieved when pre‐training each layer with an unsupervised learning algorithm. Since then, deep neural networks have been extensively studied. On the other hand, it has been revealed that singular points affect the training dynamics of the learning models such as neural networks and cause a standstill of training. Naturally, training deep neural networks suffer singular points. As described in this paper, we present a deep neural network model that has fewer singular points than the usual one. First, we demonstrate that some singular points in the deep real‐valued neural network, which is equivalent to a deep complex‐valued neural network, have been resolved as its inherent property. Such deep neural networks are less likely to become trapped in local minima or plateaus caused by critical points. Results of experiments on the two spirals problem, which has an extreme nonlinearity, support our theory. Copyright © 2017 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

In recent years, Hopfield neural networks using Clifford algebra have been studied. Clifford algebra is also referred to as geometric algebra, and is useful to deal with geometric objects. There are three kinds of Clifford algebra with degree 2; complex, hyperbolic, and dual-numbered. Complex-valued Hopfield neural networks have been studied by many researchers. Several models of hyperbolic Hopfield neural networks have also been proposed. It has been difficult to construct dual-numbered Hopfield neural networks. In this work, we propose dual-numbered Hopfield neural networks by modification of hyperbolic Hopfield neural networks with the split activation function. The stability condition and Hebbian learning rule are also provided. © 2017 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc. [ABSTRACT FROM AUTHOR]