Search

It has been reported that training deep neural networks is more difficult than training shallow neural networks. Hinton et al. proposed deep belief networks with a learning algorithm that trains one layer at a time. A much better generalization can be achieved when pre‐training each layer with an unsupervised learning algorithm. Since then, deep neural networks have been extensively studied. On the other hand, it has been revealed that singular points affect the training dynamics of the learning models such as neural networks and cause a standstill of training. Naturally, training deep neural networks suffer singular points. As described in this paper, we present a deep neural network model that has fewer singular points than the usual one. First, we demonstrate that some singular points in the deep real‐valued neural network, which is equivalent to a deep complex‐valued neural network, have been resolved as its inherent property. Such deep neural networks are less likely to become trapped in local minima or plateaus caused by critical points. Results of experiments on the two spirals problem, which has an extreme nonlinearity, support our theory. Copyright © 2017 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

In recent years, Hopfield neural networks using Clifford algebra have been studied. Clifford algebra is also referred to as geometric algebra, and is useful to deal with geometric objects. There are three kinds of Clifford algebra with degree 2; complex, hyperbolic, and dual-numbered. Complex-valued Hopfield neural networks have been studied by many researchers. Several models of hyperbolic Hopfield neural networks have also been proposed. It has been difficult to construct dual-numbered Hopfield neural networks. In this work, we propose dual-numbered Hopfield neural networks by modification of hyperbolic Hopfield neural networks with the split activation function. The stability condition and Hebbian learning rule are also provided. © 2017 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc. [ABSTRACT FROM AUTHOR]

Abstract: A critical point is a point on which the derivatives of an error function are all zero. It has been shown in the literatures that the critical points caused by the hierarchical structure of the real-valued neural network could be local minima or saddle points, whereas most of the critical points caused by the hierarchical structure are saddle points in the case of complex- valued neural networks. Several studies have demonstrated that that kind of singularity has a negative effect on learning dynamics in neural networks. In this paper, we will demonstrate via some examples that the decomposition of high- dimensional NNs into real-valued NNs equivalent to the original NNs yields the NNs that do not have critical points based on the hierarchical structure. [Copyright &y& Elsevier]

Abstract: Most of local minima caused by the hierarchical structure can be resolved by extending the real-valued neural network to complex numbers. It was proved in 2000 that a critical point of the real-valued neural network with hidden neurons always gives many critical points of the real-valued neural network with hidden neurons. These critical points consist of many lines in the parameter space which could be local minima or saddle points. Local minima cause plateaus which have a strong negative influence on learning. However, most of the critical points of complex-valued neural network are saddle points unlike those of the real-valued neural network. This is a prominent property of the complex-valued neural network. [Copyright &y& Elsevier]

This paper will prove the uniqueness theorem for 3-layered complex-valued neural networks where the threshold parameters of the hidden neurons can take non-zeros. That is, if a 3-layered complex-valued neural network is irreducible, the 3-layered complex-valued neural network that approximates a given complex-valued function is uniquely determined up to a finite group on the transformations of the learnable parameters of the complex-valued neural network. [ABSTRACT FROM AUTHOR]

A complex neural network is obtained from an ordinary network by extending the (real-valued) parameters, such as the weights and the thresholds, to complex values. Applications to problems involving complex numbers, such as communications systems, are expected. This paper presents the following uniqueness theorem. When a complex function is given, the three-layered neural network that approximates the function is uniquely determined by a certain finite group, if it is irreducible. The above finite group specifies the redundancy of the parameters in the complex neural network, but has a structure which is different from that of the real-valued neural network. The order of the finite group is examined, and it is shown that the redundancy of the complex-valued neural network is an exponent multiple of the redundancy of the real-valued neural network. Analysis of the redundancy is important in the theoretical investigation of the basic characteristics of complex-valued neural networks, such as the local minimum property. A sufficient condition is derived for the given three-layered complex-valued neural network to be minimal. The above results are shown, in essence, by extending the approach of Sussmann for real-valued neural networks. © 2003 Wiley Periodicals, Inc. Syst Comp Jpn, 34(14): 54–62, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10363 [ABSTRACT FROM AUTHOR]