Search

The three-state Ising neural network with synchronous updating and variable dilution is discussed starting from the appropriate Hamiltonians. The thermodynamic and retrieval properties are examined using replica mean-field theory. Capacity-temperature phase diagrams are derived for several values of the pattern activity and different gradations of dilution, and the information content is calculated. The results are compared with those for sequential updating. The effect of self-coupling is established. Also the dynamics is studied using the generating function technique for both synchronous and sequential updating. Typical flow diagrams for the overlap order parameter are presented. The differences with the signal-to-noise approach are outlined., Comment: 21 pages Latex, 12 eps figures and 1 ps figure

The thermodynamic and retrieval properties of the Blume-Emery-Griffiths neural network with synchronous updating and variable dilution are studied using replica mean-field theory. Several forms of dilution are allowed by pruning the different types of couplings present in the Hamiltonian. The appearance and properties of two-cycles are discussed. Capacity-temperature phase diagrams are derived for several values of the pattern activity. The results are compared with those for sequential updating. The effect of self-coupling is studied. Furthermore, the optimal combination of dilution parameters giving the largest critical capacity is obtained.

Hamiltonians for general multi-state spin-glass systems with Ising symmetry are derived for both sequential and synchronous updating of the spins. The possibly different behaviour caused by the way of updating is studied in detail for the (anti)-ferromagnetic version of the models, which can be solved analytically without any approximation, both thermodynamically via a free-energy calculation and dynamically using the generating functional approach. Phase diagrams are discussed and the appearance of two-cycles in the case of synchronous updating is examined. A comparative study is made for the Q-Ising and the Blume-Emery-Griffiths ferromagnets and some interesting physical differences are found. Numerical simulations confirm the results obtained.

The time evolution of an exactly solvable layered feedforward neural network with three-state neurons and optimizing the mutual information is studied for arbitrary synaptic noise (temperature). Detailed stationary temperature-capacity and capacity-activity phase diagrams are obtained. The model exhibits pattern retrieval, pattern-fluctuation retrieval and spin-glass phases. It is found that there is an improved performance in the form of both a larger critical capacity and information content compared with three-state Ising-type layered network models. Flow diagrams reveal that saddle-point solutions associated with fluctuation overlaps slow down considerably the flow of the network states towards the stable fixed-points., Comment: 17 pages Latex including 6 eps-figures

We introduce a spherical Hopfield-type neural network involving neurons and patterns that are continuous variables. We study both the thermodynamics and dynamics of this model. In order to have a retrieval phase a quartic term is added to the Hamiltonian. The thermodynamics of the model is exactly solvable and the results are replica symmetric. A Langevin dynamics leads to a closed set of equations for the order parameters and effective correlation and response function typical for neural networks. The stationary limit corresponds to the thermodynamic results. Numerical calculations illustrate our findings., 9 pages Latex including 3 eps figures, Addition of an author in the HTML-abstract unintentionally forgotten, no changes to the manuscript

We study supervised learning and generalisation in coupled perceptrons trained on-line using two learning scenarios. In the first scenario the teacher and the student are independent networks and both are represented by an Ashkin-Teller perceptron. In the second scenario the student and the teacher are simple perceptrons but are coupled by an Ashkin-Teller type four-neuron interaction term. Expressions for the generalisation error and the learning curves are derived for various learning algorithms. The analytic results find excellent confirmation in numerical simulations., Latex, 21 pages, 9 figures, iop style files included

A model of noise reduction for signal processing and other optimization tasks is introduced. Each noise source puts a symmetric constraint on the space of the signal vector within a tolerance bound. When the number of noise sources increases, sequences of transitions take place, causing the solution space to vanish. We find that the transition from an extended solution space to a shrunk space is retarded because of the symmetry of the constraints, in contrast with the analogous problem of pattern storage. For low tolerance, the solution space vanishes by volume reduction, whereas for high tolerance, the vanishing becomes more and more like percolation.

A complete self-control mechanism is proposed in the dynamics of neural networks through the introduction of a time-dependent threshold, determined in function of both the noise and the pattern activity in the network. Especially for sparsely coded models this mechanism is shown to considerably improve the storage capacity, the basins of attraction and the mutual information content of the network., Comment: 4 pages, 6 Postscript figures

Using a probabilistic approach, the parallel dynamics of fully connected Q-Ising neural networks is studied for arbitrary Q. A novel recursive scheme is set up to determine the time evolution of the order parameters through the evolution of the distribution of the local field, taking into account all feedback correlations. In contrast to extremely diluted and layered network architectures, the local field is no longer normally distributed but contains a discrete part. As an illustrative example, an explicit analysis is carried out for the first four time steps. For the case of the Q = 2 and Q = 3 model the results are compared with extensive numerical simulations and excellent agreement is found. Finally, equilibrium fixed-point equations are derived and compared with the thermodynamic approach based upon the replica-symmetric mean-field approximation.

We consider the behaviour of multi-state neural networks averaged over an extended monitoring period of their dynamics. Pattern reconstruction by clipping the activities is proposed, leading to an improvement in retrieval precision.

Using a signal-to-noise analysis, the effects of nonlinear modulation of the Hebbian learning rule in the multi-class proximity problem are investigated. Both random classification and classification provided by a Gaussian and a binary teacher are treated. Analytic expressions are derived for the learning and generalization rates around an old and a new prototype. For the proximity problem with binary inputs but Q′-state outputs, it is shown that the optimal modulation is a combination of a hyperbolic tangent and a linear function. As an illustration, numerical results are presented for the two-class and the Q′=3 multi-class problem.

The characteristic properties of the macroscopic retrieval dynamics of analogue neurons with Hebbian coupling strengths are studied, usign the shape of the gain function as a modeling parameter. Already at low loading a rich diversity in dynamical behaviour is observed, covering the full range from point attractors to chaotic dynamics. The attractors which are not of the fixed-point-type, are interpreted as an intermediate phase between the well-known retrieval states and the zero state, enabling a waiting-mode in the system. Using a probabilistic approach it is shown that these features persist in extremely and asymmetrically diluted systems at sufficiently low loading. A number of generic examples are worked out in detail, illustrating some properties of networks governed by nonmonotonic piecewise linear gain functions. The critical storage level above which the chaotic behaviour is absent, is numerically determined.

The effects of nonlinear modulation of the Hebbian learning rule on the performance of a perceptron are investigated. Both random classification and classification provided by a teacher perceptron are considered. It is seen that both the generalization and learning rate depend on the overlap between the teacher and the student and the signal-to-noise ratio in the local field. Furthermore, they are independent of the specific teacher distribution when the ratio between the number of training examples and the perceptron size is small. An analytic expression is obtained for the optimal modulation function for different classification schemes. For random and Gaussian teacher classifications the best choice for modulation appears to be linear. For binary teachers it is shown to be the hyperbolic tangent. The modifications on the latter from diluting the binary teacher are also obtained in analytic form.

A replica-symmetric mean-field theory approach is presented to the multi-neuron interaction model introduced by de Almeida and Iglesias (1990 Phys. Lett. 146A 239). Fixed-point equations are derived for the relevant order parameters of the model, extended to include biased patterns, without truncating the interaction. The capacity-bias and the temperature-capacity phase diagrams are discussed. Compared with the truncated version of the model, it is found that the capacity at zero temperature is infinite and that the retrieval states satisfy the de Almeida-Thouless stability condition.

The thermodynamic and retrieval properties of fully connected Q-Ising networks are studied in the replica-symmetric mean-field approximation. In particular, capacity-gain parameter and capacity-temperature phase diagrams are derived for Q=3, 4 and Q= infinity and different distributions of the stored patterns. Furthermore, the optimal gain function is determined in order to obtain the best performance. Where appropriate, the results are compared with the diluted and layered versions of these models.

The optimal storage capacity is studied for diluted networks with multi-state neurons and continuous respectively discrete couplings, within the replica-symmetric Gardner approach. The Gardner-Derrida line, the de Almeida-Thouless line and the zero-entropy line are compared and the validity of the replica-symmetric approximation is discussed in detail. The distribution of the synaptic couplings is determined. The results are analysed in terms of the number of states of the neuron, the distribution of the stored patterns, the amount of dilution and the number of discrete values for the couplings.

Using a probabilistic approach, the deterministic and the stochastic parallel dynamics of aQ-Ising neural network are studied at finiteQ and in the limitQ→∞. Exact evolution equations are presented for the first time-step. These formulas constitute recursion relations for the parallel dynamics of the extremely diluted asymmetric versions of these networks. An explicit analysis of the retrieval properties is carried out in terms of the gain parameter, the loading capacity, and the temperature. The results for theQ→∞ network are compared with those for theQ=3 andQ=4 models. Possible chaotic microscopic behavior is studied using the time evolution of the distance between two network configurations. For arbitrary finiteQ the retrieval regime is always chaotic. In the limitQ→∞ the network exhibits a dynamical transition toward chaos.

We derive exact equations that determine the spectra of undirected and directed sparsely connected regular graphs containing loops of arbitrary length. The implications of our results to the structural and dynamical properties of networks are discussed by showing how loops influence the size of the spectral gap and the propensity for synchronization. Analytical formulas for the spectrum are obtained for specific length of the loops., Comment: 4 pages, 4 figures

The presence and stability of mixture states in Q-state Potts neural networks are studied for different learning rules within the replica-symmetric mean-field-theory approach. The retrieval properties of the asymmetric mixture states are examined in the case of biased patterns. For the storage of a finite number of such patterns, these properties are compared for the usual Hebb learning rule and some variants obtained by subtracting, for a certain pattern, the average of the Potts neuron state over all the other patterns. The latter are introduced to suppress the symmetric mixture states. Furthermore, the embedding of an additional, infinite number of unbiased patterns stored with the Hebb rule is allowed

Optimal capacities of perceptrons with graded input-output relations are computed within the Gardner approach (1988). The influence of desired output precision, stability with respect to input errors, and output-pattern statistics are analysed and discussed.

Using a probabilistic approach, the parallel dynamics of theQ-state Potts andQ-Ising neural networks are studied at zero and at nonzero temperatures. Evolution equations are derived for the first time step and arbitraryQ. These formulas constitute recursion relations for the exact parallel dynamics of the extremely diluted asymmetric versions of these networks. An explicit analysis, including dynamical capacity-temperature diagrams and the temperature dependence of the overlap, is carried out forQ=3. Both types of models are compared.

A systematic study of the Q-state Potts model of neural networks, extended to include biased patterns, is made for extensive loading alpha . Mean-field equations are written down within the replica symmetric approximation, for general Q and arbitrary temperature T. For the Q=3 model and two classes of representative bias parameters, the storage capacity and retrieval quality at zero temperature are discussed as functions of the bias, taking into account the Mattis retrieval state and the lowest symmetric states. The T- alpha diagram is obtained and the stability properties of the retrieval state are analysed at finite temperatures. A comparison is made with the biased Hopfield model.

Using the Gardner approach, the optimal storage capacity is considered for diluted networks with multi-state neurons and spherical or local constraints on the couplings. The first results are discussed about its dependence upon the number of states of the neuron, the amount of dilution and the number of discrete values for the couplings.

The Q-state Potts model of neural networks, extended to include biased patterns, is studied for extensive loading α. Mean-field equations are written down for general Q and arbitrary temperature T. Within the replica symmetric approximation, the complete T-α phase diagram is discussed, especially for Q = 3. A study of the bifurcation diagrams for the overlap and the free clarifies the results obtained.

Neural networks with multi-state neurons are studied in the case of low loading. For symmetric couplings satisfying a certain positivity condition, a Lyapunov function is shown to exist in the space of overlaps between the instantaneous microscopic state of the system and the learned patterns. Furthermore, an algorithm is derived for zero temperature to determine all the fixed points. As an illustration, the three-state model is worked out explicitly for Hebbian couplings. For finite temperature the time evolution of the overlap is studied for couplings which need not be symmetric. The stability properties are discussed in detail for the three-state model. For asymmetric couplings limit-cycle behaviour is shown to be possible.

The Q-state Potts model of neural networks, extended to include biased patterns, is studied for extensive loading \ensuremath{\alpha}. Within the replica-symmetric approximation, mean-field equations are written down for general Q and arbitrary temperature T. The critical storage capacity is discussed for Q=3 and two classes of representative bias parameters. The complete T-\ensuremath{\alpha} phase diagram is presented. A tricritical point is found in the spin-glass transition for Qg6, depending on \ensuremath{\alpha}. Contrary to the Hopfield model, the critical lines do not converge to the same T as \ensuremath{\alpha}\ensuremath{\rightarrow}0. A stability analysis is made.

We study the L\'evy spin-glass model with the replica and the cavity method. In this model each spin interacts through a finite number of strong bonds and an infinite number of weak bonds. This hybrid behaviour of L\'evy spin glasses becomes transparent in our solution: the local field contains a part propagating along a backbone of strong bonds and a Gaussian noise term due to weak bonds. Our method allows to determine the complete replica symmetric phase diagram, the replica symmetry breaking line and the entropy. The results are compared with simulations and previous calculations using a Gaussian ansatz for the distribution of fields., Comment: 20 pages, 7 figures

We use the cavity method to study parallel dynamics of disordered Ising models on a graph. In particular, we derive a set of recursive equations in single site probabilities of paths propagating along the edges of the graph. These equations are analogous to the cavity equations for equilibrium models and are exact on a tree. On graphs with exclusively directed edges we find an exact expression for the stationary distribution of the spins. We present the phase diagrams for an Ising model on an asymmetric Bethe lattice and for a neural network with Hebbian interactions on an asymmetric scale-free graph. For graphs with a nonzero fraction of symmetric edges the equations can be solved for a finite number of time steps. Theoretical predictions are confirmed by simulation results. Using a heuristic method, the cavity equations are extended to a set of equations that determine the marginals of the stationary distribution of Ising models on graphs with a nonzero fraction of symmetric edges. The results of this method are discussed and compared with simulations.

Using the Gardner approach, the optimal storage capacity is discussed for a network with multi-stage neurons and a spherical constraint on the couplings. It is found that this capacity decreases with the number of states and the corresponding separation of the plateaus of the gain function. An analysis of the validity of the replica-symmetric approximation is made.

The q-state Potts glass model of neural networks is extended to include biased patterns. For a finite number of such patterns, the existence and stability properties of the Mattis states and symmetric states are discussed in detail as a function of the bias. Analytic results are presented for all q at zero temperature. For finite temperatures numerical results are obtained for q=3 and two classes of representative bias parameters. A comparison is made with the Hopfield model.

Neural networks of the q-state Potts type are considered for a finite number of biased patterns. The existence and stability properties of the Mattis states and symmetric states are discussed at zero temperature for all q and arbitrary bias and near the “critical” temperature for q = 3 and two classes of representative bias parameters. Some of these properties are illustrated through the solution of the dynamical equations governing the time evolution of the macroscopic overlap between the learned patterns and the instantaneous microscopic state of the network.

We derive critical noise levels for Gallager codes on asymmetric channels as a function of the input bias and the temperature. Using a statistical mechanics approach we study the space of codewords and the entropy in the various decoding regimes. We further discuss the relation of the convergence of the message passing algorithm with the endogeny property and complexity, characterizing solutions of recursive equations of distributions for cavity fields.

We require total invariance of a lagrangian under supersymmetry transformations and we observe that special variables are singled out. They are identical to those entering the Nicolai mapping. We show that a similarity transformation is connected with the introduction of these new variables. We give a stochastic formulation of this transformation using the Cameron-Martin formula.

We study the thermodynamic properties of spin systems with bond-disorder on small-world hypergraphs, obtained by superimposing a one-dimensional Ising chain onto a random Bethe graph with p-spin interactions. Using transfer-matrix techniques, we derive fixed-point equations describing the relevant order parameters and the free energy, both in the replica symmetric and one step replica symmetry breaking approximation. We determine the static and dynamic ferromagnetic transition and the spinglass transition within replica symmetry for all temperatures, and demonstrate corrections to these results when one step replica symmetry breaking is taken into account. The results obtained are in agreement with Monte-Carlo simulations., 9 pages, 4 figures

We study the thermodynamic properties of spin systems on small-world hypergraphs, obtained by superimposing sparse Poisson random graphs with p-spin interactions onto a one-dimensional Ising chain with nearest-neighbor interactions. We use replica-symmetric transfer-matrix techniques to derive a set of fixed-point equations describing the relevant order parameters and free energy, and solve them employing population dynamics. In the special case where the number of connections per site is of the order of the system size we are able to solve the model analytically. In the more general case where the number of connections is finite we determine the static and dynamic ferromagnetic-paramagnetic transitions using population dynamics. The results are tested against Monte-Carlo simulations., 14 pages, 7 figures; Added 2 figures. Extended results

The optimal capacity of a diluted Blume-Emery-Griffiths neural network is studied as a function of the pattern activity and the embedding stability using the Gardner entropy approach. Annealed dilution is considered, cutting some of the couplings referring to the ternary patterns themselves and some of the couplings related to the active patterns, both simultaneously (synchronous dilution) or independently (asynchronous dilution). Through the de Almeida-Thouless criterion it is found that the replica-symmetric solution is locally unstable as soon as there is dilution. The distribution of the couplings shows the typical gap with a width depending on the amount of dilution, but this gap persists even in cases where a particular type of coupling plays no role in the learning process., 9 pages Latex, 2 eps figures

For the retrieval dynamics of sparsely coded attractor associative memory models with synaptic noise the inclusion of a macroscopic time-dependent threshold is studied. It is shown that if the threshold is chosen appropriately as a function of the cross-talk noise and of the activity of the memorized patterns, adapting itself automatically in the course of the time evolution, an autonomous functioning of the model is guaranteed. This self-control mechanism considerably improves the quality of the fixed-point retrieval dynamics, in particular the storage capacity, the basins of attraction and the mutual information content., Comment: 5 pages Latex, 1 ps and 4 eps figures, to appear in the proceedings of the 2004 International Joint Conference on Neural Networks, Budapest (IEEE)

The effects of a variable amount of random dilution of the synaptic couplings in Q-Ising multi-state neural networks with Hebbian learning are examined. A fraction of the couplings is explicitly allowed to be anti-Hebbian. Random dilution represents the dying or pruning of synapses and, hence, a static disruption of the learning process which can be considered as a form of multiplicative noise in the learning rule. Both parallel and sequential updating of the neurons can be treated. Symmetric dilution in the statics of the network is studied using the mean-field theory approach of statistical mechanics. General dilution, including asymmetric pruning of the couplings, is examined using the generating functional (path integral) approach of disordered systems. It is shown that random dilution acts as additive gaussian noise in the Hebbian learning rule with a mean zero and a variance depending on the connectivity of the network and on the symmetry. Furthermore, a scaling factor appears that essentially measures the average amount of anti-Hebbian couplings., Comment: 15 pages, 5 figures, to appear in the proceedings of the Conference on Noise in Complex Systems and Stochastic Dynamics II (SPIE International)