Search

The interplay between structure and function affects the emerging properties of many natural systems. Here we use an adaptive neural network model that couples activity and topological dynamics and reproduces the experimental temporal profiles of synaptic density observed in the brain. We prove that the existence of a transient period of relatively high synaptic connectivity is critical for the development of the system under noise circumstances, such that the resulting network can recover stored memories. Moreover, we show that intermediate synaptic densities provide optimal developmental paths with minimum energy consumption, and that ultimately it is the transient heterogeneity in the network that determines its evolution. These results could explain why the pruning curves observed in actual brain areas present their characteristic temporal profiles and they also suggest new design strategies to build biologically inspired neural networks with particular information processing capabilities., The authors acknowledge financial support from the Spanish Ministry of Science and Technology, and the Agencia Espanola de Investigacion (AEI), Spain under grant FIS2017-84256-P (FEDER funds) and from the Consejeria de Conocimiento, Investigacion Universidad, Junta de Andalucia and European Regional Development Funds, Spain, Refs. SOMM17/6105/UGR and A-FQM-175UGR18. APM also acknowledges support from ``Obra Social La Caixa, Spain'' (ID 100010434 with code LCF/BQ/ES15/10360004) and from ZonMw, Netherlands and the Dutch Epilepsy Foundation, Netherlands, project number 95105006. SJ acknowledges support from the Alan Turing Institute under EPSRC, United Kingdom Grant No. EP/N510129/1.

This paper presents a deep associative neural network (DANN) based on unsupervised representation learning for associative memory. In brain, the knowledge is learnt by associating different types of sensory data, such as image and voice. The associative memory models which imitate such a learning process have been studied for decades but with simpler architectures they fail to deal with large scale complex data as compared with deep neural networks. Therefore, we define a deep architecture consisting of a perception layer and hierarchical propagation layers. To learn the network parameters, we define a probabilistic model for the whole network inspired from unsupervised representation learning models. The model is optimized by a modified contrastive divergence algorithm with a novel iterated sampling process. After training, given a new data or corrupted data, the correct label or corrupted part is associated by the network. The DANN is able to achieve many machine learning problems, including not only classification, but also depicting the data given a label and recovering corrupted images. Experiments on MNIST digits and CIFAR-10 datasets demonstrate the learning capability of the proposed DANN.

The strength of association between a pair of data vectors is represented by a nonnegative real number, called matching weight. For dimensionality reduction, we consider a linear transformation of data vectors, and define a matching error as the weighted sum of squared distances between transformed vectors with respect to the matching weights. Given data vectors and matching weights, the optimal linear transformation minimizing the matching error is solved by the spectral graph embedding of Yan et al. (2007). This method is a generalization of the canonical correlation analysis, and will be called as matching correlation analysis (MCA). In this paper, we consider a novel sampling scheme where the observed matching weights are randomly sampled from underlying true matching weights with small probability, whereas the data vectors are treated as constants. We then investigate a cross-validation by resampling the matching weights. Our asymptotic theory shows that the cross-validation, if rescaled properly, computes an unbiased estimate of the matching error with respect to the true matching weights. Existing ideas of cross-validation for resampling data vectors, instead of resampling matching weights, are not applicable here. MCA can be used for data vectors from multiple domains with different dimensions via an embarrassingly simple idea of coding the data vectors. This method will be called as cross-domain matching correlation analysis (CDMCA), and an interesting connection to the classical associative memory model of neural networks is also discussed.

In recent years, complex-valued recurrent neural networks have been developed and analysed in-depth in view of that they have good modelling performance for some applications involving complex-valued elements. In implementing continuous-time dynamical systems for simulation or computational purposes, it is quite necessary to utilize a discrete-time model which is an analogue of the continuous-time system. In this paper, we analyse a discrete-time complex-valued recurrent neural network model and obtain the sufficient conditions on its global exponential periodicity and exponential stability. Simulation results of several numerical examples are delineated to illustrate the theoretical results and an application on associative memory is also given.

How the neural system proceeds from sensory stimuli to generate appropriate behaviors is a basic question that has not yet been fully answered. In contrast to the conventional viewpoint, in which the external stimulus dominantly drives the response behavior, recent studies have revealed that not only external stimuli, but also intrinsic neural dynamics, contribute to the generation of response behavior. In particular, spontaneous activity, which is neural activity without extensive external stimuli, has been found to exhibit similar patterns to those evoked by external inputs, from time to time. In order to further understand the role of this spontaneous activity on the response, we propose a viewpoint, memories-as-bifurcations, that differs from the traditional memories-as-attractors viewpoint. According to this viewpoint, memory is recalled when spontaneous neural activity is changed to an appropriate output activity upon the application of an input. After reviewing the previous rate-coding model embodying this viewpoint, we employ a model of a spiking neuron network that can embed input/output associations, and study the dynamics of collective neural activity. The organized neural activity, which matched the target pattern, is shown to be generated even under application of stochastic input, while the spontaneous activity, which apparently shows noisy dynamics, is found to exhibit selectively higher similarity with evoked activities corresponding to embedded target patterns. These results suggest that such an intrinsic structure in the spontaneous activity might play a role in generating the higher response. The relevance of these results to biological neural processing is also discussed.

The brain functions as a spatio-temporal information processing machine. Spatio- and spectro-temporal brain data (STBD) are the most commonly collected data for measuring brain response to external stimuli. An enormous amount of such data has been already collected, including brain structural and functional data under different conditions, molecular and genetic data, in an attempt to make a progress in medicine, health, cognitive science, engineering, education, neuro-economics, Brain-Computer Interfaces (BCI), and games. Yet, there is no unifying computational framework to deal with all these types of data in order to better understand this data and the processes that generated it. Standard machine learning techniques only partially succeeded and they were not designed in the first instance to deal with such complex data. Therefore, there is a need for a new paradigm to deal with STBD. This paper reviews some methods of spiking neural networks (SNN) and argues that SNN are suitable for the creation of a unifying computational framework for learning and understanding of various STBD, such as EEG, fMRI, genetic, DTI, MEG, and NIRS, in their integration and interaction. One of the reasons is that SNN use the same computational principle that generates STBD, namely spiking information processing. This paper introduces a new SNN architecture, called NeuCube, for the creation of concrete models to map, learn and understand STBD. A NeuCube model is based on a 3D evolving SNN that is an approximate map of structural and functional areas of interest of the brain related to the modeling STBD. Gene information is included optionally in the form of gene regulatory networks (GRN) if this is relevant to the problem and the data. A NeuCube model learns from STBD and creates connections between clusters of neurons that manifest chains (trajectories) of neuronal activity. Once learning is applied, a NeuCube model can reproduce these trajectories, even if only part of the input STBD or the stimuli data is presented, thus acting as an associative memory. The NeuCube framework can be used not only to discover functional pathways from data, but also as a predictive system of brain activities, to predict and possibly, prevent certain events. Analysis of the internal structure of a model after training can reveal important spatio-temporal relationships 'hidden' in the data. NeuCube will allow the integration in one model of various brain data, information and knowledge, related to a single subject (personalized modeling) or to a population of subjects. The use of NeuCube for classification of STBD is illustrated in a case study problem of EEG data. NeuCube models result in a better accuracy of STBD classification than standard machine learning techniques. They are robust to noise (so typical in brain data) and facilitate a better interpretation of the results and understanding of the STBD and the brain conditions under which data was collected. Future directions for the use of SNN for STBD are discussed.

Smartphone video capture and transmission to the Web contributes to data pollution. In contrast, mammalian eyes sense all, capture only significant events, allowing us vividly recall the causalities. Likewise in our videos, we wish to skip redundancies and keep only significantly differences, as determined by real-time local medium filters. We construct a Picture Index (PI) of one's (center of gravity changes) among zeros (no changes) as Motion Organized Sparseness (MOS). Only non-overlapping time-ordered PI pair is admitted in the outer-product Associative Memory (AM). Another outer product between PI and its image builds Hetero-AM (HAM) for fault tolerant retrievals.

In a recent communication, Sacramento and Wichert (2011) proposed a hierarchical retrieval prescription for Willshaw-type associative networks. Through simulation it was shown that one could make use of low resolution descriptor patterns to decrease the total time requirements of recalling a learnt association. However, such a method introduced a dependence on a set of new parameters which define the structure of the hierarchy. In this work we compute the expected retrieval time for the random neural activity regime which maximises the capacity of the Willshaw model and we study the task of finding the optimal hierarchy parametrisation with respect to the derived temporal expectation. Still in regard to this performance measure, we investigate some asymptotic properties of the algorithm.

We recently employed concepts of mathematical morphology to introduce fuzzy morphological associative memories (FMAMs), a broad class of fuzzy associative memories (FAMs). We observed that many well-known FAM models can be classified as belonging to the class of FMAMs. Moreover, we developed a general learning strategy for FMAMs using the concept of adjunction of mathematical morphology. In this paper, we describe the properties of FMAMs with adjunction-based learning. In particular, we characterize the recall phase of these models. Furthermore, we prove several theorems concerning the storage capacity, noise tolerance, fixed points, and convergence of auto-associative FMAMs. These theorems are corroborated by experimental results concerning the reconstruction of noisy images. Finally, we successfully employ FMAMs with adjunction-based learning in order to implement fuzzy rule-based systems in an application to a time-series prediction problem in industry.

The focus of this paper is to propose a hybrid neural network model for associative recall of analog and digital patterns. This hybrid model consists of self-feedback neural network structures (SFNN) in parallel with generalized regression neural networks (GRNN). Using a new one-shot learning algorithm developed in the paper, pattern representations are first stored as the asymptotically stable fixed points of the SFNN. Then in the retrieving process, each pattern is applied to the GRNN to make the corresponding initial condition and to initiate the dynamical equations of the SFNN that should in turn output the corresponding representation. In this way, the corresponding stored patterns are retrieved even under high noise degradation. Moreover, contrary to many associative memories, the proposed hybrid model is without any spurious attractors and can store both binary and real-value patterns without any preprocessing. Several simulations confirm the theoretical analyses of the model. Results indicate that the performance of the hybrid model is better than that of recurrent associative memory and competitive with other classes of networks.

Spiking neural networks (SNN) are promising artificial neural network (ANN) models as they utilise information representation as trains of spikes, that adds new dimensions of time, frequency and phase to the structure and the functionality of ANN. The current SNN models though are deterministic, that restricts their applications for large scale engineering and cognitive modelling of stochastic processes. This paper proposes a novel probabilistic spiking neuron model (pSNM) and suggests ways of building pSNN for a wide range of applications including classification, string pattern recognition and associative memory. It also extends previously published computational neurogenetic models.

In this paper, we present a new recurrent bidirectional model that encompasses correlational, competitive and topological model properties. The simultaneous use of many classes of network behaviors allows for the unsupervised learning/categorization of perceptual patterns (through input compression) and the concurrent encoding of proximities in a multidimensional space. All of these operations are achieved within a common learning operation, and using a single set of defining properties. It is shown that the model can learn categories by developing prototype representations strictly from exposition to specific exemplars. Moreover, because the model is recurrent, it can reconstruct perfect outputs from incomplete and noisy patterns. Empirical exploration of the model's properties and performance shows that its ability for adequate clustering stems from: (1) properly distributing connection weights, and (2) producing a weight space with a low dispersion level (or higher density). In addition, since the model uses a sparse representation (k-winners), the size of topological neighborhood can be fixed, and no longer requires a decrease through time as was the case with classic self-organizing feature maps. Since the model's learning and transmission parameters are independent from learning trials, the model can develop stable fixed points in a constrained topological architecture, while being flexible enough to learn novel patterns.

Associative memories are brain-style devices designed to store a set of patterns as stable equilibria such that the stored patterns can be reliably retrieved with the initial probes containing sufficient information about the patterns. This paper presents a new design procedure for synthesizing associative memories based on continuous-time cellular neural networks with time delays characterized by input and output matrices obtained using two-dimensional space-invariant cloning templates. The design procedure enables hetero-associative or auto-associative memories to be synthesized by solving a set of linear inequalities with few design parameters and retrieval probes feeding from external inputs instead of initial states. The designed associative memories are robust in terms of design parameter selection. In addition, the hosting cellular neural networks are guaranteed to be globally exponentially stable. Simulation and experimental results of illustrative examples and Monte Carlo tests demonstrate the applicability and superiority of the methodology.

Since their introduction sixty years ago, cell assemblies have proved to be a powerful paradigm for brain information processing. After their introduction in artificial intelligence, cell assemblies became commonly used in computational neuroscience as a neural substrate for content addressable memories. However, the mechanisms underlying their formation are poorly understood and, so far, there is no biologically plausible algorithms which can explain how external stimuli can be online stored in cell assemblies. We addressed this question in a previous paper [Salihoglu, U., Bersini, H., Yamaguchi, Y., Molter, C., (2009). A model for the cognitive map formation: Application of the retroaxonal theory. In Proc. IEEE international joint conference on neural networks], were, based on biologically plausible mechanisms, a novel unsupervised algorithm for online cell assemblies' creation was developed. The procedure involved simultaneously, a fast Hebbian/anti-Hebbian learning of the network's recurrent connections for the creation of new cell assemblies, and a slower feedback signal which stabilized the cell assemblies by learning the feedforward input connections. Here, we first quantify the role played by the retroaxonal feedback mechanism. Then, we show how multiple cognitive maps, composed by a set of orthogonal input stimuli, can be encoded in the network. As a result, when facing a previously learned input, the system is able to retrieve the cognitive map it belongs to. As a consequence, ambiguous inputs which could belong to multiple cognitive maps can be disambiguated by the knowledge of the context, i.e. the cognitive map.

In this paper, we propose the concept of a manifold of color perception through empirical observation that the center-surround properties of images in a perceptually similar environment define a manifold in the high dimensional space. Such a manifold representation can be learned using a novel recurrent neural network based learning algorithm. Unlike the conventional recurrent neural network model in which the memory is stored in an attractive fixed point at discrete locations in the state space, the dynamics of the proposed learning algorithm represent memory as a nonlinear line of attraction. The region of convergence around the nonlinear line is defined by the statistical characteristics of the training data. This learned manifold can then be used as a basis for color correction of the images having different color perception to the learned color perception. Experimental results show that the proposed recurrent neural network learning algorithm is capable of color balance the lighting variations in images captured in different environments successfully.

Our detailed analysis has established that in addition to the advantages of computationally efficiency and easy hardware implementation, the two-level decoupled Hamming network possesses a substantially higher capacity over the single-level Hamming associative memory since the effect caused by Ikeda et al.'s uniform random noise [Ikeda, N., Watta, P., Artiklar, M., & Hassoun, M. (2001). A two-level Hamming network for high performance associative memory. Neural Networks, 14(9), 1189-1200] is much smaller than that caused by the practically more prevalent concentrated noise. We therefore conclude that the two-level decoupled Hamming network with middle-sized windows should be an elegant associative memory model in all the senses of efficiency, hardware implementation and capacity.

In this paper we provide an in-depth evaluation of the SOM as a feasible tool for nonlinear adaptive filtering. A comprehensive survey of existing SOM-based and related architectures for learning input-output mappings is carried out and the application of these architectures to nonlinear adaptive filtering is formulated. Then, we introduce two simple procedures for building RBF-based nonlinear filters using the Vector-Quantized Temporal Associative Memory (VQTAM), a recently proposed method for learning dynamical input-output mappings using the SOM. The aforementioned SOM-based adaptive filters are compared with standard FIR/LMS and FIR/LMS--Newton linear transversal filters, as well as with powerful MLP-based filters in nonlinear channel equalization and inverse modeling tasks. The obtained results in both tasks indicate that SOM-based filters can consistently outperform powerful MLP-based ones.

Neurophysiology reveals the properties of individual mirror neurons in the macaque while brain imaging reveals the presence of 'mirror systems' (not individual neurons) in the human. Current conceptual models attribute high level functions such as action understanding, imitation, and language to mirror neurons. However, only the first of these three functions is well-developed in monkeys. We thus distinguish current opinions (conceptual models) on mirror neuron function from more detailed computational models. We assess the strengths and weaknesses of current computational models in addressing the data and speculations on mirror neurons (macaque) and mirror systems (human). In particular, our mirror neuron system (MNS), mental state inference (MSI) and modular selection and identification for control (MOSAIC) models are analyzed in more detail. Conceptual models often overlook the computational requirements for posited functions, while too many computational models adopt the erroneous hypothesis that mirror neurons are interchangeable with imitation ability. Our meta-analysis underlines the gap between conceptual and computational models and points out the research effort required from both sides to reduce this gap.

This paper describes an approach for constructing a classifier which is unaffected by occlusions in images. We propose a method for integrating an auto-associative network into a simple classifier. As the auto-associative network can recall the original image from a partly occluded input image, we can employ it to detect occluded regions and complete the input image by replacing those regions with recalled pixels. By iterating this reconstruction process, the integrated network is able to classify target objects with occlusions robustly. To confirm the effectiveness of this method, we performed experiments involving face image classification. It is shown that the classification performance is not decreased, even if about 30% of the face image is occluded.

This paper addresses recurrent neural architectures based on bifurcating nodes that exhibit chaotic dynamics, with local dynamics defined by first order parametric recursions. In the studied architectures, logistic recursive nodes interact through parametric coupling, they self organize, and the network evolves to global spatio-temporal period-2 attractors that encode stored patterns. The performance of associative memories arrangements is measured through the average error in pattern recovery, under several levels of prompting noise. The impact of the synaptic connections magnitude on architecture performance is analyzed in detail, through pattern recovery performance measures and basin of attraction characterization. The importance of a planned choice of the synaptic connections scale in RPEs architectures is shown. A strategy for minimizing pattern recovery degradation when the number of stored patterns increases is developed. Experimental results show the success of such strategy. Mechanisms for allowing the studied associative networks to deal with asynchronous changes in input patterns, and tools for the interconnection between different associative assemblies are developed. Finally, coupling in heterogeneous assemblies with diverse recursive maps is analyzed, and the associated synaptic connections are equated.

Computational models of the hippocampal region link psychological theories of associative learning with their underlying physiological and anatomical substrates. Our approach to theory development began with a broad description of the computations that depend on the hippocampal region in classical conditioning (Gluck and Myers, 1993 and Gluck and Myers, 2001). In this initial model, the hippocampal region was treated as an Information-processing system that transformed stimulus representations, compressing (making more similar) representations of inputs that co-occur or are otherwise redundant, while differentiating (or making less similar) representations of inputs that predict different future events. This model led to novel predictions for the behavioral consequences of hippocampal-region lesions in rodents and of brain damage in humans who have amnesia or are in the earliest stages of Alzheimer's disease. Many of these predictions have, since been confirmed by our lab and others. Functional brain imaging studies have provided further supporting evidence. In more recent computational modeling, we have shown how some aspects of this proposed information-processing function could emerge from known anatomical and physiological characteristics of the hippocampal region, including the entorhinal cortex and the septo-hippocampal cholinergic system. The modeling to date lays the groundwork for future directions that increase the depth of detail of the biological modeling, as well as the breadth of behavioral phenomena addressed. In particular, we are working now to reconcile these kinds of incremental associative learning models with other models of the hippocampal region that account for the rapid formation of declarative memories.

Recent biological experimental findings have shown that synaptic plasticity depends on the relative timing of pre- and post-synaptic spikes and this is called spike-timing-dependent plasticity (STDP). Many authors have claimed that a precise balance between long-term potentiation (LTP) and long-term depression (LTD) of STDP is crucial in the storage of spatio-temporal patterns. Some authors have numerically investigated the impact of an imbalance between LTP and LTD on the network properties. However, the mathematical mechanism remains unknown. We analytically show that an associative memory network has the robust retrieval properties of spatio-temporal patterns, and these properties make the network less vulnerable to any deviation from a precise balance between LTP and LTD when the network contains a finite number of neurons., 24 pages, 5 figures

This paper evaluates a novel k-nearest neighbour (k-NN) classifier built from binary neural networks. The binary neural approach uses robust encoding to map standard ordinal, categorical and numeric data sets onto a binary neural network. The binary neural network uses high speed pattern matching to recall a candidate set of matching records, which are then processed by a conventional k-NN approach to determine the k-best matches. We compare various configurations of the binary approach to a conventional approach for memory overheads, training speed, retrieval speed and retrieval accuracy. We demonstrate the superior performance with respect to speed and memory requirements of the binary approach compared to the standard approach and we pinpoint the optimal configurations.

The chaotic neural network constructed with chaotic neuron shows the associative memory function, but its memory searching process cannot be stabilized in a stored state because of the chaotic motion of the network. In this paper, a pinning control method focused on the chaotic neural network is proposed. The computer simulation proves that the chaos in the chaotic neural network can be controlled with this method and the states of the network can converge in one of its stored patterns if the control strength and the pinning density are chosen suitable. It is found that in general the threshold of the control strength of a controlled network is smaller at higher pinned density and the chaos of the chaotic neural network can be controlled more easily if the pinning control is added to the variant neurons between the initial pattern and the target pattern.

This article is concerned with the representation and generalisation of continuous action spaces in reinforcement learning (RL) problems. A model is proposed based on the self-organising map (SOM) of Kohonen [Self Organisation and Associative Memory, 1987] which allows either the one-to-one, many-to-one or one-to-many structure of the desired state-action mapping to be captured. Although presented here for tasks involving immediate reward, the approach is easily extended to delayed reward. We conclude that the SOM is a useful tool for providing real-time, on-line generalisation in RL problems in which the latent dimensionalities of the state and action spaces are small. Scalability issues are also discussed.

The problem of learning correlated patterns in a neurobiologically plausible way without multiple iterations is discussed. Three guiding principles are outlined for solving this problem in a manner suggestive of how the memory modules of the human brain do it. The first, already known, principle of minimum disturbance is applied quantitatively to minimise the number of learning iterations. The second guiding principle involves a self-induced remedy to undo any damage caused to the stabilities of the old stored patterns after a new one has been learnt. The third involves localising connectivities between neurons depending on the structure of the input patterns. A neurobiologically plausible network and a learning rule are constructed based on these principles. Satisfactory use of this network in learning English words as sequences of their letters is demonstrated by means of computer simulation.

This paper presents an analysis of a two-level decoupled Hamming network, which is a high performance discrete-time/discrete-state associative memory model. The two-level Hamming memory generalizes the Hamming memory by providing for local Hamming distance computations in the first level and a voting mechanism in the second level. In this paper, we study the effect of system dimension, window size, and noise on the capacity and error correction capability of the two-level Hamming memory. Simulation results are given for both random images and human face images.

Scene analysis in the mammalian visual system, conceived as a distributed and parallel process, faces the so-called binding problem. As a possible solution, the temporal correlation hypothesis has been suggested and implemented in phase-coding models. We propose an alternative model that reproduces experimental findings of synchronized and desynchronized fast oscillations more closely. This model is based on technical considerations concerning improved pattern separation in associative memories on the one hand, and on known properties of the visual cortex on the other. It consists of two reciprocally connected areas, one corresponding to a peripheral visual area (P), the other a central association area (C). P implements the orientation-selective subsystem of the primary visual cortex, while C was modeled as an associative memory with connections formed by Hebbian learning of all assemblies corresponding to stimulus objects. Spiking neurons including habituation and correlated noise were incorporated as well as realistic synaptic delays. Three learned stimuli were presented simultaneously and correlation analysis was performed on spike recordings. Generally, we found two states of activity: (i) relatively slow and unordered oscillations at about 20-25 Hz, synchronized only within small regions; and (ii) faster and more precise oscillations around 50-60 Hz, synchronized over the whole simulated area. The neuron groups representing one stimulus tended to be simultaneously in either the slow or the fast state. At each particular time, only one assembly was found to be in the fast state. Activation of the three assemblies switched on a time scale of 100 ms. This can be interpreted as self-generated attention switching. On the time scale corresponding to gamma oscillations, cross correlations between local neuron groups were either modulated or flat. Modulated correlograms resulted if the groups coded features corresponding to a common object. Otherwise, the correlograms remained flat. This behavior is in agreement with experimental results, while phase-code models would generally predict modulated correlations also in the case of different objects. Furthermore, we derive a technical version from our biological associative memory model that accomplishes fast pattern separation parallel in O(log2 n) steps for n neurons and sparse coding.

Here, we develop and investigate a computational model of a network of cortical neurons on the base of biophysically well constrained and tested two-compartmental neurons developed by Pinsky and Rinzel [Pinsky, P. F., & Rinzel, J. (1994). Intrinsic and network rhythmogenesis in a reduced Traub model for CA3 neurons. Journal of Computational Neuroscience, 1, 39–60]. To study associative memory, we connect a pool of cells by a structured connectivity matrix. The connection weights are shaped by simple Hebbian coincidence learning using a set of spatially sparse patterns. We study the neuronal activity processes following an external stimulation of a stored memory. In two series of simulation experiments, we explore the effect of different classes of external input, tonic and flashed stimulation. With tonic stimulation, the addressed memory is an attractor of the network dynamics. The memory is displayed rhythmically, coded by phase-locked bursts or regular spikes. The participating neurons have rhythmic activity in the gamma-frequency range (30–80 Hz). If the input is switched from one memory to another, the network activity can follow this change within one or two gamma cycles. Unlike similar models in the literature, we studied the range of high memory capacity (in the order of 0.1 bit/synapse), comparable to optimally tuned formal associative networks. We explored the robustness of efficient retrieval varying the memory load, the excitation/inhibition parameters, and background activity. A stimulation pulse applied to the identical simulation network can push away ongoing network activity and trigger a phase-locked association event within one gamma period. Unlike as under tonic stimulation, the memories are not attractors. After one association process, the network activity moves to other states. Applying in close succession pulses addressing different memories, one can switch through the space of memory patterns. The readout speed can be increased up to the point where in every gamma cycle another pattern is displayed. With pulsed stimulation, bursts become relevant for coding, their occurrence can be used to discriminate relevant processes from background activity.

The influence of a macroscopic time-dependent threshold on the retrieval dynamics of attractor associative memory models with ternary neurons {−1,0,+1} is examined. If the threshold is chosen appropriately as a function of the cross-talk noise and of the activity of the memorized patterns in the model, adapting itself in the course of the time evolution, it guarantees an autonomous functioning of the model. Especially in the limit of sparse coding, it is found that this self-control mechanism considerably improves the quality of the fixed-point retrieval dynamics, in particular the storage capacity, the basins of attraction and the information content. The mutual information is shown to be the relevant parameter to study the retrieval quality of such sparsely coded models. Numerical results confirm these observations.

In conventional models for storing hierarchically correlated patterns, correlations between ancestors (first-level patterns) and their descendants (second-level ones) are assumed to be uniform, so that the descendants are distributed around their ancestors with equal distances. However, this assumption might be unnatural. We believe that objects are encoded into patterns by preserving the similarity between them. In this case, descendants are distributed around their ancestors with various distances, so that the assumption is invalid and the conventional models become inapplicable. To overcome this, we propose a model CASM3 for storing hierarchically correlated patterns with various correlations. In CASM3, critical load levels vary with the descendants, and become higher with increasing correlations. Increase in load level successively destroys the memories of the descendants in descending order of their correlations. The size of the basins of attraction depends on the range of the correlations, and becomes larger as the correlation range is shifted toward lower levels.

The Willshaw model is asymptotically the most efficient neural associative memory (NAM), but its finite version is hampered by high retrieval errors. Iterative retrieval has been proposed in a large number of different models to improve performance in auto-association tasks. In this paper, bidirectional retrieval for the hetero-associative memory task is considered: we define information efficiency as a general performance measure for bidirectional associative memory (BAM) and determine its asymptotic bound for the bidirectional Willshaw model. For the finite Willshaw model, an efficient new bidirectional retrieval strategy is proposed, the appropriate combinatorial model analysis is derived, and implications of the proposed sparse BAM for applications and brain theory are discussed. The distribution of the dendritic sum in the finite Willshaw model given by Buckingham and Willshaw [Buckingham, J., & Willshaw, D. (1992). Performance characteristics of associative nets. Network, 3, 407-414] allows no fast numerical evaluation. We derive a combinatorial formula with a highly reduced evaluation time that is used in the improved error analysis of the basic model and for estimation of the retrieval error in the naive model extension, where bidirectional retrieval is employed in the hetero-associative Willshaw model. The analysis rules out the naive BAM extension as a promising improvement. A new bidirectional retrieval algorithm - called crosswise bidirectional (CB) retrieval - is presented. The cross talk error is significantly reduced without employing more complex learning procedures or dummy augmentation in the pattern coding, as proposed in other refined BAM models [Wang, Y. F., Cruz, J. B., & Mulligan, J. H. (1990). Two coding strategies for bidirectional associative memory. IEEE Trans. Neural Networks, 1(1), 81-92; Leung, C.-S., Chan, L.-W., & Lai, E. (1995). Stability, capacity and statistical dynamics of second-order bidirectional associative memory. IEEE Trans. Syst. Man Cybern., 25(10), 1414-1424]. The improved performance of CB retrieval is shown by a combinatorial analysis of the first step and by simulation experiments: it allows very efficient hetero-associative mapping, as well as auto-associative completion for sparse patterns - the experimentally achieved information efficiency is close to the asymptotic bound. The different retrieval methods in the hetero-associative Willshaw matrix are discussed as Boolean linear optimization problems. The improved BAM model opens interesting new perspectives, for instance, in information retrieval it allows efficient data access providing segmentation of ambiguous user input, relevance feedback and relevance ranking. Finally, we discuss BAM models as functional models for reciprocal cortico-cortical pathways, and the implication of this for a more flexible version of Hebbian cell-assemblies.

The asymptotic stability of a continuous neural network is analyzed for associative memory. An optimal design method is proposed which ensures the highest associative memory speed and guarantees the storage of each desired memory with attractivity. The network asymptotic stability is analyzed by means of a new energy function, and four theorems are obtained. By comparing these theorems with existing ones, it can be shown that in some cases they are consistent, while in others they are not equivalent but complementary to each other. Further study results in two more generalized conclusions, of which the existing conclusions are special cases. The network optimal design method is proposed in terms of an optimal associative memory theorem. Two application examples are presented to demonstrate the defeffectiveness of the optimal design method, which can be used to design the network for many applications.

This paper investigates the problem of approximating a dynamical system (DS) by a recurrent neural network (RNN) as one extension of the problem of approximating orbits by an RNN. We systematically investigate how an RNN can produce a DS on the visible state space to approximate a given DS and as a first step to the generalization problem for RNNs, we also investigate whether or not a DS produced by some RNN can be identified from several observed orbits of the DS. First, it is proved that RNNs without hidden units uniquely produce a certain class of DS. Next, neural dynamical systems (NDSs) are proposed as DSs produced by RNNs with hidden units. Moreover, affine neural dynamial systems (A-NDSs) are provided as nontrivial examples of NDSs and it is proved that any DS can be finitely approximated by an A-NDS with any precision. We propose an A-NDS as a DS that an RNN can actually produce on the visible state space to approximate the target DS. For the generalization problem of RNNs, a geometric criterion is derived in the case of RNNs without hidden units. This theory is also extended to the case of RNNs with hidden units for learning A-NDSs.

In the study of associative memory networks, the updating rule remains unchanged during neuron evolution. When evolution stops at an undesired minimum, simulated annealing is used to resolve the problem. In this paper, a combined neuron evolution method based on a multipath network architecture is presented. It is shown that, by controlling the evolution path, the probability that evolution terminates in undesired, minima is significantly reduced. Once evolution is trapped in an undesired minimum with respect to one path, the method seeks an alternative path to carry on the evolution. The process continues until a desired minimum is reached. Visual examples are used to demonstrate the performance of the method.

In this paper, optimal encoding schemes for linear associative memories are derived for biased association under both the white-noise and colored-noise situations. Analysis and simulation results all show that the biased encodings thus derived are optimal and superior to existing models in their performance. Together with the Wee-Kohonen unbiased encoding, the study settles the optimality issue of linear associative memories and enhances their practicalities.

Algorithms for auto-associative memory in sparsely connected networks are commonly brittle with respect to relationships between such parameters as training set size, connectivity, synaptic strength, threshold, or inhibitory feedback. This paper describes an algorithm for auto-associative memory based on depotentiation of inhibitory synapses (disinhibition) rather than potentiation of excitatory synapses. All parameter values are robust, largely independent of one another, and independent of network architecture over a large range of random and structured architectures. The algorithm does not require thresholds which depend on the number of active neurons or the number of synapses per neuron. Architectures to which the algorithm is applicable include randomized variants of projective networks in which all links between neurons in the same layer are of path length two (i.e., through some other layer). The resulting fanout produces prodigiously large networks with correspondingly large information storage capacities relative to the number of synapses per neuron.

In this paper, we consider the problem of how to construct an artificial neuronal network such that it reproduces a given set of patterns in an exact manner. It turns out that the structure of the weight matrix of the network represents the structure of the set of patterns it is acting on, not the patterns themselves. Conditions are discussed under which the associative network memorizes a certain subset of these patterns. Our formal approach is based on the simple observation that neural networks are structured sets of neurons. By regarding recurrent neural networks as dynamical systems with symmetry, the category of G-sets and G-morphisms appears as a natural framework for evaluating their structure and functioning analytically.

We study associative and storage memories for memory traces of size N and aim to establish that both the size of the system (as measured by, e.g., the number of nodes in a network) can be of order N and the number of traces consistent with reliable operation can be exponentially large in N, so that a positive capacity (in bits per node) can be achieved. It is well known that, if the traces are generated as M random vectors, then reliability imposes a linear bound on M, in that it implies an upper bound on the asymptotic (large N) value of α = M/N. For the noise-free Hopfield net this critical bound is about 0.138. We show that, if superposition of traces is allowed, so that the M given traces constitute the random basis of a linear code, then exponential memory size and a positive capacity can be achieved. However, there is still a critical upper bound on the basis rate α = M/N, implied now, not by the condition of reliability, but by the necessity that the recursion realising the calculation should be stable. For our model we determine this critical value exactly as α c = 3 − √8 ≐ 0.172. Our model is based upon inference concepts and differs in slight but important respects from the Hopfield model. We do not use replica methods, but appeal to a generalised version of the Wigner semi-circle theorem on the asymptotic distribution of eigenvalues.

A learning algorithm for single layer perceptrons is proposed. First, a cone-like domain is derived such that all its elements can be recognized as a stored pattern in the perceptron network. The learning algorithm is obtained as a process that enlarges the cone-like domain. For autoassociative networks, it is shown that the cone-like domain becomes a domain of attraction for a stored pattern in the network. In this case, extended domains of attraction are also obtained by feeding the outputs of the network back to the input layer. In computer simulations, character recognition ability of the autoassociative network is examined.

H. Gutfreund (Neural networks with hierarchically correlated patterns. Physical Review A, 37, 570–577, 1988) has proposed a model for storing hierachically correlated patterns where ancestor patterns are correlated with descendent ones. However, there is a problem of small storage capacity. Furthermore, we must give ancestors in the learning phase, and determine the value of the parameter on which the capacity and the basins of attraction strongly depend. To overcome these problems, we present a model (CASM) consisting of the first associative memory to form ancestors from their descendants and the second associative memory to store sparse difference patterns which have only information on differences between the formed ancestors and the descendants. To evaluate the performance of CASM, extensive simulations are carried out. The results show that the capacity increases correlation between the ancestors and the descendants, and is as large as that of sparsely encoded associative memory. The basins of attraction become larger with decreasing correlation, and do not depend on loading level.

We introduce a theoretical framework for estimating the matching performance of binary correlation matrices acting as hetero-associative memories. The framework is applicable to non-recursive, fully connected systems with binary (0, 1) Hebbian weights and hard-limited threshold. It can handle both full and partial matching of single or multiple data items in non-square memories. Theoretical development takes place under a probability theory framework. Inherent uncertainties in the matching process are accommodated through the use of probability distributions to describe the numbers of correct and incorrect neuron responses during retrieval. Theoretical predictions are verified experimentally for medium-sized memories and are used to aid in the design of larger systems. The results highlight the fact that correlation-based models can act as highly efficient memories provided a small probability of retrieval error is accepted.

Human episodic memory provides a seemingly unlimited storage for everyday experiences, and a retrieval system that allows us to access the experiences with partial activation of their components. The system is believed to consist of a fast, temporary storage in the hippocampus, and a slow, long-term storage within the neocortex. This paper presents a neural network model of the hippocampal episodic memory inspired by Damasio's idea of Convergence Zones. The model consists of a layer of perceptual feature maps and a binding layer. A perceptual feature pattern is coarse coded in the binding layer, and stored on the weights between layers. A partial activation of the stored features activates the binding pattern, which in turn reactivates the entire stored pattern. For many configurations of the model, a theoretical lower bound for the memory capacity can be derived, and it can be an order of magnitude or higher than the number of all units in the model, and several orders of magnitude higher than the number of binding-layer units. Computational simulations further indicate that the average capacity is an order of magnitude larger than the theoretical lower bound, and making the connectivity between layers sparser causes an even further increase in capacity. Simulations also show that if more descriptive binding patterns are used, the errors tend to be more plausible (patterns are confused with other similar patterns), with a slight cost in capacity. The convergence-zone episodic memory therefore accounts for the immediate storage and associative retrieval capability and large capacity of the hippocampal memory, and shows why the memory encoding areas can be much smaller than the perceptual maps, consist of rather coarse computational units, and are only sparsely connected to the perceptual maps. Copyright 1997 Elsevier Science Ltd.

We present a study on the stability of the generalized Hopfield network in randomly asynchronous mode. First, the stability is investigated from the state space of the network. We introduce a concept of hole and define two kinds of stability in randomly asynchronous mode. By mathematical inductive method, we have proved that a generalized Hopfield network with non negative weights is strictly stable, that is, the network evolves to a simple hole—stable state with probability one when it starts at any initial state. For the general case, we made the simulation experiments on the networks with the number of neurons from 5 to 10. The empirical results have shown that almost any generalized Hopfield network with simple holes is strictly stable. © 1997 Elsevier Science Ltd.

New autocorrelation memory matrices which realize eigenspace separation are proposed from the geometrical viewpoint over association dynamics. These memorization methods realize narrower distributions of eigenvalues and remove self-connections in a natural way. As a result, stable memory patterns and large basins of attraction are achieved without iterative learning. Modification of recalling dynamics for the new memory matrices is discussed based on the existing modified dynamics to attain even more memory capacity. © 1997 Elsevier Science Ltd.

A sparsely encoded Hopfield-like attractor neural network is investigated analytically and by computer simulation. Informational capacity and recall quality are evaluated. Three analytical approaches are used: replica method (RM); method of statistical neurodynamics (SN); and single-step approximation (SS). Computer simulation confirmed the good accuracy of RM and SN for all levels of network activity. SS is accurate only for large sparseness. It is shown that informational capacity monotonically increases when sparseness increases, while recall quality changes nonmonotonically: initially it decreases and then increases. Computer simulation revealed the main features of network behaviour near the saturation which are not predicted by the used analytical approaches. © 1997 Elsevier Science Ltd.

Conspicuous problems confront the use of fully-connected networks (F-nets) in the construction of realistic partial models of biological memory. These problems include the high synaptic densities of F-nets, and the low information storage capacities of F-nets with simple, biologically realistic learning rules. Most auto-associative networks constructed with low connectivities have employed random projections of path length 1. Projective networks (P-nets) are nonrandom, multilayer networks which achieve extremely low connectivities by linking all neurons in the same layer through paths of length 2. In this paper we derive a lower bound on the storage capacities of a class of simple, two-layer P-nets with binary Hebbian synapses. Given a 1% tolerance for spurious neurons, we find that the P-net with 1000 synapses per neuron (2 x 10(6) neurons) will store more than 1.5 x 10(6) training vectors with 20 active neurons per vector (0.25 bits per synapse). Copyright 1997 Elsevier Science Ltd.

We consider the Hopfield associative memory for storing m patterns xi(r) in { - 1, + 1}(n), r = 1, em leader,m. The weights are given by the scalar product model w(ij)=(m/n)G,i not equal j,w(ii) identical with 0, where G:R --R is some nonlinear function, like G(x) z.tbnd6; Sgn(x), which is used in hardware implementation of associative memories. We give a rigorous lower bound for the memory size of this (ANN) by using large deviations estimates. Our main results states that retrieval without errors occurs when m(n) --infinity as n --infinity in such a way that m(n)(n/2log(n))q(G), where 0q(G): = E(NG)N))(2)/E(G(N)(2)))/= 1, N denotes a standard normal random variable, and E(.) is the expectation operator. Copyright 1997 Elsevier Science Ltd.

The contextual layered associative memory (CLAM) has been developed as a self-generating structure which implements a probabilistic encoding scheme. The training algorithms are geared towards the unsupervised generation of a layerable associative mapping ( Thacker and Mayhew, 1989 ). We show here that the resulting structure will support layers which can be trained to produce outputs that approximate conditional probabilities of classification. Unsupervised and supervised learning algorithms operate independently permitting the unsupervised representational layer to be developed before supervision is available. The system thus supports learning which is inherently more flexible than conventional node labelling schemes. © 1997 Elsevier Science Ltd. All Rights Reserved.