Search

Autonomous one‐shot on‐the‐fly learning copes with the high privacy, small dataset, and in‐stream data at the edge. Implementing such learning on digital hardware suffers from the well‐known von‐Neumann and scaling bottlenecks. The optical neural networks featuring large parallelism, low latency, and high efficiency offer a promising solution. However, ex‐situ training of conventional optical networks, where optical path configuration and deep learning model optimization are separated, incurs hardware, energy and time overheads, and defeats the advantages in edge learning. Here, we introduced a bio‐inspired material‐algorithm co‐design to construct a hydrogel‐based optical Willshaw model (HOWM), manifesting Hebbian‐rule‐based structural plasticity for simultaneous optical path configuration and deep learning model optimization thanks to the underlying opto‐chemical reactions. We first employed the HOWM as an all optical in‐sensor AI processor for one‐shot pattern classification, association and denoising. We then leveraged HOWM to function as a ternary content addressable memory (TCAM) of an optical memory augmented neural network (MANN) for one‐shot learning the Omniglot dataset. The HOWM empowered one‐shot on‐the‐fly edge learning leads to 1000× boost of energy efficiency and 10× boost of speed, which paves the way for the next‐generation autonomous, efficient, and affordable smart edge systems. [ABSTRACT FROM AUTHOR]

Abstract Autonomous one‐shot on‐the‐fly learning copes with the high privacy, small dataset, and in‐stream data at the edge. Implementing such learning on digital hardware suffers from the well‐known von‐Neumann and scaling bottlenecks. The optical neural networks featuring large parallelism, low latency, and high efficiency offer a promising solution. However, ex‐situ training of conventional optical networks, where optical path configuration and deep learning model optimization are separated, incurs hardware, energy and time overheads, and defeats the advantages in edge learning. Here, we introduced a bio‐inspired material‐algorithm co‐design to construct a hydrogel‐based optical Willshaw model (HOWM), manifesting Hebbian‐rule‐based structural plasticity for simultaneous optical path configuration and deep learning model optimization thanks to the underlying opto‐chemical reactions. We first employed the HOWM as an all optical in‐sensor AI processor for one‐shot pattern classification, association and denoising. We then leveraged HOWM to function as a ternary content addressable memory (TCAM) of an optical memory augmented neural network (MANN) for one‐shot learning the Omniglot dataset. The HOWM empowered one‐shot on‐the‐fly edge learning leads to 1000× boost of energy efficiency and 10× boost of speed, which paves the way for the next‐generation autonomous, efficient, and affordable smart edge systems.

Abstract: In a recent communication, proposed a hierarchical retrieval prescription for Willshaw-type associative networks. Through simulation it was shown that one could make use of low resolution descriptor patterns to decrease the total time requirements of recalling a learnt association. However, such a method introduced a dependence on a set of new parameters which define the structure of the hierarchy. In this work we compute the expected retrieval time for the random neural activity regime which maximises the capacity of the Willshaw model and we study the task of finding the optimal hierarchy parametrisation with respect to the derived temporal expectation. Still in regard to this performance measure, we investigate some asymptotic properties of the algorithm. [Copyright &y& Elsevier]

Previous asymptotic analyses of binary neural associative networks of Willshaw or Steinbuch type relied on a binomial approximation of the neurons' dendritic potentials. This approximation has been proven to be good only if the stored patterns are extremely sparse, for example, when the mean number of active units k per pattern vector scales with the logarithm of the vector size n. Recent promising results concerning storage capacity and retrieval efficiency for larger pattern activities k > log n have been doubted because here the binomial approximation can lead to a massive overestimation of performance. In this work I compute and characterize the exact Willshaw-Palm distribution of the dendritic potentials for hetero-association, auto-association, and fixed and random pattern activity. Comparing the raw and central moments of the Willshaw-Palm distribution to the moments of the corresponding binomial probability reveals that, asymptotically, the binomial approximation becomes exact for almost any sublinear pattern activity, including k = O(n/ log2 n). This verifies, for large networks, the existence of a wide high-performance parameter range as predicted by the approximative theory. [ABSTRACT FROM AUTHOR]

Abstract: In this study binary associative networks of the Willshaw type are analyzed with respect to the effect of clipped Hebbian learning on the distribution of postsynaptic potentials when stimulating with random activation patterns. It is shown that the variance in the postsynaptic potentials grows with the square of the stimulation strength if the synapses have been generated by Hebbian learning of many overlapping patterns, but only linearly for independent random synapses. This result bears implications both for analysis of associative memory and the detection of Hebbian cell assemblies in neurophysiological experiments. [Copyright &y& Elsevier]