Search

Dropout and DropConnect are known as effective methods to improve on the generalization performance of neural networks, by either dropping states of neural units or dropping weights of synaptic connections randomly selected at each time instance throughout the training process. In this paper, we extend on the use of these methods in the design of neuromorphic spiking neural networks (SNN) hardware to improve further on the reliability of inference as impacted by resource constrained errors in network connectivity. Such energy and bandwidth constraints arise for low-power operation in the communication between neural units, which cause dropped spike events due to timeout errors in the transmission. The Dropout and DropConnect processes during training of the network are aligned with a statistical model of the network during inference that accounts for these random errors in the transmission of neural states and synaptic connections. The use of Dropout and DropConnect during training hence allows to simultaneously meet two design objectives: improving robustness of inference to dropped spike events due to timeout communication constraints in network connectivity, while maximizing time-to-decision bandwidth and hence minimizing inference energy in the neuromorphic hardware. Simulations with 5-layer fully connected 784-500-500-500-10 SNN on the MNIST task show a 3.42-fold and 7.06-fold decrease in inference energy at 90% test accuracy, by using Dropout and DropConnect respectively during backpropagation training. Also the simulation with convolutional neural networks on the CIFAR-10 task show a 1.24-fold decrease in inference energy at 60% test accuracy by using Dropout during backpropagation training.

In order for neuromorphic computing to attain full throughput capacity, its hardware design must mitigate any inefficiencies that result from limited bandwidth to neural and synaptic information. In large-scale neuromorphic systems, synaptic memory access is typically the defining bottleneck, demanding that system design closely analyze the interdependence between the functional blocks to keep the memory as active as possible. In this paper, we formulate principles in memory organization of digital spiking neural networks, with a focus on systems with High Bandwidth Memory (HBM) as their bulk memory element. We present some of the fundamental steps and considerations required when designing a highly efficient HBM-centric system, and describe parallelization and pipelining solutions which serve as a foundational architecture for streamlined operation in any multi-port memory system. In our experiments using the Xilinx VU37P FPGA, we demonstrate random, short burst-length memory read bandwidths in excess of 400 GBps (95% relative to sequential-access peak bandwidth), supporting dynamically reconfigurable sparse synaptic connectivity. Therefore, the combination of our proposed network model with practical results suggest a promising path towards implementing highly parallel large-scale neuromorphic systems centered on HBM.

Spike-Timing-Dependent Plasticity (STDP) is a bio-inspired local incremental weight update rule commonly used for online learning in spike-based neuromorphic systems. In STDP, the intensity of long-term potentiation and depression in synaptic efficacy (weight) between neurons is expressed as a function of the relative timing between pre- and post-synaptic action potentials (spikes), while the polarity of change is dependent on the order (causality) of the spikes. Online STDP weight updates for causal and acausal relative spike times are activated at the onset of post- and pre-synaptic spike events, respectively, implying access to synaptic connectivity both in forward (pre-to-post) and reverse (post-to-pre) directions. Here we study the impact of different arrangements of synaptic connectivity tables on weight storage and STDP updates for large-scale neuromorphic systems. We analyze the memory efficiency for varying degrees of density in synaptic connectivity, ranging from crossbar arrays for full connectivity to pointer-based lookup for sparse connectivity. The study includes comparison of storage and access costs and efficiencies for each memory arrangement, along with a trade-off analysis of the benefits of each data structure depending on application requirements and budget. Finally, we present an alternative formulation of STDP via a delayed causal update mechanism that permits efficient weight access, requiring no more than forward connectivity lookup. We show functional equivalence of the delayed causal updates to the original STDP formulation, with substantial savings in storage and access costs and efficiencies for networks with sparse synaptic connectivity as typically encountered in large-scale models in computational neuroscience.

Stochastic neural networks such as Restricted Boltzmann Machines (RBMs) have been successfully used in applications ranging from speech recognition to image classification. Inference and learning in these algorithms use a Markov Chain Monte Carlo procedure called Gibbs sampling, where a logistic function forms the kernel of this sampler. On the other side of the spectrum, neuromorphic systems have shown great promise for low-power and parallelized cognitive computing, but lack well-suited applications and automation procedures. In this work, we propose a systematic method for bridging the RBM algorithm and digital neuromorphic systems, with a generative pattern completion task as proof of concept. For this, we first propose a method of producing the Gibbs sampler using bio-inspired digital noisy integrate-and-fire neurons. Next, we describe the process of mapping generative RBMs trained offline onto the IBM TrueNorth neurosynaptic processor -- a low-power digital neuromorphic VLSI substrate. Mapping these algorithms onto neuromorphic hardware presents unique challenges in network connectivity and weight and bias quantization, which, in turn, require architectural and design strategies for the physical realization. Generative performance metrics are analyzed to validate the neuromorphic requirements and to best select the neuron parameters for the model. Lastly, we describe a design automation procedure which achieves optimal resource usage, accounting for the novel hardware adaptations. This work represents the first implementation of generative RBM inference on a neuromorphic VLSI substrate., A similar version of this manuscript has been submitted to IEEE TBioCAS for revision in October 2015

Identifying specific keywords in speech is a common task present in current mobile devices. In this context, Keyword spotting (KWS) is the triggering cue for the device to start “listening in” on the uttered command. Since this cue can originate at any moment, KWS must be continuously performed. Small-footprint KWS is typically designed to directly detect the entire keyword (as opposed to sub-word components, such as phonemes), with current low-power KWS solutions relying heavily on artificial neural networks (ANNs). As a power-efficient alternative to ANNs, biologically-inspired spiking neural networks (SNNs) have achieved similar results in terms of accuracy, with the benefit of simpler and, typically, sparser computation. SNNs can be designed to operate in rate- and time-based modes, each with its own particularities. In this work, we propose a small-footprint KWS solution on six common words from the TIMIT speech dataset using fully connected, feedforward neural networks, and compare the frame error rate performance and computational cost of ANN-versus-SNN alternatives. For the cost comparison, we designed a figure of merit which takes into account the types of operations present in digital implementations of the different types of networks. Lastly, we discuss the promising results for power-efficient KWS using SNNs, particularly in the time-based domain, which shows equivalent accuracy performance yet with an 84% reduction in cost in relation to its ANN equivalent.

While unsupervised generative neural networks are attractive choices for adoption in always-on continuous-time smart sensory systems, they typically impose heavy memory requirements on the underlying computational fabric. Recent literature on binarized neural networks has not yet been extended to unsupervised generative networks and alternate strategies are required to reduce their memory footprint. This work studies unsupervised synaptic pruning strategies to reduce the memory requirements for Restricted Boltzmann Machines (RBMs). In addition to one-shot pruning, we explore alternative strategies that encompass iterative stochastic pruning as well as pruning under target probability density functions for an RBM trained over the MNIST database. Interestingly, the results presented here suggest that one-shot re-training after pruning of the least significant connections in a trained network yields improved per-formance/memory trade-off over multiple iterations of stochastic pruning and re-training on the same network.

Neural computation and communication in the brain are partitioned into the grey matter of dense local synaptic connectivity in tightly knit neuronal networks, and the white matter of sparse long-range connectivity over axonal fiber bundles across distant brain regions. This exquisite distributed multiscale organization provides inspiration to the design of scalable neuromorphic systems for deep learning and inference, with hierarchical address event-routing of neural spike events and multiscale synaptic connectivity and plasticity, and their efficient implementation in low-power mixed-signal VLSI circuits.

Spike generation and routing is typically the most energy-demanding operation in neuromorphic hardware built using spiking neurons. Spiking neural networks running on neuromorphic hardware, however, often use rate-coding where the neurons spike rate is treated as the information-carrying quantity. Rate-coding is a highly inefficient coding scheme with minimal information content in each spike, which requires the transmission of a large number of spikes. In this paper, we describe an alternative type of spiking networks based on temporal coding where neuron spiking activity is very sparse and information is encoded in the time of each spike. We implemented the proposed networks on an FPGA platform and we use these sparsely active spiking networks to classify MNIST digits. The network FPGA implementation produces the classification output using only few tens of spikes from the hidden layer, and the classification result is obtained very quickly, typically within 1–3 synaptic time constants. We describe the idealized network dynamics and how these dynamics are adapted to allow an efficient implementation on digital hardware. Our results illustrate the importance of making use of the temporal dynamics in spiking networks in order to maximize the information content of each spike, which ultimately leads to reduced spike counts, improved energy efficiency, and faster response times.

In this paper we propose a method for continuously processing and learning from data in Restricted Boltzmann Machines (RBMs). Traditionally, RBMs are trained using Contrastive Divergence (CD), which is an algorithm consisting of two phases, of which only one is driven by data. This not only prohibits training of RBMs in conjugation with continuous-time data streams, especially in event-based real-time systems, but also hinders training speed of RBMs in large-scale machine learning systems. The model we propose trades space for time and, by pipelining information propagation in the network, is capable of processing both phases of the CD learning algorithm simultaneously. Simulation results of our model on generative and discriminative tasks show convergence to the original CD algorithm. We finalize with a discussion of applying our method to other deep neural networks, resulting in continuous learning and training time reduction.

© 2016 Neftci, Pedroni, Joshi, Al-Shedivat and Cauwenberghs. Recent studies have shown that synaptic unreliability is a robust and sufficient mechanism for inducing the stochasticity observed in cortex. Here, we introduce Synaptic Sampling Machines (S2Ms), a class of neural network models that uses synaptic stochasticity as a means to Monte Carlo sampling and unsupervised learning. Similar to the original formulation of Boltzmann machines, these models can be viewed as a stochastic counterpart of Hopfield networks, but where stochasticity is induced by a random mask over the connections. Synaptic stochasticity plays the dual role of an efficient mechanism for sampling, and a regularizer during learning akin to DropConnect. A local synaptic plasticity rule implementing an event-driven form of contrastive divergence enables the learning of generative models in an on-line fashion. S2Ms perform equally well using discrete-timed artificial units (as in Hopfield networks) or continuous-timed leaky integrate and fire neurons. The learned representations are remarkably sparse and robust to reductions in bit precision and synapse pruning: removal of more than 75% of the weakest connections followed by cursory re-learning causes a negligible performance loss on benchmark classification tasks. The spiking neuron-based S2Ms outperform existing spike-based unsupervised learners, while potentially offering substantial advantages in terms of power and complexity, and are thus promising models for on-line learning in brain-inspired hardware.

In recent years the field of neuromorphic low-power systems gained significant momentum, spurring brain-inspired hardware systems which operate on principles that are fundamentally different from standard digital computers and thereby consume orders of magnitude less power. However, their wider use is still hindered by the lack of algorithms that can harness the strengths of such architectures. While neuromorphic adaptations of representation learning algorithms are now emerging, the efficient processing of temporal sequences or variable length-inputs remains difficult, partly due to challenges in representing and configuring the dynamics of spiking neural networks. Recurrent neural networks (RNN) are widely used in machine learning to solve a variety of sequence learning tasks. In this work we present a train-and-constrain methodology that enables the mapping of machine learned (Elman) RNNs on a substrate of spiking neurons, while being compatible with the capabilities of current and near-future neuromorphic systems. This “train-and-constrain” method consists of first training RNNs using backpropagation through time, then discretizing the weights and finally converting them to spiking RNNs by matching the responses of artificial neurons with those of the spiking neurons. We demonstrate our approach by mapping a natural language processing task (question classification), where we demonstrate the entire mapping process of the recurrent layer of the network on IBM's Neurosynaptic System TrueNorth, a spike-based digital neuromorphic hardware architecture. TrueNorth imposes specific constraints on connectivity, neural and synaptic parameters. To satisfy these constraints, it was necessary to discretize the synaptic weights to 16 levels, discretize the neural activities to 16 levels, and to limit fan-in to 64 inputs. Surprisingly, we find that short synaptic delays are sufficient to implement the dynamic (temporal) aspect of the RNN in the question classification task. Furthermore we observed that the discretization of the neural activities is beneficial to our train-and-constrain approach. The hardware-constrained model achieved 74% accuracy in question classification while using less than 0.025% of the cores on one TrueNorth chip, resulting in an estimated power consumption of ≈ 17µW.

Spike-timing-dependent plasticity (STDP) incurs both causal and acausal synaptic weight updates, for negative and positive time differences between pre-synaptic and post-synaptic spike events. For realizing such updates in neuromorphic hardware, current implementations either require forward and reverse lookup access to the synaptic connectivity table, or rely on memory-intensive architectures such as crossbar arrays. We present a novel method for realizing both causal and acausal weight updates using only forward lookup access of the synaptic connectivity table, permitting memory-efficient implementation. A simplified implementation in FPGA, using a single timer variable for each neuron, closely approximates exact STDP cumulative weight updates for neuron refractory periods greater than 10 ms, and reduces to exact STDP for refractory periods greater than the STDP time window. Compared to conventional crossbar implementation, the forward table-based implementation leads to substantial memory savings for sparsely connected networks supporting scalable neuromorphic systems with fully reconfigurable synaptic connectivity and plasticity., Comment: Submitted to BioCAS 2016

Restricted Boltzmann Machines and Deep Belief Networks have been successfully used in a wide variety of applications including image classification and speech recognition. Inference and learning in these algorithms uses a Markov Chain Monte Carlo procedure called Gibbs sampling. A sigmoidal function forms the kernel of this sampler which can be realized from the firing statistics of noisy integrate-and-fire neurons on a neuromorphic VLSI substrate. This paper demonstrates such an implementation on an array of digital spiking neurons with stochastic leak and threshold properties for inference tasks and presents some key performance metrics for such a hardware-based sampler in both the generative and discriminative contexts., Comment: Accepted at ISCAS 2015

Restricted Boltzmann Machines (RBMs) and Deep Belief Networks (DBNs) have been demonstrated to perform efficiently on a variety of applications such as dimensionality reduction and classification. Implementation of RBMs on neuromorphic platforms which emulate large scale networks of spiking neurons has significant advantages from concurrency and low power perspectives. This work outlines a neuromorphic adaptation of the RBM which uses a recently proposed neural sampling algorithm (Buesing et al. 2011) and examines its algorithmic efficiency. Results show the feasibility of such alterations which will serve as a guide for future implementation of such algorithms in neuromorphic very large scale integration (VLSI) platforms. © 2013 IEEE.

McEliece and Lin introduced the minimal trellis for convolutional codes, which can be considerably less complex than the conventional trellis typically used to construct Viterbi decoders. The authors state that this reduced trellis complexity can lead to less complex Viterbi decoders in practice. In this paper, we compare conventional and minimal Viterbi decoders for a rate 2/3 convolutional code considered by McEliece and Lin, which possesses a minimal trellis with half of the complexity of the conventional trellis. The decoder circuits were implemented in Altera programmable logic devices. We show that, though the complexity measure used by McEliece and Lin does not translate directly to practice, reductions in power consumption and hardware utilization, along with increased maximum operating frequency, can indeed be obtained for the decoders.