Unsupervised Synaptic Pruning Strategies for Restricted Boltzmann Machines

While unsupervised generative neural networks are attractive choices for adoption in always-on continuous-time smart sensory systems, they typically impose heavy memory requirements on the underlying computational fabric. Recent literature on binarized neural networks has not yet been extended to unsupervised generative networks and alternate strategies are required to reduce their memory footprint. This work studies unsupervised synaptic pruning strategies to reduce the memory requirements for Restricted Boltzmann Machines (RBMs). In addition to one-shot pruning, we explore alternative strategies that encompass iterative stochastic pruning as well as pruning under target probability density functions for an RBM trained over the MNIST database. Interestingly, the results presented here suggest that one-shot re-training after pruning of the least significant connections in a trained network yields improved per-formance/memory trade-off over multiple iterations of stochastic pruning and re-training on the same network.