Search

Existing convolution techniques in artificial neural networks suffer from huge computation complexity, while the biological neural network works in a much more powerful yet efficient way. Inspired by the biological plasticity of dendritic topology and synaptic strength, our method, Learnable Heterogeneous Convolution, realizes joint learning of kernel shape and weights, which unifies existing handcrafted convolution techniques in a data-driven way. A model based on our method can converge with structural sparse weights and then be accelerated by devices of high parallelism. In the experiments, our method either reduces VGG16/19 and ResNet34/50 computation by nearly 5x on CIFAR10 and 2x on ImageNet without harming the performance, where the weights are compressed by 10x and 4x respectively; or improves the accuracy by up to 1.0% on CIFAR10 and 0.5% on ImageNet with slightly higher efficiency. The code will be available on www.github.com/Genera1Z/LearnableHeterogeneousConvolution., Published in Neural Networks journal

Toward the long-standing dream of artificial intelligence, two successful solution paths have been paved: 1) neuromorphic computing and 2) deep learning. Recently, they tend to interact for simultaneously achieving biological plausibility and powerful accuracy. However, models from these two domains have to run on distinct substrates, i.e., neuromorphic platforms and deep learning accelerators, respectively. This architectural incompatibility greatly compromises the modeling flexibility and hinders promising interdisciplinary research. To address this issue, we build a unified model description framework and a unified processing architecture (Tianjic), which covers the full stack from software to hardware. By implementing a set of integration and transformation operations, Tianjic is able to support spiking neural networks, biological dynamic neural networks, multilayered perceptron, convolutional neural networks, recurrent neural networks, and so on. A compatible routing infrastructure enables homogeneous and heterogeneous scalability on a decentralized many-core network. Several optimization methods are incorporated, such as resource and data sharing, near-memory processing, compute/access skipping, and intra-/inter-core pipeline, to improve performance and efficiency. We further design streaming mapping schemes for efficient network deployment with a flexible tradeoff between execution throughput and resource overhead. A 28-nm prototype chip is fabricated with >610-GB/s internal memory bandwidth. A variety of benchmarks are evaluated and compared with GPUs and several existing specialized platforms. In summary, the fully unfolded mapping can achieve significantly higher throughput and power efficiency; the semi-folded mapping can save 30x resources while still presenting comparable performance on average. Finally, two hybrid-paradigm examples, a multimodal unmanned bicycle and a hybrid neural network, are demonstrated to show the potential of our unified architecture. This article paves a new way to explore neural computing.

3D object detection and recognition are crucial tasks for many spatiotemporal processing applications, such as computer-aided diagnosis and autonomous driving. Although prevalent 3D Convolution Nets (ConvNets) have continued to improve the accuracy and sensitivity, excessive computing resources are required. In this paper, we propose Leaky Integrate and Fire Networks (LIF-Nets) for 3D detection and recognition tasks. LIF-Nets have rich inter-frame sensing capability brought from membrane potentials, and low power event-driven mechanism, which make them excel in 3D processing and save computational cost at the same time. We also develop ResLIF Blocks to solve the degradation problem of deep LIF-Nets, and employ U-LIF structure to improve the feature representation capability. As a result, we carry out experiments on the LUng Nodule Analysis 2016 (LUNA16) public dataset for chest CT automated analysis and conclude that the LIF-Nets achieve 94.6% detection sensitivity at 8 False Positives per scan and 94.14% classification accuracy while the LIF-detection net reduces 65.45% multiplication operations, 65.12% addition operations, and 65.32% network parameters. The results show that LIF-Nets have extraordinary time-efficient and energy-saving performance while achieving comparable accuracy.

There is a growing trend to design hybrid neural networks (HNNs) by combining spiking neural networks and artificial neural networks to leverage the strengths of both. Here, we propose a framework for general design and computation of HNNs by introducing hybrid units (HUs) as a linkage interface. The framework not only integrates key features of these computing paradigms but also decouples them to improve flexibility and efficiency. HUs are designable and learnable to promote transmission and modulation of hybrid information flows in HNNs. Through three cases, we demonstrate that the framework can facilitate hybrid model design. The hybrid sensing network implements multi-pathway sensing, achieving high tracking accuracy and energy efficiency. The hybrid modulation network implements hierarchical information abstraction, enabling meta-continual learning of multiple tasks. The hybrid reasoning network performs multimodal reasoning in an interpretable, robust and parallel manner. This study advances cross-paradigm modeling for a broad range of intelligent tasks.

Bio-inspired recipes are being introduced to artificial neural networks for the efficient processing of spatio-temporal tasks. Among them, Leaky Integrate and Fire (LIF) model is the most remarkable one thanks to its temporal processing capability, lightweight model structure, and well investigated direct training methods. However, most learnable LIF networks generally take neurons as independent individuals that communicate via chemical synapses, leaving electrical synapses all behind. On the contrary, it has been well investigated in biological neural networks that the inter-neuron electrical synapse takes a great effect on the coordination and synchronization of generating action potentials. In this work, we are engaged in modeling such electrical synapses in artificial LIF neurons, where membrane potentials propagate to neighbor neurons via convolution operations, and the refined neural model ECLIF is proposed. We then build deep networks using ECLIF and trained them using a back-propagation-through-time algorithm. We found that the proposed network has great accuracy improvement over traditional LIF on five datasets and achieves high accuracy on them. In conclusion, it reveals that the introduction of the electrical synapse is an important factor for achieving high accuracy on realistic spatio-temporal tasks.

Spiking neural networks (SNNs) based on Leaky Integrate and Fire (LIF) model have been applied to energy-efficient temporal and spatiotemporal processing tasks. Thanks to the bio-plausible neuronal dynamics and simplicity, LIF-SNN benefits from event-driven processing, however, usually faces the embarrassment of reduced performance. This may because in LIF-SNN the neurons transmit information via spikes. To address this issue, in this work, we propose a Leaky Integrate and Analog Fire (LIAF) neuron model, so that analog values can be transmitted among neurons, and a deep network termed as LIAF-Net is built on it for efficient spatiotemporal processing. In the temporal domain, LIAF follows the traditional LIF dynamics to maintain its temporal processing capability. In the spatial domain, LIAF is able to integrate spatial information through convolutional integration or fully-connected integration. As a spatiotemporal layer, LIAF can also be used with traditional artificial neural network (ANN) layers jointly. Experiment results indicate that LIAF-Net achieves comparable performance to Gated Recurrent Unit (GRU) and Long short-term memory (LSTM) on bAbI Question Answering (QA) tasks, and achieves state-of-the-art performance on spatiotemporal Dynamic Vision Sensor (DVS) datasets, including MNIST-DVS, CIFAR10-DVS and DVS128 Gesture, with much less number of synaptic weights and computational overhead compared with traditional networks built by LSTM, GRU, Convolutional LSTM (ConvLSTM) or 3D convolution (Conv3D). Compared with traditional LIF-SNN, LIAF-Net also shows dramatic accuracy gain on all these experiments. In conclusion, LIAF-Net provides a framework combining the advantages of both ANNs and SNNs for lightweight and efficient spatiotemporal information processing., Comment: 14 pages, 9 figures, submitted to IEEE Transactions on Neural Networks and Learning Systems

There are two general approaches to developing artificial general intelligence (AGI)1: computer-science-oriented and neuroscience-oriented. Because of the fundamental differences in their formulations and coding schemes, these two approaches rely on distinct and incompatible platforms2–8, retarding the development of AGI. A general platform that could support the prevailing computer-science-based artificial neural networks as well as neuroscience-inspired models and algorithms is highly desirable. Here we present the Tianjic chip, which integrates the two approaches to provide a hybrid, synergistic platform. The Tianjic chip adopts a many-core architecture, reconfigurable building blocks and a streamlined dataflow with hybrid coding schemes, and can not only accommodate computer-science-based machine-learning algorithms, but also easily implement brain-inspired circuits and several coding schemes. Using just one chip, we demonstrate the simultaneous processing of versatile algorithms and models in an unmanned bicycle system, realizing real-time object detection, tracking, voice control, obstacle avoidance and balance control. Our study is expected to stimulate AGI development by paving the way to more generalized hardware platforms. The ‘Tianjic’ hybrid electronic chip combines neuroscience-oriented and computer-science-oriented approaches to artificial general intelligence, demonstrated by controlling an unmanned bicycle.

The integration of computer-science-oriented and neuroscience-oriented approaches is believed to be a promising way for the development of artificial general intelligence (AGI). Recently, a hybrid Tianjic chip that integrates both approaches has been reported, providing a general platform to facilitate the research of AGI. The control algorithm for handling various neural networks is the key to this platform; however, it is still primitive. In this work, we propose a hybrid neural state machine (H-NSM) framework that can efficiently cooperate with artificial neural networks and spiking neural networks and control the workflows to accomplish complex tasks. The H-NSM receives input from different types of networks, makes decisions according to the fusing of various information, and sends control signals to the sub-network or actuator. The H-NSM can be trained to adapt to context-aware tasks or sequential tasks, thereby improving system robustness. The training algorithm works correctly even if only 50% of the forced state information is provided. It achieved performance comparable to the optimum algorithm on the Tower of Hanoi task and achieved multiple tasks control on a self-driving bicycle. After only 50 training epochs, the transfer accuracy reaches 100% in the test case. It proves that H-NSM has the potential to advance control logic for hybrid systems, paving the way for designing complex intelligent systems and facilitating the research towards AGI.

Spiking neural networks (SNNs), stem from neuroscience, are promising for energy-efficient information processing due to the “event-driven” characteristic, whereas, they are inferior to artificial neural networks (ANNs) in real complicated tasks. However, ANNs usually suffer from expensive processing costs and a large number of parameters. Likewise, constrained to convergence speed, stability, complicated training mechanism and preprocessing setting, which is an obstacle for the SNN practitioners to expand its application scope. Inspired by the operation mechanism of human brain neurons, a brain-inspired Adaptive firing threshold Recurrent Leaky Integrate-and-Fire (ARLIF) model proposed. ARLIF and his variant ConvARLIF2D, which fuses the calculation logic of ANNs and bio-dynamic behaviors of SNNs, has a low-power dissipation since its number of weights is far less than SimpleRNN, GRU or LSTM. In this work, we present a Keras-based implementation of the layer of ARLIF and ConvARLIF2D that seamlessly fits in the contemporary deep learning framework without writing complex boilerplate code. The experiments result indicating that our ARLIF performs favorably against the state-of-the-art architectures.

The conventional methodology for designing QC-LDPC decoders is applied for fixed configurations used in wireless communication standards, and the supported largest expansion factor Z (the parallelism of the layered decoding) is a fixed number. In this paper, we study the circular-shifting network for decoding LDPC codes with arbitrary Z factor, especially for decoding large Z (Z P) codes, where P is the decoder parallelism. By buffering the P-length slices from the memory, and assembling the shifted slices in a fixed routine, the P-parallelism shift network can process Z-parallelism circular-shifting tasks. The implementation results show that the proposed network for arbitrary sized data shifting consumes only one times of additional resource cost compared to the traditional solution for only maximum P sized data shifting, and achieves significant saving on area and routing complexity.

Pedestrian attribute analysis, which is a vital component in the intelligent video surveillance area, can facilitate person retrieval, searching and indexing. However, the resolution of the surveillance video is relatively low. The ability of neural networks to learn abstract features (age, gender) and solid features (hat, backpack, clothes) on low-resolution images is limited due to the inability to accurately locate the human body and the disturbance of background noise. In this study, we adopt a Semantic Parsing Technique (SPT) as a pedestrian extractor to localize informative regions of human effectively. Experiments are conducted on the RAP, PA-100k and other datasets, we show that our proposed SAFE-Net is capable of capturing abstract features and solid features, and produces competitive performance with the state-of-the-art methods.

Compressed sensing (CS) allows a signal that is sparse in certain domain to be acquired and reconstructed accurately with only a small number of samples. In this paper, we propose an adaptive $(\ell_{0},\ell_{1})$ complex approximate message passing (CAMP) algorithm and its hardware implementation for complex-valued sparse signal recovery. Compared with the existing CAMP algorithm which solves $\ell_{1}$ -regularized least squares problems, our proposed algorithm adaptively switches between $\ell_{0}$ and $\ell_{1}$ -regularized least squares and therefore significantly outperforms the original CAMP. We implement the architecture in a medium-sized field-programmable gate array (FPGA) chip. For a sparse stepped frequency waveform radar application, we perform experiments on the simulated data and the data collected by a real radar system. According to the result, the decoder design achieves 7.2 dB improvement over the conventional CAMP architecture with less than 27.4% extra hardware cost.

There are two general approaches to developing artificial general intelligence (AGI)

Although deep neural networks (DNNs) are being a revolutionary power to open up the AI era, the notoriously huge hardware overhead has challenged their applications. Recently, several binary and ternary networks, in which the costly multiply-accumulate operations can be replaced by accumulations or even binary logic operations, make the on-chip training of DNNs quite promising. Therefore there is a pressing need to build an architecture that could subsume these networks under a unified framework that achieves both higher performance and less overhead. To this end, two fundamental issues are yet to be addressed. The first one is how to implement the back propagation when neuronal activations are discrete. The second one is how to remove the full-precision hidden weights in the training phase to break the bottlenecks of memory/computation consumption. To address the first issue, we present a multi-step neuronal activation discretization method and a derivative approximation technique that enable the implementing the back propagation algorithm on discrete DNNs. While for the second issue, we propose a discrete state transition (DST) methodology to constrain the weights in a discrete space without saving the hidden weights. Through this way, we build a unified framework that subsumes the binary or ternary networks as its special cases, and under which a heuristic algorithm is provided at the website https://github.com/AcrossV/Gated-XNOR. More particularly, we find that when both the weights and activations become ternary values, the DNNs can be reduced to sparse binary networks, termed as gated XNOR networks (GXNOR-Nets) since only the event of non-zero weight and non-zero activation enables the control gate to start the XNOR logic operations in the original binary networks. This promises the event-driven hardware design for efficient mobile intelligence. We achieve advanced performance compared with state-of-the-art algorithms. Furthermore, the computational sparsity and the number of states in the discrete space can be flexibly modified to make it suitable for various hardware platforms.

The end-to-end latency in 5th generation (5G) wireless communication is required to be less than a millisecond for delay sensitive scenarios, which becomes a serious constraint for the designing of 5G Turbo decoder. Traditional Turbo decoders primarily concern on throughput and resource consumption, which leads the latency unoptimized. This work targets on minimizing the decoding latency with the hardware cost constrains and performance constrains. A Cross Sliding Window (CSW) scheduling method is introduced which achieves both less memory capacity and lower latency. The Bit Error Rate (BER) is investigated and shows that the performance degradation of CSW method is acceptable. The hardware cost comparisons among the cross parallel window method, traditional sliding window method and the proposed CSW method suggest the hardware cost benefits and latency efficiency of CSW method.

In 5G communications, to meet the higher throughput requirement of Turbo decoding, Quadratic Permutation Polynomial (QPP) interleaver is adopted for contention-free parallel memory access. However, with the intrinsic attribute of QPP interleaver, the contention-free memory access is achieved only when the codeword length is divisible by the degree of parallelism (P), which leads the selection of P be rather inflexible especially when P is high. In this paper, we present a novel contention-free solution by introducing dual memory access and parallel window dynamic alignment. The proposed method provides more choices of contention-free parallelism. It offers throughput improvements for high parallelism (P>64) decoders with 3GPP LTE-A configurations, and also enables more P selections for QPP-interleaving Turbo decoders.

Computation intensive tasks which consist of nested short loops usually suffer from massive control overhead, or memory size increasing when employing loop unrolling. In this approach, by introducing a modified instruction fetch unit with instruction FIFO and multiple loop controllers, loops can be performed in hardware, and single execution-cycle instructions can be executed in self-loop. Therefore no loop overhead exists for the optimized processor. The flexibility and the instruction granularity are maintained. Special domains for loop and repeat indications are added in the application-specific instructions. The proposed approach achieves dramatically performance and area benefits for many nested short loop dominated programs where the loops are determinable.

Forward Error Correction (FEC) consumes excessive computation in a Software Defined Radio (SDR) system. In this work, a high-throughput flexible FEC processor is proposed for the decoding acceleration. The FEC processor enables Turbo/QC-LDPC/Convolutional Code decoding with software-hardware co-reconfigurability. A multi-algorithm unified trellis processing unit is introduced for resource sharing. A parallel architecture is proposed for high-throughput decoding. The Software Defined FEC (SD-FEC) with Application Specific Instruction-set Processor architecture is introduced for improving flexibility and enabling fast reconfiguration. The proposed SD-FEC can be applied to both low-cost low power applications and high performance applications. Results show that the proposed tri-mode FEC processor achieves high decoding efficiency and enough flexibility, which suits for the flexible SDR platforms.

Nowadays multi-standard wireless baseband, Convolutional Code (CC), Turbo code and LDPC code are widely applied and need to be integrated within one FEC module. Since memory occupies half or even more area of the decoder, memory sharing techniques for area saving purpose is valuable to consider. In this work, several memory merging techniques are proposed. A non-conflict access technique for merged path metric buffer is proposed. The results show that 41% of total memory bits are saved when integrating three different decoding schemes including CC (802.11a/g/n), LDPC (802.11n and 802.16e) and Turbo (3GPP-LTE). Synthesis result with 65nm process shows that the merged memory blocks consume merely 1.06mm 2 of the chip area.

Designing decoder for forward error correction (FEC) is more and more challenging because of the requirements on simultaneous supporting of various wireless standards within one IC module. The flexibility, silicon cost and throughput efficiency are all necessary to be traded off. In this paper, by using ASIP methodology, software-hardware co-design is introduced to offer sufficient flexibility of FEC decoding. The decoding procedure can be programmable for decoding QC-LDPC, Turbo and Convolutional Codes. Firstly, the common features from all mentioned algorithms and their corresponding datapaths are analyzed and a unified multi-standard datapath is introduced. Based on it, an application specific instruction-set is proposed and an ASIP (Application Specific Instruction-set Processor) for the FEC algorithms is designed. The firmware FEC codes are developed to adapt to standards. Synthesis results show that the proposed FEC processor is 1.54mm 2 under 65nm CMOS process. It offers QC-LDPC decoding for WiMAX, Turbo decoding for 3GPP-LTE, and 64 states Convolutional code (CC) decoding at the throughput of 193 Mbps, 62 Mbps and 60 Mbps respectively under clock frequency of 200 MHz. The proposed ASIP provides programmable high throughput compared to other tri-mode hardware modules.

In this paper, a light-weighted pipelined serial Viterbi Decoder is implemented for resource saving purpose. The trace back module of the decoder consumes fewer logical resources by employing a RAM-based Register Exchange architecture. All the metric and trace back bits are stored in the RAM to save logical resources. Synthesis results show that, the proposed architecture can save more than half of resource utilization than fabric IP core and has the minimum logic consumption than almost other schemes we can find with nearly no performance loss.