Search

Edge computing is promising to become one of the next hottest topics in artificial intelligence because it benefits various evolving domains such as real-time unmanned aerial systems, industrial applications, and the demand for privacy protection. This paper reviews recent advances on binary neural network (BNN) and 1-bit CNN technologies that are well suitable for front-end, edge-based computing. We introduce and summarize existing work and classify them based on gradient approximation, quantization, architecture, loss functions, optimization method, and binary neural architecture search. We also introduce applications in the areas of computer vision and speech recognition and discuss future applications for edge computing.

Magnetic RAM (MRAM)-based crossbar array has a great potential as a platform for in-memory binary neural network (BNN) computing. However, the number of word-lines that can be activated simultaneously is limited because of the low $I_{H}/I_{L}$ ratio of MRAM, which makes BNNs more vulnerable to the device variation. To address this issue, we propose an algorithm/hardware co-design methodology. First, we choose a promising memristor crossbar array (MCA) structure based on the sensitivity analysis to process variations. Since the selected MCA structure becomes more tolerant to the device variation when the number of 1 in input activation values decreases, we apply an input distribution regularization scheme to reduce the number of 1 in input of BNNs during training. We further improve the robustness against device variation by adopting the retraining scheme based on knowledge distillation. Experimental results show that the proposed method makes BNNs more tolerant to MRAM variation and increases the number of parallel word-line activation significantly; thereby achieving improved throughput and energy efficiency.

Magnetic skyrmions are emerging as potential candidates for next-generation non-volatile memories. In this article, we propose an in-memory binary neural network (BNN) accelerator based on the non-volatile skyrmionic memory, which we call as Skyrmionic In-Memory BNN Accelerator (SIMBA). SIMBA consumes 26.7 mJ of energy and 2.7 ms of latency when running inference on a VGG-like BNN. In addition, SIMBA saves up to 97.07% in energy consumption with $3.73\times $ speedup compared with the other accelerators in the literature at similar inference accuracy. Furthermore, we demonstrate improvements in the performance of SIMBA by optimizing material parameters, such as saturation magnetization, anisotropic energy, and damping ratio. Finally, we show that the inference accuracy of BNNs is robust against the possible stochastic behavior of SIMBA (88.5%±1%).

Due to the high computational complexity and memory storage requirement, it is hard to directly deploy a full-precision convolutional neural network (CNN) on embedded devices. The hardware-friendly designs are needed for resource-limited and energy-constrained embedded devices. Emerging solutions are adopted for the neural network compression, e.g., binary/ternary weight network, pruned network and quantized network. Among them, binary neural network (BNN) is believed to be the most hardware-friendly framework due to its small network size and low computational complexity. No existing work has further shrunk the size of BNN. In this work, we explore the redundancy in BNN and build a compact BNN (CBNN) based on the bit-level sensitivity analysis and bit-level data pruning. The input data is converted to a high dimensional bit-sliced format. In the post-training stage, we analyze the impact of different bit slices to the accuracy. By pruning the redundant input bit slices and shrinking the network size, we are able to build a more compact BNN. Our result shows that we can further scale down the network size of the BNN up to 3.9x with no more than 1% accuracy drop. The actual runtime can be reduced up to 2x and 9.9x compared with the baseline BNN and its full-precision counterpart, respectively.

In-memory computing (IMC) is a promising approach for energy cost reduction due to data movement between memory and processor for running data-intensive deep learning applications on the computing systems. Together with Binary Neural Network (BNN), IMC provides a viable solution for running deep neural networks at the edge devices with stringent memory and energy constraints. In this paper, we propose a novel 10T bit-cell with a back-end-of-line (BEOL) metal-oxide-metal (MOM) capacitor laid on pitch for in-memory computing. Our IMC bit-cell, when arranged in a memory array, performs binary convolution (XNOR followed by Bit-count operations) and binary activation generation operations. We show, when binary layers of BNN are mapped into our IMC arrays for MNIST digit classification, 98.75% accuracy with energy efficiency of 2193 TOPS/W and throughput of 22857 GOPS can be obtained. We determine the memory array size considering the word-line and bit-line nonidealities and show how these impact classification accuracy. We analyze the impact of process variations on classification accuracy and show how word-line pulse tunability provided by our design can be used to improve the robustness of classification under process variations.

As is well-known, defects precisely affect the lives and functions of the machines in which they occur, and even cause potentially catastrophic casualties. Therefore, quality assessment before mounting is an indispensable requirement for factories. Apart from the recognition accuracy, current networks suffer from excessive computing complexity, making it of great difficulty to deploy in the manufacturing process. To address these issues, this paper introduces binary networks into the area of surface defect detection for the first time, for the reason that binary networks prohibitively constrain weight and activation to +1 and −1. The proposed Bi-ShuffleNet and U-BiNet utilize binary convolution layers and activations in low bitwidth, in order to reach comparable performances while incurring much less computational cost. Extensive experiments are conducted on real-life NEU and Magnetic Tile datasets, revealing the least OPs required and little accuracy decline. When classifying the defects, Bi-ShuffleNet yields comparable results to counterpart networks, with at least 2× inference complexity reduction. Defect segmentation results indicate similar observations. Some network design rules in defect detection and binary networks are also summarized in this paper.

International audience; Voice quality is known to be an important factor for the characterization of a speaker's voice, both in terms of physiological features (mainly laryngeal and supralaryngeal) and of the speaker's habits (sociolinguistic factors). This paper is devoted to one of the main components of voice quality: phonation type. It proposes neural representations of speech followed by a cascade of two binary neural network-based classifiers, one dedicated to the detection of modal and nonmodal vowels, and one for the classification of nonmodal vowels into creaky and breathy types. This approach is evaluated on the spontaneous part of the PTSVOX database, following an expert manual labelling of the data by phonation type. The results of the proposed classifiers reaches on average 85 % accuracy at the framelevel and up to 95 % accuracy at the segment-level. Further research is planned to generalize the classifiers on more contexts and speakers, and thus pave the way for a new workflow aimed at characterizing phonation types.

Binary Neural Networks (BNN) are promising machine learning solutions for deployment on resource-limited devices. Recent approaches to training BNNs have produced impressive results, but minimizing the drop in accuracy from full precision networks is still challenging. One reason is that conventional BNNs ignore the uncertainty caused by weights that are near zero, resulting in the instability or frequent flip while learning. In this work, we investigate the intrinsic uncertainty of vanishing near-zero weights, making the training vulnerable to instability. We introduce an uncertainty-aware BNN (UaBNN) by leveraging a new mapping function called certainty-sign (c-sign) to reduce these weights' uncertainties. Our c-sign function is the first to train BNNs with a decreasing uncertainty for binarization. The approach leads to a controlled learning process for BNNs. We also introduce a simple but effective method to measure the uncertainty-based on a Gaussian function. Extensive experiments demonstrate that our method improves multiple BNN methods by maintaining stability of training, and achieves a higher performance over prior arts.

In order to make convolutional neural network (CNN) run more effectively on embedded devices, many studies about binary neural networks have appeared. The traditional binary neural networks adopt the STE-based binarization method. This method has the problem of gradient mismatch during the training process. It will cause an enormous loss of accuracy to the network. For this reason, this paper proposes an adaptive binarization method. The method selects hardtanh function as the basic binarization function. This function changes alternately as the training epoch increases. It will eventually approach the sign function. And the method adjusts the threshold of binarization adaptively by introducing a learnable parameter. Based on the two CNN models of VGG-Small and ResNet-20, this paper conducts training with the adaptive binarization method on the CIFAR-10 dataset. The results show that the proposed method is better than traditional methods (BC, BWN). And Its accuracy is very close to some advanced methods, while these methods have lower running speed. Therefore, the adaptive binarization method has higher accuracy, and it can achieve faster running speed in the application of binary neural network.

Binary Neural Networks enable smart IoT devices, as they significantly reduce the required memory footprint and computational complexity while retaining a high network performance and flexibility. This paper presents ChewBaccaNN, a 0.7 mm$^2$ sized binary convolutional neural network (CNN) accelerator designed in GlobalFoundries 22 nm technology. By exploiting efficient data re-use, data buffering, latch-based memories, and voltage scaling, a throughput of 241 GOPS is achieved while consuming just 1.1 mW at 0.4V/154MHz during inference of binary CNNs with up to 7x7 kernels, leading to a peak core energy efficiency of 223 TOPS/W. ChewBaccaNN's flexibility allows to run a much wider range of binary CNNs than other accelerators, drastically improving the accuracy-energy trade-off beyond what can be captured by the TOPS/W metric. In fact, it can perform CIFAR-10 inference at 86.8% accuracy with merely 1.3 $\mu J$, thus exceeding the accuracy while at the same time lowering the energy cost by 2.8x compared to even the most efficient and much larger analog processing-in-memory devices, while keeping the flexibility of running larger CNNs for higher accuracy when needed. It also runs a binary ResNet-18 trained on the 1000-class ILSVRC dataset and improves the energy efficiency by 4.4x over accelerators of similar flexibility. Furthermore, it can perform inference on a binarized ResNet-18 trained with 8-bases Group-Net to achieve a 67.5% Top-1 accuracy with only 3.0 mJ/frame -- at an accuracy drop of merely 1.8% from the full-precision ResNet-18., Comment: Accepted at IEEE ISCAS 2021, Daegu, South Korea, 23-26 May 2021

Batch normalization (BN) is a key facilitator and considered essential for state-of-the-art binary neural networks (BNN). However, the BN layer is costly to calculate and is typically implemented with non-binary parameters, leaving a hurdle for the efficient implementation of BNN training. It also introduces undesirable dependence between samples within each batch. Inspired by the latest advance on Batch Normalization Free (BN-Free) training [7], we extend their framework to training BNNs, and for the first time demonstrate that BNs can be completely removed from BNN training and inference regimes. By plugging in and customizing techniques including adaptive gradient clipping, scale weight standardization, and specialized bottleneck block, a BN-free BNN is capable of maintaining competitive accuracy compared to its BN-based counterpart. Extensive experiments validate the effectiveness of our proposal across diverse BNN backbones and datasets. For example, after removing BNs from the state-of-the-art ReActNets [38], it can still be trained with our proposed methodology to achieve 92.08%, 68.34%, and 68.0% accuracy on CIFAR-10, CIFAR-100, and ImageNet respectively, with marginal performance drop (0.23% ∼ 0.44% on CIFAR and 1.40% on ImageNet). Codes and pre-trained models are available at: https://github.com/VITA-Group/BNN_NoBN.

Binary Neural Networks (BNNs) are regarded as very effective approaches to reduce the high computational and memory cost of deep convolutional neural networks; however, most of the previous works only focus on the binarization of traditional large-scale networks. In this paper, to further decrease the computational and memory complexity, we propose to binarize the lightweight network: GhostNet, which can generate feature maps with fewer operations. We first modify the ghost block for reducing the information loss caused by the binarization. Moreover, to enhance the performance of binarized GhostNet, some training schemes are introduced including gradient approximation and new activation layers. Various experiments on CIFAR10 dataset show that, compared with other methods, the MicroNet can achieve better performance with fewer computational and memory cost.

This paper studies dynamic binary neural networks with simple sparse connection such that each neuron transforms three binary inputs to one binary output via three binary branches. First, it is proven theoretically that rotation-type periodic orbits can be stored into the networks and local stability of the stored orbits is very strong. If the local stability is very strong, the networks can remove any 1-bit error and make the periodic orbits robust. Second, an FPGA-based implementation is presented where the neurons are realized by simple majority decision circuits and the binary positive/negative connections are realized by direct/inverter connections. Using the hardware, typical periodic orbits are confirmed experimentally. The simple FPGA-based implementation is basic to realize engineering applications of the networks such as control signals of switching circuits.

This paper presents a development of wrist-worn device for fall detection which can be a fall detector and localizer. The system consists of a wristband and a mobile phone. The wristband acts as a fall detector and the mobile phone acts as an IoT gateway for locating and transmitting the fall information to the people involved. The fall detection method is based on a threshold-based and a binary neural network that uses to reduce resources and computational requirements. Experimental results showed that the accuracy of the proposed method was 97.23%. However, for some movement patterns such as hand clapping or hand waving, the system might be given a low accuracy.

Binary neural networks (BNNs) are adequate for energy-constrained embedded systems thanks to binarized parameters. Several researchers have proposed the compute-in-memory (CiM) SRAMs for XNOR-and-accumulation computations (XACs) in BNNs by adding additional transistors to the conventional 6T SRAM, which reduce the latency and energy of the data movements. However, due to the additional transistors, the CiM SRAMs suffer from larger area and longer wires than the conventional 6T SRAMs. Meanwhile, monolithic 3D (M3D) integration enables fine-grained 3D integration, reducing the 2D wire length in small functional units. In this paper, we propose a BNN accelerator (BNN_Accel), composed of a 9T CiM SRAM (CiM_SRAM), input buffer, and global periphery logic, to execute the computations in the binarized convolution layers of BNNs. We also propose CiM_SRAM with the subarray-level M3D integration (as well as the transistor-level M3D integration), which reduces the wire latency and energy compared to the 2D planar CiM_SRAM. Across the binarized convolution layers, our simulation results show that BNN_Accel with the 4-layer CiM_SRAM reduces the average execution time and energy by 39.9% and 23.2%, respectively, compared to BNN_Accel with the 2D planar CiM_SRAM.

We present a new PyTorch-based framework for neural network compression with fine-tuning named Neural Network Compression Framework (NNCF) (https://github.com/openvinotoolkit/nncf) . It leverages recent advances of various network compression methods and implements some of them, namely quantization, sparsity, filter pruning and binarization. These methods allow producing more hardware-friendly models that can be efficiently run on general-purpose hardware computation units (CPU, GPU) or specialized deep learning accelerators. We show that the implemented methods and their combinations can be successfully applied to a wide range of architectures and tasks to accelerate inference while preserving the original model’s accuracy. The framework can be used in conjunction with the supplied training samples or as a standalone package that can be seamlessly integrated into the existing training code with minimal adaptations.

Although deep neural networks are successful for many tasks in the speech domain, the high computational and memory costs of deep neural networks make it difficult to directly deploy highperformance Neural Network systems on low-resource embedded devices. There are several mechanisms to reduce the size of the neural networks i.e. parameter pruning, parameter quantization, etc. This paper focuses on how to apply binary neural networks to the task of speaker verification. The proposed binarization of training parameters can largely maintain the performance while significantly reducing storage space requirements and computational costs. Experiment results show that, after binarizing the Convolutional Neural Network, the ResNet34-based network achieves an EER of around 5% on the Voxceleb1 testing dataset and even outperforms the traditional real number network on the text-dependent dataset: Xiaole while having a 32x memory saving.

With the advent of the Internet of Everything, more and more devices are connected to the Internet every year. In major cities, in order to maintain normal social order, the demand for deployed cameras is also increasing. In terms of public safety, person Re-Identification (ReID) can play a big role. However, the current methods of ReID are to transfer the collected pedestrian images to the cloud for processing, which will bring huge communication costs. In order to solve this problem, we combine the recently emerging edge computing and use the edge to combine the end devices and the cloud to implement our proposed binarized segmented ResNet. Our method is mainly to divide a complete ResNet into three parts, corresponding to the end devices, the edge, and the cloud. After joint training, the corresponding segmented sub-network is deployed to the corresponding side, and inference is performed to realize ReID. In our experiments, we compared some traditional ReID methods in terms of accuracy and communication overhead. It can be found that our method can greatly reduce the communication cost on the basis of basically not reducing the recognition accuracy of ReID. In general, the communication cost can be reduced by four to eight times.

Binary Neural Networks (BNNs) offer an effective way to slash the cost of computation and memory accesses in inference. Recently, a plurality of ideas has been proposed, some of which are complementary while others are incompatible. This work presents a thorough review of state-of-the-art methods and an analysis of their computational cost based on the energy consumption of fixed-point, ternary and binary MAC vector operations. We derive an approach on how to systematically design a cost-efficient BNN. Our quantized LeNet and VGGNet architectures highlight the benefit of prudent capacity augmentation, with layer-wise ternarization providing best improvement of accuracy over μJ/dassification in BNNs.

The spiking neural network (SNN) converted from artificial neural network (ANN) usually contains many high-precision parameters. This will cause a lot of hardware resources to be consumed. A spiking binary neural network is proposed in this paper, whose weights are binary and it does not contain high-precision parameters. Experimental results show that the proposed network can be adapted to the neuromorphic chip and reduce memory consumption by 16 times without much loss of accuracy.

Advanced intelligent systems 2(10), 2000134 (2020). doi:10.1002/aisy.202000134, Published by Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim

Quantization neural networks use low-precision lowbit for both weights and activations, which are used for portable, low-power devices. And binary neural networks (BNNs) is an extreme case of quantization network in which both weights and activations are in a 1-bit representation. Binary neural networks replace expensive convolutional operations with fast bitwise operations such as xnor and popcount. While being efficient in the inference task, training BNN is very difficult because the training process is easy to be stuck on bad local minima. Besides, BNNs also suffer a problem of severe accuracy degradation. So how to train binary neural networks more effectively with higher accuracy? We answer this question by proposing a new training strategy on state-of-the-art binary neural network Bi-Real net. Our proposed strategy uses 3-steps binarization for activations, which helps to gradually reduce quantization error in the training phase. Also, we apply block-wise binarization to quantize block by block when training. Finally, we add a new regularization term into loss, to drive weights to −1 or 1. The comprehensive experiments on ImageNet show that our proposed training algorithm surpasses other binarization methods.

This paper presents the development of a binary neural network (BNN) hardware in metal-oxide thin film transistor (TFT) technology on a flexible substrate. We develop the BNN for a sweat odour application that takes data from an e-nose sensor array detecting odour, and classifies the odour. We demonstrate a fully functional BNN flexible integrated circuit (FlexIC) fabricated in $0.8 \mu \mathrm{m}$ n-type metal-oxide TFT on polyimide, consuming around 1mW power, which becomes the first neural network hardware built as a FlexIC.

Sound event detection (SED) is a hot topic in consumer and smart city applications. Existing approaches based on Deep Neural Networks are very effective, but highly demanding in terms of memory, power, and throughput when targeting ultra-low power always-on devices. Latency, availability, cost, and privacy requirements are pushing recent IoT systems to process the data on the node, close to the sensor, with a very limited energy supply, and tight constraints on the memory size and processing capabilities precluding to run state-of-the-art DNNs. In this paper, we explore the combination of extreme quantization to a small-footprint binary neural network (BNN) with the highly energy-efficient, RISC-V-based (8+1)-core GAP8 microcontroller. Starting from an existing CNN for SED whose footprint (815 kB) exceeds the 512 kB of memory available on our platform, we retrain the network using binary filters and activations to match these memory constraints. (Fully) binary neural networks come with a natural drop in accuracy of 12-18% on the challenging ImageNet object recognition challenge compared to their equivalent full-precision baselines. This BNN reaches a 77.9% accuracy, just 7% lower than the full-precision version, with 58 kB (7.2 times less) for the weights and 262 kB (2.4 times less) memory in total. With our BNN implementation, we reach a peak throughput of 4.6 GMAC/s and 1.5 GMAC/s over the full network, including preprocessing with Mel bins, which corresponds to an efficiency of 67.1 GMAC/s/W and 31.3 GMAC/s/W, respectively. Compared to the performance of an ARM Cortex-M4 implementation, our system has a 10.3 times faster execution time and a 51.1 times higher energy-efficiency., Comment: 6 pages conference

This study investigates the performance of a binary neural network, which is a lightweight neural network, for classification problems in holographic applications. We performed data classification in holographic memory using XNOR-Net as one of the binary neural networks. We compared the performance of the binary neural network with convolutional neural networks.

Adversarial neural cryptography is deemed as an encouraging area of research that could provide different perspective in the post-quantum cryptography age, specially for secure transmission of information. Nevertheless, it is still under explored with a handful of publications on the subject. This study proposes the theoretical implementation of a neuroevolved binary neural network based on boolean logic functions only (BiSUNA), with the purpose of encrypting/decrypting a payload between two agents, hiding information from a competitor.

A 10nm digital Binary Neural Network (BNN) chip implements 1b activations and weights for compute density of 418TOPS/mm2 and memory density of 414KB/mm2. The chip achieves an energy efficiency of 617TOPS/W by leveraging Compute Near Memory (CNM), parallel inner product compute, and Near-Threshold Voltage (NTV) operation. The digital BNN design approaches the energy efficiency of analog in-memory techniques while also ensuring deterministic, scalable, and precise operation.

Standard convolutional neural networks (CNNs) have large amounts of data redundancy, and the same accuracy can be obtained even in lower bit weights instead of floating-point representation. Most CNNs have to be developed and executed on high-end GPU-based workstations, for which it is hard to transplant the existing implementations onto portable edge FPGAs because of the limitation of on-chip block memory storage size and battery capacity. In this paper, we present adaptive pointwise convolution and 2D convolution joint network (AP2D-Net), an ultra-low power and relatively high throughput system combined with dynamic precision weights and activation. Our system has high performance, and we make a trade-off between accuracy and power efficiency by adopting unmanned aerial vehicle (UAV) object detection scenarios. We evaluate our system on the Zynq UltraScale+ MPSoC Ultra96 mobile FPGA platform. The target board can get the real-time speed of 30 fps under 5.6 W, and the FPGA on-chip power is only 0.6 W. The power efficiency of our system is 2.8&times, better than the best system design on a Jetson TX2 GPU and 1.9&times, better than the design on a PYNQ-Z1 SoC FPGA.

Recently, binary neural networks (BNNs) have been extensively studied since they can address the challenge of large memory footprint and power consumption caused by floating-point convolutional neural networks (CNNs) while maintaining tolerable accuracy. There have been many efforts designing BNN hardware accelerators and showed very well results. But even that will require huge improvements in reduction of memory cost and energy efficiency. In this paper, we first propose the bagged binary neural network accelerator (BBNA), which is a fully pipelined BNN accelerator with bagging ensemble unit for aggregating several BNN pipelines to achieve better model accuracy. In other words, the proposed architecture provides an opportunity for embedded devices to obtain acceptable accuracy with smaller ensemble BNNs. As a result, compared to other works, our design achieves 1.9x better energy efficiency with better performance, and the ensemble method saves more than 79% and 94% memory footprint, respectively, with nearly accuracy on MNIST dataset.

We consider the compilation of a binary neural network's decision function into tractable representations such as Ordered Binary Decision Diagrams (OBDDs) and Sentential Decision Diagrams (SDDs). Obtaining this function as an OBDD/SDD facilitates the explanation and formal verification of a neural network's behavior. First, we consider the task of verifying the robustness of a neural network, and show how we can compute the expected robustness of a neural network, given an OBDD/SDD representation of it. Next, we consider a more efficient approach for compiling neural networks, based on a pseudo-polynomial time algorithm for compiling a neuron. We then provide a case study in a handwritten digits dataset, highlighting how two neural networks trained from the same dataset can have very high accuracies, yet have very different levels of robustness. Finally, in experiments, we show that it is feasible to obtain compact representations of neural networks as SDDs., In Proceedings of the 17th International Conference on Principles of Knowledge Representation and Reasoning (KR) 2020

Mobile networks and binary neural networks are the most commonly used techniques for modern deep learning models to perform a variety of tasks on embedded systems. In this paper, we develop a technique to identify an object considering the deep learning pre-trained model MobileNet for Single Shot Multi-Box Detector (SSD). This algorithm is used for real-time detection, and for webcam feed to detect the purpose webcam which detects the object in a video stream. Therefore, we use an object detection module that can detect what is in the video stream. In order to implement the module, we combine the MobileNet and the SSD framework for a fast and efficient deep learning-based method of object detection. The main purpose of our research is to elaborate the accuracy of an object detection method SSD and the importance of pre-trained deep learning model MobileNet. The experimental results show that the Average Precision (AP) of the algorithm to detect different classes as car, person and chair is 99.76%, 97.76% and 71.07%, respectively. This improves the accuracy of behavior detection at a processing speed which is required for the real-time detection and the requirements of daily monitoring indoor and outdoor.

In this paper, we propose an efficient design to convert full-precision convolutional networks into binary neural networks. Our method approximates a full-precision convolutional filter by sum of binary filters with multiplicative and additive scaling factors. We present closed form solutions to the proposed methods. We perform experiments on binary neural networks with binary activations and pre-trained neural networks with full-precision activations. The results show an increase in accuracy compared to previous binary neural networks. Furthermore, to reduce the complexity, we prune scaling factors considering the accuracy. We show that up to a certain degree of threshold, we can prune scaling factors while maintaining accuracy comparable to full-precision convolutional neural networks.

Binarization of neural network models is considered as one of the promising methods to deploy deep neural network models on resource-constrained environments such as mobile devices. However, Binary Neural Networks (BNNs) tend to suffer from severe accuracy degradation compared to the full-precision counterpart model. Several techniques were proposed to improve the accuracy of BNNs. One of the approaches is to balance the distribution of binary activations so that the amount of information in the binary activations becomes maximum. Based on extensive analysis, in stark contrast to previous work, we argue that unbalanced activation distribution can actually improve the accuracy of BNNs. We also show that adjusting the threshold values of binary activation functions results in the unbalanced distribution of the binary activation, which increases the accuracy of BNN models. Experimental results show that the accuracy of previous BNN models (e.g. XNOR-Net and Bi-Real-Net) can be improved by simply shifting the threshold values of binary activation functions without requiring any other modification., Comment: CVPR 2021, 10 pages, 10 figures

Despite the growing popularity of deep learning technologies, high memory requirements and power consumption are essentially limiting their application in mobile and IoT areas. While binary convolutional networks can alleviate these problems, the limited bitwidth of weights is often leading to significant degradation of prediction accuracy. In this paper, we present a method for training binary networks that maintains a stable predefined level of their information capacity throughout the training process by applying Shannon entropy based penalty to convolutional filters. The results of experiments conducted on the SVHN, CIFAR and ImageNet datasets demonstrate that the proposed approach can statistically significantly improve the accuracy of binary networks.

Despite the achievements of recent binarization methods on reducing the performance degradation of Binary Neural Networks (BNNs), gradient mismatching caused by the Straight-Through-Estimator (STE) still dominates quantized networks. This paper proposes a meta-based quantizer named QuantNet, which utilizes a differentiable sub-network to directly binarize the full-precision weights without resorting to STE and any learnable gradient estimators. Our method not only solves the problem of gradient mismatching, but also reduces the impact of discretization errors, caused by the binarizing operation in the deployment, on performance. Generally, the proposed algorithm is implemented within a fully differentiable framework, and is easily extended to the general network quantization with any bits. The quantitative experiments on CIFAR-100 and ImageNet demonstrate that QuantNet achieves the significant improvements comparing with previous binarization methods, and even bridges gaps of accuracies between binarized models and full-precision models.

A novel synaptic architecture based on NAND cell strings is proposed as a high-density synapse capable of XNOR operation for binary neural networks (BNNs) for the first time. By changing the threshold voltage of NAND flash cells and input voltages in complementary fashion, the XNOR operation is successfully demonstrated. The large on/off current ratio (~7×105) of NAND flash cells can implement high-density and highly-reliable BNNs without error correction codes. It is shown that without conventional ISPP scheme, only 1 erase or program pulse can achieve sufficiently low bit-error rate. Finally, the estimated synapse area of VNAND memory with 128 stacks is ~100 times that of 2T2R synapse in RRAMs.

Binary neural networks can effectively reduce the number of required parameters but might decrease the classification accuracy. To solve the problem, we propose a dual-path binary neural network (DPBNN) in this paper. Experimental results show that our DPBNN can outperform other traditional binary neural network in CIFAR-10 and SVHN dataset. The proposed network is simple, so it is suitable to be implemented on embedded systems or SoC designs.

Research into deep neural networks has brought about architectures and models that solve problems we once thought could not be approached by machine learning. Year after year, performance improves, to the point that it is becoming difficult to differentiate between the strengths of deep neural network models given our current data sets. However, due to their significant requirements in terms of hardware resources, all but few architectures are dependent on cloud environments. Yet, there are many use cases for neural networks in a variety of areas, many of which require consumer-grade hardware or highly resource constrained embedded devices. This paper offers a comparison of selected state-of-the-art neural network miniaturization methods, and proposes a new approach, PXNOR, that achieves a noteworthy accuracy, remarkable inference speed and significant memory savings. PXNOR seeks to fully replace traditional convolutional filters with approximate operations, while replacing all multiplications and additions with simpler, much faster versions such as XNOR and bitcounting, which are implemented at hardware level on all existing platforms.

Binary Neural Networks (BNNs) show promising progress in reducing computational and memory costs, but suffer from substantial accuracy degradation compared to their real-valued counterparts on large-scale datasets, e.g., ImageNet. In this work we study existing BNN architectures and revisit the commonly used technique to include scaling factors. We suggest several architectural design principles for BNNs, based on our studies on architectures. Guided by our principles we develop a novel BNN architecture BinaryDenseNet, which is the first architecture specifically created for BNNs to the best of our knowledge. In our experiments, BinaryDenseNet achieves 18.6% and 7.6% relative improvement over the well-known XNOR-Network and the current state-of-the-art Bi-Real Net in terms of top-1 accuracy on ImageNet, respectively. Further, we show the competitiveness of our BinaryDenseNet regarding memory requirements and computational complexity.

Binary neural networks (BNNs) - neural networks with binary {-1, +1} weights and activations at run-time, have attracted much attention in recent years, which drastically reduce resource requirement and power consumption with tolerable accuracy loss. While, spiking neural networks (SNNs) are designed as a brain-inspired computing model, which can be deployed on large-scale distributed and event-based neuromorphic hardware efficiently. To enable a conversion of above two, we develop a variant of BNNs which employ binary-valued {0, +1} activations and ternary-valued {-1, 0, +1} weights and achieve start-of-the-art classification accuracy across two standard vision datasets, even better than the previous proposals using BNNs or SNNs. Furtherly, we propose an effective configuration method for network deployment on our designed programmable neuro-synaptic system with only one timestep and milliwatt level power consumption.

We report an experimental implementation of width-tunable neurons to train a binary neural network. The angle-dependent magnetic behavior in an oxide thin film highly mimics neurons with width-controllable activation window, providing an opportunity to train the activation functions and weights toward binary values. We apply this feature to train the MNIST dataset using a 684-800-10 fully connected network and achieve a high accuracy of 97.4%, thus opening an implementation strategy toward training neural networks.

Neural networks have been widely used and implemented on various hardware platforms, but high computational costs and low similarity of network structures relative to hardware structures are often obstacles to research. In this paper, we propose a novel neural network in combination with the structural features of a recently proposed memory-based programmable logic device, compare it with the standard structure, and test it on common datasets with full and binary precision, respectively. The experimental results reveal that the new structured network can provide almost consistent full-precision performance and binary-precision performance ranging from 61.0% to 78.8% after using sparser connections and about 50% reduction in the size of the weight matrix.

It is always well believed that Binary Neural Networks (BNNs) could drastically accelerate the inference efficiency by replacing the arithmetic operations in float-valued Deep Neural Networks (DNNs) with bit-wise operations. Nevertheless, there has not been open-source implementation in support of this idea on low-end ARM devices (e.g., mobile phones and embedded devices). In this work, we propose daBNN --- a super fast inference framework that implements BNNs on ARM devices. Several speed-up and memory refinement strategies for bit-packing, binarized convolution, and memory layout are uniquely devised to enhance inference efficiency. Compared to the recent open-source BNN inference framework, BMXNet, our daBNN is $7\times$$\sim$$23\times$ faster on a single binary convolution, and about $6\times$ faster on Bi-Real Net 18 (a BNN variant of ResNet-18). The daBNN is a BSD-licensed inference framework, and its source code, sample projects and pre-trained models are available on-line: https://github.com/JDAI-CV/dabnn., Accepted by 2019 ACMMM Open Source Software Competition. Source code: https://github.com/JDAI-CV/dabnn

Energy-constrained neural network processing is in high demanded for various mobile applications. Binarized neural network (BNN) aggressively enhances the computational efficiency, and in contrast, it suffers from degradation of accuracy due to its extreme approximation. We propose a neural network model using a new activation function "Delta" based on binarization of differences between weighted-sums. The "Delta" retains the magnitude relation between numerical values, and conveys richer information than ordinary binarization. We can design the hardware architecture for the proposed model with almost the same elements as BNN. The evaluation shows that it achieves higher recognition accuracy than a conventional BNN with almost the same hardware configuration.