Search

This paper provides a comprehensive study of signal integrity issues in RRAM-based neuromorphic chip crossbar arrays due to interconnect parasitic. First, the parasitic parameters of the crossbar array are calculated by the partial equivalent element circuit (PEEC) method with an efficient unit-cell approach. Numerical experiments show that for a $50\times 50$ array scale, this method consumes only 1.5% of the calculation time of the commercial software based 3D model, which translates to a calculation speed up of 72 times. Moreover, the PEEC circuit simulation results match well with those of the 3D model. Then, we investigate the effects of parasitic parameters such as capacitance and inductance, as well as the feature size of the crossbar array on signal integrity. All of them will lead to corresponding changes in parasitic effects, which in turn result in the most common signal integrity issues such as crosstalk, time delay and mutual capacitive coupling induced sneak path problem. Different from other studies, the excitation used in this paper is the neural spike signal generated by the Izhikevich neuron model, which is both rich in dynamic characteristics and high in computational efficiency. Finally, based on the study we propose a simple but effective design scheme for reduction of signal distortion, which can provide valuable design guidance for neuromorphic systems to achieve high performance and high computational accuracy. [ABSTRACT FROM AUTHOR]

Neural networks (NNs) with large scales of artificial neurons are increasingly used in recognition and classification tasks. In power-constrained scenarios, the tradeoff between performance and hardware consumptions must be carefully evaluated before silicon tape-out. In this paper, we proposed a systematic approach to design ultra-low power NN system. This work is motivated by the facts that NNs are resilient to approximation in many of the computations and NNs are outputting statistical tensors which are acceptable to less-than-perfect results. We resort to the front-back end approach with a twofold aim: (1) a fast and accurate design approach is proposed by estimating the computing quality of low-power approximate adder arrays, and it is adopted to evaluate the neural network system; (2) a quality configurable engine with different approximation degrees while processing NNs is implemented. The proposed work is demonstrated with a comprehensive keyword spotting (KWS) system as an ultra-low power NN engine. The experimental environment is setup with ten keywords from the google speech command dataset (GSCD) using an industrial 22-nm ultra-low-leakage (ULL) process. Comparing to the state-of-the-art KWS processors, the proposed approximate NN engine can demonstrate over 60% improvement in power efficiency and $1.1\times $ area efficiency while achieving similar recognition accuracy. [ABSTRACT FROM AUTHOR]

In this paper, aiming to systematically derive single-inductor multi-input multi-output (SI-MIMO) converters having favorable merits of high-density and low-cost, a basic model is firstly extracted from the working principle. Theoretically, by enumerating all possible connections of the basic model and one-by-one manually judging their effectiveness according to working criterions, multiple viable SI-MIMO topologies can be obtained, but with heavy workload. To simplify the topology derivation process, this paper also proposes a computer-aided method to replace the manual effort. By modelling the basic model and working criterions with graph theory, MATLAB codes can be developed to automatically screen out the effective topologies. With the proposed method, 8 three-port, 28 four-port and 115 five-port converters with single inductor are conveniently obtained, including both the existing and new topologies. Moreover, by configuring ports and switches in each derived topology, multiple specific converters can be further expanded. These converters can provide more options for practical engineering applications, and a new topology is also introduced and experimentally verified in the paper. [ABSTRACT FROM AUTHOR]

Instability and oscillation issues originating in high-voltage direct current (HVDC) systems comprising modular multilevel converters (MMCs) are gaining increasing interest in the research community. To detect such phenomena in advance, considerable effort has been devoted recently to developing MMC models for small-signal analysis. The derivation of such models is a challenging task due to the topology and complex control structure of MMCs, which results in them having a multi-frequency response. To address this issue, scholars developed methods based on dynamic phasors and harmonic state-space modelling, which, however, require extensive pen-and-paper computations. In this paper, the periodic small-signal analysis (PAC) is adopted to determine numerically several MMCs transfer functions. Such functions can be computed between any electrical circuit node or input/output port, without the need to recast the three-phase average MMC model to derive a linear equivalent one, possibly in the DQ-frame. We show how vector fitting allows converting these functions to equivalent algebraic representations, which can be profitably used to design MMCs and study their stability following some parameter changes. To showcase this feature, we exploit one of these transfer functions to detect DC-side instability in a point-to-point HVDC system. [ABSTRACT FROM AUTHOR]

This paper presents a sequential learning framework for sensors in a network, where a few sensors assume the role of an instructor to train other sensors in the network. The instructors provide estimated labels for measurements of new sensors. These labels are possibly noisy, because a classifier of the instructor may not be perfect. A recursive density estimator is proposed to obtain the true measurement model (i.e., the observation density conditioned on the label) in spite of the training with noisy labels. Specifically, this paper answers the question “Can a sensor train other sensors?”, provides necessary conditions for sensors to act as instructors, presents a sequential learning framework using recursive nonparametric kernel density estimation, and provides a convergence rate for the expected error in an observation density. The underlying concepts are illustrated and validated with simulation results. [ABSTRACT FROM AUTHOR]

In recent years, fast computation, low power, and small footprint are the key motivations for building SNN hardware. The unique features of SNN hardware have not been fully exploited, where the computation speed and energy efficiency of the SNN hardware can be improved according to the sparse spiking and non-uniform traffic of SNN. In this paper, we propose a 5.28-mm $^{2}~4096$ -neuron 1M-synapse energy-efficient digital SNN hardware that can achieve ultra-low energy per synaptic operation of 4.5 pJ. The proposed neuron computing unit is implemented in pipeline architecture to achieve high synaptic processing speed. The proposed spike processing unit can significantly increase the processing speed of the neuron core by $1.9\times $ and $9.4\times $ when the spike injection rate is 50% and 10%, respectively. Besides, the increase in the processing speed of the neuron core leads to a reduction in energy consumption of up to 81.5%. An event-driven clock gating circuit that can reduce the power consumption of the proposed neuron block by more than 70% is proposed in this paper. This paper proposes a supervised STDP+ algorithm for SNN training, and the classification accuracy of the MNIST digits is 89.6% with 73.6% weight sparsity of the output layer. [ABSTRACT FROM AUTHOR]

This paper studies the process scalability of pulse-mode CMOS circuits for analog 2-D convolution in computer vision systems. A simple, scalable architecture for an integrate and fire neuron is presented for implementing weighted addition of pulse-frequency modulated (PFM) signals. Sources of error are discussed and modeled in a detailed behavioral simulation and compared with equivalent transistor-level simulations. Next, the design of a 180-nm PFM chip with programmable weights is presented, and full image convolutions are demonstrated with the analog hardware. Preliminary chip measurements for a 45-nm implementation are also included to demonstrate process scalability. Design considerations for porting this architecture to nanometer processes, including FinFET technologies, are then discussed. This paper concludes with a simulation study on scaling down to 10 nm using a predictive technology model. [ABSTRACT FROM AUTHOR]

This paper is concerned with a method for design and analysis of specific neuronal networks, called central pattern generators (CPGs), which produce primary rhythmic patterns in animals. In particular, the paper is focused on synthetic CPGs made up of few basic elements and governing quadrupeds’ gaits and gait transitions, under the control of an external drive. The method combines the principles of bifurcation theory, geometric properties of symmetry, and numerical analysis based on the recently proposed toolbox CEPAGE. The method is applied to two CPGs, one bio-inspired and one purely synthetic. In both the cases, the method provides a way to obtain a desired sequence of gaits by continuously changing a bifurcation parameter related to the external drive. [ABSTRACT FROM AUTHOR]

Memristive LSTM networks have been proven as a powerful Neuromorphic Computing Architecture (NCA) for various time series forecasting tasks and are recognized as the next generation of AI. However, a lack of model explainability makes it hard to properly interpret forecasting results for existing memristive LSTM networks, which makes this NCA unreliable, unaccountable and untrustworthy. In this paper, an interpretable memristive (IM) LSTM network design is proposed for time series forecasting, where the mixture attention technique is embedded into IM-LSTM cells for characterizing the variable-wise feature and the temporal importance. The updating rules and training approach are also presented for this interpretable memristive LSTM network. We evaluate this approach on a probabilistic residential load forecasting task incorporating PV. By improving model interpretability, the most influential predictive factors can be verified by Built Environment domain experts, demonstrating the effectiveness of our design. [ABSTRACT FROM AUTHOR]

Spiking neural network (SNN) can process discrete spikes and offers a high degree of real-time performance and excellent energy efficiency ratio. However, most current neuromorphic hardware platforms lack efficient driven algorithms and only support a single type of neuron model, which has slow speed and poor scalability. This paper proposes a general FPGA-based neuromorphic hardware acceleration platform (FPGA-NHAP), supporting the effective inference and acceleration of SNN network with low power, high speed and good scalability. First, a neuron computing unit is designed to simulate the both LIF and Izhikevich (IZH) neurons with the parallel spike caching and scheduling technique. Second, a novel integrated driven update algorithm is proposed to complete the spike encoding of external data, reducing the waiting time of neuron state update effectively. Third, the proposed platform is implemented using a RISC-V processor and a Xilinx FPGA, simulating 16,384 neurons and 16.8 million synapses with a power consumption of 0.535 W. Finally, two different three-layer SNN networks are deployed on the proposed platform for recognition tasks on the MNIST and Fashion-MNIST datasets, achieving the accuracy of 97.70%, 85.14% (LIF) and 97.81%, 83.16% (IZH), frame rates of 208 frame/s, 128 frame/s (LIF) and 206 frame/s, 141 frame/s (IZH), respectively. [ABSTRACT FROM AUTHOR]

The rising popularity of intelligent mobile devices and the computational cost of deep learning-based models call for efficient and accurate on-device inference schemes. We propose a novel model compression scheme that allows inference to be carried out using bit-level sparsity, which can be efficiently implemented using in-memory computing macros. In this paper, we introduce a method called BitS-Net to leverage the benefits of bit-sparsity (where the number of zeros are more than number of ones in binary representation of weight/activation values) when applied to compute-in-memory (CIM) with resistive RAM (RRAM) to develop energy efficient DNN accelerators operating in the inference mode. We demonstrate that BitS-Net improves the energy efficiency by up to 5x for ResNet models on the ImageNet dataset. [ABSTRACT FROM AUTHOR]

This paper addresses and experimentally demonstrates a programmable linear equation solver by analog computation. A set of differential equations using transconductance devices directly translated from circuit theory converges to the linear equation solution. These energy-efficient analog techniques are experimentally demonstrated in a configurable analog platform. The resulting analog linear equation solution circuits are effectively analog filters. The paper analyzes the algorithmic issues and analog numerical analysis issues, including accuracy, convergence time, and the interpretation of condition number for analog solutions. [ABSTRACT FROM AUTHOR]

The paper considers a relevant class of networks containing memristors and (possibly) nonlinear capacitors and inductors. The goal is to unfold the nonlinear dynamics of these networks by highlighting some main features that are potentially useful for real-time signal processing and in-memory computing. In particular, an analytic treatment is provided for dynamic phenomena as the presence of invariant manifolds, the coexistence of different regimes, complex dynamics and attractors and the phenomenon of bifurcations without parameters, i.e., bifurcations due to changing the initial conditions of the state variables for a fixed set of circuit parameters. The paper also addresses the issue of how to design pulse independent voltage or current sources to steer the network dynamics through different manifolds and attractors. Two relevant examples are worked out in details, namely, a variant of Chua’s circuit with a memristor and a nonlinear capacitor and a relaxation oscillator with a memristor and a nonlinear inductor. In the latter example, the paper also studies the effect on manifolds and coexisting dynamics when real memristive devices are accounted for using a class of extended memristor models. The analysis is conducted by means of a recently developed technique named flux-charge analysis method (FCAM). Numerical simulations are presented to confirm the theoretic findings. [ABSTRACT FROM AUTHOR]

This paper provides the stability analysis for a model-free action-dependent heuristic dynamic programing (HDP) approach with an eligibility trace long-term prediction parameter ($\lambda $). HDP($\lambda $) learns from more than one future reward. Eligibility traces have long been popular in Q-learning. This paper proves and demonstrates that they are worthwhile to use with HDP. In this paper, we prove its uniformly ultimately bounded (UUB) property under certain conditions. Previous works present a UUB proof for traditional HDP [HDP($\lambda =0$)], but we extend the proof with the $\lambda $ parameter. By using Lyapunov stability, we demonstrate the boundedness of the estimated error for the critic and actor neural networks as well as learning rate parameters. Three case studies demonstrate the effectiveness of HDP($\lambda $). The trajectories of the internal reinforcement signal nonlinear system are considered as the first case. We compare the results with the performance of HDP and traditional temporal difference [TD($\lambda $)] with different $\lambda $ values. The second case study is a single-link inverted pendulum. We investigate the performance of the inverted pendulum by comparing HDP($\lambda $) with regular HDP, with different levels of noise. The third case study is a 3-D maze navigation benchmark, which is compared with state action reward state action, Q($\lambda $), HDP, and HDP($\lambda $). All these simulation results illustrate that HDP($\lambda $) has a competitive performance; thus this contribution is not only UUB but also useful in comparison with traditional HDP. [ABSTRACT FROM AUTHOR]

This paper presents an improved method for reducing high-order dynamical system models via clustering. Agglomerative hierarchical clustering based on performance evaluation (HC-PE) is introduced for model order reduction. This method computes the reduced order denominator of the transfer function model by clustering system poles in a hierarchical dendrogram. The base layer represents an $n{\textrm {th}}$ order system, which is used to calculate each successive layer to reduce the model order until finally reaching a second-order system. HC-PE uses a mean-squared error (MSE) in every reduced order, which modifies the pole placement process. The coefficients for the numerator of the reduced model are calculated by using the Padé approximation (PA) or alternatively a genetic algorithm (GA). Several numerical examples of reducing techniques are taken from the literature to compare with HC-PE. Two classes of results are shown in this paper. The first sets are single-input single-output models that range from simple models to 48th order systems. The second sets of experiments are with a multi-input multioutput model. We demonstrate the best performance for HC-PE through minimum MSEs compared with other methods. Furthermore, the robustness of HC-PE combined with PA or GA is confirmed by evaluating the third-order reduced model for the triple-link inverted pendulum model by adding a disturbance impulse signal and by changing model parameters. The relevant stability proofs are provided in Appendixes A and B in the supplementary material. HC-PE with PA slightly outperforms its performance with GA, but both approaches are attractive alternatives to other published methods. [ABSTRACT FROM AUTHOR]

A growing number of applications, e.g., video surveillance and medical image analysis, require training recognition systems from large amounts of weakly annotated data, while some targeted interactions with a domain expert are allowed to improve the training process. In such cases, active learning (AL) can reduce labeling costs for training a classifier by querying the expert to provide the labels of most informative instances. This paper focuses on AL methods for instance classification problems in multiple instance learning (MIL), where data are arranged into sets, called bags, which are weakly labeled. Most AL methods focus on single-instance learning problems. These methods are not suitable for MIL problems because they cannot account for the bag structure of data. In this paper, new methods for bag-level aggregation of instance informativeness are proposed for multiple instance AL (MIAL). The aggregated informativeness method identifies the most informative instances based on classifier uncertainty and queries bags incorporating the most information. The other proposed method, called cluster-based aggregative sampling, clusters data hierarchically in the instance space. The informativeness of instances is assessed by considering bag labels, inferred instance labels, and the proportion of labels that remain to be discovered in clusters. Both proposed methods significantly outperform reference methods in extensive experiments using benchmark data from several application domains. Results indicate that using an appropriate strategy to address MIAL problems yields a significant reduction in the number of queries needed to achieve the same level of performance as single-instance AL methods. [ABSTRACT FROM AUTHOR]

This paper introduces rational polynomial chaos expansions for the stochastic modeling of the frequency-domain responses of linear electrical networks. The proposed method models stochastic network responses as a ratio of polynomial chaos expansions, rather than the standard single polynomial expansion. This approach is motivated by the fact that network responses are best represented by rational functions of both frequency and parameters. In particular, it is proven that the rational stochastic model is exact for lumped networks. The model coefficients are computed via an iterative re-weighted linear least-square regression. Several application examples, concerning both lumped and a distributed systems, illustrate and validate the advocated methodology. [ABSTRACT FROM AUTHOR]

The computation-intensive circuit simulation makes the analog circuit sizing challenging for large-scale/complicated analog/RF circuits. A Bayesian optimization approach has been proposed recently for the optimization problems involving the evaluations of black-box functions with high computational cost in either objective functions or constraints. In this paper, we propose a weighted expected improvement-based Bayesian optimization approach for automated analog circuit sizing. Gaussian processes (GP) are used as the online surrogate models for circuit performances. Expected improvement is selected as the acquisition function to balance the exploration and exploitation during the optimization procedure. The expected improvement is weighted by the probability of satisfying the constraints. In this paper, we propose a complete Bayesian optimization framework for the optimization of analog circuits with constraints for the first time. The existing GP model-based optimization methods for analog circuits take the GP models as either offline models or as assistance for the evolutionary algorithms. We also extend the Bayesian optimization algorithm to handle multi-objective optimization problems. Compared with the state-of-the-art approaches listed in this paper, the proposed Bayesian optimization method achieves better optimization results with significantly less number of simulations. [ABSTRACT FROM PUBLISHER]

Artificial neural networks (ANNs) have traditionally been seen as black-box models, because, although they are able to find “hidden” relations between inputs and outputs with a high approximation capacity, their structure seldom provides any insights on the structure of the functions being approximated. Several research papers have tried to debunk the black-box nature of ANNs, since it limits the potential use of ANNs in many research areas. This paper is framed in this context and proposes a methodology to determine the individual and collective effects of the input variables on the outputs for classification problems based on the ANOVA-functional decomposition. The method is applied after the training phase of the ANN and allows researchers to rank the input variables according to their importance in the variance of the ANN output. The computation of the sensitivity indices for product unit neural networks is straightforward as those indices can be calculated analytically by evaluating the integrals in the ANOVA decomposition. Unfortunately, the sensitivity indices associated with ANNs based on sigmoidal basis functions or radial basis functions cannot be calculated analytically. In this paper, the indices for those kinds of ANNs are proposed to be estimated by the (quasi-) Monte Carlo method. [ABSTRACT FROM PUBLISHER]

In this paper, we present a compact memristor model for bipolar neuromorphic ReRAM devices. The proposed model focuses on the high level description of the device, and it reproduces some of the most important characteristics (i.e. conductance, energy dissipation) without needing a detailed electrical simulation. Its functionality is shown by using it to model the behavior of three different ReRAM devices that were fabricated and measured at the CNR-IMM, MDM Laboratory. The parameters extraction procedure is also discussed. The obtained results clearly show that the model proposed in this paper is able to capture the influence of the programming pulse parameters (i.e. pulse duration and height) changes. This is accomplished by the introduction of a parameter, which is related to the specific device technology. [ABSTRACT FROM AUTHOR]

This paper presents an iterative decimation approach to significantly alleviate the computational burden of centralized controllers applying real-time recursive system identification algorithms in multi-rail power converters. The proposed approach uses an adaptive update rate as opposed to the fixed update rate used in conventional adaptive filters. Also, the step size/forgetting factors vary at different iteration stages. As a result, a reduced computational burden and faster model update can be achieved. Besides, recursive algorithms, such as Recursive Least Square (RLS), Fast Affine Projection (FAP) and Kalman Filter (KF), contain two important updates per iteration cycle; Covariance Matrix Approximation (CMA) update and Gradient Vector (GV) update. Usually, the CMA update requires the greater computational effort than the GV update. Therefore, in circumstances where the sampled data in the regressor does not experience significant fluctuations, re-using the CMA, calculated from the last iteration cycle for the current update can result in computational cost savings for real-time system identification. In this paper, both iteration rate adjustment and CMA re-cycling are combined and applied to simultaneously identify the power converter models in a three-rail power conversion architecture. [ABSTRACT FROM AUTHOR]

SRAM based computing-in-memory (SRAM-CIM) techniques have been widely studied for neural networks (NNs) to solve the “Von Neumann bottleneck”. However, as the scale of the NN model increasingly expands, the weight cannot be fully stored on-chip owing to the big device size (limited capacity) of SRAM. In this case, the NN weight data have to be frequently loaded from external memories, such as DRAM and Flash memory, which results in high energy consumption and low efficiency. In this paper, we propose a hybrid-device computing-in-memory (HD-CIM) architecture based on SRAM and MRAM (magnetic random-access memory). In our HD-CIM, the NN weight data are stored in on-chip MRAM and are loaded into SRAM-CIM core, significantly reducing energy and latency. Besides, in order to improve the data transfer efficiency between MRAM and SRAM, a high-speed pipelined MRAM readout structure is proposed to reduce the BL charging time. Our results show that the NN weight data loading energy in our design is only 0.242 pJ/bit, which is 289 $\times $ less in comparison with that from off-chip DRAM. Moreover, the energy breakdown and efficiency are analyzed based on different NN models, such as VGG19, ResNet18 and MobileNetV1. Our design can improve $\mathbf {58\times \,\,to\,\,124\times }$ energy efficiency. [ABSTRACT FROM AUTHOR]

When deploying deep neural networks (DNNs), edge devices training is practical to improve model adaptivity for various user-specific scenarios while avoiding privacy disclosure. However, the training computation is intolerable for edge devices. It inspires sparse DNN training (SDT) into the limelight, which reduces training computation by dynamic weight pruning. Generally, SDT has two strategies based on the pruning granularity: the structured or the unstructured. Unfortunately, both of them suffer from limited training efficiency due to the gap between pruning granularity and hardware implementation. The former is hardware-friendly but has a low pruning ratio, indicating limited computation reduction. The latter has a high pruning ratio, but the unbalanced workload decreases utilization and irregular sparsity distribution causes considerable sparsity processing overhead. This paper proposes a software-hardware co- design to bridge the gap for improving the efficiency of SDT. On the algorithm side, a sub-structured pruning method, achieved with hybrid shape-wise and line-wise pruning, generates a high sparsity ratio and keeps the hardware-friendly property. On the hardware side, a sub-structured weight processing unit (SWPU) effectively handles the hybrid sparsity with three techniques. First, SWPU dynamically reorders the computation sequence with hamming-distance-based clustering, balancing the irregular workload. Second, SWPU performs runtime scheduling by exploiting the feature of sub-structured sparse convolution through a detect-before-load controller, which skips redundant memory access and sparsity processing. Third, SWPU performs sparse convolution by compressing operands with spatial disconnect log-based routing and recovers their location with bi-directional switching, avoiding the power-consumed routing logic. Synthesized with 28nm CMOS technology, SWPU can enable 0.56V-to-1.0V supply voltage with a maximum frequency of 675 MHz. It achieves a 50.1% higher pruning ratio than structured pruning and $1.53\times $ higher energy efficiency than unstructured pruning. The peak energy efficiency of SWPU is 126.04TFLOPS/W, outperforming the state-of-the-art training processor by $1.67\times $. When training a ResNet-18 model, SWPU reduces $3.72\times $ energy and offers $4.69\times $ speedup than previous sparse training processors. [ABSTRACT FROM AUTHOR]

Edge device deep neural network (DNN) training is practical to improve model adaptivity for unfamiliar datasets while avoiding privacy disclosure and huge communication cost. Nevertheless, apart from feed-forward (FF) as inference, DNN training still requires back-propagation (BP) and weight gradient (WG), introducing power-consuming floating-point computing requirements, hardware underutilization, and energy bottleneck from excessive memory access. This paper proposes a DNN training processor named PL-NPU to solve the above challenges with three innovations. First, a posit-based logarithm-domain processing element (PE) adapts to various training data requirements with a low bit-width format and reduces energy by transferring complicated arithmetics into simple logarithm domain operation. Second, a reconfigurable inter-intra-channel-reuse dataflow dynamically adjusts the PE mapping with a regrouping omega network to improve the operands reuse for higher hardware utilization. Third, a pointed-stake-shaped codec unit adaptively compresses small values to variable-length data format while compressing large values to fixed-length 8b posit format, reducing the memory access for breaking the training energy bottleneck. Simulated with 28nm CMOS technology, the proposed PL-NPU achieves a maximum frequency of 1040MHz with 343mW and 5.28mm $\mathbf {^{2}}$. The peak energy efficiency is 3.87TFLOPS/W for 0.6V at 60MHz. Compared with the state-of-the-art training processor, PL-NPU reaches $3.75\times $ higher energy efficiency and offers $1.68\times $ speedup when training ResNet18. [ABSTRACT FROM AUTHOR]

In this paper, we consider nonlinear stochastic systems and intersect ideas from nonlinear control theory and numerical analysis. In particular, we use the idea of relative degree. This concept guarantees smoothness properties of the output and this, in turn, allows one to establish properties that are unique to the control-theoretic perspective. The contributions of the current paper are threefold. Firstly, we define different error measures that extend the ideas of local and global approximation errors for nonlinear stochastic systems. Secondly, we demonstrate that the concept of relative degree plays a key role in obtaining higher order of accuracy for integration procedures compared to Euler-Maruyama integration. We show that a particular state-space model, named STTS model, has an improved order of accuracy when compared to an Euler-Maruyama approximation, at no significant extra computational cost. Thirdly, we show that a further approximation to the STTS model, named MSTTS model, while retaining the order of local errors, has explicit sampling zero dynamics, associated with the noise processes, that have no continuous-time counterpart. The extra zero dynamics are shown to be a function of the Euler-Frobenius polynomials. To the best of the authors' knowledge, this is the first reference to sampling zero dynamics for stochastic nonlinear systems. [ABSTRACT FROM AUTHOR]

This paper presents a novel analysis of nonlinear circuits useful for the computation of the reactive power from only terminal measurements. The method computes the reactive power of unknown circuits based on the physical understanding of Maxwell equations. The proposed method is able to identify instantaneous quantities (circuit parameters, powers, etc.) for two physical equivalent models: series and parallel circuits. The parallel equivalent model shows advantages for multi-subsystems. Several numerical examples are provided for validation of the proposed method and to illustrate step-by-step the calculation details. To be compatible with digital instrumentation, a time discrete formulation is used for all calculations. All examples are simulated in MATLAB and the EMTP (Electro-Magnetic Transients Program). [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNN) have been proved to be an effective method in the field of artificial intelligence (AI), and large-scale deploying CNN to embedded devices, no doubt, will greatly promote the development and application of AI into the practical industry. However, mainly due to the space-time complexity of CNN, computing power, memory bandwidth and flexibility are performance bottlenecks. In this paper, a framework containing model compression and hardware acceleration is proposed to solve the above problems. This framework consists of a mixed pruning method, data storage optimization for efficient memory utilization and an accelerator for mapping CNN on field programmable gate array (FPGA). The mixed pruning method is used to compress the model, and data bit-width is reduced to 8-bit by data quantization. Accelerator based on FPGA makes it flexible, configurable and efficient for CNN implementation. The model compression is evaluated on NVIDIA RTX2080Ti, and the results illustrate that the VGG16 is compressed by $30\times $ and the fully convolutional network (FCN) is compressed by $11\times $ within 1% accuracy loss. The compressed model is deployed and accelerated on ZCU102, which is up to $1.7\times $ and $24.5\times $ better in energy efficiency compared with RTX2080Ti and Intel i7 7700. [ABSTRACT FROM AUTHOR]

This paper presents an advanced offline scheduling scheme to improve the accelerator efficiency, especially for the highly-pruned convolutional neural networks (HP-CNNs). Based on the existing outlier-aware accelerator design, we demonstrate the efficiency drop of HP-CNN processing for the first time, and element-wise channel merging is proposed to make a dense processing sequence even for the highly-pruned model. The dedicated hardware architecture is also presented to process the merged channels with the minimum hardware-level overheads, improving the energy efficiency for handling HP-CNNs by preserving the hardware utilization. We further investigate the optimal size of accumulator and multiplexer, in addition to the number of merged channels, exploiting the attractive energy-performance trade-offs. As a result, unlike the practical HP-CNNs for the on-device solutions, the proposed method enhances the overall efficiency by up to 33% compared to the state-of-the-art schemes. [ABSTRACT FROM AUTHOR]

Fully homomorphic encryption (FHE) allows arbitrary computation on encrypted data and thus has potential in privacy-preserving computing. However, efficiency is still the bottleneck. In this paper we present an area-efficient and highly unified reconfigurable multi-core architecture (named ReMCA) for full Residue Number System (RNS) variant of Fan-Vercauteren variant of Brakerski’s scheme (RNS-BFV), which employs a variable number of reconfigurable processing elements (PEs) and RNS channels. The PE unit can be flexibly configured as NTT, INTT or modular multiplier, thereby avoiding the need of other extra computational units. To reduce the computational complexity, ReMCA merges the pre/post-processing into NTT/INTT and unifies the read/write structure of NTT and INTT. Also, a conflict-free memory access pattern that doesn’t need separate bit-reversal operation is proposed to optimize the memory access. Furthermore, targeting different computational requirements, a unified hardware architecture mapping model and data memory organization model are introduced, and all the computing units that RNS-BFV involved are optimized and mapped on ReMCA. ReMCA is evaluated on a Xilinx Virtex-7 FPGA platform. Running at 250MHz, it can perform 2260 homomorphic multiplication per second. When normalized to the same parameter set, the throughput and Area-Time-Products (ATPs) of ReMCA achieve $1.45\times \sim 5.51\times $ and $1.58\times \sim 5.12\times $ improvements. [ABSTRACT FROM AUTHOR]

Learning under concept drift is a novel and promising research area aiming at designing learning algorithms able to deal with nonstationary data-generating processes. In this research field, most of the literature focuses on learning nonstationary probabilistic frameworks, while some extensions about learning graphs and signals under concept drift exist. For the first time in the literature, this paper addresses the problem of learning discrete-time Markov chains (DTMCs) under concept drift. More specifically, following a hybrid active/passive approach, this paper introduces both a family of change-detection mechanisms (CDMs), differing in the required assumptions and performance, for detecting changes in DTMCs and an adaptive learning algorithm able to deal with DTMCs under concept drift. The effectiveness of both the proposed CDMs and the adaptive learning algorithm has been extensively tested on synthetically generated experiments and real data sets. [ABSTRACT FROM AUTHOR]

Low-cost accurate digital realization of the spiking neural networks is crucial to investigate the behaviors of human brain performance. This paper presents a multiplierless burst-mode Fitz–Hugh Nagumo (MBM-FHN) model, which is used for generating the burst-mode of the FHN neuron model. Using extra parameters (time-varying function) in neuron equations and estimating it by linear function, efficient low-cost and high-speed implementation is achieved. The simulation results show that the MBM-FHN model generates the similar bursting patterns of the original FHN neuron. The comparison shows that the MBM-FHN model has a better performance and best cost reduction related to the original neuron model especially in overall saving and speed-up parameters. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) are predominantly used for several challenging computer vision tasks achieving state-of-the-art performance. However, CNNs are complex models that require the use of powerful hardware, both for training and deploying them. To this end, a quantization-based pooling method is proposed in this paper. The proposed method is inspired from the bag-of-features model and can be used for learning more lightweight deep neural networks. Trainable radial basis function neurons are used to quantize the activations of the final convolutional layer, reducing the number of parameters in the network and allowing for natively classifying images of various sizes. The proposed method employs differentiable quantization and aggregation layers leading to an end-to-end trainable CNN architecture. Furthermore, a fast linear variant of the proposed method is introduced and discussed, providing new insight for understanding convolutional neural architectures. The ability of the proposed method to reduce the size of CNNs and increase the performance over other competitive methods is demonstrated using seven data sets and three different learning tasks (classification, regression, and retrieval). [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) are widely employed in modern computer vision algorithms, where the input image is convolved iteratively by many filters to extract the knowledge behind it. However, while the depth of convolutional layers gets deeper and deeper in recent years, the enormous computational complexity makes it difficult to be deployed on embedded systems with limited hardware resources. In this paper, inspired by rate-distortion optimization in image and video coding, we propose a computation-performance optimization (CPO) method to remove the redundant convolution filters in a CNN with performance constraints. To prove the effectiveness of the proposed method, CPO is applied to the networks for image super-resolution and image classification. Under almost the same PSNR drop and accuracy drop for performance evaluation in these two tasks, we can achieve the best parameter and computation reduction when compared with previous works. [ABSTRACT FROM AUTHOR]

The technology for oxide resistive memories offers nowadays a number of different implementations and solutions. As a consequence, many models have been presented so far, generally technology-specific, representing a problem for the memory design standardization. Moreover, the computational effort can be reduced using memristive models with lower complexity but that grant at the same time good speed and numerical stability. In this paper, the authors try to find a compromise between accuracy and computational costs, proposing a simulation procedure based on a linear approximation of the memristive device model. The description, formally derived from the traditional linear model, has been implemented in Verilog-A and, thanks to the analysis of practical design cases on a reference architecture, has been demonstrated to be useful in highlighting critical parasitic effects related to the interaction of the memristive cells with the memory circuit periphery. Thanks to the approximated method, a computer simulation time saving up to 40% is achieved when simulating a full cross-bar memory device. The model parameters have been extracted from experimental curves obtained from HfO2 devices implemented in a 250-nm BiCMOS IHP technology. [ABSTRACT FROM AUTHOR]

Active learning (AL) is a machine-learning approach for reducing the data labeling effort. Given a pool of unlabeled samples, it tries to select the most useful ones to label so that a model built from them can achieve the best possible performance. This paper focuses on pool-based sequential AL for regression (ALR). We first propose three essential criteria that an ALR approach should consider in selecting the most useful unlabeled samples: informativeness, representativeness, and diversity, and compare four existing ALR approaches against them. We then propose a new ALR approach using passive sampling, which considers both the representativeness and the diversity in both the initialization and subsequent iterations. Remarkably, this approach can also be integrated with other existing ALR approaches in the literature to further improve the performance. Extensive experiments on 11 University of California, Irvine, Carnegie Mellon University StatLib, and University of Florida Media Core data sets from various domains verified the effectiveness of our proposed ALR approaches. [ABSTRACT FROM AUTHOR]

This paper considers a Markov decision process (MDP) model with safety state constraints, which specify polytopic invariance constraints on the state probability distribution (pd) for all time epochs. Typically, in the MDP framework, safety is addressed indirectly by penalizing failure states through the reward function. However, such an approach does not allow imposing hard constraints on the state pd, which could be an issue for practical applications where the chance of failure must be limited to prescribed bounds. In this paper, we explicitly separate state constraints from the reward function. We provide analysis and synthesis methods to impose generalized safety constraints at all time epochs, unlike current constrained MDP approaches where such constraints can only be imposed on the stationary distributions. We show that, contrary to the unconstrained MDP policies, optimal safe MDP policies depend on the initial state pd. We present novel algorithms for both finite- and infinite-horizon MDPs to synthesize feasible decision-making policies that satisfy safety constraints for all time epochs and ensure that the performance is above a computable lower bound. Linear programming implementations of the proposed algorithms are developed, which are formulated by using the duality theory of convex optimization. A swarm control simulation example is also provided to demonstrate the use of proposed algorithms. [ABSTRACT FROM AUTHOR]

This paper presents a family of methods for the design of adaptive kernels for tree-structured data that exploits the summarization properties of hidden states of hidden Markov models for trees. We introduce a compact and discriminative feature space based on the concept of hidden states multisets and we discuss different approaches to estimate such hidden state encoding. We show how it can be used to build an efficient and general tree kernel based on Jaccard similarity. Furthermore, we derive an unsupervised convolutional generative kernel using a topology induced on the Markov states by a tree topographic mapping. This paper provides an extensive empirical assessment on a variety of structured data learning tasks, comparing the predictive accuracy and computational efficiency of state-of-the-art generative, adaptive, and syntactical tree kernels. The results show that the proposed generative approach has a good tradeoff between computational complexity and predictive performance, in particular when considering the soft matching introduced by the topographic mapping. [ABSTRACT FROM AUTHOR]

Incremental learning with concept drift has often been tackled by ensemble methods, where models built in the past can be retrained to attain new models for the current data. Two design questions need to be addressed in developing ensemble methods for incremental learning with concept drift, i.e., which historical (i.e., previously trained) models should be preserved and how to utilize them. A novel ensemble learning method, namely, Diversity and Transfer-based Ensemble Learning (DTEL), is proposed in this paper. Given newly arrived data, DTEL uses each preserved historical model as an initial model and further trains it with the new data via transfer learning. Furthermore, DTEL preserves a diverse set of historical models, rather than a set of historical models that are merely accurate in terms of classification accuracy. Empirical studies on 15 synthetic data streams and 5 real-world data streams (all with concept drifts) demonstrate that DTEL can handle concept drift more effectively than 4 other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In this paper, we propose a new approach to implementing hierarchical graph neuron (HGN), an architecture for memorizing patterns of generic sensor stimuli, through the use of vector symbolic architectures. The adoption of a vector symbolic representation ensures a single-layer design while retaining the existing performance characteristics of HGN. This approach significantly improves the noise resistance of the HGN architecture, and enables a linear (with respect to the number of stored entries) time search for an arbitrary subpattern. [ABSTRACT FROM PUBLISHER]

This paper describes the learning and control capabilities of a biologically constrained bottom-up model of the mammalian cerebellum. Results are presented from six tasks: 1) eyelid conditioning; 2) pendulum balancing; 3) proportional–integral–derivative control; 4) robot balancing; 5) pattern recognition; and 6) MNIST handwritten digit recognition. These tasks span several paradigms of machine learning, including supervised learning, reinforcement learning, control, and pattern recognition. Results over these six domains indicate that the cerebellar simulation is capable of robustly identifying static input patterns even when randomized across the sensory apparatus. This capability allows the simulated cerebellum to perform several different supervised learning and control tasks. On the other hand, both reinforcement learning and temporal pattern recognition prove problematic due to the delayed nature of error signals and the simulator’s inability to solve the credit assignment problem. These results are consistent with previous findings which hypothesize that in the human brain, the basal ganglia is responsible for reinforcement learning, while the cerebellum handles supervised learning. [ABSTRACT FROM PUBLISHER]

Distributed averaging (also known as average consensus) is an algorithm that builds on neighbor to neighbor interactions with the ultimate goal of convergence to the average of all initial node values or to some value close to that average. We analyze in this paper the performance of distributed averaging algorithms where the information exchanged between neighboring agents is subject to deterministic uniform quantization (i.e., real values sent by nodes to their neighbors are truncated). With such quantization, convergence to the precise average cannot be achieved in general, but the convergence would be to some value close to it, called quantized consensus. Using Lyapunov stability analysis, we characterize the convergence properties of the resulting nonlinear quantized system. We show that in finite time and depending on initial conditions, the algorithm on any static and connected graph will either cause all agents to reach a quantized consensus where the consensus value is the largest quantized value not greater than the average of their initial values, or will lead all variables to oscillate in a small neighborhood around the average. In the latter case, we identify tight bounds for the size of the neighborhood and we further show that the error can be made arbitrarily small by adjusting the algorithm's parameters in a distributed manner. [ABSTRACT FROM AUTHOR]

In this paper, a low complexity moving average nested generalized memory polynomial model (MAN-GMP) is proposed for digital predistortion (DPD) of broadband power amplifiers (PAs). As the signal bandwidth increases drastically, the strong nonlinear distortions, especially those induced by the memory effect, are generated from the highly efficient PAs. To compensate for the strong memory effect, a moving average nested envelope memory polynomial (MAN-EMP) model is derived from an accuracy-enhanced GMP model, which offers reduced complexity while suffering from degraded modeling accuracy. The MAN-GMP model is further proposed to improve the modeling accuracy by connecting several memory branches of the MAN-EMP model in parallel. An iterative algorithm is designed to extract the model coefficients efficiently through only one or two iterations. Experimental measurements are carried out on two sub-7 GHz broadband GaN Doherty PAs with up to 200 MHz bandwidth OFDM signals to benchmark the proposed MAN-GMP model against the GMP, the parallel-LUT-MP-EMP (PLUME), the augmented complexity-reduced GMP (ACR-GMP), the generalized twin-nonlinear two-box (GTNTB), and the enhanced Wiener models. The experimental results show that the MAN-GMP model can effectively compensate for the nonlinear distortion of broadband PAs with significant complexity reduction. [ABSTRACT FROM AUTHOR]

Second order memristors are two terminal devices that present a conductance depending on two orders of variables, namely the geometric parameters and the internal temperature. They have shown to be able to mimic some specific features of neuron synapses, specifically Spike-Timing-Dependent-Plasticity (STDP), and consequently to be good candidates for neuromorphic computing. In particular, memristor crossbar structures appear to be suitable for implementing locally competitive algorithms and for tackling classification problems by exploiting temporal learning techniques. On the other hand, neuromorphic studies and experiments have revealed the existence of different kinds of plasticity and have shown the effect of calcium concentration on synaptic changes. Computational studies have investigated the behavior of spiking networks in the context of supervised, unsupervised, and reinforcement learning. In this paper, we first derive a simplified, almost analytical, model of a second-order memristor, only involving two variables, the mem-conductance, and the temperature, directly attributable to the synaptic efficacy and to the calcium concentration. Then we study in detail the response of a single memristive synapse to the most relevant plasticity models, including cycles of spike pairs, triplets, and quadruplets at different frequencies. Finally, we accurately characterize memristor spiking networks as discrete nonlinear dynamic systems, with mem-conductances as state variables and pre and postsynaptic spikes as inputs and outputs, respectively. The result shows that the model developed in this manuscript can explain and accurately reproduce a significant portion of observed synaptic behaviors, including those not captured by classical spike pair-based STDP models. Furthermore, under such an approach, the global dynamic behavior of memristor networks and the related learning mechanisms can be deeply analyzed by employing advanced nonlinear dynamic techniques. [ABSTRACT FROM AUTHOR]

The dynamics of modern hybrid power systems are characterized by behaviors at different timescales, typically separated by several orders of magnitude. In this paper we apply the envelope-following method to hybrid power systems simulations, in which single-phase and three-phase representations of a power system co-occur. This simulation method is well suited for handling coexisting behaviors occurring at different timescales. Compared to the simulation approaches present in the literature, the envelope-following method does not need to continuously switch between two separate simulation engines, but rather automatically handles instantaneous variations in the dynamics of the system, such as faults or topology changes. Additionally, given the vast penetration of power electronic components in modern electricity networks, employing the envelope-following method allows using a vast array of numerical algorithms that have been developed over the years to simulate electrical and electronic circuits. The proposed approach is validated by means of power systems case studies of increasing complexity. [ABSTRACT FROM AUTHOR]

Solving continuous problems on digital computers gives generally only approximations of the continuous solutions. It is therefore crucial that the error between the continuous solution and the digital approximation can effectively be controlled. This paper investigates the possibility of simulating linear, time-invariant (LTI) systems on Turing machines. It is shown that there exist elementary LTI systems for which an admissible and computable input signal results in a non-computable output signal. For these LTI systems, the paper gives sharp characterizations of input spaces such that the output is guaranteed to be computable. The second part of the paper discusses the computability of the impulse and step response of LTI systems. It is shown that the computability of the step response implies not the computability of the impulse response. Moreover, there exist impulse responses which cannot be computed from the transfer function using the inverse Laplace transform. Finally, the paper gives a stronger version of a classical non-computability result, showing that there exist admissible and computable initial values for the wave equation so that the solution cannot be computed at certain points in space and time. [ABSTRACT FROM AUTHOR]

Quantization is a promising technique to compress the size of Convolutional Neural Network (CNN) models. Recently, various precision-scalable designs have been presented to reduce the computational complexity in CNNs. However, most of them adopt straightforward calculation scheme to implement the CNN, which causes high bandwidth requirement and low hardware utilization efficiency. This paper proposes a new precision-scalable architecture which can fully reduce the computational complexity in CNN inference and meanwhile has a finely simplified calculation scheme. Based on the proposed scheme, a well-optimized multiplier called Compositional Processing Element (C-PE) is devised. Compared with the previous multipliers, the new C-PE requires less area and power. Furthermore, two levels of optimization are introduced to the design to relieve the bandwidth problem and increase the hardware utilization efficiency. Implemented under the TSMC 90nm CMOS technology, the whole design achieves 6-68.1 fps in various precisions on VGG16 benchmark and a 49.8TOPS/W energy efficiency at 500MHz when scaled to 28nm, which is much better than previous precision-scalable ones. [ABSTRACT FROM AUTHOR]

With the increasing popularity of social media and smart devices, the face as one of the key biometrics becomes vital for person identification. Among those face recognition algorithms, video-based face recognition methods could make use of both temporal and spatial information just as humans do to achieve better classification performance. However, they cannot identify individuals when certain key facial areas, such as eyes or nose, are disguised by heavy makeup or rubber/digital masks. To this end, we propose a novel deep spiking neural network architecture in this paper. It takes dynamic facial movements, the facial muscle changes induced by speaking or other activities, as the sole input. An event-driven continuous spike-timing-dependent plasticity learning rule with adaptive thresholding is applied to train the synaptic weights. The experiments on our proposed video-based disguise face database (MakeFace DB) demonstrate that the proposed learning method performs very well, i.e., it achieves from 95% to 100% correct classification rates under various realistic experimental scenarios. [ABSTRACT FROM AUTHOR]

This paper proposes a novel end-to-end learning model, called skip-connected covariance (SCCov) network, for remote sensing scene classification (RSSC). The innovative contribution of this paper is to embed two novel modules into the traditional convolutional neural network (CNN) model, i.e., skip connections and covariance pooling. The advantages of newly developed SCCov are twofold. First, by means of the skip connections, the multi-resolution feature maps produced by the CNN are combined together, which provides important benefits to address the presence of large-scale variance in RSSC data sets. Second, by using covariance pooling, we can fully exploit the second-order information contained in such multi-resolution feature maps. This allows the CNN to achieve more representative feature learning when dealing with RSSC problems. Experimental results, conducted using three large-scale benchmark data sets, demonstrate that our newly proposed SCCov network exhibits very competitive or superior classification performance when compared with the current state-of-the-art RSSC techniques, using a much lower amount of parameters. Specifically, our SCCov only needs 10% of the parameters used by its counterparts. [ABSTRACT FROM AUTHOR]

Robustness as a system property describes the degree to which a system is able to function correctly in the presence of disturbances, i.e., unforeseen or erroneous inputs. In this paper, we introduce a notion of robustness termed input-output dynamical stability for cyber-physical systems (CPS) which merges existing notions of robustness for continuous systems and discrete systems. The notion captures two intuitive aims of robustness: bounded disturbances have bounded effects and the consequences of a sporadic disturbance disappear over time. We present a design methodology for robust CPS which is based on an abstraction and refinement process. We suggest several novel notions of simulation relations to ensure the soundness of the approach. In addition, we show how such simulation relations can be constructed compositionally. The different concepts and results are illustrated throughout the paper with examples. [ABSTRACT FROM AUTHOR]

In complex-valued Hopfield neural networks (CHNNs), the neuron states are complex numbers whose amplitudes are: 1) they can also be described in special orthogonal matrices of order and 2) here, we propose a new Hopfield model, the $O(2)$ -valued Hopfield neural network [ $O(2)$ -HNN], whose neuron states are extended to orthogonal matrices. Its neuron states are embedded in 4-D space, while those of CHNNs are embedded in 2-D space. Computer simulations were conducted to compare the noise tolerance (NT) and storage capacity (SC) of CHNNs, $O(2)$ -HNNs, and rotor Hopfield neural networks. In terms of SC, $O(2)$ -HNNs outperformed the others, while in NT, they outdid CHNNs. [ABSTRACT FROM AUTHOR]