Search

Highlights • Development of a biological neural network model that allows long term simulation of brain activity. • Optimization of the model using multiple techniques that led to a speed-up of X200. • Presentation of alternative simulation frameworks for long term simulations., Simulations of neural networks can be used to study the direct effect of internal or external changes on brain dynamics. However, some changes are not immediate but occur on the timescale of weeks, months, or years. Examples include effects of strokes, surgical tissue removal, or traumatic brain injury but also gradual changes during brain development. Simulating network activity over a long time, even for a small number of nodes, is a computational challenge. Here, we model a coupled network of human brain regions with a modified Wilson-Cowan model representing dynamics for each region and with synaptic plasticity adjusting connection weights within and between regions. Using strategies ranging from different models for plasticity, vectorization and a different differential equation solver setup, we achieved one second runtime for one second biological time.

In the process of Human-robot skill transfer, we require the robot to reproduce the trajectory of teacher and expect that the robot can generalize the learned trajectory. For the trajectory after generalization, we expect that the robot arm can accurately track. However, because the model of the robot can not be accurately obtained, some researchers have proposed using a neural network to approximate the unknown term. The parameters of the traditional RBF neural network are usually selected through the empirical and trial-and-error method, which maybe biased and inefficient. In addition, due to the end-effector of the mechanical arm trajectory will be constantly changing according to the needs of the task, when the neural network of compact set cannot contain the whole input vector, the neural network cannot achieve the ideal approximation effect. In this paper, the broad neural network is used to approximate the unknown terms of the robot. This method can reuse the motion controller that has been learned and complete other motions in the robot operating space without relearning its weight parameters. In this paper, the effectiveness of the proposed method is proved by the ultrasound scanning task.

This paper proposes a hybrid architecture based on neural networks, fuzzy systems, and n-uninorms for solving pattern classification problems, termed as ENFS-Uni0 (short for evolving neuro-fuzzy system based on uni-nullneurons). The model can produce knowledge in an on-line (single-pass) and evolving learning context in a particular form of neuro-fuzzy rules representing the dependencies among input features through IF-THEN type relations. The rules antecedents are thereby realized through uni-nullneurons, which are constructed from n-uninorms, leading to the possibility to express both, AND- and OR-connections (and a mixture of these) among the single antecedent parts of a rule (and thus achieving an advanced interpretability aspect of the rules). The neurons’ evolution is done through an extended version of an autonomous data partition method (ADPA). On-line interpretation of the timely evolution of rules is addressed by (i) a concept for tracking the degree of changes of the rules over data stream samples, which may indicate experts/operators how much dynamics is in the process and may be used as a structural active learning component to request operator’s feedback in the case of significant changes and (ii) a concept for updating feature weights incrementally. These weights express the (possibly changing) impact degrees of features on the classification problem: features with low weights can be seen as unimportant and masked out when showing rules to an expert ( → rule length reduction). The rules’ consequents are represented by certainty vectors and are recursively updated by an indicator-based recursive weighted least squares (I-RWLS) approach (one RWLS estimator per class) where the weights are given through the neuron activation levels in order to gain stable local learning. The model proposed in this paper was successfully compared to related hybrid and evolving approaches in the literature for classifying binary and multi-class patterns. The results obtained by the proposed model show an outperformance of the related works in terms of higher accuracy trend lines over time, while offering a high degree of interpretability through coherent neuro-fuzzy rules to solve the classification problems.

It is well known that weighted sum of node states plays an essential role in function implementation of neural networks. Therefore, this paper proposes a new weighted sum synchronization model for memristive neural networks. Unlike the existing synchronization models of memristive neural networks which control each network node to reach synchronization, the proposed model treats the networks as dynamic entireties by weighted sum of node states and makes the entireties instead of each node reach expected synchronization. In this paper, weighted sum complete synchronization and quasi-synchronization are both investigated by designing feedback controller and aperiodically intermittent controller, respectively. Meanwhile, a flexible control scheme is designed for the proposed model by utilizing some switching parameters and can improve anti-interference ability of control system. By applying Lyapunov method and some differential inequalities, some effective criteria are derived to ensure the synchronizations of memristive neural networks. Moreover, the error level of the quasi-synchronization is given. Finally, numerical simulation examples are used to certify the effectiveness of the derived results.

In this paper, we focus on synchronization issue of coupled multistable memristive neural networks (CMMNNs) with time delay under multiple stable equilibrium states. First, we build delayed CMMNNs consisting of one master subnetwork without controller and N−1 identical slave subnetworks with controllers, and every subnetwork has n nodes. Moreover, this paper investigates multistability of delayed CMMNNs with continuous nonmonotonic piecewise linear activation function (PLAF) owning 2 r + 2 corner points. By using the theorems of differential inclusion and fixed point, sufficient conditions are derived such that master subnetwork of CMMNNs can acquire ( r + 2 ) n exponentially stable equilibrium points, stable periodic orbits or hybrid stable equilibrium states. Then, this paper proposes hybrid multisynchronization of delayed CMMNNs related with various external inputs under multiple stable equilibrium states for the first time. There exist ( r + 2 ) n hybrid multisynchronization manifolds in CMMNNs with different initial conditions and external inputs. Finally, two numerical simulations are given to illustrate the effectiveness of the obtained results.

Twin Support Vector Machines (TWSVMs) have emerged as an efficient alternative to Support Vector Machines (SVM) for learning from imbalanced datasets. The TWSVM learns two non-parallel classifying hyperplanes by solving a couple of smaller sized problems. However, it is unsuitable for large datasets, as it involves matrix operations. In this paper, we discuss a Twin Neural Network (Twin NN) architecture for learning from large unbalanced datasets. The objective functions of the networks in the Twin NN are designed to realize the idea of the Twin SVM with non-parallel decision boudaries for the respective classes, while also being able to reduce model complexity. The Twin NN optimizes the feature map, allowing for better discrimination between classes. The paper also discusses an extension of the Twin NN for multiclass datasets. This architecture trains as many neural networks as the number of classes, and has the additional advantage that it does not have any hyper-parameter which requires tuning. Results presented in the paper demonstrate that the Twin NN generalizes well and scales well on large unbalanced datasets.

Activation function is crucial to the recent successes of deep neural networks. In this paper, we first propose a new activation function, Multiple Parametric Exponential Linear Units (MPELU), aiming to generalize and unify the rectified and exponential linear units. As the generalized form, MPELU shares the advantages of Parametric Rectified Linear Unit (PReLU) and Exponential Linear Unit (ELU), leading to better classification performance and convergence property. In addition, weight initialization is very important to train very deep networks. The existing methods laid a solid foundation for networks using rectified linear units but not for exponential linear units. This paper complements the current theory and extends it to the wider range. Specifically, we put forward a way of initialization, enabling training of very deep networks using exponential linear units. Experiments demonstrate that the proposed initialization not only helps the training process but leads to better generalization performance. Finally, utilizing the proposed activation function and initialization, we present a deep MPELU residual architecture that achieves state-of-the-art performance on the CIFAR-10/100 datasets. The code is available at https://github.com/Coldmooon/Code-for-MPELU .

Pedestrian detection and tracking have become an important field in the computer vision research area. This growing interest, started in the last decades, might be explained by the multitude of potential applications that could use the results of this research field, e.g. robotics, entertainment, surveillance, care for the elderly and disabled, and content-based indexing. In this survey paper, vision-based pedestrian detection systems are analysed based on their field of application, acquisition technology, computer vision techniques and classification strategies. Three main application fields have been individuated and discussed: video surveillance, human-machine interaction and analysis. Due to the large variety of acquisition technologies, this paper discusses both the differences between 2D and 3D vision systems, and indoor and outdoor systems. The authors reserved a dedicated section for the analysis of the Deep Learning methodologies, including the Convolutional Neural Networks in pedestrian detection and tracking, considering their recent exploding adoption for such a kind systems. Finally, focusing on the classification point of view, different Machine Learning techniques have been analysed, basing the discussion on the classification performances on different benchmark datasets. The reported results highlight the importance of testing pedestrian detection systems on different datasets to evaluate the robustness of the computed groups of features used as input to classifiers.

The paper considers a class of nonsmooth neural networks where hard-limiter saturation nonlinearities are used to constrain solutions of a linear system with concentrated and distributed delays to evolve within a closed hypercube of R n . Such networks are termed delayed linear systems in saturated mode (D-LSSMs) and they are a generalization to the delayed case of a relevant class of neural networks previously introduced in the literature. The paper gives a rigorous foundation to the D-LSSM model and then it provides a fundamental result on convergence of solutions toward equilibrium points in the case where there are nonsymmetric cooperative (nonnegative) interconnections between neurons. The result ensures convergence for any finite value of the maximum delay and is physically robust with respect to perturbations of the interconnections. More importantly, it encompasses situations where there exist multiple stable equilibria, thus guaranteeing multistability of cooperative D-LSSMs. From an application viewpoint the delays in combination with the property of multistability make D-LSSMs potentially useful in the fields of associative memories, motion detection and processing of temporal patterns.

Computer simulation, which has been proved to be an effective approach to problem solving, is nowadays widely used in modern science. However, it requires a lot of computing resources, which are difficult for general users to acquire. In this paper, we design a Web based system to implement on-line simulation system for ordinary users. As a useful example, the simulation of one type of collision detection model is presented in this paper. Moreover, the software application of simulation is offered as a service on Web. Meanwhile, the incorporation of general vector machine (GVM, a type of neural network) to intelligently predict the relationship between simulation parameters and computation resources is presented, which could further provide more information for system monitoring and scheduling. The system has demonstrated efficiency and intuitiveness for users of this type of applications.

In this paper, a self-organizing map (SOM) neural network is used to visualize corrective actions of failure modes and effects analysis (FMEA). SOM is a popular unsupervised neural network model that aims to produce a low-dimensional map (typically a two-dimensional map) for visualizing high-dimensional data. With regards to FMEA, it is a popular methodology to identify potential failure modes for a product or a process, to assess the risk associated with those failure modes, also, to identify and carry out corrective actions to address the most serious concerns. Despite the popularity of FMEA in a wide range of industries, two well-known shortcomings are the complexity of the FMEA worksheet and its intricacy of use. To the best of our knowledge, the use of computation techniques for solving the aforementioned shortcomings is limited. The use of SOM in FMEA is new. In this paper, corrective actions in FMEA are described in their severity, occurrence and detect scores. SOM is then used as a visualization aid for FMEA users to see the relationship among corrective actions via a map. Color information from the SOM map is then included to the FMEA worksheet for better visualization. In addition, a Risk Priority Number Interval is used to allow corrective actions to be evaluated and ordered in groups. Such approach provides a quick and easily understandable framework to elucidate important information from a complex FMEA worksheet; therefore facilitating the decision-making tasks by FMEA users. The significance of this study is two-fold, viz., the use of SOM as an effective neural network learning paradigm to facilitate FMEA implementations, and the use of a computational visualization approach to tackle the two well-known shortcomings of FMEA.

SCARA robot is one of the most popularly used robots in industry. The obstacle avoidance feature of multiple SCARA robot collaboration is essential and prominent, which can be used to support multiple robots to accomplish not only more sophisticated tasks but also more efficient than individual robot. This paper mainly focuses on studying the problem of simultaneous multi-robot coordination and obstacle avoidance. A cooperative kinematic control problem of multiple robot manipulators, collision avoidance is taken into account to be the primary task as an inequality constraint and trajectory planning task is considered to be the secondary objective as to ensure the priority of safety, is described as a quadratic programming (QP) problem. Then, a recurrent neural network (RNN) based dynamic controller is designed to solve the formulated QP problem recursively. The convergence of the designed neural network is proved through Lyapunov analysis. With three SCARA planar robots, the effectiveness of the proposed controller is validated through numerical simulations. As observed in the results, when the minimal distance between robots is less than the setting safety distance, the collision avoidance strategy reacts to impel robots to avoid collision, which achieves the primary objective for obstacle avoidance; otherwise, the robot performs the desired trajectory tracking task.

Knowledge distillation (KD) is widely applied in the training of efficient neural network. A compact model, which is trained to mimic the representation of a cumbersome model for the same task, generally obtains a better performance compared with being trained with the ground truth label. Previous KD-based works mainly focus on two aspects: (1) designing various feature representation for knowledge transfer; (2) introducing different training mechanism such as progressive learning or adversarial learning. In this paper, we revisit the standard KD and observe that training with teacher’s logits might suffer from incorrect and uncertain supervision. To tackle these problems, we propose two novel approaches to deal with incorrect logits and uncertain logits respectively, which are called Logits Adjustment (LA) and Dynamic Temperature Distillation (DTD). To be specific, LA rectifies the incorrect logits according to ground truth label and certain rules. While DTD treats the temperature of KD as a dynamic sample wise parameter rather than a static and global hyper-parameter, which actually notes the uncertainty for each sample’s logits. With iteratively updating the sample wise temperature, the student model could pay more attention on the samples that confuse the teacher model. Experiments on CIFAR-10/100, CINIC-10 and Tiny ImageNet verify that the proposed methods yield encouraging improvement compared with the standard KD. Furthermore, considering the simple implementations, LA and DTD can be easily attached to many KD-based frameworks and bring improvements without extra cost of training time and computing resources.

Quantized neural networks (NN) with reduced bit precision are practical solutions to minimize computational and memory resource requirements and play a vital role in machine learning. However, it is still challenging to avoid significant accuracy degradation due to numerical approximation and lower redundancy. In this paper, a novel robustness-aware 2-bit quantization scheme (RAQ) is proposed for NN, based on binary NN and generative adversarial networks (GAN), which improve performance by enriching binary NN information, extracting the structural information and considering the robustness of the quantized NN. Specifically, using a shift-add operation to replace the multiply-accumulate in the quantization process can speed the NN. A structural loss is proposed to represent the difference between the original NN and quantized NN, such that the structural information of data is preserved after quantization. The structural information learned from NN plays an important role in improving the performance and allows for further fine-tuning of the quantized NN by applying the Lipschitz constraint to the structural loss. For the first time, we consider the robustness of the quantized NN and propose a non-sensitive perturbation loss function by introducing an extraneous term of the spectral norm. The experiments were conducted on CIFAR-10, SVHN and ImageNet datasets with popular NN (such as MobileNetV2, ResNet20, etc.). Extensive experiments show that our new 2-bit quantization scheme is more efficient than the state-of-the-art quantization methods. Our scheme effectively reduced the latency by 2 × and the accuracy decline by 1–4%. Meanwhile, the experimental results also demonstrate that the RAQ is robust with adversarial attacks, we not only eliminate the robustness gap between full-precision and quantized models, but also improve the robustness over full-precision ones by 10%.

At present, multi-oriented text detection methods based on deep neural network have achieved promising performances on various benchmarks. Nevertheless, there are still some difficulties for arbitrary shape text detection, especially for a simple and proper representation of arbitrary shape text instances. In this paper, a pixel-based text detector is proposed to facilitate the representation and prediction of text instances with arbitrary shapes in a simple manner. Firstly, to alleviate the influence of the target vertex sorting and achieve the direct regression of arbitrary shape text instances, the starting-point-independent coordinates regression loss is proposed. Furthermore, to predict more accurate text instances, the text instance accuracy loss is proposed as an assistant task to refine the predicted coordinates under the guidance of IoU. To evaluate the effectiveness of our detector, extensive experiments have been carried on public benchmarks which contain arbitrary shape text instances and multi-oriented text instances. We obtain 84.8% of F-measure on Total-Text benchmark. The results show that our method can reach state-of-the-art performance.

Facing the challenge of distributed computing on processing large-scale data, this paper proposes a consensus-based decentralized training method with communication compression. First, the decentralized training method is designed based on the decentralized topology to reduce the communication burden on the busiest agent and avoid any agent revealing its locally stored data. The convergence of the decentralized training algorithm is then analyzed, which demonstrates that the decentralized trained model can reach the minimal empirical risk on the whole dataset, without the sharing of data samples. Furthermore, model compression combined with the error-compensated method is considered to reduce communication costs during the decentralized training process. At last, the simulation study shows that the proposed decentralized training with error-compensated communication compression is applicable for both IID and non-IID datasets, and exhibits much better performance than the local training method. Besides, the proposed algorithm with an appropriate compression rate shows comparable performance with decentralized training and centralized training, while saving a lot of communication costs.

Spiking Neural Networks (SNNs) are brain-inspired, event-driven machine learning algorithms that have been widely recognized in producing ultra-high-energy-efficient hardware. Among existing SNNs, unsupervised SNNs based on synaptic plasticity, especially Spike-Timing-Dependent Plasticity (STDP), are considered to have great potential in imitating the learning process of the biological brain. Nevertheless, the existing STDP-based SNNs have limitations in constrained learning capability and/or slow learning speed. Most STDP-based SNNs adopted a slow-learning Fully-Connected (FC) architecture and used a sub-optimal vote-based scheme for spike decoding. In this paper, we overcome these limitations with: 1) a design of high-parallelism network architecture, inspired by the Inception module in Artificial Neural Networks (ANNs); 2) use of a Vote-for-All (VFA) decoding layer as a replacement to the standard vote-based spike decoding scheme, to reduce the information loss in spike decoding and, 3) a proposed adaptive repolarization (resetting) mechanism that accelerates SNNs' learning by enhancing spiking activities. Our experimental results on two established benchmark datasets (MNIST/EMNIST) show that our network architecture resulted in superior performance compared to the widely used FC architecture and a more advanced Locally-Connected (LC) architecture, and that our SNN achieved competitive results with state-of-the-art unsupervised SNNs (95.64%/80.11% accuracy on the MNIST/EMNISE dataset) while having superior learning efficiency and robustness against hardware damage. Our SNN achieved great classification accuracy with only hundreds of training iterations, and random destruction of large numbers of synapses or neurons only led to negligible performance degradation., Published at Neurocomputing

In this paper, we propose a novel Attentive Multi-View Deep Subspace Nets (AMVDSN), which deeply explores underlying consistent and view-specific information from multiple views and fuse them by considering each view’s dynamic contribution obtained by attention mechanism. Unlike most multi-view subspace learning methods that they directly reconstruct data points on raw data or only consider consistency or complementarity when learning representation in deep or shallow space, our proposed method seeks to find a joint latent representation that explicitly considers both consensus and view-specific information among multiple views, and then performs subspace clustering on learned joint latent representation. Besides, different views contribute differently to representation learning, we therefore introduce attention mechanism to derive dynamic weight for each view, which performs much better than previous fusion methods in the field of multi-view subspace clustering. The proposed algorithm is intuitive and can be easily optimized just by using Stochastic Gradient Descent (SGD) because of the neural network framework, which also provides strong non-linear characterization capability compared with traditional subspace clustering approaches. The experimental results on seven real-world data sets have demonstrated the effectiveness of our proposed algorithm against some state-of-the-art subspace learning approaches.

The excellent performance of deep neural networks mainly relies on the attributes of the dataset, such as the size, diversity, completeness. However, it is usually difficult to obtain a high-qualified training dataset for some scenarios. Inspired by the human visual cognition process with few sample learning, and strong robustness, we believe the experience knowledge is more powerful than large-scale data. To combine the power of knowledge and data, we propose a knowledge-guided semantic computing network (SCN) in this paper, which is constructed with a primary knowledge-guided semantic tree module and an auxiliary data-driven lightweight neural network module. The semantic tree module can calculate the classification results by a forward computing process rapidly. The lightweight neural network module can aid the semantic tree module for higher classification ability. We also propose a hinge cross-entropy loss function to train the SCN, which enables the SCN to focus on those misclassified training samples and further improve the classification accuracy. The experimental results on MNIST and GTSRB data sets prove that the SCN achieves excellent classification accuracy comparable to the state-of-the-art methods on the original training samples and higher classification accuracy than the state-of-the-art methods on few training samples. What is more, at BIM eps = 0.3 on MNIST and FGSM eps = 0.03 on GTSRB adversarial test samples, the proposed SCN(1/4) and SCN(1/8) obtain over 75% and 14% accuracy improvement than the original CapsNet.

This paper presents CentroidNetV2, a novel hybrid Convolutional Neural Network (CNN) that has been specifically designed to segment and count many small and connected object instances. This complete redesign of the original CentroidNet uses a CNN backbone to regress a field of centroid-voting vectors and border-voting vectors. The segmentation masks of the individual object instances are produced by decoding centroid votes and border votes. A loss function that combines cross-entropy loss and Euclidean-distance loss achieves high quality centroids and borders of object instances. Several backbones and loss functions are tested on three different datasets ranging from precision agriculture to microbiology and pathology. CentroidNetV2 is compared to the state-of-the art networks You Only Look Once Version 3 (YOLOv3) and Mask Recurrent Convolutional Neural Network (MRCNN). On two out of three datasets CentroidNetV2 achieves the highest F1 score and on all three datasets CentroidNetV2 achieves the highest recall. CentroidNetV2 demonstrates the best ability to detect small objects although the best segmentation masks for larger objects are produced by MRCNN. (c) 2020 Elsevier B.V. All rights reserved.

Facial expression recognition is a hot research topic and can be applied in many computer vision fields, such as human–computer interaction, affective computing and so on. In this paper, we propose a novel end-to-end network with attention mechanism for automatic facial expression recognition. The new network architecture consists of four parts, i.e., the feature extraction module, the attention module, the reconstruction module and the classification module. The LBP features extract image texture information and then catch the small movements of the faces, which can improve the network performance. Attention mechanism can make the neural network pay more attention to useful features. We combine LBP features and attention mechanism to enhance the attention model to obtain better results. In addition, we collected and labelled a new facial expression dataset of seven expressions from 35 subjects aged from 20 to 25. For each subject, we captured both RGB images and depth images with a Microsoft Kinect sensor. For each image type, there are 245 image sequences, each of which contains 110 images, resulting in 26,950 images in total. We apply the newly proposed method to our own dataset and four representative expression datasets, i.e., JAFFE, CK+, FER2013 and Oulu-CASIA. The experimental results demonstrate the feasibility and effectiveness of the proposed method.

This paper investigates the synchronization problem of inertial memristive neural networks (IMNNs) with time-varying delays via event-triggered control (ETC) scheme and state feedback controller for the first time. First, two types of state feedback controllers are designed; the first type of controller is added to the transformational first-order system, and the second type of controller is added to the original second-order system. Next, based on each feedback controller, static event-triggered control (SETC) condition and dynamic event-triggered control (DETC) condition are presented to significantly reduce the update times of controller and decrease the computing cost. Then, some sufficient conditions are given such that synchronization of IMNNs with time-varying delays can be achieved under ETC schemes. Finally, a numerical simulation and some data analyses are given to verify the validity of the proposed results.

Fully memristive spiking-neuron learning framework, which uses drift and diffusion memristor models as axon and dendrite respectively, becomes a hot topic recently with the development of memristor devices. Normally, some other devices like resistor or capacitor are still necessary on recent works of fully memristive learning framework. However, theoretically, one neuron needs axon and dendrite only, which makes technique process simpler and learning framework more similar to biologic brain. In this paper, a fully memristive spiking-neuron learning framework is introduced, in which a neuron structure is just built of one drift and one diffusion memristive models. To verify it merits, a feedforward neural network for pattern recognition and a cellular neural network for edge detection are designed. Experiment results show that compared to other memristive neural networks, our framework's the processing speed is much faster and the hardware resource is saved in pattern recognition due to its simple structure. Further due to the dynamic filtering function of diffusion memristor model in our learning framework, its peak signal noise ratio (PSNR) is much higher than traditional algorithms in edge detection., Comment: Accepted by Neurocomputing

The optimal generation of initial connection weight parameters and dynamic updating strategies of connection weights are critical for adjusting the performance of back-propagation (BP) neural networks. This paper presents an adaptive fractional-order BP neural network abbreviated as PEO-FOBP for handwritten digit recognition problems by combining a competitive evolutionary algorithm called population extremal optimization and a fractional-order gradient descent learning mechanism. Population extremal optimization is introduced to optimize a large number of initial connection weight parameters and fractional-order gradient descent learning mechanism is designed to update these connection weight parameters adaptively during the evolutionary process of fractional-order BP neural network. The extensive experimental results for a well-known MNIST handwritten digits dataset have demonstrated that the proposed PEO-FOBP outperforms the original fractional-order BP neural network and the traditional integer-order BP neural network in terms of training and testing accuracies.

It is well known that any kind of time series algorithm requires past information to model the inherent temporal relationship between past and future. This temporal dependency (i.e. number of past samples required for a good prediction) is generally addressed by feeding a number of past instances to the model in an empirical manner. Conventional approaches mostly rely on offline model, making them impractical to be adopted in the online or streaming context. Hence, a novel method of online temporality analysis is proposed in this paper. The estimated temporality is then employed to form an Adaptive Temporal Neural Network (ATNN) with an elastic memory capable of automatically selecting number of past samples to be used. Temporality change or drift can be a common occurrence in data streams. Hence a drift detection mechanism is also proposed. Once such drift is detected, a drift handling mechanism kicks in which utilizes the rate of drift, making our solution truly autonomous. The entire mechanism is termed as LEMON: Learning Elastic Memory Online. LEMON although not a time series model in itself, can work with any predictive models to improve their performance. Synthetic datasets are used here as proof of correct temporality estimation and drift detection whereas real world datasets are employed to demonstrate how LEMON improves the predictive performance and speed of an existing model with the knowledge of temporality and drift. Nanyang Technological University We would like to sincerely thank Energy Research Institute at Nanyang Technological University (ERI@N), Singapore for their continued support. The work was funded by their SMES project.

© 2019 Elsevier B.V. Robot learning from demonstration (LfD) enables robots to be fast programmed. This paper presents a novel LfD framework involving a teaching phase, a learning phase and a reproduction phase, and proposes methods in each of these phases to guarantee the overall system performance. An adaptive admittance controller is developed to take into account the unknown human dynamics so that the human tutor can smoothly move the robot around in the teaching phase. The task model in this controller is formulated by the Gaussian mixture regression to extract the human-related motion characteristics. In the learning and reproduction phases, the dynamic movement primitive is employed to model a robotic motion that is generalizable. A neural network-based controller is designed for the robot to track the trajectories generated from the motion model, and a radial basis function neural network is used to compensate for the effect caused by the dynamic environments. Experiments have been performed using a Baxter robot and the results have confirmed the validity of the proposed robot learning framework.

This paper addresses the problem of composite adaptive learning and tracking control for discrete-time nonlinear uncertain systems in general normal form. To deal with the system’s unstructured nonlinear uncertain dynamics, novel adaptive neural network (NN) learning control strategies are proposed by extending methodologies from the continuous-time deterministic learning theory. Both state-feedback and output-feedback cases are considered. The proposed control schemes are compelling in the sense that (i) they are capable of rendering not only stable tracking control, but also locally-accurate learning/identification of unknown system dynamics during stable closed-loop control; (ii) the learned knowledge can be effectively represented and stored as constant NN models, whose weights are guaranteed to partially converge to ideal/optimal values. Based on this, experience-based controllers are also developed to pursue improved tracking control performance without online adaptation. In particular for the output-feedback control case, a new observer-less adaptive NN learning control scheme is proposed without resorting to high-gain observers, followed by an observer-less output-feedback experience-based controller. Numerical simulations have been conducted to demonstrate the effectiveness of the proposed approaches.

To obtain the superiority property of solving time-varying linear matrix inequalities (LMIs), three novel finite-time convergence zeroing neural network (FTCZNN) models are designed and analyzed in this paper. First, to make the Matlab toolbox calculation processing more conveniently, the matrix vectorization technique is used to transform matrix-valued FTCZNN models into vector-valued FTCZNN models. Then, considering the importance of nonlinear activation functions on the conventional zeroing neural network (ZNN), the sign-bi-power activation function (AF), the improved sign-bi-power AF and the tunable sign-bi-power AF are explored to establish the FTCZNN models. Theoretical analysis shows that the FTCZNN models not only can accelerate the convergence speed, but also can achieve finite-time convergence. Computer numerical results ulteriorly confirm the effectiveness and advantages of the FTCZNN models for finding the solution set of time-varying LMIs.

General fuzzy min-max (GFMM) neural network is a generalization of fuzzy neural networks formed by hyperbox fuzzy sets for classification and clustering problems. Two principle algorithms are deployed to train this type of neural network, i.e., incremental learning and agglomerative learning. This paper presents a comprehensive empirical study of performance influencing factors, advantages, and drawbacks of the general fuzzy min-max neural network on pattern classification problems. The subjects of this study include (1) the impact of maximum hyperbox size, (2) the influence of the similarity threshold and measures on the agglomerative learning algorithm, (3) the effect of data presentation order, (4) comparative performance evaluation of the GFMM with other types of fuzzy min-max neural networks and prevalent machine learning algorithms. The experimental results on benchmark datasets widely used in machine learning showed overall strong and weak points of the GFMM classifier. These outcomes also informed potential research directions for this class of machine learning algorithms in the future., Comment: 18 pages, 7 figures, 12 tables

Expressivity is one of the most significant issues in assessing neural networks. In this paper, we provide a quantitative analysis of the expressivity for the deep neural network (DNN) from its dynamic model, where the Hilbert space is employed to analyze the convergence and criticality. We study the feature mapping of several widely used activation functions obtained by Hermite polynomials, and find sharp declines or even saddle points in the feature space, which stagnate the information transfer in DNNs. We then present a new activation function design based on the Hermite polynomials for better utilization of spatial representation. Moreover, we analyze the information transfer of DNNs, emphasizing the convergence problem caused by the mismatch between input and topological structure. We also study the effects of input perturbations and regularization operators on critical expressivity. Our theoretical analysis reveals that DNNs use spatial domains for information representation and evolve to the edge of chaos as depth increases. In actual training, whether a particular network can ultimately arrive the edge of chaos depends on its ability to overcome convergence and pass information to the required network depth. Finally, we demonstrate the empirical performance of the proposed hypothesis via multivariate time series prediction and image classification examples.

High-dimensional data in many machine learning applications leads to computational and analytical complexities. Feature selection provides an effective way for solving these problems by removing irrelevant and redundant features, thus reducing model complexity and improving accuracy and generalization capability of the model. In this paper, we present a novel teacher-student feature selection (TSFS) method in which a 'teacher' (a deep neural network or a complicated dimension reduction method) is first employed to learn the best representation of data in low dimension. Then a 'student' network (a simple neural network) is used to perform feature selection by minimizing the reconstruction error of low dimensional representation. Although the teacher-student scheme is not new, to the best of our knowledge, it is the first time that this scheme is employed for feature selection. The proposed TSFS can be used for both supervised and unsupervised feature selection. This method is evaluated on different datasets and is compared with state-of-the-art existing feature selection methods. The results show that TSFS performs better in terms of classification and clustering accuracies and reconstruction error. Moreover, experimental evaluations demonstrate a low degree of sensitivity to parameter selection in the proposed method., 28 pages

This paper aims at developing a faster and more accurate solution to the amodal 3D object detection problem for indoor scenarios. The solution is achieved through a novel neural network structure which takes a pair of RGB-D images as input and delivers oriented 3D bounding boxes as the output. Such network, named 3D-SSD, has two components: hierarchical feature fusion and multi-layer prediction. The hierarchical feature fusion combines multi-scale appearance and geometric features learned from RGB-D images, which is later utilized in the multi-layer prediction for object detection. Both the accuracy and the efficiency can be improved by exploiting 2.5D representations in a synergistic way. To specifically address the shape variance of different objects, a set of 3D anchor boxes with varying physical sizes are attached to every location on the prediction layers. While testing, the category scores for 3D anchor boxes are generated with adjusted positions, sizes and orientations, leading to the final detections using non-maximum suppression. Comprehensive experiments have been performed on publicly accessible dataset of SUN RGB-D and NYUV2. The results show the proposed algorithm is the first 3D detector that runs in near real-time on the challenging datasets with competitive performance to the state-of-the-art methods. The 3D-SSD gets 37.1% mAP on the SUN RGB-D dataset at around 5.6 fps, which outperforms the state-of-the-art Deep Sliding Shape by 10.2% mAP and around 109 × faster. For an efficient model setting with a rate of 9.3 fps, 3D-SSD still gets an accuracy of 37% on mAP. Further, experiments also suggest the proposed approach achieves comparable accuracy and is about 477 × faster than the state-of-art method on the NYUv2 dataset even with a smaller input image size.

It is important to improve the classification accuracy and reduce the storage space when probabilistic neural networks are used for pattern classification tasks. Based on a unit hyperspherical crown mapping and adaptive kernel coverage strategy, this paper presents an optimized hyperspherical crown probabilistic neural network(HCPNN). To overcome the separability problem caused by the fusion of heterogeneous samples, we adopt an unconventional unit hyperspherical crown mapping model in the sample space. Theoretical analysis indicates that nonlinear mapping can improve the separability of the original sample set under certain conditions. In addition, to optimize the pattern layer structure of probabilistic neural networks, we adopt an adaptive kernel coverage method for the training sample space to generate initial pattern nodes. The accumulation potential of the sample in each training subclass is used to measure the distribution density of different classes, and an adaptive update mechanism of potential values is established. In each iteration, nodes with high accumulation potential values are searched as pattern nodes from the dense to sparse regions. The precise position of each pattern node and the corresponding kernel width are adjusted by the Expected Maximum algorithm. Experiments show that HCPNN outperforms other algorithms with respect to the classification performance.

In this paper, a hybrid learning approach, which combines the extreme learning machine (ELM) with a genetic algorithm (GA), is proposed. The utilization of this hybrid algorithm enables the creation of heterogeneous single layer neural networks (SLNNs) with better generalization ability than traditional ELM in terms of lower mean square error (MSE) for regression problems or higher accuracy for classification problems. The architecture of this method is not limited to traditional linear neurons, where each input participates equally to the neuron’s activation, but is extended to support higher order neurons which affect the network’s generalization ability. Initially, the proposed heterogeneous hybrid extreme learning machine (He-HyELM) algorithm creates a number of custom created neurons with different structure, which are used for the creation of homogeneous SLNNs. These networks are trained with ELM and an application specific GA evolves them into heterogeneous networks according to a fitness criterion utilizing the uniform crossover operator for the recombination process. After the completion of the evolution process, the network with the best fitness is selected as the most optimal. Experimental results demonstrate that the proposed learning algorithm can get better results than traditional ELM, homogeneous hybrid extreme learning machine (Ho-HyELM) and optimally pruned extreme learning machine (OP-ELM) for homogeneous and heterogeneous SLNNs.

This paper presents a two-stage approach, denoted as CBRDE-LM, to evolve the architecture and weights of feedforward artificial neural networks. In the first stage, a collaborative binary-real differential evolution (CBRDE) is used to optimize simultaneously network architecture and connection weights of an ANN by a specific individual representation and evolutionary scheme, in which the structure is indirectly represented as binary coding and connection weights are directly encoded by real-valued coding. In the second stage, based on the resulting architecture and weights of an ANN, Levenberg-Marquardt (LM) backpropagation algorithm is adopted for fine-tuning ANN weights. The performance of the two-stage approach has been evaluated on several benchmarks. The results demonstrate that the two-stage approach can fast produce compact ANNs with good generalization ability at low computational cost.

Bayesian inference provides a rigorous approach for neural learning with knowledge representation via the posterior distribution that accounts for uncertainty quantification. Markov Chain Monte Carlo (MCMC) methods typically implement Bayesian inference by sampling from the posterior distribution. This not only provides point estimates of the weights, but the ability to propagate and quantify uncertainty in decision making. However, these techniques face challenges in convergence and scalability, particularly in settings with large datasets and neural network architectures. This paper addresses these challenges in two ways. First, parallel tempering MCMC sampling method is used to explore multiple modes of the posterior distribution and implemented in multi-core computing architecture. Second, we make within-chain sampling scheme more efficient by using Langevin gradient information for creating Metropolis–Hastings proposal distributions. We demonstrate the techniques using time series prediction and pattern classification applications. The results show that the method not only improves the computational time, but provides better decision making capabilities when compared to related methods.

Person re-identification (ReID) under occlusions is a challenging problem in video surveillance. Most of existing person ReID methods take advantage of local features to deal with occlusions. However, these methods usually independently extract features from the local regions of an image without considering the relationship among different local regions. In this paper, we propose a novel person ReID method, which learns the spatial dependencies between the local regions and extracts the discriminative feature representation of the pedestrian image based on Long Short-Term Memory (LSTM), dealing with the problem of occlusions. In particular, we propose a novel loss (termed the adaptive nearest neighbor loss) based on the classification uncertainty to effectively reduce intra-class variations while enlarging inter-class differences within the adaptive neighborhood of the sample. The proposed loss enables the deep neural network to adaptively learn discriminative metric embeddings, which significantly improve the generalization capability of recognizing unseen person identities. Extensive comparative evaluations on challenging person ReID datasets demonstrate the significantly improved performance of the proposed method compared with several state-of-the-art methods., 6 pages, 3 figures

Computer Aided Diagnosis (CAD) systems can support physicians in classifying different kinds of breast cancer, liver cancer and blood tumours also revealed by images acquired via Computer Tomography, Magnetic Resonance, and Blood Smear systems. In this regard, this survey focuses on papers dealing with the description of existing CAD frameworks for the classification of the three mentioned diseases, by detailing existing CAD workflows based on the same steps for supporting the diagnosis of these tumours. In detail, after an appropriate acquisition of the images, the fundamental steps carried out by a CAD framework can be identified as image segmentation, feature extraction and classification. In particular, in this work, specific CAD frameworks are considered, where the task of feature extraction is performed by using both traditional handcrafted strategies and Convolutional Neural Networks-based innovative methodologies, whereas the final supervised pattern classification is based on neural/non-neural machine learning methods. The cited methodology is focused on sharing and reviewing an amount of specific works. Then, the performance of three selected case studies are carefully reported, designed with the aim of showing how final outcomes can vary according to different choices in each step of the adopted workflow. More in detail, these case studies concern with breast images acquired by Tomosynthesis and Magnetic Resonance, hepatocellular carcinoma images acquired by Computed Tomography and enhanced by a triphasic protocol with a contrast medium, peripheral blood smear images for cellular blood tumours and are used to compare their performance.

Extreme learning machine (ELM) is a new single hidden layer feedback neural network. The weights of the input layer and the biases of neurons in hidden layer are randomly generated; the weights of the output layer can be analytically determined. ELM has been achieved good results for a large number of classification tasks. In this paper, a new extreme learning machine called rough extreme learning machine (RELM) was proposed. RELM uses rough set to divide data into upper approximation set and lower approximation set, and the two approximation sets are utilized to train upper approximation neurons and lower approximation neurons. In addition, an attribute reduction is executed in this algorithm to remove redundant attributes. The experimental results showed, comparing with the comparison algorithms, RELM can get a better accuracy and a simpler neural network structure on most data sets; RELM cannot only maintain the advantages of fast speed, but also effectively cope with the classification task for high-dimensional data.

© 2018 Elsevier B.V. The loss of coolant accident (LOCA) of a nuclear power plant (NPP) is a severe accident in the nuclear energy industry. Nowadays, neural networks have been trained on nuclear simulation transient datasets to detect LOCA. This paper proposes a constraint-based genetic algorithm (GA) to find optimised 2-hidden layer network architectures for detecting LOCA of a NPP. The GA uses a proposed constraint satisfaction algorithm called random walk heuristic to create an initial population of neural network architectures of high performance. At each generation, the GA population is split into a sub-population of feature subsets and a sub-population of 2-hidden layer architectures to breed offspring from each sub-population independently in order to generate a wide variety of network architectures. During breeding 2-hidden layer architectures, a constraint-based nearest neighbor search algorithm is proposed to find the nearest neighbors of the offspring population generated by mutation. The results showed that for LOCA detection, the GA-optimised network outperformed a random search, an exhaustive search and a RBF kernel support vector regression (SVR) in terms of generalization performance. For the skillcraft dataset of the UCI machine learning repository, the GA-optimised network has a similar performance to the RBF kernel SVR and outperformed the other approaches.

A Deep Boltzmann Machine is a model of a Deep Neural Network formed from multiple layers of neurons with nonlinear activation functions. The structure of a Deep Boltzmann Machine enables it to learn very complex relationships between features and facilitates advanced performance in learning of high-level representation of features, compared to conventional Artificial Neural Networks. Feature selection at the input level of Deep Neural Networks has not been well studied, despite its importance in reducing the input features processed by the deep learning model, which facilitates understanding of the data. This paper proposes a novel algorithm, Deep Feature Selection (Deep-FS), which is capable of removing irrelevant features from large datasets in order to reduce the number of inputs which are modelled during the learning process. The proposed Deep-FS algorithm utilizes a Deep Boltzmann Machine, and uses knowledge which is acquired during training to remove features at the beginning of the learning process. Reducing inputs is important because it prevents the network from learning the associations between the irrelevant features which negatively impact on the acquired knowledge of the network about the overall distribution of the data. The Deep-FS method embeds feature selection in a Restricted Boltzmann Machine which is used for training a Deep Boltzmann Machine. The generative property of the Restricted Boltzmann Machine is used to reconstruct eliminated features and calculate reconstructed errors, in order to evaluate the impact of eliminating features. The performance of the proposed approach was evaluated with experiments conducted using the MNIST, MIR-Flickr, GISETTE, MADELON and PANCAN datasets. The results revealed that the proposed Deep-FS method enables improved feature selection without loss of accuracy on the MIR-Flickr dataset, where Deep-FS reduced the number of input features by removing 775 features without reduction in performance. With regards to the MNIST dataset, Deep-FS reduced the number of input features by more than 45%; it reduced the network error from 0.97% to 0.90%, and also reduced processing and classification time by more than 5.5%. Additionally, when compared to classical feature selection methods, Deep-FS returned higher accuracy. The experimental results on GISETTE, MADELON and PANCAN showed that Deep-FS reduced 81%, 57% and 77% of the number of input features, respectively. Moreover, the proposed feature selection method reduced the classifier training time by 82%, 70% and 85% on GISETTE, MADELON and PANCAN datasets, respectively. Experiments with various datasets, comprising a large number of features and samples, revealed that the proposed Deep-FS algorithm overcomes the main limitations of classical feature selection algorithms. More specifically, most classical methods require, as a prerequisite, a pre-specified number of features to retain, however in Deep-FS this number is identified automatically. Deep-FS performs the feature selection task faster than classical feature selection algorithms which makes it suitable for deep learning tasks. In addition, Deep-FS is suitable for finding features in large and big datasets which are normally stored in data batches for faster and more efficient processing.

In this paper, our main goal is to solve optimal control problem by using reinforcement learning (RL) algorithm for marine surface vessel system with known dynamic. And this algorithm is an optimal control algorithm based on policy iteration (PI), and it can obtain the suitable approximations of cost function and the optimized control policy. There are two neural networks (NNs), where critic NN aims to estimate the cost-to-go and actor NN is utilized to design suitable input controller and minimize the tracking error. A novel tuning method is given for critic NN and actor NN. The stability and convergence are proven by Lyapunov’s direct method. Finally, the numerical simulations are conducted to demonstrate the feasibility and superiority of presented algorithm.

In this paper, we propose a novel graph-based approach for semi-supervised learning problems, which considers an adaptive adjacency of the examples throughout the unsupervised portion of the training. Adjacency of the examples is inferred using the predictions of a neural network model which is first initialized by a supervised pretraining. These predictions are then updated according to a novel unsupervised objective which regularizes another adjacency, now linking the output nodes. Regularizing the adjacency of the output nodes, inferred from the predictions of the network, creates an easier optimization problem and ultimately provides that the predictions of the network turn into the optimal embedding. Ultimately, the proposed framework provides an effective and scalable graph-based solution which is natural to the operational mechanism of deep neural networks. Our results show comparable performance with state-of-the-art generative approaches for semi-supervised learning on an easier-to-train, low-cost framework.

The purpose of this paper is to introduce and investigate a new system of global fractional-order interval implicit projection neural networks. An existence and uniqueness theorem of the equilibrium point for the system of global fractional-order interval implicit projection neural networks is obtained under some suitable assumptions. Moreover, Mittag–Leffler stability for the system of global fractional-order interval implicit projection neural networks is also proved. Finally, two numerical examples are given to illustrate the validity of our results.

Class imbalance presents a major hurdle in the application of classification methods. A commonly taken approach is to learn ensembles of classifiers using rebalanced data. Examples include bootstrap averaging (bagging) combined with either undersampling or oversampling of the minority class examples. However, rebalancing methods entail asymmetric changes to the examples of different classes, which in turn can introduce their own biases. Furthermore, these methods often require specifying the performance measure of interest a priori, i.e., before learning. An alternative is to employ the threshold moving technique, which applies a threshold to the continuous output of a model, offering the possibility to adapt to a performance measure a posteriori, i.e., a plug-in method. Surprisingly, little attention has been paid to this combination of a bagging ensemble and threshold-moving. In this paper, we study this combination and demonstrate its competitiveness. Contrary to the other resampling methods, we preserve the natural class distribution of the data resulting in well-calibrated posterior probabilities. Additionally, we extend the proposed method to handle multiclass data. We validated our method on binary and multiclass benchmark data sets by using both, decision trees and neural networks as base classifiers. We perform analyses that provide insights into the proposed method.

Navigation on unmarked and possible poorly delineated roads where the boundaries between the road and the non-road surfaces are not clearly indicated is a particularly challenging task for autonomous vehicles. The results of this study show that fairly robust navigation strategies can be generated by a robot equipped with a dynamic active-vision based control system represented by an artificial neural network synthesized using evolutionary computation techniques. In the experiments described in this paper, a simulated Pioneer robot is required to visually navigate multiple poorly delineated roads that differ in terms of variations in luminance and/or chrominance between the road and the adjacent non-road areas. Low resolution camera images are processed by a mechanism that continuously adjusts the contribution of each component of a three dimensional colour model (e.g., R, G and B) to the generation of the robot perceptual experience. We show that the best controller can successfully drive a simulated Pioneer robot in environments with colour characteristics never encountered during the design phase, and operate with colour models never used during training. We show that the dynamic differential weighting of the colour components is underpinned by a complex pattern of neural activity that allows the robot to successfully adapt its perceptual system to the colour characteristics of different visual scenes. We also show that the controller can be easily ported onto real hardware, by showing the results of a series of tests with a physical Pioneer robot required to navigate various poorly delineated pedestrian roads.

The integration of symbolic reasoning systems based on logic and connectionist systems based on the functioning of living neurons is a vivid research area in computer science. In the literature, one can find many efforts where different reasoning systems based on different logics are linked to classic artificial neural networks. In this paper, we study the relation between the semantics of reasoning systems based on propositional logic and the connectionist model in the framework of membrane computing, namely, spiking neural P systems. We prove that the fixed point semantics of deductive databases without negation can be implemented in the spiking neural P systems model and such a model can also deal with negation if it is endowed with anti-spikes and annihilation rules.

Presently, there is an increasing interest in the deployment of unmanned surface vehicles (USVs) to support complex ocean operations. In order to carry out these missions in a more efficient way, an intelligent hybrid multi-task allocation and path planning algorithm is required and has been proposed in this paper. In terms of the multi-task allocation, a novel algorithm based upon a self-organising map (SOM) has been designed and developed. The main contribution is that an adaptive artificial repulsive force field has been constructed and integrated into the SOM to achieve collision avoidance capability. The new algorithm is able to fast and effectively generate a sequence for executing multiple tasks in a cluttered maritime environment involving numerous obstacles. After generating an optimised task execution sequence, a path planning algorithm based upon fast marching square (FMS) is utilised to calculate the trajectories. Because of the introduction of a safety parameter, the FMS is able to adaptively adjust the dimensional influence of an obstacle and accordingly generate the paths to ensure the safety of the USV. The algorithms have been verified and evaluated through a number of computer based simulations and has been proven to work effectively in both simulated and practical maritime environments.

Deep neural networks (DNNs) have attracted significant attention for their excellent accuracy especially in areas such as computer vision and artificial intelligence. To enhance their performance, technologies for their hardware acceleration are being studied. FPGA technology is a promising choice for hardware acceleration, given its low power consumption and high flexibility which makes it suitable particularly for embedded systems. However, complex DNN models may need more computing and memory resources than those available in many current FPGAs. This paper presents FP-BNN, a binarized neural network (BNN) for FPGAs, which drastically cuts down the hardware consumption while maintaining acceptable accuracy. We introduce a Resource-Aware Model Analysis (RAMA) method, and remove the bottleneck involving multipliers by bit-level XNOR and shifting operations, and the bottleneck of parameter access by data quantization and optimized on-chip storage. We evaluate the FP-BNN accelerator designs for MNIST multi-layer perceptrons (MLP), Cifar-10 ConvNet, and AlexNet on a Stratix-V FPGA system. An inference performance of Tera opartions per second with acceptable accuracy loss is obtained, which shows improvement in speed and energy efficiency over other computing platforms.

Using neural networks to handle intractability problems and solve complex computation equations is becoming common practices in academia and industry. It has been shown that, although complicated, these problems can be formulated as a set of equations and the key is to find the zeros of them. Zeroing neural networks (ZNN), as a class of neural networks particularly dedicated to find zeros of equations, have played an indispensable role in the online solution of time-varying problem in the past years and many fruitful research outcomes have been reported in the literatures. The aim of this paper is to provide a comprehensive survey of the research on ZNNs, including continuous-time and discrete-time ZNN models for various problems solving as well as their applications in motion planning and control of redundant manipulators, tracking control of chaotic systems, or even populations control in mathematical biosciences. By considering the fact that real-time performance is highly demanded for time-varying problems in practice, stability and convergence analyses of different continuous-time ZNN models are reviewed in detail in a unified way. For the case of discrete-time problems solving, the procedures on how to discretize a continuous-time ZNN model and the techniques on how to obtain an accuracy solution are summarized. Concluding remarks and future directions of ZNN are pointed out and discussed.