Search

• Explore two neural network-based methods to aggregate noisy sequence labels. • Provide a detailed comparison of different methods: probabilistic graphical models, neural networks and methods that combine graphical and neural network models. • Improve true label prediction performance on three real-world datasets from different domains and tasks. • Present a simple but effective method to learn meaningful annotator embeddings. • Demonstrate our annotator embedding is helpful for studying annotators' reliability and behaviour. Labelled data for training sequence labelling models can be collected from multiple annotators or workers in crowdsourcing. However, these labels could be noisy because of the varying expertise and reliability of annotators. In order to ensure high quality of data, it is crucial to infer the correct labels by aggregating noisy labels. Although label aggregation is a well-studied topic, only a number of studies have investigated how to aggregate sequence labels. Recently, neural network models have attracted research attention for this task. In this paper, we explore two neural network-based methods. The first method combines Hidden Markov Models with networks while also learning distributed representations of annotators (i.e., annotator embedding); the second method combines BiLSTM with autoencoders. The experimental results on three real-world datasets demonstrate the effectiveness of using neural networks for sequence label aggregation. Moreover, our analysis shows that annotators' embeddings not only make our model applicable to real-time applications, but also useful for studying the behaviour of annotators. [ABSTRACT FROM AUTHOR]

This paper presents the neural dynamical network to compute the generalized and restricted singular value decompositions (GSVD/RSVD) in the regularization methods for ill-posed problems. The neural network model is defined by ordinary differential equations (ODE) which can be solved by many state-of-the-art techniques. The main purpose of the paper is to develop two neural network models for finding approximations of the GSVD and the RSVD. The globally asymptotic stability analyses are provided and numerical experiments illustrate our theory and methods. For small scale problems, the estimation can be as accurate as O (10 - 15). For ill-posed problems, the truncation regularization method implementing the GSVD/RSVD algorithms also produces accurate results. [ABSTRACT FROM AUTHOR]

Neural network compression is a widely used technique when deploying the neural network in energy-constrained and computation-constrained devices. To guarantee the compressed neural network is still usable without too much accuracy loss, the equivalence between the compressed networks and the original ones must be verified. However, the current verification approach can only verify the equivalence of the two neural networks in the same structure which is limited and unseen in real-world scenarios. In this paper, we proposed an equivalence verification method named VeriPrune, which can verify the equivalence of the deep neural networks without the structure limitation. In detail, we proposed an innovative virtual node completing method to solve the problem that node pruning causes structure change and invalidates the existing equivalence verification approaches. To demonstrate the feasibility and efficiency of the proposed approach, we conducted experiments on the public dataset with 49 DNNs and 1272 properties. The results show that the 83.9% properties, the 1067 of the total 1272 properties can be verified. The proposed VeriPrune can be further developed as a CASE tool for the industry settings. [Display omitted] [ABSTRACT FROM AUTHOR]

Community question answering (CQA) is a popular platform where users can ask questions or solve the questions proposed by other users. The expert recommendation aims at providing high-quality answers for the newly proposed questions in time, which is the key to a successful CQA. Questions in CQA usually consist of two parts, a subject which describes the main point, and a body which gives the details of the question. In previous studies, researchers usually ignore the differences between the subject and the body and concatenate them as a whole. In this paper, we propose a multi-hop interactive attention based classification network (MIACN) to recommend experts for newly proposed questions. In our model, the subject and the body are seen as two separate parts. A multi-hop attention is used to capture the multiple latent interactions among the two parts. Then, a high-level representation of the question is generated from the interactions. Experiment results on two real-world datasets demonstrate the effectiveness of our model. [ABSTRACT FROM AUTHOR]

In this paper, a novel control scheme is developed to solve an optimal containment control problem of unknown continuous-time multi-agent systems. Different from traditional adaptive dynamic programming (ADP) algorithms, this paper proposes an internal reinforcement ADP algorithm (IR-ADP), in which the internal reinforcement signals are added in order to facilitate the learning process. Then a distributed containment control law is designed for each agent with the internal reinforcement signal. The convergence of this IR-ADP algorithm and the stability of the closed-loop multi-agent system are analyzed theoretically. For the implementation of the optimal controllers, three neural networks (NNs), namely internal reinforcement NNs, critic NNs and actor NNs, are utilized to approximate the internal reinforcement signals, the performance indices and optimal control laws, respectively. Finally, some simulation results are provided to demonstrate the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Artificial neural networks have achieved great success in artificial intelligence. Back propagation is one of the most widely used parameter search techniques for artificial neural networks, and it has been widely used in many deep learning approaches such as convolutional neural networks (CNNs), generator adversarial networks (GANs) and recurrent neural networks (RNNs) to train their fully-connected layers. However, the fully-connected layers of the neural networks with back propagation technique depends on the gradient descent strategy, which has the disadvantage of easily falling into local optima. Meanwhile, many evolutionary computation (EC) techniques are applied to the parameters optimization of neural networks. They take advantage of populations to achieve global search. But their convergence speed is relatively slow since they use their original evolutionary search strategies. In this paper, we propose a self-adaptive gradient descent search algorithm (SaGDSA) to search the parameters for fully-connected neural networks. Different from the existing adaptive gradient descent algorithm, the proposed algorithm does not require the user to have any prior knowledge and manually design the learning rate of different stages to match the different search stages. In addition, four kinds of gradient descent strategies are used to optimize parameters of fully-connected neural networks. All used datasets are collected from the University of California Irvine (UCI) machine learning repository. The experimental results indicate that SaGDSA is better than the comparison algorithms on most of the datasets. By introducing the self-adaptive mechanism, SaGDSA not only has excellent global search capabilities from evolutionary computation but also has local search capabilities from gradient descent. [ABSTRACT FROM AUTHOR]

• A novel supervisory information is adopted to solve reviewer assignment problem. • The supervisory information helps machine to understand field relationships. • Reviewer assignment problem is treated as a sentence pair modeling issue. Assigning appropriate reviewers to a manuscript from a pool of candidate reviewers is an important task in the academic community. Recent researches have focused mainly on text processing methods based on natural language processing, such as topic model, word embedding and so on. However, it is difficult for the computer to understand the research fields of reviewers and manuscripts. In this paper, a novel supervisory information that is expressed as sentence pairs constructed by titles and abstracts is adopted to solve reviewer assignment problem. We propose a sentence pair modeling-based reviewer assignment (SPM-RA) method, which models the relationship of sentence pairs by supervising information. The supervisory information makes the model accurately learn the field features of reviewers and manuscripts. Firstly, we construct the training set by the field relationship between title and abstract. We use TF-IDF sampling to solve the problem of unbalanced data set. Then, we use neural network models, such as the BERT (bidirectional encoder representations from transformers), CNN (convolution neural network), biLSTM (bidirectional long short term memory), or various combinations of them to do modeling sentence pair and learn the field features of the paper through the training set. Finally, we predict the similarity between reviewers and manuscripts by training model. We evaluate SPM-RA on two real datasets and compare its performance to that of seven existing methods. The experimental results show that SPM-RA improves the recommendation precision by at least 18% on the public dataset. [ABSTRACT FROM AUTHOR]

In this paper, we consider a family of neural networks for solving nonlinear complementarity problems (NCP). The neural networks are constructed from the merit functions based on three classes of NCP-functions: the generalized natural residual function and its two symmetrizations. In this paper, we first characterize the stationary points of the induced merit functions. Growth behavior of the complementarity functions is also described, as this will play an important role in describing the level sets of the merit functions. In addition, the stability of the steepest descent-based neural network model for NCP is analyzed. We provide numerical simulations to illustrate the theoretical results, and also compare the proposed neural networks with existing neural networks based on other well-known NCP-functions. Numerical results indicate that the performance of the neural network is better when the parameter p associated with the NCP-function is smaller. The efficiency of the neural networks in solving NCPs is also reported. [ABSTRACT FROM AUTHOR]

Unsupervised dependency parsing is the task of learning a dependency grammar from a set of unannotated sentences. This paper describes a neural network (NN) based probabilistic model, Neural Dependency Model with Valence (NDMV), that combines the dependency parsing with the rich nonlinear featurization of NN approaches. NDMV follows the traditional DMV model and computes the probability of a grammar rule via a feed-forward NN. A strength of this proposed approach is the ability to learn the underlying features of input tokens. In our experiments, this capability leads to gains in both NDMV and its extension version, Lexicalized Neural Dependency Model with Valence (L-NDMV): NDMV achieves better performance on WSJ and datasets of eight additional languages in comparison with previous approaches in the basic setting; L-NDMV achieves a result that is competitive with the current state-of-the-art. 1 1 Part of this paper been presented in a conference: Jiang et al. 2016 [1] and Han et al. 2017 [2]. [ABSTRACT FROM AUTHOR]

Recently, various normalization layers have been proposed to stabilize the training of deep neural networks. Among them, group normalization is a generalization of layer normalization and instance normalization by allowing a degree of freedom in the number of groups it uses. However, to determine the optimal number of groups, trial-and-error-based hyperparameter tuning is required, and such experiments are time-consuming. In this study, we discuss a reasonable method for setting the number of groups. First, we find that the number of groups influences the gradient behavior of the group normalization layer. Based on this observation, we derive the ideal number of groups, which calibrates the gradient scale to facilitate gradient descent optimization. This paper is the first to propose an optimal number of groups that is theoretically grounded, architecture-aware, and can provide a proper value in a layer-wise manner for all layers. The proposed method exhibited improved performance over existing methods in numerous neural network architectures, tasks, and datasets. • We propose a method to determine the number of groups of group normalization. • A theoretical analysis of group normalization with activation function is provided. • The proposed method is validated against various practical problems. [ABSTRACT FROM AUTHOR]

In this paper, Harris' Hawks weight optimization-guided artificial neural learning-based quicker session key coordination for Industrial Internet-of-Things (IIoT) to enhance the security of Critical Energy Infrastructure (CEI) is proposed. Transportation, telecommunications, healthcare, finance, and defense are all being revolutionized by the energy industry's digitization. CEI is widely dispersed, resulting in complex cyber-physical networks that require constant monitoring and quick recovery to avoid cyberattacks. Substantial efforts were made in this regard to tackle the key exchange problem in IIoT devices, the majority of which have depended on traditional approaches. Existing solutions fail to adequately resolve the security and privacy issues that IIoT systems face. This study proposes a Triple Layer Vector-Valued Neural Network (TLVVNN) to cope with the problem. However, research into optimizing the value of neural weights for quicker neural synchronization is rare. In this case, Harris' Hawks is used to optimizing the neural network's weight vector for quicker coordination. The coordinated weight becomes the session key once this process is accomplished. This technique has several advantages, including (1) Generation of session key via mutual neural synchronization over the public channel. (2) It enables Harris' Hawks-based neural weight vector optimization for faster neural synchronization across public channels. (3) Vector inputs and weights are taken into consideration for TLVVNN networks. (4) The internal structure of the TLVVNN is complex by three hidden layers. As a result, the attacker might have a lot of difficulties determining the internal architecture. (5) Several pairs of variable-length session keys are generated by TLVVNN. (6) It prevents impersonation, geometric, brute force, and majority assaults. Tests to validate the performance of the proposed methodology are carried out, and the results show that the proposed methodology outperforms similar approaches already in use. [ABSTRACT FROM AUTHOR]

Fast and accurate structural dynamics analysis is important for structural design and damage assessment. Structural dynamics analysis leveraging machine learning techniques has become a popular research focus in recent years. Although the basic neural network provides an alternative approach for structural dynamics analysis, the lack of physics law inside the neural network limits the model accuracy and fidelity. In this paper, a new family of the energy-conservation neural network is introduced, which respects the physical laws. The neural network is explored from a fundamental single-degree-of-freedom system to a complicated multiple-degrees-of-freedom system. The damping force and external forces are also considered step by step. To improve the parallelization of the algorithm, the derivatives of the structural states are parameterized with the novel energy-conservation neural network instead of specifying the discrete sequence of structural states. The proposed model uses the system energy as the last layer of the neural network and leverages the underlying automatic differentiation graph to incorporate the system energy naturally, which ultimately improves the accuracy and long-term stability of structural dynamics response calculation under an earthquake impact. The trade-off between computation accuracy and speed is discussed. As a case study, a 3-story building earthquake simulation is conducted with realistic earthquake records. [ABSTRACT FROM AUTHOR]

Twin Support Vector Machines (TWSVMs) have emerged as an efficient alternative to Support Vector Machines (SVM) for learning from imbalanced datasets. The TWSVM learns two non-parallel classifying hyperplanes by solving a couple of smaller sized problems. However, it is unsuitable for large datasets, as it involves matrix operations. In this paper, we discuss a Twin Neural Network (Twin NN) architecture for learning from large unbalanced datasets. The objective functions of the networks in the Twin NN are designed to realize the idea of the Twin SVM with non-parallel decision boudaries for the respective classes, while also being able to reduce model complexity. The Twin NN optimizes the feature map, allowing for better discrimination between classes. The paper also discusses an extension of the Twin NN for multiclass datasets. This architecture trains as many neural networks as the number of classes, and has the additional advantage that it does not have any hyper-parameter which requires tuning. Results presented in the paper demonstrate that the Twin NN generalizes well and scales well on large unbalanced datasets. [ABSTRACT FROM AUTHOR]

This paper presents a method to identify an unknown discrete-time nonlinear system, using high-order neural networks and high-order sliding mode algorithms, which may be subject to internal and external disturbances. Based on the information obtained from available system states, a high-order neural network model is proposed to approximate the system dynamics. Neural network weights are trained by means of the unscented Kalman filter and high-order sliding mode observer. A simulation example is included to illustrate effectiveness of the proposed scheme. [ABSTRACT FROM AUTHOR]

Regularization is a very effective algorithm to solve overfitting problem in neural network, which improves the generalization ability of the model. However, their working mechanisms and the impact on the model performance have not been fully explored. In this paper, we study and analyze them using information bottleneck theory and one theory from human brain sensory system. We propose a metric to characterise the encoding length of hidden layers, named as AEntry value. Then, we implement extensive experiments on MNIST and FashionMNIST datasets with several commonly used regularization algorithms, and calculate the corresponding AEntry values. We analyze these results and obtain three conclusions. (1) The introduction of regularization influences the encoding of relative features with prediction task in neural network. The early stopping technique avoids introducing unrelated information with the task into the model by stopping the training process as an appropriate iterations. Laplace, Gaussian and Sparse Response regularizations compress the related representation and improve the performance of neural network by introducing the prior information into the model. In contrast, Dropout, Batch Normalization, and Layer Normalization increase the encoding length of features by adopting redundant representation to improve the performance. (2) The encoding of neural network does not satisfy the data processing inequality of information theory, which is mainly caused by redundant coding of extracted features. (3) The overfitting is caused by introducing irrelative information with the target. These results can give us insight into building more efficient regularization algorithm to improve the performance of neural network model. [ABSTRACT FROM AUTHOR]

Robot learning from demonstration (LfD) enables robots to be fast programmed. This paper presents a novel LfD framework involving a teaching phase, a learning phase and a reproduction phase, and proposes methods in each of these phases to guarantee the overall system performance. An adaptive admittance controller is developed to take into account the unknown human dynamics so that the human tutor can smoothly move the robot around in the teaching phase. The task model in this controller is formulated by the Gaussian mixture regression to extract the human-related motion characteristics. In the learning and reproduction phases, the dynamic movement primitive is employed to model a robotic motion that is generalizable. A neural network-based controller is designed for the robot to track the trajectories generated from the motion model, and a radial basis function neural network is used to compensate for the effect caused by the dynamic environments. Experiments have been performed using a Baxter robot and the results have confirmed the validity of the proposed robot learning framework. [ABSTRACT FROM AUTHOR]

This paper proposes a new model for a theoretical neural network that can be used as a guide for the design of future (quantum or optical) computational technologies. The model utilizes a uniformly connected nodal structure, where the connections are discretely adaptive and the nodes act as simple gatekeepers. The model replicates all known logics used in current electronics, such as AND, OR, XNOR, XOR, NOR, XNOR, NAND and NOT. Additionally, by using recurrent negating connections the model easily creates XOR gates, and adds novel sided and favoured gates. This model also facilitates the creation of ternary to n-ary gates, and it simplifies the creation of a number of majority functions (especially for an odd number of inputs). Also, as a multi-layered neural network, the model allows learning back propagation through the use of its negating connections. Finally, because of its adaptive connections, parts of the network can be used as internal memory. Overall, the model provides backward compatibility to existing CMOS circuitry, while opening up a number of new logics and architectures for neural computing. [ABSTRACT FROM AUTHOR]

• We propose an end-to-end method for Multispectral Photometric Stereo, without any extra information. • For the first time, our MPS-Net takes the initial surface normal into account, which provides a state-of-the-art estimation. • We design a localized convolutional neural network to establish flexible mapping considering the adjacent structural feature. Multispectral Photometric Stereo (MPS) estimates per-pixel surface normals from one single image captured under three colored (red, green and blue) light sources. Unlike traditional Photometric Stereo, MPS can therefore be used in dynamic scenes for single frame reconstruction. However, MPS is challenging due to the tangle of the illumination, surface reflectance and camera response, causing inaccurate estimation of surface normal. Existing approaches rely on either extra depth information or materials calibration strategies, thus limiting its usage in practical applications. In this paper, we propose a Multispectral Photometric Stereo Network (MPS-Net) to solve this under-determined system. The MPS-Net takes the single multispectral image and an initial surface normal estimation obtained from this image itself, and outputs an accurate surface normal map, where no extra depth or materials calibration information is required. We show that the MPS-Net is not constrained to Lambertian surfaces and can be applied to surfaces with complex reflectance. We evaluated the MPS-Net using both synthetic and real objects of various materials. Our experiment results show that the MPS-Net outperforms the state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

In this paper, we propose continuous time neural network and modified continuous time neural networks for solving a multi-linear system with M -tensors. Theoretically, we prove that the presented neural networks are stable in the sense of Lyapunov stability theory. Numerical simulations are presented to show the effectiveness of the proposed neural networks. [ABSTRACT FROM AUTHOR]

This paper focuses on the state estimation issue of neural networks with two additive time-varying delay components. By constructing a suitable augmented Lyapunov–Krasovskii functional, more time delay information is considered. Using the recently developed reciprocally convex inequality, some reciprocally convex combinations can be estimated more closely. The design conditions of state estimators are expressed as linear matrix inequalities (LMIs). The delay-product-type Lyapunov–Krasovskii functional is used to further reduce the conservativeness. With the help of state estimation method, the stability conditions of neural networks with two additive time-varying delay components are also developed. Finally, two numerical examples widely used in the literature show the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

In this paper, a decentralized fault tolerant control (DFTC) scheme is proposed for a class of large-scale nonlinear systems based on self-tuned local feedback gain against partial loss of actuator effectiveness (PLOAE). Consider a large-scale nonlinear system as a set of interconnected subsystems, a decentralized control method is proposed by employing two radial basis function neural networks (RBFNNs) for the fault-free system. Then, the unknown system is identified using RBFNNs. By establishing a decentralized observer, the derived self-tuned local feedback gain is placed before the proposed decentralized controller to guarantee control performance for the subsystem suffering from PLOAE fault. Finally, simulation examples are provided to demonstrate the effectiveness of the present DFTC scheme. The main contributions of this paper are: i) The unknown large-scale nonlinear system can be identified using locally measured states, so the actuator fault can be handled in its local subsystem. It implies that the performance degradation of the faulty subsystem cannot affect the fault-free subsystems. ii) The estimated effectiveness factor is placed before the proposed decentralized scheme. The fault tolerant control structure is simple since it does not need to be redesigned in the case of PLOAE. [ABSTRACT FROM AUTHOR]

This paper addresses the trajectory-tracking control problem of mobile robot systems with nonholonomic constraints, in the presence of time-varying parametric uncertainties and external disturbances. This necessitates an accurate controller to meet as much control objectives, as possible. Therefore, this paper deals with an artificial intelligence control technique to design a robust controller to meet the control objectives. The design of the intelligent controller is based on the optimal control theory, the adaptive neural network system, and the robust control technique. The trajectory-tracking problem is solved using the optimal control methodology. Since the nonholonomic wheeled mobile robot is strongly nonlinear, the neural network system is applied to approximate the nonlinear function in the optimal control law. The robust controller, for his part, is then applied to adaptively estimate an unknown upper bound of the time-varying parametric uncertainties, external disturbances and approximation error of the neural network system. The stability of the closed-loop robot system is proven using the optimal control theory and Lyapunov stability analysis. The results of the simulation studies on three typical nonholonomic mobile robots are provided to demonstrate the effectiveness of the proposed controller. In addition, a comparative study with a recent robust adaptive controller shows that our proposed intelligent controller gives better results, in the sense that the output trajectory converges to the steady state faster with smaller tracking error. [ABSTRACT FROM AUTHOR]

This paper proposes a neural network based adaptive constrained control scheme for a solid oxide fuel cell (SOFC). First, a radial basis function (RBF) neural network is designed for the identification of SOFC dynamic model. The Jacobian information can be obtained through the identified RBF model. Then, a back propagation (BP) neural network based PID controller is designed to tune the parameters that BP neural network has strong self-learning and adaptive capabilities. At same time, in order to solve the control input saturation and fuel utilization problems of SOFC, a dynamic anti-windup compensator is proposed for accommodating the reference. Moreover this paper theoretically proves the stability of the proposed method based on Lyapunov stability analysis. Finally, the simulation results for SOFC are provided to demonstrate the effectiveness of the proposed constrained control approach. [ABSTRACT FROM AUTHOR]

Abstract Although adaptive control with neural networks has been widely studied for robotic systems, the classical adaptive laws have been derived by using the gradient algorithm to minimize the tracking error, and thus their sluggish convergence may lead to performance degradation or even affect the operation safety. In this paper, we propose an adaptive neural control strategy for nonlinear robot manipulators with new adaptive laws. We first reformulate the robotic model by defining a set of auxiliary variables to avoid using the joint acceleration signals. Then an adaptive law with the extracted estimation error being new leakage terms is developed to update the unknown weight of the radial basis function neural network (RBFNN) used to address the unknown dynamics. With this new learning algorithm, both the tracking error and estimation error will exponentially converge to a small set around zero, whose size solely depends on the RBFNN approximation error. Moreover, we incorporate the sliding mode technique into the design of adaptive law and feedback control, such that finite-time convergence can be proved. The robustness of the proposed estimation and control methods is also investigated. Finally, simulation results based on a SCARA robot model are provided to demonstrate the effectiveness of the proposed approaches, and illustrate their superior response over classical adaptive control schemes with σ -modification method. [ABSTRACT FROM AUTHOR]

Abstract In this paper, a novel robust control strategy based on adaptive dynamics programming (ADP) technique is proposed for multi-player nonlinear systems with input disturbances. A pair of robust control policies is constructed by multiplying appropriate coupling gains to the Nash solution of nominal nonlinear nonzero-sum game with predefined cost functions accounting for system uncertain disturbances. Sufficient conditions for the existence of robust control strategy are derived, and it is proved that the robust control strategy can guarantee the multi-player nonlinear systems to be stable in the sense of uniform ultimate boundedness (UUB) with disturbance rejection. The single-network ADP algorithm is employed to solve the coupled Hamilton–Jacobi equations, where only requires to online tune the weights of critic neural networks (NN) for each player. By utilizing Lyapunov theory, the NN weight estimation errors are proved to be uniformly ultimately bounded, while the stability of the closed-loop nonzero-sum game system is also guaranteed. Two numerical experiments are given to demonstrate the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

Abstract Aspect-level sentiment classification, as a fine-grained task in sentiment classification, aiming to extract sentiment polarity from opinions towards a specific aspect word, has been made tremendous improvements in recent years. There are three key factors for aspect-level sentiment classification: contextual semantic information towards aspect words, correlations between aspect words and their context words, and location information of context words with regard to aspect words. In this paper, two models named AE-DLSTMs (Attention-Enabled Double LSTMs) and AELA-DLSTMs (Attention-Enabled and Location-Aware Double LSTMs) are proposed for aspect-level sentiment classification. AE-DLSTMs takes full advantage of the DLSTMs (Double LSTMs) which can capture the contextual semantic information in both forward and backward directions towards aspect words. Meanwhile, a novel attention weights generating method that combines aspect words with their contextual semantic information is designed so that those weights can make better use of the correlations between aspect words and their context words. Besides, we observe that context words with different distances or different directions towards aspect words have different contributions in sentiment polarity. Based on AE-DLSTMs, the location information of context words by assigning different weights is incorporated in AELA-DLSTMs to improve the accuracy. Experiments are conducted on two English datasets and one Chinese dataset. The experimental results have confirmed that our models can make remarkable improvements and outperform all the baseline models in all datasets, improving the accuracy of 1.67 percent to 4.77 percent in different datasets compared with baseline models. 1 1 Our code is open-source and available at https://github.com/rxt2012kc/AELA-DLSTMs. [ABSTRACT FROM AUTHOR]

Abstract Inspired by the concept of memory mechanism and predictive coding from the cognitive neuroscience, this paper presents a deep memory and prediction neural network (DMPNet) for video prediction. Correspondingly, memory and error propagation units are designed in DMPNet to capture the previous spatial-temporal information and compute current predictive error which is forwarded to the prediction unit for correcting the subsequent video prediction. Subsequently, prediction unit takes the information stored in memory unit and predictive error of previous frame as input to predict the next frame. We evaluate our method on two public real-world datasets and demonstrate that the proposed DMPNet outperforms some state-of-the-art methods quantitatively and qualitatively. [ABSTRACT FROM AUTHOR]

Abstract This paper presents a neural network for solving least absolute deviation problems with equality and box constraints. Compared with some existing models, the proposed neural network has fewer state variables and only one-layer structure. The proposed model is proved to be Lyapunov stable and converge to an exact optimal solution of the original problem. Some simulation results show the validity and transient behavior of the proposed neural network. [ABSTRACT FROM AUTHOR]

Abstract Extreme learning machine (ELM) is a new single hidden layer feedback neural network. The weights of the input layer and the biases of neurons in hidden layer are randomly generated; the weights of the output layer can be analytically determined. ELM has been achieved good results for a large number of classification tasks. In this paper, a new extreme learning machine called rough extreme learning machine (RELM) was proposed. RELM uses rough set to divide data into upper approximation set and lower approximation set, and the two approximation sets are utilized to train upper approximation neurons and lower approximation neurons. In addition, an attribute reduction is executed in this algorithm to remove redundant attributes. The experimental results showed, comparing with the comparison algorithms, RELM can get a better accuracy and a simpler neural network structure on most data sets; RELM cannot only maintain the advantages of fast speed, but also effectively cope with the classification task for high-dimensional data. [ABSTRACT FROM AUTHOR]

Abstract In this paper, based on the Lagrange equation, and considering the n th order elastic mode, the dynamics of a flexible beam equation with unknown spatiotemporally varying disturbance are derived in tangent coordinates system using assumed mode method (AMM). Subsequently, a disturbance observer is developed to solve the unknown spatiotemporally varying disturbance. And neural network (NN) control with full-state feedback and output feedback is proposed to approximate the dynamic uncertainty of the flexible beam system and achieve vibration suppression. It is proved by Lyapunov’s stability that the elastic vibration of the flexible beam system can be effectively suppressed. Extensive numerical simulation results verify the performance of the compounded disturbance observer-based adaptive neural network strategy from the perspective of practice. [ABSTRACT FROM AUTHOR]

Highlights • System parameter-induced bifurcation criteria are precisely established for a fractional delayed neural network. • Effects of fractional order on bifurcation are numerically illustrated via prudential computation. • Bifurcation diagrams are elaborately plotted. Abstract The aim of this paper is to analyze the bifurcation for a fractional neural network with self-connection delay involving three neurons. First, by employing the characteristic equation, the bifurcation points of the proposed network are exactly calculated using time delay or the system parameter as a bifurcation parameter, respectively. Second, it is detected that fractional order is conducive to the bifurcation point for such network, and the onset of bifurcation can be markedly deferred or advanced. Third, the bifurcation diagrams are leerily plotted. Finally, to confirm the efficacy of the analytical results, some numerical simulations are rendered. [ABSTRACT FROM AUTHOR]

Abstract Feed-forward neural networks are efficient at solving various types of problems. However, finding efficient training algorithms for feed-forward neural networks is challenging. The dynamic group optimisation (DGO) algorithm is a recently proposed half-swarm half-evolutionary algorithm, which exhibits a rapid convergence rate and good performance in searching and avoiding local optima. In this paper, we propose a new hybrid algorithm, FNNDGO that integrates the DGO algorithm into a feed-forward neural network. DGO plays an optimisation role in training the neural network, by tuning parameters to their optimal values and configuring the structure of feed-forward neural networks. The performance of the proposed algorithm was determined by comparing its performance with those of other training methods in solving two types of problems. The experimental results show that our proposed algorithm exhibits promising performance for solving real-world problems. [ABSTRACT FROM AUTHOR]

This paper focuses on the parameter pattern during the initialization of Extreme Learning Machines (ELMs). According to the algorithm, model performance is highly dependent on the matrix rank of its hidden layer. Previous research has already proved that the sigmoid activation function can transform input data to a full rank hidden matrix with probability 1, which secures the stability of ELM solution. In recent study, we notice that, under full-rank condition, the hidden matrix possibly has very small eigenvalue, which seriously affects the model generalization ability. Our study indicates such a negative impact is caused by the discontinuity of generalized inverse at the boundary of full and waning rank. Experiments show that each phase of ELM modeling possibly leads to this rank deficient phenomenon, which harms the test accuracy. [ABSTRACT FROM AUTHOR]

In this paper, we consider a general 2-neuron network model with reaction–diffusion term and time delay. We study the effect of time delay in kinetic terms of reaction–diffusive system. We mainly investigate the effects of time delay and diffusion term on the stability of the neural network model. Later, we present an algorithm to determine the existence of Hopf bifurcation for the delayed system with reaction–diffusion term along with Neumann boundary conditions. We determine the conditions on the delay parameter for the Hopf bifurcation to exist corresponding to the characteristic equation obtained by linearization of system. At the end, we give some numerical examples along with simulation results to show effectiveness of our analytic findings. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of efficiently simulating a believable virtual crowd. Our method is the first one that uses the Neural Network (NN) model to fit behaviors from real crowd data to a crowd simulation. Unlike several rule-based approaches that often result in ‘walking robots’, our model can learn motion rules derived from real data and later simulate human walking motions. Additionally, unlike the existing data-driven crowd simulation methods that have to perform search operations on the bound dataset simultaneously during the simulation, our model directly uses the NN model to generate the proper motion for each crowd member. The proposed method is being tested on various scenarios and compared with state-of-the-art state-action-based methods that are commonly employed in data-driven crowd simulation systems. The results demonstrate a significant increase in speed, as well as better simulation quality. [ABSTRACT FROM AUTHOR]

Biologically–compatible methods are not commonly used for face recognition. Complex computational approaches are preferred and dominate the state of the art. However, we know that the human brain is very efficient at processing faces, without explicitly depending on advanced mathematics. In this paper we focus on evaluating the performance of an expression-invariant face recognition system, which is based on the most widely-accepted biological model of stereo vision: the Disparity Energy Model (DEM), which has been shown to deliver precise but inaccurate results. We show that the DEM can provide 3D disparity maps which are suitable for both identity recognition and verification, even coping with a wide range of facial expressions. We test disparity information, both alone and in combination with luminance data, achieving state-of-the-art results. We also compare DEM results with those obtained by precise and accurate laser range maps, concluding that the differences in performance are very small. [ABSTRACT FROM AUTHOR]

This paper firstly treats of the anti-synchronization of a class of memristor-based neural networks with mixed time delays and reaction–diffusion terms. An adaptive controller is designed such that the response system can be synchronized with the drive system by utilizing Lyapunov stability theory. Furthermore, the synchronization criteria derived in this paper are associated with the time switching. In addition, the H ∞ anti-synchronization of memristive neural networks has been studied as well. Therefore, the results obtained in this paper are more general and extend the previous work. Finally, two numerical examples and their simulations are given to show the effectiveness of the obtained results. [ABSTRACT FROM AUTHOR]

This paper deals with the robust H ∞ control problem for a class of uncertain neural networks with discrete interval and distributed time-varying delays. The main purpose of this paper is to estimate robust asymptotic stability of the given neural network with H ∞ performance analysis γ . By constructing novel Lyapunov–Krasovskii functionals with triple integral terms, several new less conservative delay-dependent stability conditions for H ∞ control are obtained in terms of linear matrix inequalities. Numerical examples are given to illustrate the effectiveness of the proposed theoretical results. The method given in this paper shows less conservative results when comparing with some existing methods. [ABSTRACT FROM AUTHOR]

This paper presents a recurrent neural network to support vector machine (SVM) learning in pattern classification arising widespread applications in a variety of setting. The SVM learning problem in classification is first converted into an equivalent quadratic programming (QP) formulation, and then a recurrent neural network for SVM learning is proposed. The proposed neural network is guaranteed to obtain the optimal solution of support vector classification. It is also shown that the proposed neural network model is stable in the sense of Lyapunov and it is globally convergent to an exact optimal solution of the QP problem. Several illustrative examples are provided to show the feasibility and the efficiency of the proposed method in this paper. [ABSTRACT FROM AUTHOR]

In this paper we propose a novel Computational Attention Models (CAM) that fuses bottom-up, top-down and salient motion visual cues to compute visual salience in surveillance videos. When dealing with a number of visual features/cues in a system, it is always challenging to combine or fuse them. As there is no commonly agreed natural way of combining different conspicuity maps obtained from different features: face and motion for example, the challenge is thus to find the right mix of visual cues to get a salience map that is the closest to a corresponding gaze map? In the literature many CAMs have used fixed weights for combining different visual cues. This is computationally attractive but is a very crude way of combining the different cues. Furthermore, the weights are typically set in an ad hoc fashion. Therefore in this paper, we propose a machine learning approach, using an Artificial Neural Network (ANN) to estimate these weights. The ANN is trained using gaze maps, obtained by eye tracking in psycho-physical experiments. These weights are then used to combine the conspicuities of the different visual cues in our CAM, which is later applied to surveillance videos. The proposed model is designed in a way to consider important visual cues typically present in surveillance videos, and to combine their conspicuities via ANN. The obtained results are encouraging and show a clear improvement over state-of-the-art CAMs. [ABSTRACT FROM AUTHOR]

Creation of a computationally efficient and biologically plausible neuron and neural network models is an exceptionally important occupation, the results of which are much needed for the neuroscience in general or for the neurophysiology in partial. Besides, the applying neural network technological areas are always looking for novel improvements of the methodological approaches, algorithms, computational facilities. In this paper, a new (in some aspects) hybrid integrate-and-fire (HIF) neuron model, based on N -shaped Na + and K + ionic current–voltage characteristics and a double discrete reset mechanism, is proposed. In the results of computational experiments, the spike series from a single spiking to the burst and chaotic burst trains are demonstrated. In the other part of the paper, analogous FitzHugh–Nagumo neuron model as a generalized neural networks system with an expansion to an inclusion of a coupling kernel function has been considered. The mean field approximation of neuronal potentials and recovery currents inside a neuron cluster was used. The biologically more realistic nonlinear N-shaped Na + characteristic and kernel functions (one - or two-dimensional) were applied. A possibility to present the nonlinear integral differential equations with kernel functions under the Fourier transformation by partial differential equations was used that allowed us to overcome the analytical and numerical modeling difficulties. The equivalence of two kinds of solutions was additionally confirmed based on the error evaluation. The approach of the equivalent partial differential equations was successfully employed to solve the system with the heterogeneous synaptic functions. The analytical studies are corroborated by many numerical modeling experiments. [ABSTRACT FROM AUTHOR]

This paper is concerned with finite-time synchronization for a class of delayed neural networks with Cohen-Grossberg type. Different from the existing related results, the time-delayed feedback strategy is utilized to investigate finite-time synchronization of delayed Cohen-Grossberg neural networks. By constructing Lyapunov functions and using differential inequalities, several new and effective criteria are derived to realize local and global synchronization in finite time of the addressed neural networks based on two different time-delayed feedback controllers. Besides, the upper bounds of the settling time of synchronization are estimated. Furthermore, as corollaries, some sufficient conditions are given to achieve finite-time synchronization of delayed cellular neural networks. Finally, some numerical examples are provided to verify the theoretical results established in this paper. [ABSTRACT FROM AUTHOR]

Robot manipulators are playing increasingly significant roles in scientific researches and engineering applications in recent years. Using manipulators to save labors and increase accuracies are becoming common practices in industry. Neural networks, which feature high-speed parallel distributed processing, and can be readily implemented by hardware, have been recognized as a powerful tool for real-time processing and successfully applied widely in various control systems. Particularly, using neural networks for the control of robot manipulators have attracted much attention and various related schemes and methods have been proposed and investigated. In this paper, we make a review of research progress about controlling manipulators by means of neural networks. The problem foundation of manipulator control and the theoretical ideas on using neural network to solve this problem are first analyzed and then the latest progresses on this topic in recent years are described and reviewed in detail. Finally, toward practical applications, some potential directions possibly deserving investigation in controlling manipulators by neural networks are pointed out and discussed. [ABSTRACT FROM AUTHOR]

This paper presents a self-learning interval type-2 fuzzy neural network (SLIT2FNN) control scheme for trajectory tracking problem of a Delta robot. This intelligent control scheme is computationally efficient and can be easily applied to existing equipment. The controller has a parallel structure, which combines an interval type-2 fuzzy neural network (IT2FNN) controller and a traditional proportional-derivative (PD) controller. We use the PD controller to compensate the transient performance, and use the IT2FNN to learn the dynamic characteristics of the system. A novel arrangement of trapezoidal interval type-2 fuzzy membership functions (IT2MF) is proposed, the arrangement enables the adaptation laws to have an analytical form. A learning algorithm based on sliding mode control (SMC) theory is proposed for the parameter training of the IT2FNN system. The control algorithm learns from the feedback error online and tunes the parameters of the IT2FNN, and will become the main source of the control signal after several learning iterations. Unlike model-based control, this control method has no requirement of prior information and constraint conditions from the robot plant. Lyapunov stability method is employed to prove asymptotic stability of the proposed approach. The structure of the SLIT2FNN and the operations in each layer are introduced in detail. The performance and robustness of the proposed controller is demonstrated on the Delta robot trajectory tracking problems in the presence of structured and unstructured uncertainties. Simulation results illustrate that the proposed SLIT2FNN control approach produces higher trajectory tracking accuracy and more robust to uncertainties as compared to its counterparts. [ABSTRACT FROM AUTHOR]

Sequence labeling is the widely used method for CCG supertagging task where a supertag (lexical category) is assigned to each word in an input sentence. In CCG supertagging the major challenging problem is due to the large number of lexical categories. To address this, machine learning and deep learning methods have been used and achieved promising results. However, these models whether use many hand-crafted features case of machine learning methods or use sentence level representation processing a sequence without any correlations between labels in neighborhoods which have great influences on predicting the current label case of deep learning models. More recently, there is a marriage of machine learning and deep learning models. In this paper, we use the combination of Conditional Random Field and Bidirectional Long Short-Term Memory models. So first the model learns sentence representation where we can gain from both past and future input features thanks to Bidirectional Long Short-Term Memory Networks architecture. Afterward, the model uses sentence level tag information thanks to Conditional Random Field model. By combining Bidirectional Long Short-Term Memory and Conditional Random Field (BLSTM-CRF) models, we evaluate our model on in-domain and out-of-domain datasets, and in both cases achieve (or close to) state-of-the-art results on CCG supertagging task. [ABSTRACT FROM AUTHOR]

This paper investigates the general decay synchronization (GDS) of a type of neural networks (NNs) with general neuron activation functions and varying-time delays. By introducing suitable Lyapunov functional and employing useful inequality techniques, some simple and useful sufficient conditions ensuring the GDS of considered NNs are established via designing a novel nonlinear feedback controller. In addition, two examples are presented to show the effectiveness of the established theoretical results. The polynomial synchronization, asymptotical synchronization, and exponential synchronization can be seen the special cases of the GDS. [ABSTRACT FROM AUTHOR]

In this paper, a boundary control scheme is developed for a vibrating flexible string system subject to input deadzone and external disturbance. First, a basic boundary control scheme is proposed to suppress the string’s vibration based on the backstepping method. Subsequently, a radial basis function neural network (RBFNN) is utilized to handle the effect of the input deadzone and a disturbance observer is exploited to track the external disturbance. Under the proposed control, the uniformly ultimately bounded stability of the closed loop system is achieved through rigorous Lyapunov analysis without any discretization or simplification of the dynamics in the time and space. Finally, simulation results are demonstrated for control performance verification. [ABSTRACT FROM AUTHOR]

This paper is concerned with the exponential stability of delayed neural networks. Combined the Legendre polynomials with free-weighting matrices technique, an improved free-matrix-based single integral inequality is given, which includes the general single integral inequality and the free-matrix-based single integral inequality as special cases. Furthermore, a free-matrix-based double integral inequality which improves the existing results is derived. As applications of these novel free-matrix-based integral inequalities, several exponential stability criteria with less conservatism for the delayed neural networks are obtained. The effectiveness of our main results is illustrated by three numerical examples from the literatures. [ABSTRACT FROM AUTHOR]

This paper introduces a new formal test of the significance of neural network inputs. It is simple, accurate, and powerful and is based on a linear relationship between the output of a neural network when all of the input variables are fixed at their mean values—other than the input variable, which is subject to significance testing—and the target values of the network. Simulation results show that as the number of observations increases, the power of the test tends to 1 in all cases, and that the empirical size approaches the nominal size in some cases. The results, based on the ordinary least squares (OLS) estimation of parameters, are very encouraging, but using a heteroscedasticity and autocorrelation consistent covariance matrix and fast double bootstrap improves the speed of the convergence to the nominal size. The test can also be used for nonlinear models with nuisance parameters. [ABSTRACT FROM AUTHOR]

Motion detection plays an important role in most static-camera video surveillance systems, yet video communications over wireless networks can easily suffer from network congestion or unstable bandwidth, especially for embedded applications. A rate control scheme produces variable bit rate video streams to match the available network bandwidth. However, effectively detecting moving objects in a variable bit rate video stream is a considerable challenge. This paper proposes an advanced approach based on a counter-propagation artificial neural network to achieve effective moving-object detection in such conditions. Qualitative and quantitative tests over real-world limited bandwidth networks show that the proposed method substantially outperforms other state-of-the-art methods. [ABSTRACT FROM AUTHOR]