Search

This paper proposes a deep learning approach for solving non-smooth convex optimization problems (NCOPs), which have broad applications in computer science, engineering, and physics. Our approach combines neurodynamic optimization with physics-informed neural networks (PINNs) to provide an efficient and accurate solution. We first use neurodynamic optimization to formulate an initial value problem (IVP) that involves a system of ordinary differential equations for the NCOP. We then introduce a modified PINN as an approximate state solution to the IVP. Finally, we develop a dedicated algorithm to train the model to solve the IVP and minimize the NCOP objective simultaneously. Unlike existing numerical integration methods, a key advantage of our approach is that it does not require the computation of a series of intermediate states to produce a prediction of the NCOP. Our experimental results show that this computational feature results in fewer iterations being required to produce more accurate prediction solutions. Furthermore, our approach is effective in finding feasible solutions that satisfy the NCOP constraint. [ABSTRACT FROM AUTHOR]

Artificial neural networks (ANNs) have been widely adopted as general computational tools both in computer science as well as many other engineering fields. Stochastic gradient descent (SGD) and adaptive methods such as Adam are popular as robust optimization algorithms used to train the ANNs. However, the effectiveness of these algorithms is limited because they calculate a search direction based on a first-order gradient. Although higher-order gradient methods such as Newton's method have been proposed, they require the Hessian matrix to be semi-definite, and its inversion incurs a high computational cost. Therefore, in this paper, we propose a variable three-term conjugate gradient (VTTCG) method that approximates the Hessian matrix to enhance search direction and uses a variable step size to achieve improved convergence stability. To evaluate the performance of the VTTCG method, we train different ANNs on benchmark image classification and generation datasets. We also conduct a similar experiment in which a grasp generation and selection convolutional neural network (GGS-CNN) is trained to perform intelligent robotic grasping. After considering a simulated environment, we also test the GGS-CNN with a physical grasping robot. The experimental results show that the performance of the VTTCG method is superior to that of four conventional methods, including SGD, Adam, AMSGrad, and AdaBelief. [ABSTRACT FROM AUTHOR]

Finding the saddle point of a matrix game is a classical problem that arises in various fields, e.g., economics, computer science, and engineering. The standard problem-solving methods consist of formulating the problem as a linear program (LP). However, this approach seems less efficient when many instances need to be solved. In this paper, we propose a Convolutional Neural Network based approach, which is able to predict both the strategy profile (x , y) and the optimal value v of the game. We call this approach Matrix Game-Conventional Neural Network or MG-CNN for short. Thanks to a global pooling technique, MG-CNN can solve matrix games with different shapes. We propose a specialized algorithm to train MG-CNN, which includes both data generation and model training. Our numerical experiments show that MG-CNN outperforms standard LP solvers in terms of computational CPU time and provides a high-quality prediction. [ABSTRACT FROM AUTHOR]

Many real-world applications suffer from the class imbalance problem, in which some classes have significantly fewer examples compared to the other classes. In this paper, we focus on online sequential learning methods, which are considerably more preferable to tackle the large size imbalanced classification problems effectively. For example, weighted online sequential extreme learning machine (WOS-ELM), voting based weighted online sequential extreme learning machine (VWOS-ELM) and weighted online sequential extreme learning machine with kernels (WOS-ELMK), etc. handle the imbalanced learning effectively. One of our recent works class-specific extreme learning machine (CS-ELM) uses class-specific regularization and has been shown to perform better for imbalanced learning. This work proposes a novel online sequential class-specific extreme learning machine (OSCSELM), which is a variant of CS-ELM. OSCSELM supports online learning technique in both chunk-by-chunk and one-by-one learning mode. It targets to handle the class imbalance problem for both small and larger datasets. The proposed work has less computational complexity in contrast with WOS-ELM for imbalanced learning. The proposed method is assessed by utilizing benchmark real-world imbalanced datasets. Experimental results illustrate the effectiveness of the proposed approach as it outperforms the other methods for imbalanced learning. [ABSTRACT FROM AUTHOR]

Abstract: This paper proposes a technique, called Evolving Probabilistic Neural Network (), that presents many interesting features, including incremental learning, evolving architecture, the capacity to learn continually throughout its existence and requiring that each training sample be used only once in the training phase without reprocessing. A series of experiments was performed on data sets in the public domain; the results indicate that is superior or equal to the other incremental neural networks evaluated in this paper. These results also demonstrate the advantage of the small architecture and show that its architecture is more stable than the other incremental neural networks evaluated. thus appears to be a promising alternative for a quick learning system and a fast classifier with a low computational cost. [Copyright &y& Elsevier]

Abstract: The ability to detect social contingencies plays an important role in the social and emotional development of infants. Analyzing this problem from a computational perspective may provide important clues for understanding social development, as well as for the synthesis of social behavior in robots. In this paper, we show that the turn-taking behaviors observed in infants during contingency detection situations are tuned to optimally gather information as to whether a person is responsive to them. We show that simple reinforcement learning mechanisms can explain how infants acquire these efficient contingency detection schemas. The key is to use the reduction of uncertainty (information gain) as a reward signal. The result is an interesting form of learning in which the learner rewards itself for conducting actions that help reduce its own sense of uncertainty. This paper illustrates the possibilities of an emerging area of computer science and engineering that focuses on the computational understanding of human behavior and on its synthesis in robots. We believe that the theory of stochastic optimal control will play a key role providing a formal mathematical foundation for this newly emerging discipline. [Copyright &y& Elsevier]

Abstract: In an earlier paper [Fukushima, K., & Tohyama, K. (2005). Analysis of optic flow: a neural network model. In ICONIP 2005, International Conference on Neural Information Processing (pp. 312–317); Tohyama, K., & Fukushima, K. (2005). Neural network model for extracting 98 optic flow. Neural Networks, 18(5–6), 549–556], we proposed a neural network extracting optic flow based on vector field hypothesis. We started discussion there, however, from the stage where local velocities of visual objects have already been extracted. This paper proposes a new mechanism of extracting local velocity from retinal images, and adds it to the previous network. The network has a hierarchical multilayered architecture. X- and Y-cells of the retina extract spatial and temporal contrast of brightness. The network contains several types of V1 cells, namely, S- C- and V-cells. S- and C-cells extract orientated edges. V-cells extract local velocities, based on the signals from Y- and S-cells. MT cells extract relative velocities between adjoining small visual fields. MST cells add the responses of MT cells to extract a specific optic flow, such as rotation and expansion/contraction. The difference in type of optic flow extracted by MST cells can be created simply by the difference in relative locations of inhibitory to excitatory areas in the receptive fields of MT cells. [Copyright &y& Elsevier]

This paper aims to characterize whether a multi-layer cellular neural network is of deep architecture; namely, when can an n -layer cellular neural network be replaced by an m -layer cellular neural network for m < n yet still preserve the same output phenomena? From a mathematical point of view, such characterization involves investigating whether the topological structure of two (or multiple) layers is conjugate. A decision procedure that addresses the necessary and sufficient condition for the topological conjugacy between two layers in a network is revealed. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, a class of fractional-order neural networks is investigated. First, -exponential stability is introduced as a new type of stability and some effective criteria are derived for such kind of stability of the addressed networks by handling a new fractional-order differential inequality. Based on the results, the existence and -exponential stability of the equilibrium point are considered. Besides, the synchronization of fractional chaotic networks is also proposed. Finally, several examples with numerical simulations are given to show the effectiveness of the obtained results. [Copyright &y& Elsevier]

Abstract: This paper describes a redundant robot arm that is capable of learning to reach for targets in space in a self-organized fashion while avoiding obstacles. Self-generated movement commands that activate correlated visual, spatial and motor information are used to learn forward and inverse kinematic control models while moving in obstacle-free space using the Direction-to-Rotation Transform (DIRECT). Unlike prior DIRECT models, the learning process in this work was realized using an online Fuzzy ARTMAP learning algorithm. The DIRECT-based kinematic controller is fault tolerant and can handle a wide range of perturbations such as joint locking and the use of tools despite not having experienced them during learning. The DIRECT model was extended based on a novel reactive obstacle avoidance direction (DIRECT-ROAD) model to enable redundant robots to avoid obstacles in environments with simple obstacle configurations. However, certain configurations of obstacles in the environment prevented the robot from reaching the target with purely reactive obstacle avoidance. To address this complexity, a self-organized process of mental rehearsals of movements was modeled, inspired by human and animal experiments on reaching, to generate plans for movement execution using DIRECT-ROAD in complex environments. These mental rehearsals or plans are self-generated by using the Fuzzy ARTMAP algorithm to retrieve multiple solutions for reaching each target while accounting for all the obstacles in its environment. The key aspects of the proposed novel controller were illustrated first using simple examples. Experiments were then performed on real robot platforms to demonstrate successful obstacle avoidance during reaching tasks in real-world environments. [Copyright &y& Elsevier]

Abstract: This paper is concerned with the exponential state estimation for Markovian jumping neural networks with time-varying discrete and distributed delays. The parameters of the neural networks are subject to the switching from one mode to another according to a Markov chain. By constructing a novel Lyapunov–Krasovskii functional and developing a new convex combination technique, a new delay-dependent exponential stability condition is proposed, such that for all admissible delay bounds, the resulting estimation error system is mean-square exponentially stable with a prescribed noise attenuation level in the sense. It is also shown that the design of the desired state estimator is achieved by solving a set of linear matrix inequalities (LMIs). The obtained condition implicitly establishes the relations among the maximum delay bounds, noise attenuation level and the exponential decay rate of the estimation error system. Finally, a numerical example is given to show the effectiveness of the proposed result. [Copyright &y& Elsevier]

Abstract: Many hardware-based solutions now exist for the simulation of bio-like neural networks. Less conventional than software-based systems, these types of simulators generally combine digital and analog forms of computation. In this paper we present a mixed hardware–software platform, specifically designed for the simulation of spiking neural networks, using conductance-based models of neurons and synaptic connections with dynamic adaptation rules (Spike-Timing-Dependent Plasticity). The neurons and networks are configurable, and are computed in ‘biological real time’ by which we mean that the difference between simulated time and simulation time is guaranteed lower than 50 μs. After presenting the issues and context involved in the design and use of hardware-based spiking neural networks, we describe the analog neuromimetic integrated circuits which form the core of the platform. We then explain the organization and computation principles of the modules within the platform, and present experimental results which validate the system. Designed as a tool for computational neuroscience, the platform is exploited in collaborative research projects together with neurobiology and computer science partners. [Copyright &y& Elsevier]

Abstract: Data splitting is an important consideration during artificial neural network (ANN) development where hold-out cross-validation is commonly employed to ensure generalization. Even for a moderate sample size, the sampling methodology used for data splitting can have a significant effect on the quality of the subsets used for training, testing and validating an ANN. Poor data splitting can result in inaccurate and highly variable model performance; however, the choice of sampling methodology is rarely given due consideration by ANN modellers. Increased confidence in the sampling is of paramount importance, since the hold-out sampling is generally performed only once during ANN development. This paper considers the variability in the quality of subsets that are obtained using different data splitting approaches. A novel approach to stratified sampling, based on Neyman sampling of the self-organizing map (SOM), is developed, with several guidelines identified for setting the SOM size and sample allocation in order to minimize the bias and variance in the datasets. Using an example ANN function approximation task, the SOM-based approach is evaluated in comparison to random sampling, DUPLEX, systematic stratified sampling, and trial-and-error sampling to minimize the statistical differences between data sets. Of these approaches, DUPLEX is found to provide benchmark performance with good model performance, with no variability. The results show that the SOM-based approach also reliably generates high-quality samples and can therefore be used with greater confidence than other approaches, especially in the case of non-uniform datasets, with the benefit of scalability to perform data splitting on large datasets. [Copyright &y& Elsevier]

Abstract: In this paper, improved criteria for the passivity of neural networks with time delay are proposed. The improvement is achieved by exploiting the finer relationships between the time-varying delay and its size bound which allow the use of more slack variables. It is illustrated via numerical examples that the proposed results are less conservative than those available in the literature. [Copyright &y& Elsevier]

The temporal character of the input is, generally, not taken into account in the neural models. This paper presents an extension of the FasArt model focused on the treatment of temporal signals. FasArt model is proposed as an integration of the characteristic elements of the Fuzzy System Theory in an ART architecture. A duality between the activation concept and membership function is established. FasArt maintains the structure of the Fuzzy ARTMAP architecture, implying a static character since the dynamic response of the input is not considered. The proposed novel model, dynamic FasArt (dFasArt), uses dynamic equations for the processing stages of FasArt: activation, matching and learning. The new formulation of dFasArt includes time as another characteristic of the input. This allows the activation of the units to have a history-dependent character instead of being only a function of the last input value. Therefore, dFasArt model is robust to spurious values and noisy inputs. As experimental work, some cases have been used to check the robustness of dFasArt. A possible application has been proposed for the detection of variations in the system dynamics. [Copyright &y& Elsevier]

Artificial neural networks (ANN) have demonstrated good predictive performance in a wide range of applications. They are, however, not considered sufficient for knowledge representation because of their inability to represent the reasoning process succinctly. This paper proposes a novel methodology Gyan that represents the knowledge of a trained network in the form of restricted first-order predicate rules. The empirical results demonstrate that an equivalent symbolic interpretation in the form of rules with predicates, terms and variables can be derived describing the overall behaviour of the trained ANN with improved comprehensibility while maintaining the accuracy and fidelity of the propositional rules. [Copyright &y& Elsevier]

In this paper a model called symbolic function network (SFN) is introduced; that is based on using elementary functions (for example powers, the exponential function, and the logarithm) as building blocks. The proposed method uses these building blocks to synthesize a function that best fits the training data in a regression framework. The resulting network is of the form of a tree, where adding nodes horizontally means having a summation of elementary functions and adding nodes vertically means concatenating elementary functions. Several new algorithms were proposed to construct the tree based on the concepts of forward greedy search and backward greedy search, together with applying the steepest descent concept. The method is tested on a number of examples and it is shown to exhibit good performance. [Copyright &y& Elsevier]

This paper introduces an effective approach to studying the stability of recurrent neural networks with a time-invariant delay. By employing a new Lyapunov¿Krasovskii functional form based on delay partitioning, novel delay-dependent stability criteria are established to guarantee the global asymptotic stability of static neural networks. These conditions are expressed in the framework of linear matrix inequalities, which can be verified easily by means of standard software. It is shown, by comparing with existing approaches, that the delay-partitioning projection approach can largely reduce the conservatism of the stability results. Finally, two examples are given to show the effectiveness of the theoretical results. [Copyright &y& Elsevier]

Abstract: The Kohonen self-organization map is usually considered as a classification or clustering tool, with only a few applications in time series prediction. In this paper, a particular time series forecasting method based on Kohonen maps is described. This method has been specifically designed for the prediction of long-term trends. The proof of the stability of the method for long-term forecasting is given, as well as illustrations of the utilization of the method both in the scalar and vectorial cases. [Copyright &y& Elsevier]

Classical Computer Science approaches to navigation by autonomous robots continue to make good progress. However, we have only a limited understanding of how navigation is implemented in the neural networks of animals, which still perform very much better in navigational tasks than robots. In this paper we explore the implementation of neural network based navigation in a simple robot. We use a modular navigation system that contains separate representations of visual input and the path integration process. These representations are combined to influence the behavior of a robot. Both representations are encoded within recurrent neuronal networks. The outputs of the representations are vectors of polar values that encode the location of the nearest object, or of a specific place in the environment. The robot manoeuvres in relation to these attended locations, in the context of its egocentric spatial map. During ego-motion towards a goal, the network representation of the goal moves in a counter-movement due to applied motor feedback. The robot''s position is continuously compared against its visual input, and mismatches between the visually perceived goal position and its spatial representation are corrected. [Copyright &y& Elsevier]

In this paper, we address the problem of stable tracking control of a flexible macro–micro manipulator (M3) system. A two-layer neural network is utilized to approximate the nonlinear robot dynamic behavior of the M3 system, and the controllers for the macro and micro arms are developed without any need for prior knowledge of the dynamic model of the controlled M3 system. A learning algorithm for the neural network using Lyapunov stability theory is derived. It is shown that both the tracking error and the weight-tuning error are uniformly ultimately bounded under this new control scheme. Simulation results are presented and compared to those obtained using a PD controller. [Copyright &y& Elsevier]

A neural network model called lateral interaction in accumulative computation for detection of non-rigid objects from motion of any of their parts in indefinite sequences of images is presented. Some biological evidences inspire the model. After introducing the model, the complete multi-layer neural architecture is offered in this paper. The architecture consists of four layers that perform segmentation by gray level bands, accumulative charge computation, charge redistribution by gray level bands and moving object fusion. The lateral interaction in accumulative computation associated learning algorithm is also introduced. Some examples that explain the usefulness of the system we propose are shown at the end of this article. [Copyright &y& Elsevier]