Search

Neuroevolution is the name given to a field of computer science that applies evolutionary computation for evolving some aspects of neural networks. After the AI Winter came to an end, neural networks reemerged to solve a great variety of problems. However, their usage requires designing their topology, a decision with a potentially high impact on performance. Whereas many works have tried to suggest rules-of-thumb for designing topologies, the truth is that there are not analytic procedures for determining the optimal one for a given problem, and trial-and-error is often used instead. Neuroevolution arose almost 3 decades ago, with some works focusing on the evolutionary design of the topology and most works describing techniques for learning connection weights. Since then, evolutionary computation has been proved to be a convenient approach for determining the topology and weights of neural networks, and neuroevolution has been applied to a great variety of fields. However, for more than 2 decades neuroevolution has mainly focused on simple artificial neural networks models, far from today's deep learning standards. This is insufficient for determining good architectures for modern networks extensively used nowadays, which involve multiple hidden layers, recurrent cells, etc. More importantly, deep and convolutional neural networks have become a de facto standard in representation learning for solving many different problems, and neuroevolution has only focused in this kind of networks in very recent years, with many works being presented in 2017 onward. In this paper, we review the field of neuroevolution during the last 3 decades. We will put the focus on very recent works on the evolution of deep and convolutional neural networks, which is a new but growing field of study. To the best of our knowledge, this is the best survey reviewing the literature in this field, and we have described the features of each work as well as their performance on well-known databases when available. This work aims to provide a complete reference of all works related to neuroevolution of convolutional neural networks up to the date. Finally, we will provide some future directions for the advancement of this research area. [ABSTRACT FROM AUTHOR]

In evolutionary computation, the fitness function normally measures progress toward an objective in the search space, effectively acting as an objective function. Through deception, such objective functions may actually prevent the objective from being reached. While methods exist to mitigate deception, they leave the underlying pathology untreated: Objective functions themselves may actively misdirect search toward dead ends. This paper proposes an approach to circumventing deception that also yields a new perspective on open-ended evolution. Instead of either explicitly seeking an objective or modeling natural evolution to capture open-endedness, the idea is to simply search for behavioral novelty. Even in an objective-based problem, such novelty search ignores the objective. Because many points in the search space collapse to a single behavior, the search for novelty is often feasible. Furthermore, because there are only so many simple behaviors, the search for novelty leads to increasing complexity. By decoupling open-ended search from artificial life worlds, the search for novelty is applicable to real world problems. Counterintuitively, in the maze navigation and biped walking tasks in this paper, novelty search significantly outperforms objective-based search, suggesting the strange conclusion that some problems are best solved by methods that ignore the objective. The main lesson is the inherent limitation of the objective-based paradigm and the unexploited opportunity to guide search through other means. [ABSTRACT FROM AUTHOR]

Abstract: In an earlier paper [Fukushima, K., & Tohyama, K. (2005). Analysis of optic flow: a neural network model. In ICONIP 2005, International Conference on Neural Information Processing (pp. 312–317); Tohyama, K., & Fukushima, K. (2005). Neural network model for extracting 98 optic flow. Neural Networks, 18(5–6), 549–556], we proposed a neural network extracting optic flow based on vector field hypothesis. We started discussion there, however, from the stage where local velocities of visual objects have already been extracted. This paper proposes a new mechanism of extracting local velocity from retinal images, and adds it to the previous network. The network has a hierarchical multilayered architecture. X- and Y-cells of the retina extract spatial and temporal contrast of brightness. The network contains several types of V1 cells, namely, S- C- and V-cells. S- and C-cells extract orientated edges. V-cells extract local velocities, based on the signals from Y- and S-cells. MT cells extract relative velocities between adjoining small visual fields. MST cells add the responses of MT cells to extract a specific optic flow, such as rotation and expansion/contraction. The difference in type of optic flow extracted by MST cells can be created simply by the difference in relative locations of inhibitory to excitatory areas in the receptive fields of MT cells. [Copyright &y& Elsevier]

In this paper, the Selected Objects Extraction (SOE) CNN is generalized to Directional Extraction (DE) CNN which enhances the capabilities of the SOE CNN. Using analytical approaches, a theorem to design robust templates for DE CNN is established. The theorem provides parameter inequalities to determine parameter intervals in which the templates can implement the corresponding functions. Based on the theorem, an optimal model is set up to design optimally robust templates for DE CNN. Two examples are provided to illustrate the effectiveness of the theorem. [ABSTRACT FROM AUTHOR]

UCS is a supervised learning classifier system that was introduced in 2003 for classification in data mining tasks. The representation of a rule in UCS as a univariate classification rule is straightforward for a human to understand. However, the system may require a large number of rules to cover the input space. Artificial neural networks (NNs), on the other hand, normally provide a more compact representation. However, it is not a straightforward task to understand the network. In this paper, we propose a novel way to incorporate NNs into UCS. The approach offers a good compromise between compactness, expressiveness, and accuracy. By using a simple artificial NN as the classifier's action, we obtain a more compact population size, better generalization, and the same or better accuracy while maintaining a reasonable level of expressiveness. We also apply negative correlation learning (NCL) during the training of the resultant NN ensemble. NCL is shown to improve the generalization of the ensemble. [ABSTRACT FROM AUTHOR]

Abstract: Many man-made objects are composed of a number of some simple geometric shapes such as lines, circles, rectangles, etc. Therefore, the detection of rectangular objects is an important issue to some practical applications such as the detection of buildings and vehicles in aerial imagery, the detection of license plates in car images, etc. Several methods have been proposed for solving the problem of the detection of rectangular objects. While some approaches are based on the detecting lines, some approaches are based on the Hough transform. Each approach has its own advantages and disadvantages (e.g., computational load). In this paper, we propose a class of neural networks with a special type of neural junctions for the detection of rectangular objects. The proposed neural networks can be trained in either an unsupervised mode or a batch mode. In contrast to some popular clustering algorithms such as the fuzzy -means algorithm and the fuzzy -rectangular shells algorithm, our approach is not based on minimizing an objective function but based on the idea of competitive learning. Based on the idea of competitive learning, the computational load can be decreased. Several data sets were tested to illustrate the effectiveness of our proposed approach. [Copyright &y& Elsevier]

Abstract: Automatic birdsong recognition using prediction-based singleton-type recurrent neural fuzzy networks (SRNFNs) is proposed in this paper. The recognition task consists of two stages. The first stage segments a significant portion from a birdsong sequence and the second stage performs recognition. For birdsong segmentation, an easy but effective segmentation approach based on time domain energy is proposed. For recognition, the linear predictive coding (LPC) coefficients of each frame in a segmented birdsong are extracted and used as features. These features are fed as inputs to SRNFN recognizers. The SRNFN is constructed by recurrent fuzzy if–then rules with fuzzy singletons in the consequences, and its recurrent aspect makes it suitable for processing patterns with temporal characteristics. In birdsong recognition, the sample prediction technique is used, where one SRNFN is responsible for learning the temporal birdsong relationships of only one species. The prediction error of each SRNFN is then used as a criterion for recognition. Experiments with 10 species of birds and their songs are performed, and a high recognition rate is achieved. Comparisons with a Takagi–Sugeno–Kang (TSK)-type recurrent fuzzy network (TRFN) and backpropagation neural network are also made in the experiments. [Copyright &y& Elsevier]

Abstract: Several implementations of Feedforward Neural Networks have been reported in scientific papers. These implementations do not allow the direct use of off-line trained networks. Usually, the problem is the lower precision (compared to the software used for training) or modifications in the activation function. In the present work, a hardware solution called Artificial Neural Network Processor, using a FPGA, fits the requirements for a direct implementation of Feedforward Neural Networks, because of the high precision and accurate activation function that were obtained. The resulting hardware solution is tested with data from a real system to confirm that it can correctly implement the models prepared off-line with MATLAB. [Copyright &y& Elsevier]

Abstract: This article presents a series of hardware implementations of a general regression neural network (GRNN) using FPGAs. The paper describes the study of this neural network using different fixed and floating point implementations. The implementation includes training as well as testing of the network. It is focused on precision loss and area and speed results of the resulting neural network coprocessor that can be used in a System on Programmable Chip. A GRNN is able to approximate functions and it has been used in control, prediction, fault diagnosis, engine management among others. They are mainly implemented as software entities because they require a great amount of complex mathematical operations. With the increasing power and capabilities of current FPGAs, now it is possible not only to translate them into hardware but, due to the reconfigurable feature of these devices, to explore different hardware/software partitions as well. These hardware implementations increase both the speed and performance of these neural networks and the designer can select the area–speed trade-off that best fits the application. [Copyright &y& Elsevier]

In this paper, a bibliometric study of the computational intelligence field is presented. Bibliometric maps showing the associations between the main concepts in the field are provided for the periods 1996–2000 and 2001–2005. Both the current structure of the field and the evolution of the field over the last decade are analyzed. In addition, a number of emerging areas in the field are identified. It turns out that computational intelligence can best be seen as a field that is structured around four important types of problems, namely control problems, classification problems, regression problems, and optimization problems. Within the computational intelligence field, the neural networks and fuzzy systems subfields are fairly intertwined, whereas the evolutionary computation subfield has a relatively independent position. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, a model is considered to describe the dynamics of a class of non-autonomous neural networks with time-varying delays. By applying the properties of M-matrix, the techniques of inequality analysis and the Banach fixed point theorem, we obtain a series of new criteria on the dissipativity and existence of periodic attractor. Our results can extend and improve earlier ones. [Copyright &y& Elsevier]

The generalization error bounds found by current error models using the number of effective parameters of a classifier and the number of training samples are usually very loose. These bounds are intended for the entire input space. However, support vector machine (SVM), radial basis function neural network (RBFNN), and multilayer perceptron neural network (MLPNN) are local learning machines for solving problems and treat unseen samples near the training samples to be more important. In this paper, we propose a localized generalization error model which bounds from above the generalization error within a neighborhood of the training samples using stochastic sensitivity measure. It is then used to develop an architecture selection technique for a classifier with maximal coverage of unseen samples by specifying a generalization error threshold. Experiments using 17 University of California at Irvine (UCI) data sets show that, in comparison with cross validation (CV), sequential learning, and two other ad hoc methods, our technique consistently yields the best testing classification accuracy with fewer hidden neurons and less training time. [ABSTRACT FROM AUTHOR]

This is the second part of a two-part paper where we propose, model theoretically and study a general notion of recombination for fixed-length strings where homologous recombination, inversion, gene duplication, gene deletion, diploidy and more are just special cases. In Part I, we derived both microscopic and coarse-grained evolution equations for strings and schemata for a selecto-recombinative GA using generalised recombination, and we explained the hierarchical nature of the schema evolution equations. In this part, we provide a variety of fixed points for evolution in the case where recombination is used alone, thereby generalising Geiringer's theorem. In addition, we numerically integrate the infinite-population schema equations for some interesting problems, where selection and recombination are used together to illustrate how these operators interact. Finally, to assess by how much genetic drift can make a system deviate from the infinite-population-model predictions we discuss the results of real GA runs for the same model problems with generalised recombination, selection and finite populations of different sizes. [ABSTRACT FROM AUTHOR]

The application of artificial neural networks (ANNs) to solve wind engineering problems has received increasing interests in recent years. This paper is concerned with developing two ANN approaches (a backpropagation neural network [BPNN] and a fuzzy neural network [FNN]) for the prediction of mean, root-mean-square (rms) pressure coefficients and time series of wind-induced pressures on a large gymnasium roof. In this study, simultaneous pressure measurements are made on a large gymnasium roof model in a boundary layer wind tunnel and parts of the model test data are used as the training sets for developing two ANN models to recognize the input–output patterns. Comparisons of the prediction results by the two ANN approaches and those from the wind tunnel test are made to examine the performance of the two ANN models, which demonstrates that the two ANN approaches can successfully predict the pressures on the entire surfaces of the large roof on the basis of wind tunnel pressure measurements from a certain number of pressure taps. Moreover, the FNN approach is found to be superior to the BPNN approach. It is shown through this study that the developed ANN approaches can be served as an effective tool for the design and analysis of wind effects on large roof structures. [Copyright &y& Elsevier]

Abstract: This paper proposes a new algorithm for a class of binary images thinning by using two PCNNs (pulse coupled neural networks). Once the travelling pulses of the two PCNNs meet, the thinning result is obtained. The criterion of pulses meeting is given, and the parameters of the PCNNs are also specified, which make the implementation of the proposed thinning algorithm easier. The algorithm is used to thin such a class of binary images, which separate the original images into two regions, as circularity-like images and ribbon-like shapes. Experimental results show that the proposed algorithm is efficient in extracting the skeleton of images (such as circularity-like images, ribbon-like shapes, handwriting ‘0’, etc.). [Copyright &y& Elsevier]

This paper describes two versions of a novel approach to developing binary classifiers, based on two evolutionary computation paradigms: cellular programming and genetic programming. Such an approach achieves high computation efficiency both during evolution and at runtime. Evolution speed is optimized by allowing multiple solutions to be computed in parallel. Runtime performance is optimized explicitly using parallel computation in the case of cellular programming or implicitly taking advantage of the intrinsic parallelism of bitwise operators on standard sequential architectures in the case of genetic programming. The approach was tested on a digit recognition problem and compared with a reference classifier. [ABSTRACT FROM AUTHOR]

Temporal and sequential information is essential to any agent continually interacting with its environment. In this paper, we test whether it is possible to evolve a recurrent neural network controller to match the dynamic requirement of the task. As a benchmark, we consider a sequential navigation task where the agent has to alternately visit two rewarding sites to obtain food and water after first visiting the nest. To achieve a better fitness, the agent must select relevant sensory inputs and update its working memory to realize a non-Markovian sequential behavior in which the preceding state alone does not determine the next action. We compare the performance of a feed-forward and recurrent neural control architectures in different environment settings and analyze the neural mechanisms and environment features exploited by the agents to achieve their goal. Simulation and experimental results using the Cyber Rodent robot show that a modular architecture with a locally excitatory recurrent layer outperformed the general recurrent controller. [ABSTRACT FROM AUTHOR]

This paper presents architectures for comparator trees capable of finding the minimum value of a large number of inputs. Such circuits are of general applicability although the intended application for which the circuits were designed is the winner-take-all function of a digital implementation of a neural network based on the self organising map. Mechanisms for reducing delay based on look-ahead logic within individual comparators and mechanisms based on multiplexor architectures of a comparator are compared for both propagation delay and area. [Copyright &y& Elsevier]

In this papers, we analyze some possible functions of networks where delays of pulse propagation and asynchronous events of pulse occurrence are considered, on the basic of a model asynchronous chaotic neural networks. First, the model is proposed and response characteristics of the single neuron are numerically analyzed, Second, it is shown that the network model with loop structure can select time interval sequence peculiar to the network structure even if random noise is superimposed. [ABSTRACT FROM AUTHOR]