Search

In today's busy world, people are shifting to online shopping for their basic needs like groceries, clothes, etc. As a result, modern applications are incorporating image search as part of their application. However, retrieving accurate fashion images is challenging due to the low inter-class separability. In this paper, we aim to propose and implement three deep learning denoising autoencoder image retrieval models trained on Fashion-MNIST dataset and MNIST dataset. Fashion-MNIST is widely used datasets to evaluate model because of its low inter-class separability. It is difficult to get a good accuracy on Fashion-MNIST dataset. The proposed models train to learn significant features of fashion images and improve the precision of image retrieval. These models are compared based on the Label Ranking Average Precision (LRAP) values, with the third model bagging the highest LRAP value. These models also trained and tested on MNIST dataset, which have similar configuration as Fashion-MNIST dataset but with high inter-class separability. Intriguingly, the second and third models trained on MNIST dataset demonstrate neck-to-neck performance, both achieving comparable accuracy. This choice of dataset ensures that the models' accuracy is more noticeable and significant. The outcomes of this study validate the efficacy of denoising autoencoders in fashion image retrieval and emphasize their prospective in improving online shopping experiences. [ABSTRACT FROM AUTHOR]

In the rapidly changing world of new technologies, AI and ML have been delivered to the network's edge, allowing edge devices to compute simple models that can be applied later in practical applications. Edge computing allows the deployment of cloudlike services to the network edge. Moreover, it allows network devices to contribute to Federated Learning (FL) by using their processing power. Federated learning was developed when privacy and security concerns emerged among people. Here, multiple clients work together to solve machine-learning problems. Instead of collecting users' raw data, this technique protects user privacy by aggregating model parameters from each client to a central server. Even though FL techniques have several benefits, such as scalability and data privacy, when used with heterogeneous devices, they can introduce significant problems in processing complexity and speed. This paper thoroughly surveyed the important and efficient technological advancements based on edge federated learning. We review the widely used FL frameworks that effectively facilitate communication between clients and servers. We also present a case study on handwritten digit recognition using the MNIST dataset to state the effectiveness of federated learning. On applying this approach, we were able to achieve 94.56% accuracy. [ABSTRACT FROM AUTHOR]

The MNIST dataset is considered a challenging problem for machine learning algorithms. The present paper introduces a novel approach based on a truncated SVD and kernel density estimation that outputs an error rate on MNIST comparable to classical machine learning approaches. In addition to that, a hierarchical classification framework is proposed, that allows to enhance the algorithm accuracy. The resulting algorithm outperforms the reproducible SVM accuracy, which is regarded as a state-of-the-art machine learning algorithm for MNIST. The key advantage of the proposed framework consists in a lower computational cost: training on MNIST takes 142 times less and prediction takes 4 times less than for SVM on a single run. Thus, the research results state that TSVD-KDE algorithm has the potential for being an efficient classification algorithm. The source code for the experiment is released on GitHub: https://github.com/ekplesovskaya/MNIST-Classification-Using-TSVD-and-KDE. [ABSTRACT FROM AUTHOR]

To improve the accuracy of the parallax maps obtained by the binocular stereo matching network for better 3D reconstruction, this thesis is based on the AnyNet network, with the corresponding optimization to get MyNet. The Hourglass structure allows for the acquisition of multiscale image information. The original 3D Conv is replaced by a 3D residual block in the parallax network to avoid the gradient disappearance or explosion and model degradation problems of traditional convolutional neural network models, thus further improving the cost volume obtained. The experiments were tested on the KITTI2012 and KITTI2015 datasets and showed some improvement compared to the original algorithm. The experimental results show that the accuracy of the parallax map has improved significantly with almost the same time spent. [ABSTRACT FROM AUTHOR]

With the development of deep learning technology, handwritten character recognition has become the basic hotspot of research. At the same time, the recognition of handwriting in minority languages is gradually worthy of attention. In this paper, a robust and novel improved deep convolutional neural network method is proposed to recognize normal digital handwriting images and minority digital handwritten images (Yi language). To improve the recognition accuracy of handwritten fonts, there are four improvements in our work. Firstly, separable convolution is adopted in the network to cope with the problem of large parameters. Secondly, data augmentation techniques (Gaussian noise, color perturbation, and random rotation) are utilized to improve the expression ability of the dataset and the generalization capabilities of the model. Finally, the shortcut block and the weight initialization are introduced in our network for the problems of network degradation and gradient saturation. Based on deep learning technology, this paper studies the classification and recognition of handwritten digits, and improves the accuracy of the recognition in numbers such as invoices in financial bills. The recognition accuracy of 99.82% has been achieved on MNIST dataset, and 98.79% on the self-building dataset of Yi handwritten digits (YHDD). [ABSTRACT FROM AUTHOR]

An approach is presented that generalize OCR task including polish letters using deep learning technique. The paper extends EMNIST dataset so that two new classes of polish diacritics "Ą" and "Ć" are attached. Using this new dataset and deep learning technique one can analyze sensitives of standard as well as presented extended approach with different parameters of convolutional neural network. The analysis of the results shows that the shadows and noises that Polish characters, leave with their hooks leads, can be properly recognized and two similar letters "A" and "Ą" can be distinguished by convolutional neural network. On the other hand a neural network trained on dataset without Polish characters do not treat letters "Ą" and "Ć" properly. [ABSTRACT FROM AUTHOR]

Most capsule network designs rely on traditional matrix multiplication between capsule layers and computationally expensive routing mechanisms to deal with the capsule dimensional entanglement that the matrix multiplication introduces. By using Homogeneous Vector Capsules (HVCs), which use element-wise multiplication rather than matrix multiplication, the dimensions of the capsules remain unentangled. In this work, we study HVCs as applied to the highly structured MNIST dataset in order to produce a direct comparison to the capsule research direction of Geoffrey Hinton, et al. In our study, we show that a simple convolutional neural network using HVCs performs as well as the prior best performing capsule network on MNIST using 5.5× fewer parameters, 4× fewer training epochs, no reconstruction sub-network, and requiring no routing mechanism. The addition of multiple classification branches to the network establishes a new state of the art for the MNIST dataset with an accuracy of 99.87% for an ensemble of these models, as well as establishing a new state of the art for a single model (99.83% accurate). [ABSTRACT FROM AUTHOR]

This paper proposes a parameter training method via the fractional-order momentum for convolutional neural networks (CNNs). To update the parameters of CNNs more smoothly, the parameter training method via the fractional-order momentum is proposed based on the Grünwald-Letnikov (G-L) difference operation. The stochastic classical momentum (SCM) algorithm and adaptive moment (Adam) estimation algorithm are improved by replacing the integer-order difference with the fractional-order difference. Meanwhile, the linear and the nonlinear methods are discussed to adjust the fractional-order. Therefore, the proposed methods can improve the flexibility and the adaptive ability of CNN parameters. We analyze the validity of the methods by using MNIST dataset and CIFAR-10 dataset, and the experimental results show that the proposed methods can improve the recognition accuracy and the learning convergence speed of CNNs compared with the traditional SCM and Adam methods. [ABSTRACT FROM AUTHOR]

The aim of this research is to derive new, more effective equations to adjust parameters of a neural network, which have advantage over formulas of the backpropagation method. For this purpose, an objective function of the neural network training is proposed to be modified using expansion of the abstract ideal output of each network layer into a Taylor series. Such decomposition is to be implemented at the point, which is close to the current output of the layer under consideration. The modified objective function gradients are calculated with respect to the difference between ideal and real values of the network parameters. As a result, the obtained weights and biases adjustment equations (for each layer) depend not only on the output of the previous layer, but also on its derivatives. Comparing to the backpropagation method, they allow to reduce the offline neural network training time. The derived formulas can be reduced to the ones of the backpropagation method. This proves the correctness of their development. The efficiency of the new training equations is shown using MNIST handwritten character recognition task. Considering the multilayer structures with different activation functions and number of layers, as far as new formulas are concerned, the same recognition accuracy is achieved, but faster comparing to the conventional backpropagation. [ABSTRACT FROM AUTHOR]

We analyse mathematically the constraints on weights resulting from Hebbian and STDP learning rules applied to a spiking neuron with weight normalisation. In the case of pure Hebbian learning, we find that the normalised weights equal the promotion probabilities of weights up to correction terms that depend on the learning rate and are usually small. A similar relation can be derived for STDP algorithms, where the normalised weight values reflect a difference between the promotion and demotion probabilities of the weight. These relations are practically useful in that they allow checking for convergence of Hebbian and STDP algorithms. Another application is novelty detection. We demonstrate this using the MNIST dataset. [ABSTRACT FROM AUTHOR]

• It provides an overview of the most popular ensemble methods. • It analyzes several fusion schemes using MNIST as guiding thread. • It introduces MNIST-NET10, a complex heterogeneous fusion architecture. • MNIST-NET10 reaches a new record in MNIST with only 10 misclassified images. Ensemble methods have been widely used for improving the results of the best single classification model. A large body of works have achieved better performance mainly by applying one specific ensemble method. However, very few works have explored complex fusion schemes using heterogeneous ensembles with new aggregation strategies. This paper is three-fold: 1) It provides an overview of the most popular ensemble methods, 2) analyzes several fusion schemes using MNIST as guiding thread and 3) introduces MNIST-NET10, a complex heterogeneous fusion architecture based on a degree of certainty aggregation approach; it combines two heterogeneous schemes from the perspective of data, model and fusion strategy. MNIST-NET10 reaches a new record in MNIST with only 10 misclassified images. Our analysis shows that such complex heterogeneous fusion architectures based on the degree of certainty can be considered as a way of taking benefit from diversity. [ABSTRACT FROM AUTHOR]

Forgetting is an essential phenomenon in human brain to help people get away from chaos. Such phenomenon could be emulated in the electronic synaptic device, which is the critical unit for hardware implementation of Spiking Neural Network (SNN). To the best of our knowledge, however, forgetting phenomenon has rarely been applied in weights update of traditional SNNs based on spike-timing-dependent plasticity (STDP). In this work, we propose a novel SNN training algorithm using forgetting phenomenon. The weights update procedures are composed of potentiation and forgetting, and implemented by single-polarity pulses and time intervals between training samples. Benchmarked with the MNIST handwriting dataset, we demonstrate the algorithm's performance by a single-layer perceptron with 784 × 10 synapses. Besides, the influence of some non-ideal factors are taken into consideration to analyze the robust performance of enhanced SNN. The simulation result indicates the proposed SNN with forgetting phenomenon exhibits faster convergence speed and higher recognition rate (88.07%) than traditional SNN with similar network scale. Moreover, it shows a good tolerance to the non-linear conductance response and variation while it has less endurance requirement of electronic synaptic devices. [ABSTRACT FROM AUTHOR]

Spiking neural network (SNN) system that uses rank order coding (ROC) as input spike encoding, generally suffers from low recognition accuracy and unnecessary computations that increase complexities. In this paper, we present a Spiking convolutional neural network (Spiking CNN) architecture that significantly improves recognition accuracy as well as computation efficiencies based on a novel ROC and modified kernel sizes. The proposed ROC generates spike trains based on maximum input value without sorting operations. In addition, as the recognition accuracy is affected by the reduced number of spikes as layers become deeper, the proposed ROC is inserted just before the final layer to increase the number of input spikes. The 2 × 2 pooling kernels are also replaced with 4 × 4 to reduce the network size. The hardware architecture of the proposed Spiking CNN has been implemented using 65 nm CMOS process. Neuron-centric membrane voltage update approach is also efficiently exploited in convolutional and fully connected layers to improve the hardware energy efficiencies. The Spiking CNN processor is seamlessly processing 2.85 K classifications per second with 6.79 uJ/classification. It also achieves 90.2% of recognition accuracy for MNIST dataset using unsupervised learning with STDP. [ABSTRACT FROM AUTHOR]

In the following paper we present a new type of optimization algorithms adapted for neural network training. These algorithms are based upon sequential operator splitting technique for some associated dynamical systems. Furthermore, we investigate through numerical simulations the empirical rate of convergence of these iterative schemes toward a local minimum of the loss function, with some suitable choices of the underlying hyper-parameters. We validate the convergence of these optimizers using the results of the accuracy and of the loss function on the MNIST, MNIST-Fashion and CIFAR 10 classification datasets. [ABSTRACT FROM AUTHOR]

Neural networks have enabled great advances in recent times due mainly to improved parallel computing capabilities in accordance to Moore's Law, which allowed reducing the time needed for the parameter learning of complex, multi-layered neural architectures. However, with silicon technology reaching its physical limits, new types of computing paradigms are needed to increase the power efficiency of learning algorithms, especially for dealing with deep spatio-temporal knowledge on embedded applications. With the goal of mimicking the brain's power efficiency, new hardware architectures such as the SpiNNaker board have been built. Furthermore, recent works have shown that networks using spiking neurons as learning units can match classical neural networks in supervised tasks. In this paper, we show that the implementation of state-of-the-art models on both the MNIST and the event-based NMNIST digit recognition datasets is possible on neuromorphic hardware. We use two approaches, by directly converting a classical neural network to its spiking version and by training a spiking network from scratch. For both cases, software simulations and implementations into a SpiNNaker 103 machine were performed. Numerical results approaching the state of the art on digit recognition are presented, and a new method to decrease the spike rate needed for the task is proposed, which allows a significant reduction of the spikes (up to 34 times for a fully connected architecture) while preserving the accuracy of the system. With this method, we provide new insights on the capabilities offered by networks of spiking neurons to efficiently encode spatio-temporal information. [ABSTRACT FROM AUTHOR]

This work presents three hardware architectures for convolutional neural networks with high degree of parallelism and component reuse implemented in a programmable device. The first design, which is termed memoryful architecture, uses as much memory as necessary to store the input data and intermediate results. The second design, which is termed memoryless architecture, defines and explores a specific input sequencing pattern to completely avoid the use of memory. The third design, which is termed cache memory-based architecture, is an intermediate solution, where the input sequence is further explored. In this case, a cache memory is used to store some intermediate results and, consequently, to improve processing performance. We compare the three designs in terms of power, area, and processing time. Allowing memory usage in the memoryful architecture increases the overall hardware cost, but reduces processing time. Preventing all memory usage in memoryless architecture increases operation parallelism, but compromises processing time. A trade-off between memory usage and processing performance is achieved in cache memory-based architecture. It provides a processing time 3 × shorter than memoryful architecture, running at a clock frequency about 20% higher. When compared to the memoryless design, the cache-based architecture achieves about 13 × less processing time even running at a clock frequency about 1.5% smaller. The improvement in clock frequency and processing performance comes at a cost in terms of hardware resources for the cached memory architecture. Its design, depending on the cache size, may require up to 25% more logic elements than the memoryful and memoryless architectures. • Three hardware architectures for CNN with high degree of parallelism and reuse. • LeNet-5 hardware architecture using local memories and 16-bit fixed-point format. • Specific scanning pattern for operation execution to eliminate the use of memories. • Cache memory implementation for balancing performance and memory use. [ABSTRACT FROM AUTHOR]

Training deep neural networks with the error backpropagation algorithm is considered implausible from a biological perspective. Numerous recent publications suggest elaborate models for biologically plausible variants of deep learning, typically defining success as reaching around 98% test accuracy on the MNIST data set. Here, we investigate how far we can go on digit (MNIST) and object (CIFAR10) classification with biologically plausible, local learning rules in a network with one hidden layer and a single readout layer. The hidden layer weights are either fixed (random or random Gabor filters) or trained with unsupervised methods (Principal/Independent Component Analysis or Sparse Coding) that can be implemented by local learning rules. The readout layer is trained with a supervised, local learning rule. We first implement these models with rate neurons. This comparison reveals, first, that unsupervised learning does not lead to better performance than fixed random projections or Gabor filters for large hidden layers. Second, networks with localized receptive fields perform significantly better than networks with all-to-all connectivity and can reach backpropagation performance on MNIST. We then implement two of the networks – fixed, localized, random & random Gabor filters in the hidden layer – with spiking leaky integrate-and-fire neurons and spike timing dependent plasticity to train the readout layer. These spiking models achieve > 98.2% test accuracy on MNIST, which is close to the performance of rate networks with one hidden layer trained with backpropagation. The performance of our shallow network models is comparable to most current biologically plausible models of deep learning. Furthermore, our results with a shallow spiking network provide an important reference and suggest the use of data sets other than MNIST for testing the performance of future models of biologically plausible deep learning. [ABSTRACT FROM AUTHOR]

This study evaluates the sensitivity of FPGA-based CNN systems and the efficacy of selective hardening approaches with laser injection and proton irradiation. The LeNet-5 CNN architecture was taken as a case study. First, the entire CNN with block random access memory (BRAM) attached is evaluated by laser test scanning, and errors generated from each layer are detected to determine the most critical layer. Second, the same network is evaluated without BRAM, considering only the configuration RAM (CRAM) and errors generated from each layer are detected. Third, the impact of laser scan errors from BRAM and CRAM are compared, while tracing the layers in which these errors are generated. Then, proton testing is performed on the entire CNN, and errors from each layer are detected and compared to validate the laser scanning test. Experimental results from laser and proton demonstrate that CRAM errors have a larger impact on CNN layers than BRAM errors. Regarding layer criticality, the second convolutional layer (C2) appears to be the most critical one because it generates most CRAM errors. This was validated by serial and parallel designs, where both confirm similar trends. Accordingly, the partial triple modular redundancy (TMR) approach is only applied to the critical portions, leading to around 40 % reliability improvement with less than 20 % overhead. • Evaluates the sensitivity of FPGA-based Convolutional neural networks with laser injection and proton irradiation • The impact of BRAM and CRAM within CNN layers • Layer criticality identification • Carry out radiation hardened by design approach [ABSTRACT FROM AUTHOR]

Extensive research and evaluations have been conducted on neural networks to improve classification accuracy and training time. Many classical architectures of neural networks have been modified in several different ways for advancement in design. We propose a new architecture, the hierarchical autoassociative polynomial neural network (HAP Net), which is a formulation of different neural network concepts. HAP Net is a combination of polynomial networks, which provides the network with nonlinear weighting, deep belief networks, which obtains higher level abstraction of the incoming data, and convolutional neural networks, which localizes regions of neurons. By incorporating all of these concepts together along with a derivation of a standard backpropagation algorithm, we produce a strong neural network that has the strengths of each concept. Evaluations have been conducted on the MNIST Database, which is a well-known character database tested by many state of the art classification algorithms, and have found the HAP Net to have one of the lowest test error rates among many leading algorithms. [ABSTRACT FROM AUTHOR]

We present a method for synthesising deep neural networks using Extreme Learning Machines (ELMs) as a stack of supervised autoencoders. We test the method using standard benchmark datasets for multi-class image classification (MNIST, CIFAR-10 and Google Streetview House Numbers (SVHN)), and show that the classification error rate can progressively improve with the inclusion of additional autoencoding ELM modules in a stack. Moreover, we found that the method can correctly classify up to 99.19% of MNIST test images, which surpasses the best error rates reported for standard 3-layer ELMs or previous deep ELM approaches when applied to MNIST. The approach simultaneously offers a significantly faster training algorithm to achieve its best performance (in the order of 5 min on a four-core CPU for MNIST) relative to a single ELM with the same total number of hidden units as the deep ELM, hence offering the best of both worlds: lower error rates and fast implementation. [ABSTRACT FROM AUTHOR]

In this paper, we show various memory characteristics of the Ag/AlN/TiN devices for neuromorphic systems. We verified the thickness and the components of the device stack by transmission electron microscopy (TEM) and energy-dispersive X-ray spectroscopy (EDS). We investigated the long-term memory (LTM) characteristics, and short-term memory (STM) characteristics can be determined by compliance current (CC). It shows LTM characteristics when CC is high and STM characteristics when CC is low. I-V curves for each characteristic were investigated, and potentiation and depression for LTM characteristics. The switching and conduction mechanisms of Ni/Ag/AlN/TiN devices are studied using the schematic drawing of the conducting filament and the energy band diagram, including the work function, electron affinity, and bandgap energy of each layer. The linearity of potentiation and depression was compared for an identical pulse and an incremental pulse. Finally, we investigated Modified National Institute of Standards and Technology (MNIST) pattern accuracy depending on the linearity of potentiation and depression. • AlN based device is proposed to implement synaptic properties. • TEM and EDS analysis provide the chemical and material information of device. • The improved synaptic properties are achieved by controlling pulse scheme. • Neuromorphic system simulation is conducted to evaluate the conductance update. [ABSTRACT FROM AUTHOR]

• The solution-processed thin-film-transistor of lithium ion doped electrolyte-gate transistors. • The synaptic transistor with inhibitory/excitatory postsynaptic current (IPSC/EPSC), long-term potentiation/depression (LTP/LTD), short-term potentiation (STP), and paired-pulse facilitation (PPF). • Image recognition for the neuromorphic computing. • The large G max /G min ratio and appropriate nonlinearity value which be extracted from the LTP/LTD curve. Artificial synaptic devices are the critical component for large-scale neuromorphic computing, which surpasses the limitations of von Neumann's structure. Recently the emerging electrolyte gated transistor (EGT) has proven to be a promising neuromorphic application due to the conductance can be updated by the gate voltage stimulation. This paper presents a new low-temperature solution-based oxide thin film transistor with lithium (Li) ion dope dinto dielectric layer. We have proposed the indium oxide (InO x)/ zirconia (ZrO x) transistor with large hysteresis. The synaptic plasticity of EGTs demonstrate the potential to simulate the biological neuron and calculation function. Moreover, the inhibitory/excitatory postsynaptic current (IPSC/EPSC), long-term potentiation/depression (LTP/LTD), short-term potentiation (STP), and paired-pulse facilitation (PPF) are confirmed through the electrical stimulation. The suitable ion doping concentration is obtained by the synaptic electrical characteristic. The synaptic transistor also has a low-noise linear conductance update and a relatively high G max /G min ratio. According to the G max /G min ratio and nonlinearity, the weight update process can be simulated for neuromorphic computing. [ABSTRACT FROM AUTHOR]

Two problems that burden the learning process of Artificial Neural Networks with Back Propagation are the need of building a full and representative learning data set, and the avoidance of stalling in local minima. Both problems seem to be closely related when working with the handwritten digits contained in the MNIST dataset. Using a modest sized ANN, the proposed combination of input data transformations enables the achievement of a test error as low as 0.43%, which is up to standard compared to other more complex neural architectures like Convolutional or Deep Neural Networks. [ABSTRACT FROM AUTHOR]

The paper proposes a novel supervised algorithm for data classification based on fuzzy vector quantization. The novelty here lies in including an optimization procedure to determine the step size of the quantization process. Adoption of fuzzy space for data exploration eliminates the problem of class-overlapping, while dealing with high dimensional non-linear data. In the proposed approach, the input data are first been transformed from the Euclidean space to the fuzzy space with three membership grades. Next, we have used uniform manifold approximation algorithm ( UMAP ), for projecting the data in visual space by selection of the most contrasting features from the fuzzy vectors. Then these features are passed through a quantization step with a novel step-optimization technique. Optimizing the quantization step and making it independent of data sets significantly speeds up the process. As the class information of all the feature vectors obtained by UMAP is known, majority voting principle has been used to locate the class-centroids in the subsequent step which in turn represent the class labels of the test samples. In test phase, after obtaining the test vectors, the Hyperspherical Direction Cosines (HDC) between the test vectors and the previously obtained class-centroids are evaluated. The test sample is finally assigned that class label where the sum of absolute differences (SAD) of these direction cosines is minimum. We have validated our classifier on various benchmark data sets and achieved higher accuracy with significantly low computation time than the existing state-of-the-art algorithms. • A novel pattern classifier algorithm using Fuzzy Vector Quantization is proposed. • Fuzzy-space representation of data eliminates the problem of class-overlapping. • Introducing optimization to determine the step size of the quantization process. • Validation of the classifier algorithm on various high dimensional benchmark datasets. • Achieved higher accuracy with significantly low computation cost. [ABSTRACT FROM AUTHOR]

Abstract: Recently there have been renewed interests in single-hidden-layer neural networks (SHLNNs). This is due to its powerful modeling ability as well as the existence of some efficient learning algorithms. A prominent example of such algorithms is extreme learning machine (ELM), which assigns random values to the lower-layer weights. While ELM can be trained efficiently, it requires many more hidden units than is typically needed by the conventional neural networks to achieve matched classification accuracy. The use of a large number of hidden units translates to significantly increased test time, which is more valuable than training time in practice. In this paper, we propose a series of new efficient learning algorithms for SHLNNs. Our algorithms exploit both the structure of SHLNNs and the gradient information over all training epochs, and update the weights in the direction along which the overall square error is reduced the most. Experiments on the MNIST handwritten digit recognition task and the MAGIC gamma telescope dataset show that the algorithms proposed in this paper obtain significantly better classification accuracy than ELM when the same number of hidden units is used. For obtaining the same classification accuracy, our best algorithm requires only 1/16 of the model size and thus approximately 1/16 of test time compared with ELM. This huge advantage is gained at the expense of 5 times or less the training cost incurred by the ELM training. [Copyright &y& Elsevier]

• The proposed approach can determine number of hidden layers in DBN and hidden neurons in RBM. • It removes the need of trial and error for discovering the reasonable (or optimal) network structure. • It is low cost from terms of time and computation. • It is the self-organizing deep model. • It acts as a regularization method and prevents over-fitting. Recently, deep learning is proliferating in the field of representation learning. A deep belief network (DBN) consists of a deep network architecture that can generate multiple features of input patterns, using restricted Boltzmann machines (RBMs) as a building block of DBN. A deep learning model can achieve extremely high accuracy in many applications that depend on the model structure. However, specifying various parameters of deep network architecture like the number of hidden layers and neurons is a difficult task even for expert designers. Besides, the number of hidden layers and neurons is typically set manually, while this method is costly in terms of time and computational cost, especially in big data. In this paper, we introduce an approach to determine the number of hidden layers and neurons of the deep network automatically during the learning process. To this end, the input vector is transformed from the feature space with a low dimension into the new feature space with a high dimension in a hidden layer of RBM. In the following, new features are ranked according to their discrimination power between classes in the new space, using the Separability-correlation measure for feature importance ranking algorithm. The algorithm uses the mean of weights as a threshold, so the neurons whose weights exceed the threshold are retained, and the others are removed in the hidden layer. The number of retained neurons is presented as a reasonable number of neurons. The number of layers is also determined in the deep model, using the validation data. The proposed approach acts as a regularization method since the neurons whose weights are lower than the threshold are removed; thus, RBM learns to copy input merely approximate. It also prevents over-fitting with a suitable number of hidden layers and neurons. Eventually, DBN can determine its structure according to the input data and is the self-organizing model. The experimental results on benchmark datasets confirm the proposed method. [ABSTRACT FROM AUTHOR]