Search

Backpropagation is the most popular and common algorithm for training of traditional deep neural networks. Herewe propose temporal version of backpropagation to directly train spiking neural networks with deep structure andsingle spikebased temporal coding scheme (DS4NN). We consider a convolutional spiking neural networkconsisting of simple non-leaky integrate-and-fire (IF) neurons, and a temporal coding known as timeto-first-spikecoding. These together lead to lower computational cost and higher inference speed. We use surrogate gradient atfiring times to solve the non-differentiability of spike times with respect to membrane potential of spiking neurons,and to prevent the emergence of dead neurons in deep layers, we propose a relative encoding scheme fordetermining desired firing times. Evaluations on two classification tasks of MNIST and FashionMNIST datasetsconfirm the capability of DS4NN on deep SNNs. It achieves the accuracy of 99.3% and 91.6% on respectivelyMNIST and FashionMNIST datasets with the mean required number of 1126 and 1863 spikes in the whole network.This shows that the proposed approach can make fast decisions with low cost and high accuracy.

International audience; In recent years, deep learning has revolutionized the field of machine learning, for computer vision in particular. In this approach, a deep (multilayer) artificial neural network (ANN) is trained, most often in a supervised manner using backpropagation. Vast amounts of labeled training examples are required, but the resulting classification accuracy is truly impressive, sometimes outperforming humans.Neurons in an ANN are characterized by a single, static, continuous-valued activation. Yet biological neurons use discrete spikes to compute and transmit information, and the spike times, in addition to the spike rates, matter. Spiking neural networks (SNNs) are thus more biologically realistic than ANNs, and are arguably the only viable option if one wants to understand how the brain computes at the neuronal description level. The spikes of biological neurons are sparse in time and space, and event-driven. Combined with bio-plausible local learning rules, this makes it easier to build low-power, neuromorphic hardware for SNNs. However, training deep SNNs remains a challenge. Spiking neurons’ transfer function is usually non-differentiable, which prevents using backpropagation.Here we review recent supervised and unsupervised methods to train deep SNNs, and compare them in terms of accuracy and computational cost. The emerging picture is that SNNs still lag behind ANNs in terms of accuracy, but the gap is decreasing, and can even vanish on some tasks, while SNNs typically require many fewer operations and are the better candidates to process spatio-temporal data.

Error backpropagation is the most common approach for direct training of spiking neural networks. However, the non-differentiability of spiking neurons makes the backpropagation of error a challenge. In this paper, we introduce a new temporal learning algorithm, STiDi-BP, in which we ignore backward recursive gradient computation, and to avoid the non-differentiability of SNNs, we use a linear approximation to compute the derivative of latency with respect to the potential. We apply gradient descent to each layer independently based on an estimation of the temporal error in that layer. To do so, we calculate the desired firing time of each neuron and compare it to its actual firing time. In STiDi-BP, we employ the time-to-first-spike temporal coding, one spike per neuron, and use spiking neuron models with piecewise linear postsynaptic potential which provide large computational benefits. To evaluate the performance of the proposed learning rule, we run three experiments on the XOR problem, the face/motorbike categories of the Caltech 101 dataset, and the MNIST dataset. Experimental results show that the STiDi-BP outperforms traditional BP in terms of accuracy and/or computational cost.

We propose a new learning algorithm to train spiking neural networks (SNN) using conventional artificial neural networks (ANN) as proxy. We couple two SNN and ANN networks, respectively, made of integrate-and-fire (IF) and ReLU neurons with the same network architectures and shared synaptic weights. The forward passes of the two networks are totally independent. By assuming IF neuron with rate-coding as an approximation of ReLU, we backpropagate the error of the SNN in the proxy ANN to update the shared weights, simply by replacing the ANN final output with that of the SNN. We applied the proposed proxy learning to deep convolutional SNNs and evaluated it on two benchmarked datasets of Fashion-MNIST and Cifar10 with 94.56% and 93.11% classification accuracy, respectively. The proposed networks could outperform other deep SNNs trained with tandem learning, surrogate gradient learning, or converted from deep ANNs. Converted SNNs require long simulation times to reach reasonable accuracies while our proxy learning leads to efficient SNNs with much smaller simulation times. The source codes of the proposed method are publicly available at https://github.com/SRKH/ProxyLearning.

We recently proposed the S4NN algorithm, essentially an adaptation of backpropagation to multilayer spiking neural networks that use simple non-leaky integrate-and-fire neurons and a form of temporal coding known as time-to-first-spike coding. With this coding scheme, neurons fire at most once per stimulus, but the firing order carries information. Here, we introduce BS4NN, a modification of S4NN in which the synaptic weights are constrained to be binary (+1 or -1), in order to decrease memory (ideally, one bit per synapse) and computation footprints. This was done using two sets of weights: firstly, real-valued weights, updated by gradient descent, and used in the backward pass of backpropagation, and secondly, their signs, used in the forward pass. Similar strategies have been used to train (non-spiking) binarized neural networks. The main difference is that BS4NN operates in the time domain: spikes are propagated sequentially, and different neurons may reach their threshold at different times, which increases computational power. We validated BS4NN on two popular benchmarks, MNIST and Fashion-MNIST, and obtained reasonable accuracies for this sort of network (97.0% and 87.3% respectively) with a negligible accuracy drop with respect to real-valued weights (0.4% and 0.7%, respectively). We also demonstrated that BS4NN outperforms a simple BNN with the same architectures on those two datasets (by 0.2% and 0.9% respectively), presumably because it leverages the temporal dimension. The source codes of the proposed BS4NN are publicly available at https://github.com/SRKH/BS4NN.

We propose a new supervised learning rule for multilayer spiking neural networks (SNNs) that use a form of temporal coding known as rank-order-coding. With this coding scheme, all neurons fire exactly one spike per stimulus, but the firing order carries information. In particular, in the readout layer, the first neuron to fire determines the class of the stimulus. We derive a new learning rule for this sort of network, named S4NN, akin to traditional error backpropagation, yet based on latencies. We show how approximated error gradients can be computed backward in a feedforward network with any number of layers. This approach reaches state-of-the-art performance with supervised multi-fully connected layer SNNs: test accuracy of 97.4% for the MNIST dataset, and 99.2% for the Caltech Face/Motorbike dataset. Yet, the neuron model that we use, nonleaky integrate-and-fire, is much simpler than the one used in all previous works. The source codes of the proposed S4NN are publicly available at https://github.com/SRKH/S4NN .

Background Artificial neural networks (ANNs) and adaptive neural-fuzzy Inference system (ANFIS) are the best solutions to finding the correlation between some water parameters and human hormones. The correlation between thyroid stimulating hormone (TSH) and drinking water fluoride studied by ANNS and ANFIS models in Yazd city. Method In this study, eighty people with thyroid gland disorder and 213 healthy people invited. Their thyroid hormones and fluoride drinking water analyzed. Results The result of ANFIS showed R2 = 0.81 for test and R2 = 0.85 for train in all cases and controls data. This results were R2 = 0.73 and R2 = 0.81 for ANNs respectively. Conclusion This models can be used as an alternative for show correlation between Drinking Water Fluoride and TSH Hormone and R2 = 0.85 gained from ANFIS was the best.

The elevated fluoride from drinking water impacts on T3, T4 and TSH hormones. The aim was study impacts of drinking water fluoride on T3, T4 and TSH hormones inYGA (Yazd Greater Area). In this case- control study 198 cases and 213 controls were selected. Fluoride was determined by the SPADNS Colorimetric Method. T3, T4 and TSH hormones tested in the Yazd central laboratory by RIA (Radio Immuno Assay) method. The average amount of TSH and T3 hormones based on the levels of fluoride in two concentration levels 0–0.29 and 0.3–0.5 (mg/L) was statistically significant (P = 0.001 for controls and P = 0.001 for cases). In multivariate regression logistic analysis, independent variable associated with Hypothyroidism were: gender (odds ratio: 2.5, CI 95%: 1.6–3.9), family history of thyroid disease (odds ratio: 2.7, CI 95%: 1.6–4.6), exercise (odds ratio: 5.34, CI 95%: 3.2–9), Diabetes (odds ratio: 3.7, CI 95%: 1.7–8), Hypertension (odds ratio: 3.2, CI 95%: 1.3–8.2), water consumption (odds ratio: 4, CI 95%: 1.2–14). It was found that fluoride has impacts on TSH, T3 hormones even in the standard concentration of less than 0.5 mg/L. Application of standard household water purification devices was recommended for hypothyroidism.

The effects of acute fluoride toxicity have been well documented in the literature. Drinking water is an important source of fluoride intake by humans, hence studies need to be carried out to determine the concentration of Fluoride in water. Therefore, this study tends to demonstrate the fluoride concentration in drinking water in thirty-one provinces of Iran during 2014. This cross-sectional study on drinking water was conducted in 2014. SPADNS method was determined for fluoride concentration examination according to instruction of Standard Methods. The minimum concentration of fluoride in provinces, such as Fars, Kermanshah, Kohgiluyeh and Boyer-Ahmad, Markazi, and Hormozgan, was observed to be 0.01 mg/L, while the maximum concentrations were observed to be 3.72 and 3.52 mg/L for Bushehr and Fars, respectively. The minimum and maximum average mean concentrations were 0.193 (SD = 0.11) and 0.889 (SD = 0.31) for Kermanshah and Bushehr, respectively. Due to the disadvantages of fluoride and because of the existence of different ecological conditions in Iran, there are different concentrations of water fluoride in the country. Therefore, proper policies should be made for water treatment plants based on the regional conditions in order to achieve a desirable fluoride concentration standard.

The paper aims to investigate the transient response of an air-filled multilayer hollow functionally graded (FGM) cylinder with interlaminar bonding imperfection in the presence of load which is extensively put to use in aerospace structure. The material properties of each layer are assumed to vary continuously within the cylinder along the thickness direction with arbitrary grading pattern. A linear spring model is used to define imperfectly bonded interfaces of the multilayer cylinder. The solution of problem is obtained by means of the laminate approximation theory along with the Durbin's numerical Laplace transform inversion with regard to the time coordinate. Detailed numerical study of transient response of multilayer FGM cylinder with imperfect bonding under a pulse excitation are presented. In the following, the effect of imperfection on radial and circumferential stresses is presented and in view of the lack of any data, only the obtained results with the perfect bond are compared to those of other researchers have published in the literature. Also, by displaying contours of the internal pressure field, dynamic features in the fluid-structure interaction are investigated. Finally the effect of load duration and various loads, including step load and exponential load on radial and circumferential stresses are examined in detailed and the results obtained show the effect of step load is more critical than exponential load.

Reinforcement learning (RL) has recently regained popularity, with major achievements such as beating the European game of Go champion. Here, for the first time, we show that RL can be used efficiently to train a spiking neural network (SNN) to perform object recognition in natural images without using an external classifier. We used a feedforward convolutional SNN and a temporal coding scheme where the most strongly activated neurons fire first, while less activated ones fire later, or not at all. In the highest layers, each neuron was assigned to an object category, and it was assumed that the stimulus category was the category of the first neuron to fire. If this assumption was correct, the neuron was rewarded, i.e. spike-timing-dependent plasticity (STDP) was applied, which reinforced the neuron's selectivity. Otherwise, anti-STDP was applied, which encouraged the neuron to learn something else. As demonstrated on various image datasets (Caltech, ETH-80, and NORB), this reward modulated STDP (R-STDP) approach extracted particularly discriminative visual features, whereas classic unsupervised STDP extracts any feature that consistently repeats. As a result, R-STDP outperformed STDP on these datasets. Furthermore, R-STDP is suitable for online learning, and can adapt to drastic changes such as label permutations. Finally, it is worth mentioning that both feature extraction and classification were done with spikes, using at most one spike per neuron. Thus the network is hardware friendly and energy efficient., Comment: supplementary materials are added, Caltech face/motorbike demonstration figure is updated, some parts of the main manuscript are moved to the supplementary materials, additional network analysis and performance comparison with deep nets are added

Previous studies have shown that spike-timing-dependent plasticity (STDP) can be used in spiking neural networks (SNN) to extract visual features of low or intermediate complexity in an unsupervised manner. These studies, however, used relatively shallow architectures, and only one layer was trainable. Another line of research has demonstrated - using rate-based neural networks trained with back-propagation - that having many layers increases the recognition robustness, an approach known as deep learning. We thus designed a deep SNN, comprising several convolutional (trainable with STDP) and pooling layers. We used a temporal coding scheme where the most strongly activated neurons fire first, and less activated neurons fire later or not at all. The network was exposed to natural images. Thanks to STDP, neurons progressively learned features corresponding to prototypical patterns that were both salient and frequent. Only a few tens of examples per category were required and no label was needed. After learning, the complexity of the extracted features increased along the hierarchy, from edge detectors in the first layer to object prototypes in the last layer. Coding was very sparse, with only a few thousands spikes per image, and in some cases the object category could be reasonably well inferred from the activity of a single higher-order neuron. More generally, the activity of a few hundreds of such neurons contained robust category information, as demonstrated using a classifier on Caltech 101, ETH-80, and MNIST databases. We also demonstrate the superiority of STDP over other unsupervised techniques such as random crops (HMAX) or auto-encoders. Taken together, our results suggest that the combination of STDP with latency coding may be a key to understanding the way that the primate visual system learns, its remarkable processing speed and its low energy consumption. These mechanisms are also interesting for artificial vision systems, particularly for hardware solutions.

Repeating spatiotemporal spike patterns exist and carry information. Here we investigated how a single spiking neuron can optimally respond to one given pattern (localist coding), or to either one of several patterns (distributed coding, i.e., the neuron’s response is ambiguous but the identity of the pattern could be inferred from the response of multiple neurons), but not to random inputs. To do so, we extended a theory developed in a previous paper (Masquelier, 2017), which was limited to localist coding. More specifically, we computed analytically the signal-to-noise ratio (SNR) of a multi-pattern-detector neuron, using a threshold-free leaky integrate-and-fire (LIF) neuron model with non-plastic unitary synapses and homogeneous Poisson inputs. Surprisingly, when increasing the number of patterns, the SNR decreases slowly, and remains acceptable for several tens of independent patterns. In addition, we investigated whether spike-timing-dependent plasticity (STDP) could enable a neuron to reach the theoretical optimal SNR. To this aim, we simulated a LIF equipped with STDP, and repeatedly exposed it to multiple input spike patterns, embedded in equally dense Poisson spike trains. The LIF progressively became selective to every repeating pattern with no supervision, and stopped discharging during the Poisson spike trains. Furthermore, tuning certain STDP parameters, the resulting pattern detectors were optimal. Tens of independent patterns could be learned by a single neuron using a low adaptive threshold, in contrast with previous studies, in which higher thresholds led to localist coding only. Taken together these results suggest that coincidence detection and STDP are powerful mechanisms, fully compatible with distributed coding. Yet we acknowledge that our theory is limited to single neurons, and thus also applies to feed-forward networks, but not to recurrent one

The physical and chemical quality of the public drinking water supply (tap water), bottled water, and point of use (PoU) treated water was studied comparatively. The analyzed parameters were: turbidity, electrical conductivity, total dissolved solids, pH, hardness, sodium, potassium, chloride, and alkalinity. The samples were taken and analyzed based on standard methods references for the examination of the water and wastewater. The data analysis was conducted by SPSS 16 software. The results show that the concentration of the chemical and physical parameters in all waters is below limits as allowed by national and international drinking water guidelines and standards, although there is significant difference between three types of water. The quality of the tap water is consistent to mentioned guidelines and standards; therefore, the tap water is safe for drinking and it is no need to use other water resources instead of this water. The distrust to public water supply has caused a large number of ...

Mixture of Experts is one of the most popular ensemble methods in pattern recognition systems. Although, diversity between the experts is one of the necessary conditions for the success of combining methods, ensemble systems based on Mixture of Experts suffer from the lack of enough diversity among the experts caused by unfavorable initial parameters. In the conventional Mixture of Experts, each expert receives the whole feature space. To increase diversity among the experts, solve the structural issues of Mixture of Experts such as zero coefficient problem, and improve efficiency in the system, we intend to propose a model, entitled Mixture of Feature Specified Experts, in which each expert gets a different subset of the original feature set. To this end, we first select a set of feature subsets which lead to a set of diverse and efficient classifiers. Then the initial parameters are infused to the system with training classifiers on the selected feature subsets. Finally, we train the expert and the gating networks using the learning rule of classical Mixture of Experts to organize collaboration between the members of system and aiding the gating network to find the best partitioning of the problem space. To evaluate our proposed method, we have used six datasets from the UCI repository. In addition the generalization capability of our proposed method is considered on real-world database of EEG based Brain-Computer Interface. The performance of our method is evaluated with various appraisal criteria and significant improvement in recognition rate of our proposed method is indicated in all practical tests.

Behavioral studies in humans indicate that peripheral vision can do object recognition to some extent. Moreover, recent studies have shown that some information from brain regions retinotopic to visual periphery is somehow fed back to regions retinotopic to the fovea and disrupting this feedback impairs object recognition in human. However, it is unclear to what extent the information in visual periphery contributes to human object categorization. Here, we designed two series of rapid object categorization tasks to first investigate the performance of human peripheral vision in categorizing natural object images at different eccentricities and abstraction levels (superordinate, basic, and subordinate). Then, using a delayed foveal noise mask, we studied how modulating the foveal representation impacts peripheral object categorization at any of the abstraction levels. We found that peripheral vision can quickly and accurately accomplish superordinate categorization, while its performance in finer categorization levels dramatically drops as the object presents further in the periphery. Also, we found that a 300-ms delayed foveal noise mask can significantly disturb categorization performance in basic and subordinate levels, while it has no effect on the superordinate level. Our results suggest that human peripheral vision can easily process objects at high abstraction levels, and the information is fed back to foveal vision to prime foveal cortex for finer categorizations when a saccade is made toward the target object.

We present a new classifier fusion method to combine soft-level classifiers with a new approach, which can be considered as a generalized decision templates method. Previous combining methods based on decision templates employ a single prototype for each class, but this global point of view mostly fails to properly represent the decision space. This drawback extremely affects the classification rate in such cases: insufficient number of training samples, island-shaped decision space distribution, and classes with highly overlapped decision spaces. To better represent the decision space, we utilize a prototype selection method to obtain a set of local decision prototypes for each class. Afterward, to determine the class of a test pattern, its decision profile is computed and then compared to all decision prototypes. In other words, for each class, the larger the numbers of decision prototypes near to the decision profile of a given pattern, the higher the chance for that class. The efficiency of our proposed method is evaluated over some well-known classification datasets suggesting superiority of our method in comparison with other proposed techniques.

In this article, the propagation of one-dimensional stress waves in a plate made of functionally graded materials excited by a harmonic force is studied. The material properties of the functionally graded material plate are assumed to be graded in the thickness direction according to a power law distribution in terms of the volume fractions of the constituents. The governing equations are based on stress–strain relation and the equation of motion. Keeping generality, the functionally graded material plate is assumed as a multilayer with linear material property in each layer while arbitrary exponential material property through the thickness. A plate made of aluminum and alumina is considered as an example to illustrate the effects of the volume fraction exponent and number of layers on the wave propagation characteristics. Results indicate that by changing the exponent values ( M), stress distribution can be controlled. Also at every certain power law ( M), there exist a number of layers beyond which no variation in stress can be detected on the plate response. Furthermore, the wave in time domain is also investigated and the effects of material distribution on the wave speed are examined.

Deformation modeling of an infinite plate of functionally graded materials (FGMs) loaded by normal force to the plate surface is studied. The material properties of FGM plate are assumed to be graded in the thickness direction according to a simple power-law distribution in terms of the volume fractions of the constituents. The governing equations are based on stress-strain relation and the equilibrium force equation. Keeping generality, FGM plate has been assumed as a multilayer with linear material property in each layer while arbitrary exponential material property through the thickness. A plate made of Aluminum and Alumina is considered as an example to illustrate the effects of the volume fraction exponent and number of layers on the plate deformation response. Effects of number of layers on the accuracy of the plate behavior under external load are examined. Results indicate that at every certain power-law (M), there exist a number of layers beyond which no variation can be detected on the plate deformation response.

Retinal image of surrounding objects varies tremendously due to the changes in position, size, pose, illumination condition, background context, occlusion, noise, and non-rigid deformations. But despite these huge variations, our visual system is able to invariantly recognize any object in just a fraction of a second. To date, various computational models have been proposed to mimic the hierarchical processing of the ventral visual pathway, with limited success. Here, we show that the association of both biologically inspired network architecture and learning rule significantly improves the models' performance when facing challenging invariant object recognition problems. Our model is an asynchronous feedforward spiking neural network. When the network is presented with natural images, the neurons in the entry layers detect edges, and the most activated ones fire first, while neurons in higher layers are equipped with spike timing-dependent plasticity. These neurons progressively become selective to intermediate complexity visual features appropriate for object categorization. The model is evaluated on 3D-Object and ETH-80 datasets which are two benchmarks for invariant object recognition, and is shown to outperform state-of-the-art models, including DeepConvNet and HMAX. This demonstrates its ability to accurately recognize different instances of multiple object classes even under various appearance conditions (different views, scales, tilts, and backgrounds). Several statistical analysis techniques are used to show that our model extracts class specific and highly informative features.