Search

Background: This paper addresses specific challenges in predictive modeling, namely transparency issues, susceptibility to data manipulation, and fairness concerns. To overcome these obstacles, the study introduces DeepRing, approach that combines Convolutional Neural Networks (CNNs) and blockchain technology. Objective: DeepRing aims to enhance prediction integrity, data security, and fairness, thereby improving the ethical considerations, reliability, and accountability of predictive models. Methods: involves iterative training of a CNN model on five diverse datasets, including CIFAR-10, Fashion-MNIST, MNIST, CIFAR-100, and a Hands dataset. The CNN architecture incorporates Conv2D layers, MaxPooling2D layers, and Dense layers. Training metrics such as accuracy and sparse categorical cross-entropy loss are monitored, with the Adam optimizer employed. While achieving high accuracy on Plam (0.5300), MNIST (0.9978) and Fashion MNIST (0.9673), DeepRing exhibits moderate performance on CIFAR-10 (0.9296) and lower accuracy on CIFAR-100 (0.5973). Results: demonstrate the effectiveness of DeepRing in improving accuracy and enhancing model performance across various datasets. However, further development and validation are essential for successful model implementation, further development and validation are essential for successful model implementation. Conclusions: Introduces DeepRing as an innovative solution to address key challenges in predictive modeling, specifically focusing on transparency issues, susceptibility to data manipulation, and fairness concerns. By combining Convolutional Neural Networks (CNNs) with blockchain technology, DeepRing aims to elevate prediction integrity, enhance data security, and promote fairness, thereby contributing to the improvement of ethical considerations, reliability, and accountability in predictive modelling. [ABSTRACT FROM AUTHOR]

In today's busy world, people are shifting to online shopping for their basic needs like groceries, clothes, etc. As a result, modern applications are incorporating image search as part of their application. However, retrieving accurate fashion images is challenging due to the low inter-class separability. In this paper, we aim to propose and implement three deep learning denoising autoencoder image retrieval models trained on Fashion-MNIST dataset and MNIST dataset. Fashion-MNIST is widely used datasets to evaluate model because of its low inter-class separability. It is difficult to get a good accuracy on Fashion-MNIST dataset. The proposed models train to learn significant features of fashion images and improve the precision of image retrieval. These models are compared based on the Label Ranking Average Precision (LRAP) values, with the third model bagging the highest LRAP value. These models also trained and tested on MNIST dataset, which have similar configuration as Fashion-MNIST dataset but with high inter-class separability. Intriguingly, the second and third models trained on MNIST dataset demonstrate neck-to-neck performance, both achieving comparable accuracy. This choice of dataset ensures that the models' accuracy is more noticeable and significant. The outcomes of this study validate the efficacy of denoising autoencoders in fashion image retrieval and emphasize their prospective in improving online shopping experiences. [ABSTRACT FROM AUTHOR]

In the rapidly changing world of new technologies, AI and ML have been delivered to the network's edge, allowing edge devices to compute simple models that can be applied later in practical applications. Edge computing allows the deployment of cloudlike services to the network edge. Moreover, it allows network devices to contribute to Federated Learning (FL) by using their processing power. Federated learning was developed when privacy and security concerns emerged among people. Here, multiple clients work together to solve machine-learning problems. Instead of collecting users' raw data, this technique protects user privacy by aggregating model parameters from each client to a central server. Even though FL techniques have several benefits, such as scalability and data privacy, when used with heterogeneous devices, they can introduce significant problems in processing complexity and speed. This paper thoroughly surveyed the important and efficient technological advancements based on edge federated learning. We review the widely used FL frameworks that effectively facilitate communication between clients and servers. We also present a case study on handwritten digit recognition using the MNIST dataset to state the effectiveness of federated learning. On applying this approach, we were able to achieve 94.56% accuracy. [ABSTRACT FROM AUTHOR]

Background: This paper addresses specific challenges in predictive modeling, namely transparency issues, susceptibility to data manipulation, and fairness concerns. To overcome these obstacles, the study introduces DeepRing, approach that combines Convolutional Neural Networks (CNNs) and blockchain technology. Objective: DeepRing aims to enhance prediction integrity, data security, and fairness, thereby improving the ethical considerations, reliability, and accountability of predictive models. Methods: involves iterative training of a CNN model on five diverse datasets, including CIFAR-10, Fashion-MNIST, MNIST, CIFAR-100, and a Hands dataset. The CNN architecture incorporates Conv2D layers, MaxPooling2D layers, and Dense layers. Training metrics such as accuracy and sparse categorical cross-entropy loss are monitored, with the Adam optimizer employed. While achieving high accuracy on Plam (0.5300), MNIST (0.9978) and Fashion MNIST (0.9673), DeepRing exhibits moderate performance on CIFAR-10 (0.9296) and lower accuracy on CIFAR-100 (0.5973). Results: demonstrate the effectiveness of DeepRing in improving accuracy and enhancing model performance across various datasets. However, further development and validation are essential for successful model implementation, further development and validation are essential for successful model implementation. Conclusions: Introduces DeepRing as an innovative solution to address key challenges in predictive modeling, specifically focusing on transparency issues, susceptibility to data manipulation, and fairness concerns. By combining Convolutional Neural Networks (CNNs) with blockchain technology, DeepRing aims to elevate prediction integrity, enhance data security, and promote fairness, thereby contributing to the improvement of ethical considerations, reliability, and accountability in predictive modelling.

Nowadays, deep learning methods are widely used in many fields such as robotics, sound processing, medicine and imagery. In this study, studies on image classification and analysis in the literature were examined in detail. In addition, performance analyzes of deep learning methods were carried out using CNN on 4 different data sets named MNIST, Fashion MNIST, CIFAR-10 and CIFAR-100. The structures of the created deep learning models, the parameter values used for training, the layers used, the complexity matrices obtained for the validation data, accuracy and loss graphs are shown in detail. Our study was carried out with a different network structure consisting of 3 convolution layers, 3 batch normalization layers, 2 maxpooling layers, 1 flatten, 2 droupout, 2 dense layers. In addition, the performance of image classification on different data has been increased with our proposed model. At the end of the test, the accuracy results of our study were compared according to various evaluation metrics. For all the data sets used, the best and worst images were determined. With the proposed CNN model, high accuracy rates were observed for MNIST and Fashion MNIST datasets, with these values being 99.22% and 99.21%, respectively. [ABSTRACT FROM AUTHOR]

The technology of handwritten digit and character recognition is the process of identifying handwritten numbers using computers or other devices, machine train itself to recognize the handwritten digits or characters from various sources like bank cheque, mails, images, etc. This paper is about CNN, Logistic Regression, and Random Forest algorithm in handwritten digit or character recognition system, the system works on MNIST dataset for training and testing the models, to get the best accuracy this work rewrites CNN, Logistic Regression, and Random Forest with python libraries. Finally, these algorithms are analyzed by comparing the accuracy and recognition duration. Where CNN got better accuracy than other algorithms. [ABSTRACT FROM AUTHOR]

Interpretation of neural activity in response to stimulations received from the surrounding environment is necessary to realize automatic brain decoding. Analyzing the brain recordings corresponding to visual stimulation helps to infer the effects of perception occurring by vision on brain activity. In this paper, the impact of arithmetic concepts on vision-related brain records has been considered and an efficient convolutional neural network-based generative adversarial network (CNN-GAN) is proposed to map the electroencephalogram (EEG) to salient parts of the image stimuli. The first part of the proposed network consists of depth-wise one-dimensional convolution layers to classify the brain signals into 10 different categories according to Modified National Institute of Standards and Technology (MNIST) image digits. The output of the CNN part is fed forward to a fine-tuned GAN in the proposed model. The performance of the proposed CNN part is evaluated via the visually provoked 14-channel MindBigData recorded by David Vivancos, corresponding to images of 10 digits. An average accuracy of 95.4% is obtained for the CNN part for classification. The performance of the proposed CNN-GAN is evaluated based on saliency metrics of SSIM and CC equal to 92.9% and 97.28%, respectively. Furthermore, the EEG-based reconstruction of MNIST digits is accomplished by transferring and tuning the improved CNN-GAN's trained weights. [ABSTRACT FROM AUTHOR]

Introduction: Spiking neural networks (SNNs) are a model of computation that mimics the behavior of biological neurons. SNNs process event data (spikes) and operate more sparsely than artificial neural networks (ANNs), resulting in ultra-low latency and small power consumption. This paper aims to adapt and evaluate gradient-based explainability methods for SNNs, which were originally developed for conventional ANNs. Methods: The adapted methods aim to create input feature attribution maps for SNNs trained through backpropagation that process either event-based spiking data or real-valued data. The methods address the limitations of existing work on explainability methods for SNNs, such as poor scalability, limited to convolutional layers, requiring the training of another model, and providing maps of activation values instead of true attribution scores. The adapted methods are evaluated on classification tasks for both real-valued and spiking data, and the accuracy of the proposedmethods is confirmed through perturbation experiments at the pixel and spike levels. Results and discussion: The results reveal that gradient-based SNN attribution methods successfully identify highly contributing pixels and spikes with significantly less computation time than model-agnostic methods. Additionally, we observe that the chosen coding technique has a noticeable effect on the input features that will be most significant. These findings demonstrate the potential of gradient-based explainability methods for SNNs in improving our understanding of how these networks process information and contribute to the development of more efficient and accurate SNNs. [ABSTRACT FROM AUTHOR]

In neuromorphic computing, the coding method of spiking neurons serves as the foundation and is crucial for various aspects of network operation. Existing mainstream coding methods, such as rate coding and temporal coding, have different focuses, and each has its own advantages and limitations. This paper proposes a novel coding scheme called activeness coding that integrates the strengths of both rate and temporal coding methods. It encompasses precise timing information of the most recent neuronal spike as well as the historical firing rate information. The results of basic characteristic tests demonstrate that this encoding method accurately expresses input information and exhibits robustness. Furthermore, an unsupervised learning method based on activeness-coding triplet spike-timing dependent plasticity (STDP) is introduced, with the MNIST classification task used as an example to assess the performance of this encoding method in solving cognitive tasks. Test results show an improvement in accuracy of approximately 4.5%. Additionally, activeness coding also exhibits potential advantages in terms of resource conservation. Overall, activeness offers a promising approach for spiking neural network encoding with implications for various applications in the field of neural computation. [ABSTRACT FROM AUTHOR]

TADS are a novel, concise white-box representation of neural networks. In this paper, we apply TADS to the problem of neural network verification, using them to generate either proofs or concise error characterizations for desirable neural network properties. In a case study, we consider the robustness of neural networks to adversarial attacks, i.e., small changes to an input that drastically change a neural networks perception, and show that TADS can be used to provide precise diagnostics on how and where robustness errors a occur. We achieve these results by introducing Precondition Projection, a technique that yields a TADS describing network behavior precisely on a given subset of its input space, and combining it with PCA, a traditional, well-understood dimensionality reduction technique. We show that PCA is easily compatible with TADS. All analyses can be implemented in a straightforward fashion using the rich algebraic properties of TADS, demonstrating the utility of the TADS framework for neural network explainability and verification. While TADS do not yet scale as efficiently as state-of-the-art neural network verifiers, we show that, using PCA-based simplifications, they can still scale to medium-sized problems and yield concise explanations for potential errors that can be used for other purposes such as debugging a network or generating new training samples. [ABSTRACT FROM AUTHOR]

Advanced artificial intelligence (AI) techniques have led to significant developments in optical character recognition (OCR) technologies. OCR applications, using AI techniques for transforming images of typed text, handwritten text, or other forms of text into machine-encoded text, provide a fair degree of accuracy for general text. However, even after decades of intensive research, creating OCR with human-like abilities has remained evasive. One of the challenges has been that OCR models trained on general text do not perform well on localized or personalized handwritten text due to differences in the writing style of alphabets and digits. This study aims to discuss the steps needed to create an adaptive framework for OCR models, with the intent of exploring a reasonable method to customize an OCR solution for a unique dataset of English language numerical digits were developed for this study. We develop a digit recognizer by training our model on the MNIST dataset with a convolutional neural network and contrast it with multiple models trained on combinations of the MNIST and custom digits. Using our methods, we observed results comparable with the baseline and provided recommendations for improving OCR accuracy for localized or personalized handwritten text. This study also provides an alternative perspective to generating data using conventional methods, which can serve as a gold standard for custom data augmentation to help address the challenges of scarce data and data imbalance. [ABSTRACT FROM AUTHOR]

The paper considers the problem of modelling the distribution of data with noise in the input data. In this paper, we consider encoders and decoders, which solve the problem of modelling data distribution. The improvement of variational autoencoders (VAEs) is discussed. Practical implementation is performed using the Python programming language and the Keras framework. Generative adversarial networks (GANs) and VAEs with noisy data are demonstrated. [ABSTRACT FROM AUTHOR]

Introduction/purpose: This paper provides a survey of handwritten digit recognition methods tested on the MNIST dataset. Methods: The paper analyzes, synthesizes and compares the development of different classifiers applied to the handwritten digit recognition problem, from linear classifiers to convolutional neural networks. Results: Handwritten digit recognition classification accuracy tested on the MNIST dataset while using training and testing sets is now higher than 99.5% and the most successful method is a convolutional neural network. Conclusions: Handwritten digit recognition is a problem with numerous real-life applications. Accurate recognition of various handwriting styles, specifically digits is a task studied for decades and this paper summarizes the achieved results. The best results have been achieved with convolutional neural networks while the worst methods are linear classifiers. The convolutional neural networks give better results if the dataset is expended with data augmentation.

Stephen Wolfram (2002) proposed the concept of computational equivalence, which implies that almost any dynamical system can be considered as a computation, including programmable matter and nonlinear materials such as, so called, quantum matter. Memristors are often used in building and evaluating hardware neural networks. Ukil (2011) demonstrated a theoretical relationship between piezoelectrical materials and memristors. We review that work as a necessary background prior to our work on exploring a piezoelectric material for neural network computation. Our method consisted of using a cubic block of unpoled lead zirconate titanate (PZT) ceramic, to which we have attached wires for programming the PZT as a programmable substrate. We then, by means of pulse trains, constructed on-the-fly internal patterns of regions of aligned polarization and unaligned, or disordered regions. These dynamic patterns come about through constructive and destructive interference and may be exploited as a type of reservoir network. Using MNIST data we demonstrate a learning machine.

In this article, both machine learning techniques and deep learning methods were applied on the digit datasets created using the Arabic and Latin alphabets, and the performances of the methods were compared. Each method was tested with various parameters and the results were analyzed. In addition, with this study, the recognizability of handwritten numeral datasets created using different alphabets was also observed. For experiments, an Arabic alphabet handwritten digit dataset (60,000 training and 10,000 testings) and a Latin alphabet handwritten digit dataset (60,000 training and 10,000 testings) were used. When the results of the experiment are examined, it is seen that successful results are obtained in the classification made with the MADBase dataset in some methods and in the classification made with the MNIST dataset in some methods. As a result, it can be stated that the handwriting character recognition success of a method cannot be measured only by the classification made on a dataset. Also, the digits written in the Arabic alphabet appear to be almost more recognizable than the digits written in the Latin alphabet.

Capsule Networks (CapsNets) is a great approach for understanding data in the field of computer vision. CapsNets allow a deeper understanding of images compared to the traditional Convolutional Neural Networks. The first test for CapsNet was in digits recognition on the 'MNIST' dataset, where it successfully achieved high accuracy. CapsNets are reliable at deciphering overlapping digits. Deep Capsule Networks achieved state-of-the-art accuracy in CIFAR10 which isn't achieved by shallow capsule networks. Despite all these accomplishments, Deep Capsule Networks are very slow due to the 'Dynamic Routing' algorithm. In this paper, Fast Embedded Capsule Network and Deep Fast Embedded Capsule Network are introduced, representing novel capsule network architectures that uses 1D convolution based dynamic routing with a fast element-wise multiplication transformation process. These architectures not only compete with the state-of-the-art methods in terms of accuracy in the capsule domain, but also excels in terms of speed, and reduced complexity. This is shown by the 58% reduction in the number of trainable parameters and 64% reduction in the average epoch time in the training process. Experimental results shows excellent and verified properties. [ABSTRACT FROM AUTHOR]

Image classification, a pivotal task in multiple industries, faces computational challenges due to the burgeoning volume of visual data. This research addresses these challenges by introducing two quantum machine learning models that leverage the principles of quantum mechanics for effective computations. Our first model, a hybrid quantum neural network with parallel quantum circuits, enables the execution of computations even in the noisy intermediate-scale quantum era, where circuits with a large number of qubits are currently infeasible. This model demonstrated a record-breaking classification accuracy of 99.21% on the full MNIST dataset, surpassing the performance of known quantum–classical models, while having eight times fewer parameters than its classical counterpart. Also, the results of testing this hybrid model on a Medical MNIST (classification accuracy over 99%), and on CIFAR-10 (classification accuracy over 82%), can serve as evidence of the generalizability of the model and highlights the efficiency of quantum layers in distinguishing common features of input data. Our second model introduces a hybrid quantum neural network with a Quanvolutional layer, reducing image resolution via a convolution process. The model matches the performance of its classical counterpart, having four times fewer trainable parameters, and outperforms a classical model with equal weight parameters. These models represent advancements in quantum machine learning research and illuminate the path towards more accurate image classification systems.

Interpretation of neural activity in response to stimulations received from the surrounding environment is necessary to realize automatic brain decoding. Analyzing the brain recordings corresponding to visual stimulation helps to infer the effects of perception occurring by vision on brain activity. In this paper, the impact of arithmetic concepts on vision-related brain records has been considered and an efficient convolutional neural network-based generative adversarial network (CNN-GAN) is proposed to map the electroencephalogram (EEG) to salient parts of the image stimuli. The first part of the proposed network consists of depth-wise one-dimensional convolution layers to classify the brain signals into 10 different categories according to Modified National Institute of Standards and Technology (MNIST) image digits. The output of the CNN part is fed forward to a fine-tuned GAN in the proposed model. The performance of the proposed CNN part is evaluated via the visually provoked 14-channel MindBigData recorded by David Vivancos, corresponding to images of 10 digits. An average accuracy of 95.4% is obtained for the CNN part for classification. The performance of the proposed CNN-GAN is evaluated based on saliency metrics of SSIM and CC equal to 92.9% and 97.28%, respectively. Furthermore, the EEG-based reconstruction of MNIST digits is accomplished by transferring and tuning the improved CNN-GAN’s trained weights.

The convolutional neural network (CNN) is a technique that is often used in deep learning. Various models have been proposed and improved for learning on CNN. When learning with CNN, it is important to determine the optimal parameters. This paper proposes an optimization of CNN parameters using logarithm decreasing inertia weight (LogDIW). This paper is used two datasets, i.e., MNIST and CIFAR-10 dataset. The MNIST learning experiment, the CIFAR-10 dataset, compared its accuracy with the CNN standard based on the LeNet-5 architectural model. When using the MNIST dataset, CNN's baseline was 94.02% at the 5th epoch, compared to CNN's LogDIWPSO, which improves accuracy. When using the CIFAR-10 dataset, the CNN baseline was 28.07% at the 10th epoch, compared to the LogDIWPSO CNN accuracy of 69.3%, which increased the accuracy.

Introduction/purpose: This paper provides a survey of handwritten digit recognition methods tested on the MNIST dataset. Methods: The paper analyzes, synthesizes and compares the development of different classifiers applied to the handwritten digit recognition problem, from linear classifiers to convolutional neural networks. Results: Handwritten digit recognition classification accuracy tested on the MNIST dataset while using training and testing sets is now higher than 99.5% and the most successful method is a convolutional neural network. Conclusions: Handwritten digit recognition is a problem with numerous real-life applications. Accurate recognition of various handwriting styles, specifically digits is a task studied for decades and this paper summarizes the achieved results. The best results have been achieved with convolutional neural networks while the worst methods are linear classifiers. The convolutional neural networks give better results if the dataset is expended with data augmentation. [ABSTRACT FROM AUTHOR]

Stephen Wolfram (2002) proposed the concept of computational equivalence, which implies that almost any dynamical system can be considered as a computation, including programmable matter and nonlinear materials such as, so called, quantum matter. Memristors are often used in building and evaluating hardware neural networks. Ukil (2011) demonstrated a theoretical relationship between piezoelectrical materials and memristors. We review that work as a necessary background prior to our work on exploring a piezoelectric material for neural network computation. Our method consisted of using a cubic block of unpoled lead zirconate titanate (PZT) ceramic, to which we have attached wires for programming the PZT as a programmable substrate. We then, by means of pulse trains, constructed on-the-fly internal patterns of regions of aligned polarization and unaligned, or disordered regions. These dynamic patterns come about through constructive and destructive interference and may be exploited as a type of reservoir network. Using MNIST data we demonstrate a learning machine. [ABSTRACT FROM AUTHOR]

Deep neural networks, which is a field of artificial intelligence, have been used in various fields. Deep learning is processed on high-performance GPUs or TPUs. It requires high cost as much as its good performance. Recently, as the demand for edge computing increases, many studies have been conducted to perform complex deep learning operations in a low-computing processor. Among them, a typical study is to lighten the deep learning network. In this paper, we propose a handwritten digit recognition hardware accelerator suitable for edge computing using MNIST database. After setting the correction rate for MNIST to 94% and performing network lighting processes, a hardware structure that can reduce the area of hardware and minimize memory access is proposed. Basically, the network is set as a two-layer fully connected network. The network is modeled with Python and lighten while checking the performance. Network parameters, weighs and biases, are quantized. The pixel number and bit number of MNIST input data are also reduced. The number of MAC units and the processing order of the hardware accelerator are determined so that there is no not used MACs while performing the MAC operations in parallel. It is designed with Verilog HDL and its functions are checked in Modelsim. And then it is implemented in Xilinx Zynq ZC-702 to verify the operations. The designed number recognition accelerator is expected to be widely used in edge devices by reducing the area and memory access.

Handwritten character recognition is a computer-vision-system problem that is still critical and challenging in many computer-vision tasks. With the increased interest in handwriting recognition as well as the developments in machine-learning and deep-learning algorithms, researchers have made significant improvements and advances in developing English-handwriting-recognition methodologies; however, Arabic handwriting recognition has not yet received enough interest. In this work, several deep-learning and hybrid models were created. The methodology of the current study took advantage of machine learning in classification and deep learning in feature extraction to create hybrid models. Among the standalone deep-learning models trained on the two datasets used in the experiments performed, the best results were obtained with the transfer-learning model on the MNIST dataset, with 0.9967 accuracy achieved. The results for the hybrid models using the MNIST dataset were good, with accuracy measures exceeding 0.9 for all the hybrid models; however, the results for the hybrid models using the Arabic character dataset were inferior. [ABSTRACT FROM AUTHOR]

The MNIST dataset is considered a challenging problem for machine learning algorithms. The present paper introduces a novel approach based on a truncated SVD and kernel density estimation that outputs an error rate on MNIST comparable to classical machine learning approaches. In addition to that, a hierarchical classification framework is proposed, that allows to enhance the algorithm accuracy. The resulting algorithm outperforms the reproducible SVM accuracy, which is regarded as a state-of-the-art machine learning algorithm for MNIST. The key advantage of the proposed framework consists in a lower computational cost: training on MNIST takes 142 times less and prediction takes 4 times less than for SVM on a single run. Thus, the research results state that TSVD-KDE algorithm has the potential for being an efficient classification algorithm. The source code for the experiment is released on GitHub: https://github.com/ekplesovskaya/MNIST-Classification-Using-TSVD-and-KDE. [ABSTRACT FROM AUTHOR]

This work reports 2-bits/cell hafnium oxide-based stacked resistive random access memory devices fabricated on flexible polyimide substrates for neuromorphic applications considering the high thermal budget. The ratio of low-resistance state current (${I}_{ \mathrm{\scriptscriptstyle ON}}$) to high-resistance state current (${I}_{ \mathrm{\scriptscriptstyle OFF}}$) or ${I}_{ \mathrm{\scriptscriptstyle ON}}/{I}_{ \mathrm{\scriptscriptstyle OFF}}$ for the fabricated devices was above $1.4\times10$ 3 with a low device-to-device variation at $100 \boldsymbol {\mu }\text{A}$ current compliance. The mechanical stability over 104 bending cycles at a 5 mm bending radius and endurance over 106 WRITE cycles makes these devices suitable for online neural network training. The data retention capability over 104s at 125°C also infuses these devices’ long-term inference capability. Furthermore, the performance of the devices has been verified for neuromorphic applications by system-level simulations with experimentally calibrated data. The system-level simulation reveals only a 2% loss in inference accuracy over ten years from the baseline. [ABSTRACT FROM AUTHOR]

The paper considers the problem of modelling the distribution of data with noise in the input data. In this paper, we consider encoders and decoders, which solve the problem of modelling data distribution. The improvement of variational autoencoders (VAEs) is discussed. Practical implementation is performed using the Python programming language and the Keras framework. Generative adversarial networks (GANs) and VAEs with noisy data are demonstrated.

Advanced artificial intelligence (AI) techniques have led to significant developments in optical character recognition (OCR) technologies. OCR applications, using AI techniques for transforming images of typed text, handwritten text, or other forms of text into machine-encoded text, provide a fair degree of accuracy for general text. However, even after decades of intensive research, creating OCR with human-like abilities has remained evasive. One of the challenges has been that OCR models trained on general text do not perform well on localized or personalized handwritten text due to differences in the writing style of alphabets and digits. This study aims to discuss the steps needed to create an adaptive framework for OCR models, with the intent of exploring a reasonable method to customize an OCR solution for a unique dataset of English language numerical digits were developed for this study. We develop a digit recognizer by training our model on the MNIST dataset with a convolutional neural network and contrast it with multiple models trained on combinations of the MNIST and custom digits. Using our methods, we observed results comparable with the baseline and provided recommendations for improving OCR accuracy for localized or personalized handwritten text. This study also provides an alternative perspective to generating data using conventional methods, which can serve as a gold standard for custom data augmentation to help address the challenges of scarce data and data imbalance.

To improve the accuracy of the parallax maps obtained by the binocular stereo matching network for better 3D reconstruction, this thesis is based on the AnyNet network, with the corresponding optimization to get MyNet. The Hourglass structure allows for the acquisition of multiscale image information. The original 3D Conv is replaced by a 3D residual block in the parallax network to avoid the gradient disappearance or explosion and model degradation problems of traditional convolutional neural network models, thus further improving the cost volume obtained. The experiments were tested on the KITTI2012 and KITTI2015 datasets and showed some improvement compared to the original algorithm. The experimental results show that the accuracy of the parallax map has improved significantly with almost the same time spent. [ABSTRACT FROM AUTHOR]

With the development of deep learning technology, handwritten character recognition has become the basic hotspot of research. At the same time, the recognition of handwriting in minority languages is gradually worthy of attention. In this paper, a robust and novel improved deep convolutional neural network method is proposed to recognize normal digital handwriting images and minority digital handwritten images (Yi language). To improve the recognition accuracy of handwritten fonts, there are four improvements in our work. Firstly, separable convolution is adopted in the network to cope with the problem of large parameters. Secondly, data augmentation techniques (Gaussian noise, color perturbation, and random rotation) are utilized to improve the expression ability of the dataset and the generalization capabilities of the model. Finally, the shortcut block and the weight initialization are introduced in our network for the problems of network degradation and gradient saturation. Based on deep learning technology, this paper studies the classification and recognition of handwritten digits, and improves the accuracy of the recognition in numbers such as invoices in financial bills. The recognition accuracy of 99.82% has been achieved on MNIST dataset, and 98.79% on the self-building dataset of Yi handwritten digits (YHDD). [ABSTRACT FROM AUTHOR]

Neural network classifiers (NNCs) are known to be vulnerable to malicious adversarial perturbations of inputs including those modifying a small fraction of the input features named sparse or L0 attacks. Effective and fast L0 attacks, such as the widely used Jacobian-based Saliency Map Attack (JSMA) are practical to fool NNCs but also to improve their robustness. In this paper, we show that penalising saliency maps of JSMA by the output probabilities and the input features of the NNC leads to more powerful attack algorithms that better take into account each input’s characteristics. This leads us to introduce improved versions of JSMA, named Weighted JSMA (WJSMA) and Taylor JSMA (TJSMA), and demonstrate through a variety of white-box and black-box experiments on three different datasets (MNIST, CIFAR-10 and GTSRB), that they are both significantly faster and more efficient than the original targeted and non-targeted versions of JSMA. Experiments also demonstrate, in some cases, very competitive results of our attacks in comparison with the Carlini-Wagner (CW) L0 attack, while remaining, like JSMA, significantly faster (WJSMA and TJSMA are more than 50 times faster than CW L0 on CIFAR-10). Therefore, our new attacks provide good trade-offs between JSMA and CW for L0 real-time adversarial testing on datasets such as the ones previously cited.

An approach is presented that generalize OCR task including polish letters using deep learning technique. The paper extends EMNIST dataset so that two new classes of polish diacritics "Ą" and "Ć" are attached. Using this new dataset and deep learning technique one can analyze sensitives of standard as well as presented extended approach with different parameters of convolutional neural network. The analysis of the results shows that the shadows and noises that Polish characters, leave with their hooks leads, can be properly recognized and two similar letters "A" and "Ą" can be distinguished by convolutional neural network. On the other hand a neural network trained on dataset without Polish characters do not treat letters "Ą" and "Ć" properly. [ABSTRACT FROM AUTHOR]

Las redes neuronales convolucionales cada vez son más populares en aplicaciones de aprendizaje profundo, como por ejemplo en clasificación de imágenes, reconocimiento de voz, medicina, entre otras. Sin embargo, estas redes son computacionalmente costosas y requieren altos recursos de memoria. En este trabajo se propone un acelerador en hardware para el proceso de inferencia de la red Lenet-5, un esquema de coprocesamiento hardware/software. El objetivo de la implementación es reducir el uso de recursos de hardware y obtener el mejor rendimiento computacional posible durante el proceso de inferencia. El diseño fue implementado en la tarjeta de desarrollo Digilent Arty Z7-20, la cual está basada en el System on Chip (SoC) Zynq-7000 de Xilinx. Nuestra implementación logró una precisión del 97,59 % para la base de datos MNIST utilizando tan solo 12 bits en el formato de punto fijo. Los resultados muestran que el esquema de co-procesamiento, el cual opera a una velocidad de 100 MHz, puede identificar aproximadamente 441 imágenes por segundo, que equivale aproximadamente a un 17% más rápido que una implementación de software a 650 MHz. Es difícil comparar nuestra implementación con otras implementaciones similares, porque las implementaciones encontradas en la literatura no son exactamente como la que realizó en este trabajo. Sin embargo, algunas comparaciones, en relación con el uso de recursos lógicos y la precisión, sugieren que nuestro trabajo supera a trabajos previos.

The article is an overview. We carry out the comparison of actual machine learning libraries that can be used the neural networks development. The first part of the article gives a brief description of TensorFlow, PyTorch, Theano, Keras, SciKit Learn libraries, SciPy library stack. An overview of the scope of these libraries and the main technical characteristics, such as performance, supported programming languages, the current state of development is given. In the second part of the article, a comparison of five libraries is carried out on the example of a multilayer perceptron, which is applied to the problem of handwritten digits recognizing. This problem is well known and well suited for testing different types of neural networks. The study time is compared depending on the number of epochs and the accuracy of the classifier. The results of the comparison are presented in the form of graphs of training time and accuracy depending on the number of epochs and in tabular form.

The purpose of this paper is to compare classical machine learning algorithms for handwritten number classification. The following algorithms were chosen for comparison: Logistic Regression, SVM, Decision Tree, Random Forest and k-NN. MNIST handwritten digit database is used in the task of training and testing the above algorithms. The dataset consists of 70,000 images of numbers from 0 to 9. The algorithms are compared considering such criteria as the learning speed, prediction construction speed, host machine load, and classification accuracy. Each algorithm went through the training and testing phases 100 times, with the desired KPIs retained at each iteration. The results were averaged to reach reliable outcomes.

Spiking neural network (SNN), known as the third generation of neural networks, is attracting more and more researchers’ attention because of its high energy efficiency. However, due to the spatiotemporal characteristics, SNN involved a few complicated computations such as exponential function and logarithmic function, making it hard to implement on hardware. In this brief, we propose a SNN processor, which balances the trade-offs between cost and performance. In terms of cost, through software and hardware co-design, a high-robustness SNN model with 2-bit weights and a novel synapse delay management mechanism are adopted to reduce memory utilization. In terms of performance, a spike encoder and a VFA (Vote-For-All) decoder are used to reduce latency and improve inference accuracy respectively. We implement the design on the Xilinx ZCU102 FPGA board and apply it to the MNIST handwritten digit classification, which achieves 90.53% classification accuracy. Compared to a previous proposed SNN processor of a similar neuron scale, our design achieves a 156 × inference speed-up and consumes 0,32 × hardware resources. [ABSTRACT FROM AUTHOR]

Handwritten digit recognition allows the computer to recognize the digits written by humans. In general, it is a hard task for the computers to recognize the digits because the handwriting varies with the person and there is no standard handwriting. In some languages the letters overlap making the computer hard to recognize the characters. So, Using Handwritten digit recognition technique we train the machine with different handwritten digits making the computer task easy. In this project a Convolutional Neural Network model was built. Inorder to train the model MNIST dataset is used as it consists of 60,000 training datasets. A graphical user interface is created which captures the image placed in front of the camera and passes the image as the input to the model so that the model predicts the output and the output is displayed in the result section of the graphical user interface. [ABSTRACT FROM AUTHOR]

Handwritten character recognition is a computer-vision-system problem that is still critical and challenging in many computer-vision tasks. With the increased interest in handwriting recognition as well as the developments in machine-learning and deep-learning algorithms, researchers have made significant improvements and advances in developing English-handwriting-recognition methodologies; however, Arabic handwriting recognition has not yet received enough interest. In this work, several deep-learning and hybrid models were created. The methodology of the current study took advantage of machine learning in classification and deep learning in feature extraction to create hybrid models. Among the standalone deep-learning models trained on the two datasets used in the experiments performed, the best results were obtained with the transfer-learning model on the MNIST dataset, with 0.9967 accuracy achieved. The results for the hybrid models using the MNIST dataset were good, with accuracy measures exceeding 0.9 for all the hybrid models; however, the results for the hybrid models using the Arabic character dataset were inferior.

With the growing demand for water resources, the need for monitoring has become a necessity for rational and sustainable use of this resource. Water meter data collection is an essential step toward this goal. In Morocco, this task is performed manually at most once a month due to constraints related to the cost and time. In general, the consumption is estimated and calculated based on the average consumption recorded in the previous months. This causes many claims from customers because of higher invoices, which does not reflect reality. In this paper, we propose a fully AI-based system to automate water meter data collection, which is composed of a Recognition System (RS) and a web services platform. This framework offers multiple services for both customers and water service providers, such as consumption monitoring, detecting water leaks, visualizing water consumption, and potable water coverage in a geographic map. It also provides a powerful tool to help ensure accurate decision making with multiple reporting services. The main component of the RS is the Convolutional Neural Network model trained on a proposed MR-AMR (Moroccan Automatic Meter Reading) dataset. In the model test phase, we achieved an accuracy of 98.70%. Our system was tested and validated by experiments.

The aim of this research is to derive new, more effective equations to adjust parameters of a neural network, which have advantage over formulas of the backpropagation method. For this purpose, an objective function of the neural network training is proposed to be modified using expansion of the abstract ideal output of each network layer into a Taylor series. Such decomposition is to be implemented at the point, which is close to the current output of the layer under consideration. The modified objective function gradients are calculated with respect to the difference between ideal and real values of the network parameters. As a result, the obtained weights and biases adjustment equations (for each layer) depend not only on the output of the previous layer, but also on its derivatives. Comparing to the backpropagation method, they allow to reduce the offline neural network training time. The derived formulas can be reduced to the ones of the backpropagation method. This proves the correctness of their development. The efficiency of the new training equations is shown using MNIST handwritten character recognition task. Considering the multilayer structures with different activation functions and number of layers, as far as new formulas are concerned, the same recognition accuracy is achieved, but faster comparing to the conventional backpropagation. [ABSTRACT FROM AUTHOR]

The analysis of convolutional neural model is done. The software is developed, allowing to train and test convolutional neural networks of base architecture LeNet-5. Efficiency of technique multi training and distortions of training images is shown. The qualifier of images of the isolated figures is constructed. The estimation of stability of its characteristics on examples of known hand-written and font databases is done.

The stability problem of memristor may affect the performance of memristive neural network. In order to explore it, a memristive back propagation(BP) neural network, in which the memristors are the synapses, is constructed based on the equivalent resistance topology memristor model. And it’s trained and tested on the MNIST dataset. The stability problem of memristor is simulated by setting fluctuations of the parameters in the model. Finally, it is found that the performance fluctuation of the memristor to a certain extent will promote the convergence of the neural network, but the excessive fluctuation will reduce the convergence speed of the network. To characterize this criticality, the maximum fluctuation range of each parameter in the model is obtained. Meanwhile, the fluctuations’ ranges of device parameters are obtained by tracing back to the memristor model. The result provides a reference for the fabrication and selection of memristor devices during hardware application.

Recently, many applications are developed in numerous domains with various environments. Since some environments require hands-free applications, new technology is needed for the input interfaces other than the mouse and keyboard. Therefore, to meet the needs, many researchers have begun to investigate the gaze and voice for the input technology. In particular, there are many approaches to render virtual keyboards with the gaze. However, since the virtual keyboards hide the screen space, this technique can only be applied in limited environments. In this paper, we propose a classification technique for gaze-written numbers as the hands-free interface. Since the gaze-writing is less accurate compared to the virtual keyboard typing, we apply the convolutional neural network (CNN) deep learning algorithm to recognize the gaze-writing and improve the classification accuracy. Besides, we create new gaze-writing datasets for training, gaze MNIST (gMNIST), by modifying the MNIST data with features of the gaze movement patterns. For the evaluation, we compare our approach with the basic CNN structures using the original MNIST dataset. Our study will allow us to have more options for the input interfaces and expand our choices in hands-free environments.

A new spintronic nonvolatile memory cell analogous to 1T DRAM with non-destructive READ is proposed. The cells can be used as neural computing units. A dual-circuit neural network architecture is proposed to leverage these devices against the complex operations involved in convolutional networks. Simulations based on HSPICE and MATLAB were performed to study the performance of this architecture when classifying images as well as the effect of varying the size and stability of the nanomagnets. The spintronic cells outperform a purely charge-based implementation of the same network, consuming ≈100-pJ total energy per image processed.

In this paper we present a Competitive Rate-Based Algorithm (CRBA) that approximates operation of a Competitive Spiking Neural Network (CSNN). CRBA is based on modeling of the competition between neurons during a sample presentation, which can be reduced to ranking of the neurons based on a dot product operation and the use of a discrete Expectation Maximization algorithm; the latter is equivalent to the spike time-dependent plasticity rule. CRBA's performance is compared with that of CSNN on the MNIST and Fashion-MNIST datasets. The results show that CRBA performs on par with CSNN, while using three orders of magnitude less computational time. Importantly, we show that the weights and firing thresholds learned by CRBA can be used to initialize CSNN's parameters that results in its much more efficient operation.

Spiking neural networks (SNNs) are a computational tool in which the information is coded into spikes, as in some parts of the brain, differently from conventional neural networks (NNs) that compute over real-numbers. Therefore, SNNs can implement intelligent information extraction in real-time at the edge of data acquisition and correspond to a complementary solution to conventional NNs working for cloud-computing. Both NN classes face hardware constraints due to limited computing parallelism and separation of logic and memory. Emerging memory devices, like resistive switching memories, phase change memories, or memristive devices in general are strong candidates to remove these hurdles for NN applications. The well-established training procedures of conventional NNs helped in defining the desiderata for memristive device dynamics implementing synaptic units. The generally agreed requirements are a linear evolution of memristive conductance upon stimulation with train of identical pulses and a symmetric conductance change for conductance increase and decrease. Conversely, little work has been done to understand the main properties of memristive devices supporting efficient SNN operation. The reason lies in the lack of a background theory for their training. As a consequence, requirements for NNs have been taken as a reference to develop memristive devices for SNNs. In the present work, we show that, for efficient CMOS/memristive SNNs, the requirements for synaptic memristive dynamics are very different from the needs of a conventional NN. System-level simulations of a SNN trained to classify hand-written digit images through a spike timing dependent plasticity protocol are performed considering various linear and non-linear plausible synaptic memristive dynamics. We consider memristive dynamics bounded by artificial hard conductance values and limited by the natural dynamics evolution toward asymptotic values (soft-boundaries). We quantitatively analyze the impact of resolution and non-linearity properties of the synapses on the network training and classification performance. Finally, we demonstrate that the non-linear synapses with hard boundary values enable higher classification performance and realize the best trade-off between classification accuracy and required training time. With reference to the obtained results, we discuss how memristive devices with non-linear dynamics constitute a technologically convenient solution for the development of on-line SNN training.

In this paper we present a Competitive Rate-Based Algorithm (CRBA) that approximates operation of a Competitive Spiking Neural Network (CSNN). CRBA is based on modeling of the competition between neurons during a sample presentation, which can be reduced to ranking of the neurons based on a dot product operation and the use of a discrete Expectation Maximization algorithm; the latter is equivalent to the spike time-dependent plasticity rule. CRBA's performance is compared with that of CSNN on the MNIST and Fashion-MNIST datasets. The results show that CRBA performs on par with CSNN, while using three orders of magnitude less computational time. Importantly, we show that the weights and firing thresholds learned by CRBA can be used to initialize CSNN's parameters that results in its much more efficient operation. [ABSTRACT FROM AUTHOR]

Experimental studies support the notion of spike-based neuronal information processing in the brain, with neural circuits exhibiting a wide range of temporally-based coding strategies to rapidly and efficiently represent sensory stimuli. Accordingly, it would be desirable to apply spike-based computation to tackling real-world challenges, and in particular transferring such theory to neuromorphic systems for low-power embedded applications. Motivated by this, we propose a new supervised learning method that can train multilayer spiking neural networks to solve classification problems based on a rapid, first-to-spike decoding strategy. The proposed learning rule supports multiple spikes fired by stochastic hidden neurons, and yet is stable by relying on first-spike responses generated by a deterministic output layer. In addition to this, we also explore several distinct, spike-based encoding strategies in order to form compact representations of presented input data. We demonstrate the classification performance of the learning rule as applied to several benchmark datasets, including MNIST. The learning rule is capable of generalizing from the data, and is successful even when used with constrained network architectures containing few input and hidden layer neurons. Furthermore, we highlight a novel encoding strategy, termed "scanline encoding," that can transform image data into compact spatiotemporal patterns for subsequent network processing. Designing constrained, but optimized, network structures and performing input dimensionality reduction has strong implications for neuromorphic applications. [ABSTRACT FROM AUTHOR]

Spiking neural networks (SNNs) are a computational tool in which the information is coded into spikes, as in some parts of the brain, differently from conventional neural networks (NNs) that compute over real-numbers. Therefore, SNNs can implement intelligent information extraction in real-time at the edge of data acquisition and correspond to a complementary solution to conventional NNs working for cloud-computing. Both NN classes face hardware constraints due to limited computing parallelism and separation of logic and memory. Emerging memory devices, like resistive switching memories, phase change memories, or memristive devices in general are strong candidates to remove these hurdles for NN applications. The well-established training procedures of conventional NNs helped in defining the desiderata for memristive device dynamics implementing synaptic units. The generally agreed requirements are a linear evolution of memristive conductance upon stimulation with train of identical pulses and a symmetric conductance change for conductance increase and decrease. Conversely, little work has been done to understand the main properties of memristive devices supporting efficient SNN operation. The reason lies in the lack of a background theory for their training. As a consequence, requirements for NNs have been taken as a reference to develop memristive devices for SNNs. In the present work, we show that, for efficient CMOS/memristive SNNs, the requirements for synaptic memristive dynamics are very different from the needs of a conventional NN. System-level simulations of a SNN trained to classify hand-written digit images through a spike timing dependent plasticity protocol are performed considering various linear and non-linear plausible synaptic memristive dynamics. We consider memristive dynamics bounded by artificial hard conductance values and limited by the natural dynamics evolution toward asymptotic values (soft-boundaries). We quantitatively analyze the impact of resolution and non-linearity properties of the synapses on the network training and classification performance. Finally, we demonstrate that the non-linear synapses with hard boundary values enable higher classification performance and realize the best trade-off between classification accuracy and required training time. With reference to the obtained results, we discuss how memristive devices with non-linear dynamics constitute a technologically convenient solution for the development of on-line SNN training. [ABSTRACT FROM AUTHOR]

State-of-the-art neural network models are actively used in various fields, but it is well-known that they are vulnerable to adversarial example attacks. Throughout the efforts to make the models robust against adversarial example attacks, it has been found to be a very difficult task. While many defense approaches were shown to be not effective, adversarial training remains as one of the promising methods. In adversarial training, the training data are augmented by "adversarial" samples generated using an attack algorithm. If the attacker uses a similar attack algorithm to generate adversarial examples, the adversarially trained network can be quite robust to the attack. However, there are numerous ways of creating adversarial examples, and the defender does not know what algorithm the attacker may use. A natural question is: Can we use adversarial training to train a model robust to multiple types of attack? Previous work have shown that, when a network is trained with adversarial examples generated from multiple attack methods, the network is still vulnerable to white-box attacks where the attacker has complete access to the model parameters. In this paper, we study this question in the context of black-box attacks, which can be a more realistic assumption for practical applications. Experiments with the MNIST dataset show that adversarially training a network with an attack method helps defending against that particular attack method, but has limited effect for other attack methods. In addition, even if the defender trains a network with multiple types of adversarial examples and the attacker attacks with one of the methods, the network could lose accuracy to the attack if the attacker uses a different data augmentation strategy on the target network. These results show that it is very difficult to make a robust network using adversarial training, even for black-box settings where the attacker has restricted information on the target network. [ABSTRACT FROM AUTHOR]