Search

A mixed-signal binarized neural network (BNN) processor for the MNIST image classification is demonstrated based on analogue circuit networks. BNN algorithm for training neural networks with binary weights and activations reduces power consumption and memory size. This algorithm is a design to perform core operations of multi-layer perceptron (MLP) to reduce complexity and power consumption using analogue circuits. The mixed-signal BNN processor employs a current mirror neuron to perform multiply-and-accumulate (MAC) operations and sign activation functions. The near-threshold current mirror neuron that computes the key operations of the BNN algorithm is used to achieve low power consumption. The design occupies 0.065 mm2 in 28-nm CMOS with 560B of on-chip SRAM. The 28-nm CMOS test-chip achieves energy efficiency of 24.1 TOPS/W and 94% accuracy on the MNIST image classification. This design with binary weights and activations exhibits only 5% degradation in accuracy compared with the models with floating-point precision. [ABSTRACT FROM AUTHOR]

Echo state networks (ESNs) belong to the class of recurrent neural networks and have demonstrated robust performance in time series prediction tasks. In this study, we investigate the capability of different ESN architectures to capture spatial relationships in images without transforming them into temporal sequences. We begin with three pre-existing ESN-based architectures and enhance their design by incorporating multiple output layers, customising them for a classification task. Our investigation involves an examination of the behaviour of these modified networks, coupled with a comprehensive performance comparison against the baseline vanilla ESN architecture. Our experiments on the MNIST data set reveal that a network with multiple independent reservoirs working in parallel outperforms other ESN-based architectures for this task, achieving a classification accuracy of 98.43%. This improvement on the classical ESN architecture is accompanied by reduced training times. While the accuracy of ESN-based architectures lags behind that of convolutional neural network-based architectures, the significantly lower training times of ESNs with multiple reservoirs operating in parallel make them a compelling choice for learning spatial relationships in scenarios prioritising energy efficiency and rapid training. This multi-reservoir ESN architecture overcomes standard ESN limitations regarding memory requirements and training times for large networks, providing more accurate predictions than other ESN-based models. These findings contribute to a deeper understanding of the potential of ESNs as a tool for image classification. [ABSTRACT FROM AUTHOR]

Activation functions are used to extract meaningful relationships from real-world problems with the help of deep learning models. Thus, the development of activation functions which affect deep learning models' performances is of great interest to researchers. In the literature, mostly, nonlinear activation functions are preferred since linear activation functions limit the learning performances of the deep learning models. Non-linear activation functions can be classified as fixed-parameter and trainable activation functions based on whether the activation function parameter is fixed (i.e., user-given) or modified during the training process of deep learning models. The parameters of the fixed-parameter activation functions should be specified before the deep learning model training process. However, it takes too much time to determine appropriate function parameter values and can cause the slow convergence of the deep learning model. In contrast, trainable activation functions whose parameters are updated in each iteration of deep learning models training process achieve faster and better convergence by obtaining the most suitable parameter values for the datasets and deep learning architectures. This study proposes parametric RSigELU (P+RSigELU) trainable activation functions, such as P+RSigELU Single (P+RSigELUS) and P+RSigELU Double (P+RSigELUD), to improve the performance of fixed-parameter activation function of RSigELU. The performances of the proposed trainable activation functions were evaluated on the benchmark datasets of MNIST, CIFAR-10, and CIFAR-100 datasets. Results show that the proposed activation functions outperforms PReLU, PELU, ALISA, P+FELU, PSigmoid, and GELU activation functions found in the literature. To access the codes of the activation function; https://github.com/serhatklc/P-RsigELU-Activation-Function. [ABSTRACT FROM AUTHOR]

Background: This paper addresses specific challenges in predictive modeling, namely transparency issues, susceptibility to data manipulation, and fairness concerns. To overcome these obstacles, the study introduces DeepRing, approach that combines Convolutional Neural Networks (CNNs) and blockchain technology. Objective: DeepRing aims to enhance prediction integrity, data security, and fairness, thereby improving the ethical considerations, reliability, and accountability of predictive models. Methods: involves iterative training of a CNN model on five diverse datasets, including CIFAR-10, Fashion-MNIST, MNIST, CIFAR-100, and a Hands dataset. The CNN architecture incorporates Conv2D layers, MaxPooling2D layers, and Dense layers. Training metrics such as accuracy and sparse categorical cross-entropy loss are monitored, with the Adam optimizer employed. While achieving high accuracy on Plam (0.5300), MNIST (0.9978) and Fashion MNIST (0.9673), DeepRing exhibits moderate performance on CIFAR-10 (0.9296) and lower accuracy on CIFAR-100 (0.5973). Results: demonstrate the effectiveness of DeepRing in improving accuracy and enhancing model performance across various datasets. However, further development and validation are essential for successful model implementation, further development and validation are essential for successful model implementation. Conclusions: Introduces DeepRing as an innovative solution to address key challenges in predictive modeling, specifically focusing on transparency issues, susceptibility to data manipulation, and fairness concerns. By combining Convolutional Neural Networks (CNNs) with blockchain technology, DeepRing aims to elevate prediction integrity, enhance data security, and promote fairness, thereby contributing to the improvement of ethical considerations, reliability, and accountability in predictive modelling. [ABSTRACT FROM AUTHOR]

Current mainstream deep learning optimization algorithms can be classified into two categories: non-adaptive optimization algorithms, such as Stochastic Gradient Descent with Momentum (SGDM), and adaptive optimization algorithms, like Adaptive Moment Estimation with Weight Decay (AdamW). Adaptive optimization algorithms for many deep neural network models typically enable faster initial training, whereas non-adaptive optimization algorithms often yield better final convergence. Our proposed Adaptive Learning Rate Burst (Adaburst) algorithm seeks to combine the strengths of both categories. The update mechanism of Adaburst incorporates elements from AdamW and SGDM, ensuring a seamless transition between the two. Adaburst modifies the learning rate of the SGDM algorithm based on a cosine learning rate schedule, particularly when the algorithm encounters an update bottleneck, which is called learning rate burst. This approach helps the model to escape current local optima more effectively.The results of the Adaburst experiment underscore its enhanced performance in image classification and generation tasks when compared with alternative approaches, characterized by expedited convergence and elevated accuracy. Notably, on the MNIST, CIFAR-10, and CIFAR-100 datasets, Adaburst attained accuracies that matched or exceeded those achieved by SGDM. Furthermore, in training diffusion models on the DeepFashion dataset, Adaburst achieved convergence in fewer epochs than a meticulously calibrated AdamW optimizer while avoiding abrupt blurring or other training instabilities. Adaburst augmented the final training set accuracy on the MNIST, CIFAR-10, and CIFAR-100 datasets by 0.02%, 0.41%, and 4.18%, respectively. In addition, the generative model trained on the DeepFashion dataset demonstrated a 4.62-point improvement in the Frechet Inception Distance (FID) score, a metric for assessing generative model quality. Consequently, this evidence suggests that Adaburst introduces an innovative optimization algorithm that simultaneously updates AdamW and SGDM and incorporates a learning rate burst mechanism. This mechanism significantly enhances deep neural networks’ training speed and convergence accuracy. [ABSTRACT FROM AUTHOR]

In today's busy world, people are shifting to online shopping for their basic needs like groceries, clothes, etc. As a result, modern applications are incorporating image search as part of their application. However, retrieving accurate fashion images is challenging due to the low inter-class separability. In this paper, we aim to propose and implement three deep learning denoising autoencoder image retrieval models trained on Fashion-MNIST dataset and MNIST dataset. Fashion-MNIST is widely used datasets to evaluate model because of its low inter-class separability. It is difficult to get a good accuracy on Fashion-MNIST dataset. The proposed models train to learn significant features of fashion images and improve the precision of image retrieval. These models are compared based on the Label Ranking Average Precision (LRAP) values, with the third model bagging the highest LRAP value. These models also trained and tested on MNIST dataset, which have similar configuration as Fashion-MNIST dataset but with high inter-class separability. Intriguingly, the second and third models trained on MNIST dataset demonstrate neck-to-neck performance, both achieving comparable accuracy. This choice of dataset ensures that the models' accuracy is more noticeable and significant. The outcomes of this study validate the efficacy of denoising autoencoders in fashion image retrieval and emphasize their prospective in improving online shopping experiences. [ABSTRACT FROM AUTHOR]

In the rapidly changing world of new technologies, AI and ML have been delivered to the network's edge, allowing edge devices to compute simple models that can be applied later in practical applications. Edge computing allows the deployment of cloudlike services to the network edge. Moreover, it allows network devices to contribute to Federated Learning (FL) by using their processing power. Federated learning was developed when privacy and security concerns emerged among people. Here, multiple clients work together to solve machine-learning problems. Instead of collecting users' raw data, this technique protects user privacy by aggregating model parameters from each client to a central server. Even though FL techniques have several benefits, such as scalability and data privacy, when used with heterogeneous devices, they can introduce significant problems in processing complexity and speed. This paper thoroughly surveyed the important and efficient technological advancements based on edge federated learning. We review the widely used FL frameworks that effectively facilitate communication between clients and servers. We also present a case study on handwritten digit recognition using the MNIST dataset to state the effectiveness of federated learning. On applying this approach, we were able to achieve 94.56% accuracy. [ABSTRACT FROM AUTHOR]

Background: This paper addresses specific challenges in predictive modeling, namely transparency issues, susceptibility to data manipulation, and fairness concerns. To overcome these obstacles, the study introduces DeepRing, approach that combines Convolutional Neural Networks (CNNs) and blockchain technology. Objective: DeepRing aims to enhance prediction integrity, data security, and fairness, thereby improving the ethical considerations, reliability, and accountability of predictive models. Methods: involves iterative training of a CNN model on five diverse datasets, including CIFAR-10, Fashion-MNIST, MNIST, CIFAR-100, and a Hands dataset. The CNN architecture incorporates Conv2D layers, MaxPooling2D layers, and Dense layers. Training metrics such as accuracy and sparse categorical cross-entropy loss are monitored, with the Adam optimizer employed. While achieving high accuracy on Plam (0.5300), MNIST (0.9978) and Fashion MNIST (0.9673), DeepRing exhibits moderate performance on CIFAR-10 (0.9296) and lower accuracy on CIFAR-100 (0.5973). Results: demonstrate the effectiveness of DeepRing in improving accuracy and enhancing model performance across various datasets. However, further development and validation are essential for successful model implementation, further development and validation are essential for successful model implementation. Conclusions: Introduces DeepRing as an innovative solution to address key challenges in predictive modeling, specifically focusing on transparency issues, susceptibility to data manipulation, and fairness concerns. By combining Convolutional Neural Networks (CNNs) with blockchain technology, DeepRing aims to elevate prediction integrity, enhance data security, and promote fairness, thereby contributing to the improvement of ethical considerations, reliability, and accountability in predictive modelling.

The technology of handwritten digit and character recognition is the process of identifying handwritten numbers using computers or other devices, machine train itself to recognize the handwritten digits or characters from various sources like bank cheque, mails, images, etc. This paper is about CNN, Logistic Regression, and Random Forest algorithm in handwritten digit or character recognition system, the system works on MNIST dataset for training and testing the models, to get the best accuracy this work rewrites CNN, Logistic Regression, and Random Forest with python libraries. Finally, these algorithms are analyzed by comparing the accuracy and recognition duration. Where CNN got better accuracy than other algorithms. [ABSTRACT FROM AUTHOR]

Interpretation of neural activity in response to stimulations received from the surrounding environment is necessary to realize automatic brain decoding. Analyzing the brain recordings corresponding to visual stimulation helps to infer the effects of perception occurring by vision on brain activity. In this paper, the impact of arithmetic concepts on vision-related brain records has been considered and an efficient convolutional neural network-based generative adversarial network (CNN-GAN) is proposed to map the electroencephalogram (EEG) to salient parts of the image stimuli. The first part of the proposed network consists of depth-wise one-dimensional convolution layers to classify the brain signals into 10 different categories according to Modified National Institute of Standards and Technology (MNIST) image digits. The output of the CNN part is fed forward to a fine-tuned GAN in the proposed model. The performance of the proposed CNN part is evaluated via the visually provoked 14-channel MindBigData recorded by David Vivancos, corresponding to images of 10 digits. An average accuracy of 95.4% is obtained for the CNN part for classification. The performance of the proposed CNN-GAN is evaluated based on saliency metrics of SSIM and CC equal to 92.9% and 97.28%, respectively. Furthermore, the EEG-based reconstruction of MNIST digits is accomplished by transferring and tuning the improved CNN-GAN's trained weights. [ABSTRACT FROM AUTHOR]

Introduction: Spiking neural networks (SNNs) are a model of computation that mimics the behavior of biological neurons. SNNs process event data (spikes) and operate more sparsely than artificial neural networks (ANNs), resulting in ultra-low latency and small power consumption. This paper aims to adapt and evaluate gradient-based explainability methods for SNNs, which were originally developed for conventional ANNs. Methods: The adapted methods aim to create input feature attribution maps for SNNs trained through backpropagation that process either event-based spiking data or real-valued data. The methods address the limitations of existing work on explainability methods for SNNs, such as poor scalability, limited to convolutional layers, requiring the training of another model, and providing maps of activation values instead of true attribution scores. The adapted methods are evaluated on classification tasks for both real-valued and spiking data, and the accuracy of the proposedmethods is confirmed through perturbation experiments at the pixel and spike levels. Results and discussion: The results reveal that gradient-based SNN attribution methods successfully identify highly contributing pixels and spikes with significantly less computation time than model-agnostic methods. Additionally, we observe that the chosen coding technique has a noticeable effect on the input features that will be most significant. These findings demonstrate the potential of gradient-based explainability methods for SNNs in improving our understanding of how these networks process information and contribute to the development of more efficient and accurate SNNs. [ABSTRACT FROM AUTHOR]

In neuromorphic computing, the coding method of spiking neurons serves as the foundation and is crucial for various aspects of network operation. Existing mainstream coding methods, such as rate coding and temporal coding, have different focuses, and each has its own advantages and limitations. This paper proposes a novel coding scheme called activeness coding that integrates the strengths of both rate and temporal coding methods. It encompasses precise timing information of the most recent neuronal spike as well as the historical firing rate information. The results of basic characteristic tests demonstrate that this encoding method accurately expresses input information and exhibits robustness. Furthermore, an unsupervised learning method based on activeness-coding triplet spike-timing dependent plasticity (STDP) is introduced, with the MNIST classification task used as an example to assess the performance of this encoding method in solving cognitive tasks. Test results show an improvement in accuracy of approximately 4.5%. Additionally, activeness coding also exhibits potential advantages in terms of resource conservation. Overall, activeness offers a promising approach for spiking neural network encoding with implications for various applications in the field of neural computation. [ABSTRACT FROM AUTHOR]

TADS are a novel, concise white-box representation of neural networks. In this paper, we apply TADS to the problem of neural network verification, using them to generate either proofs or concise error characterizations for desirable neural network properties. In a case study, we consider the robustness of neural networks to adversarial attacks, i.e., small changes to an input that drastically change a neural networks perception, and show that TADS can be used to provide precise diagnostics on how and where robustness errors a occur. We achieve these results by introducing Precondition Projection, a technique that yields a TADS describing network behavior precisely on a given subset of its input space, and combining it with PCA, a traditional, well-understood dimensionality reduction technique. We show that PCA is easily compatible with TADS. All analyses can be implemented in a straightforward fashion using the rich algebraic properties of TADS, demonstrating the utility of the TADS framework for neural network explainability and verification. While TADS do not yet scale as efficiently as state-of-the-art neural network verifiers, we show that, using PCA-based simplifications, they can still scale to medium-sized problems and yield concise explanations for potential errors that can be used for other purposes such as debugging a network or generating new training samples. [ABSTRACT FROM AUTHOR]

Advanced artificial intelligence (AI) techniques have led to significant developments in optical character recognition (OCR) technologies. OCR applications, using AI techniques for transforming images of typed text, handwritten text, or other forms of text into machine-encoded text, provide a fair degree of accuracy for general text. However, even after decades of intensive research, creating OCR with human-like abilities has remained evasive. One of the challenges has been that OCR models trained on general text do not perform well on localized or personalized handwritten text due to differences in the writing style of alphabets and digits. This study aims to discuss the steps needed to create an adaptive framework for OCR models, with the intent of exploring a reasonable method to customize an OCR solution for a unique dataset of English language numerical digits were developed for this study. We develop a digit recognizer by training our model on the MNIST dataset with a convolutional neural network and contrast it with multiple models trained on combinations of the MNIST and custom digits. Using our methods, we observed results comparable with the baseline and provided recommendations for improving OCR accuracy for localized or personalized handwritten text. This study also provides an alternative perspective to generating data using conventional methods, which can serve as a gold standard for custom data augmentation to help address the challenges of scarce data and data imbalance. [ABSTRACT FROM AUTHOR]

The paper considers the problem of modelling the distribution of data with noise in the input data. In this paper, we consider encoders and decoders, which solve the problem of modelling data distribution. The improvement of variational autoencoders (VAEs) is discussed. Practical implementation is performed using the Python programming language and the Keras framework. Generative adversarial networks (GANs) and VAEs with noisy data are demonstrated. [ABSTRACT FROM AUTHOR]

Introduction/purpose: This paper provides a survey of handwritten digit recognition methods tested on the MNIST dataset. Methods: The paper analyzes, synthesizes and compares the development of different classifiers applied to the handwritten digit recognition problem, from linear classifiers to convolutional neural networks. Results: Handwritten digit recognition classification accuracy tested on the MNIST dataset while using training and testing sets is now higher than 99.5% and the most successful method is a convolutional neural network. Conclusions: Handwritten digit recognition is a problem with numerous real-life applications. Accurate recognition of various handwriting styles, specifically digits is a task studied for decades and this paper summarizes the achieved results. The best results have been achieved with convolutional neural networks while the worst methods are linear classifiers. The convolutional neural networks give better results if the dataset is expended with data augmentation.

Stephen Wolfram (2002) proposed the concept of computational equivalence, which implies that almost any dynamical system can be considered as a computation, including programmable matter and nonlinear materials such as, so called, quantum matter. Memristors are often used in building and evaluating hardware neural networks. Ukil (2011) demonstrated a theoretical relationship between piezoelectrical materials and memristors. We review that work as a necessary background prior to our work on exploring a piezoelectric material for neural network computation. Our method consisted of using a cubic block of unpoled lead zirconate titanate (PZT) ceramic, to which we have attached wires for programming the PZT as a programmable substrate. We then, by means of pulse trains, constructed on-the-fly internal patterns of regions of aligned polarization and unaligned, or disordered regions. These dynamic patterns come about through constructive and destructive interference and may be exploited as a type of reservoir network. Using MNIST data we demonstrate a learning machine.

In this article, both machine learning techniques and deep learning methods were applied on the digit datasets created using the Arabic and Latin alphabets, and the performances of the methods were compared. Each method was tested with various parameters and the results were analyzed. In addition, with this study, the recognizability of handwritten numeral datasets created using different alphabets was also observed. For experiments, an Arabic alphabet handwritten digit dataset (60,000 training and 10,000 testings) and a Latin alphabet handwritten digit dataset (60,000 training and 10,000 testings) were used. When the results of the experiment are examined, it is seen that successful results are obtained in the classification made with the MADBase dataset in some methods and in the classification made with the MNIST dataset in some methods. As a result, it can be stated that the handwriting character recognition success of a method cannot be measured only by the classification made on a dataset. Also, the digits written in the Arabic alphabet appear to be almost more recognizable than the digits written in the Latin alphabet.

Capsule Networks (CapsNets) is a great approach for understanding data in the field of computer vision. CapsNets allow a deeper understanding of images compared to the traditional Convolutional Neural Networks. The first test for CapsNet was in digits recognition on the 'MNIST' dataset, where it successfully achieved high accuracy. CapsNets are reliable at deciphering overlapping digits. Deep Capsule Networks achieved state-of-the-art accuracy in CIFAR10 which isn't achieved by shallow capsule networks. Despite all these accomplishments, Deep Capsule Networks are very slow due to the 'Dynamic Routing' algorithm. In this paper, Fast Embedded Capsule Network and Deep Fast Embedded Capsule Network are introduced, representing novel capsule network architectures that uses 1D convolution based dynamic routing with a fast element-wise multiplication transformation process. These architectures not only compete with the state-of-the-art methods in terms of accuracy in the capsule domain, but also excels in terms of speed, and reduced complexity. This is shown by the 58% reduction in the number of trainable parameters and 64% reduction in the average epoch time in the training process. Experimental results shows excellent and verified properties. [ABSTRACT FROM AUTHOR]

Convolutional Neural Networks (CNN) are being widely practised in computer vision applications, where pooling indicates as an integral part. Pooling significantly reduces the training time and computational cost of any network. The pooling operations generally being used are max, average, or stochastic. The performance of the CNN architecture can be increased by using more than one pooling operations. Herein, a novel mixed fuzzy pooling is proposed for image classification in the CNN architecture. In the proposed mixed fuzzy pooling, max pooling and fuzzy pooling are combined together to boost the classification accuracy. The proposed mixed fuzzy pooling is designed as a function of max pooling, fuzzy pooling, and α parameter where the learning of the proposed pooling function takes place with the value of α. In fuzzy pooling, Type-2 fuzzy logic is used to get the convolved dominant features. This is obtained by the estimation of a threshold for image region to be pooled. Thereafter, Type-1 fuzzy logic is used to get the pooling output by a weighted average of convolved dominant features. The performance of various pooling strategies on MNIST dataset for handwritten digits classification and CIFAR-10 dataset for RGB images classification have been compared and analysed. The experimental results show better image classification accuracy for the proposed mixed fuzzy pooling than conventional pooling operations. [ABSTRACT FROM AUTHOR]

While Generative Adversarial Networks (GAN) are popular for their higher sample quality as opposed to other generative models like Variational Autoencoders (VAE) and Boltzmann machines, it is still difficult to evaluate their overall performance. Various aspects must be kept in mind, such as the quality of generated samples, the diversity of classes (within a class and among classes), avoiding overfitting, the use of disentangled latent spaces, agreement of said evaluation metric with human perception, etc. In this paper, we propose a new score, namely, GM Score, which takes into various factors such as sample quality, intra-class and inter-class diversity, overfitting, and other metrics such as precision, recall, and F1 score is employed for discriminability of latent space of Deep Belief Network (DBN) and Restricted Boltzmann Machine (RBM). The evaluation is done for different GANs (GAN, DCGAN, BiGAN, CGAN, CoupledGAN, LSGAN, SGAN, WGAN, and WGAN Improved) trained on the benchmark MNIST dataset. [ABSTRACT FROM AUTHOR]

Image classification, a pivotal task in multiple industries, faces computational challenges due to the burgeoning volume of visual data. This research addresses these challenges by introducing two quantum machine learning models that leverage the principles of quantum mechanics for effective computations. Our first model, a hybrid quantum neural network with parallel quantum circuits, enables the execution of computations even in the noisy intermediate-scale quantum era, where circuits with a large number of qubits are currently infeasible. This model demonstrated a record-breaking classification accuracy of 99.21% on the full MNIST dataset, surpassing the performance of known quantum–classical models, while having eight times fewer parameters than its classical counterpart. Also, the results of testing this hybrid model on a Medical MNIST (classification accuracy over 99%), and on CIFAR-10 (classification accuracy over 82%), can serve as evidence of the generalizability of the model and highlights the efficiency of quantum layers in distinguishing common features of input data. Our second model introduces a hybrid quantum neural network with a Quanvolutional layer, reducing image resolution via a convolution process. The model matches the performance of its classical counterpart, having four times fewer trainable parameters, and outperforms a classical model with equal weight parameters. These models represent advancements in quantum machine learning research and illuminate the path towards more accurate image classification systems.

Interpretation of neural activity in response to stimulations received from the surrounding environment is necessary to realize automatic brain decoding. Analyzing the brain recordings corresponding to visual stimulation helps to infer the effects of perception occurring by vision on brain activity. In this paper, the impact of arithmetic concepts on vision-related brain records has been considered and an efficient convolutional neural network-based generative adversarial network (CNN-GAN) is proposed to map the electroencephalogram (EEG) to salient parts of the image stimuli. The first part of the proposed network consists of depth-wise one-dimensional convolution layers to classify the brain signals into 10 different categories according to Modified National Institute of Standards and Technology (MNIST) image digits. The output of the CNN part is fed forward to a fine-tuned GAN in the proposed model. The performance of the proposed CNN part is evaluated via the visually provoked 14-channel MindBigData recorded by David Vivancos, corresponding to images of 10 digits. An average accuracy of 95.4% is obtained for the CNN part for classification. The performance of the proposed CNN-GAN is evaluated based on saliency metrics of SSIM and CC equal to 92.9% and 97.28%, respectively. Furthermore, the EEG-based reconstruction of MNIST digits is accomplished by transferring and tuning the improved CNN-GAN’s trained weights.

Machine Learning is an important field of research in current trends. The extended field of machine learning is Deep Learning and is used for various research areas such as neural networks, image and signal processing, pattern recognition, etc. The handwritten digit recognition is an important task or process included in various applications such as car number plate recognition, staff identity number detection, etc. This paper proposed the design and analysis of various deep learning algorithms such as deep neural networks, convolutional neural networks, LeNet-5, AlexNet and MiniVGGNet for handwritten digit recognition using MNIST dataset. [ABSTRACT FROM AUTHOR]

Introduction/purpose: This paper provides a survey of handwritten digit recognition methods tested on the MNIST dataset. Methods: The paper analyzes, synthesizes and compares the development of different classifiers applied to the handwritten digit recognition problem, from linear classifiers to convolutional neural networks. Results: Handwritten digit recognition classification accuracy tested on the MNIST dataset while using training and testing sets is now higher than 99.5% and the most successful method is a convolutional neural network. Conclusions: Handwritten digit recognition is a problem with numerous real-life applications. Accurate recognition of various handwriting styles, specifically digits is a task studied for decades and this paper summarizes the achieved results. The best results have been achieved with convolutional neural networks while the worst methods are linear classifiers. The convolutional neural networks give better results if the dataset is expended with data augmentation. [ABSTRACT FROM AUTHOR]

In this paper, based on the architecture of an artificial neural network using the metric recognition method, experiments are carried out to determine the weights and thresholds of the neural network with an electrostatic field parameter (potential) without any additional analytical calculations or learning algorithms. The simulation of the electrostatic field is implemented in the Builder C++ software environment, which evaluates the total potential of the electrostatic field at the points of the proposed model where potentiometer sensors are located. The same software module enables the creation of a neural network based on metrical recognition methods for which the weights of the first-layer neurons are determined based on the potentials of the simulated electrostatic field. The effectiveness of the resulting neural network is checked against the Modified National Institute of Standards and Technology (MNIST) database. [ABSTRACT FROM AUTHOR]

MFF-Net (a multi-scale feature fusion convolutional neural network) was designed to improve the recognition rate of handwritten digits. The low-level, middle-level and high-level features of the image were first extracted through the convolution operation, and then the low-level and intermediate features were further extracted through different convolutional layers, later directly fused with the high-level features of the image with a certain weight, and then processed by the full connection layer. By adding a batch normalization layer before the activation layer, and a dropout layer between the full connection layers, the accuracy and generalization capacity of the network are improved. At the same time, a dynamic learning rate algorithm was designed, with which, the trained network accuracy was significantly improved as shown in the experiments on the MNIST data set. The accurate rate could reach 99.66% through only 30 epochs training. The comparison indicated that the accuracy of the network model is significantly higher than that of others. [ABSTRACT FROM AUTHOR]

Stephen Wolfram (2002) proposed the concept of computational equivalence, which implies that almost any dynamical system can be considered as a computation, including programmable matter and nonlinear materials such as, so called, quantum matter. Memristors are often used in building and evaluating hardware neural networks. Ukil (2011) demonstrated a theoretical relationship between piezoelectrical materials and memristors. We review that work as a necessary background prior to our work on exploring a piezoelectric material for neural network computation. Our method consisted of using a cubic block of unpoled lead zirconate titanate (PZT) ceramic, to which we have attached wires for programming the PZT as a programmable substrate. We then, by means of pulse trains, constructed on-the-fly internal patterns of regions of aligned polarization and unaligned, or disordered regions. These dynamic patterns come about through constructive and destructive interference and may be exploited as a type of reservoir network. Using MNIST data we demonstrate a learning machine. [ABSTRACT FROM AUTHOR]

Deep neural networks, which is a field of artificial intelligence, have been used in various fields. Deep learning is processed on high-performance GPUs or TPUs. It requires high cost as much as its good performance. Recently, as the demand for edge computing increases, many studies have been conducted to perform complex deep learning operations in a low-computing processor. Among them, a typical study is to lighten the deep learning network. In this paper, we propose a handwritten digit recognition hardware accelerator suitable for edge computing using MNIST database. After setting the correction rate for MNIST to 94% and performing network lighting processes, a hardware structure that can reduce the area of hardware and minimize memory access is proposed. Basically, the network is set as a two-layer fully connected network. The network is modeled with Python and lighten while checking the performance. Network parameters, weighs and biases, are quantized. The pixel number and bit number of MNIST input data are also reduced. The number of MAC units and the processing order of the hardware accelerator are determined so that there is no not used MACs while performing the MAC operations in parallel. It is designed with Verilog HDL and its functions are checked in Modelsim. And then it is implemented in Xilinx Zynq ZC-702 to verify the operations. The designed number recognition accelerator is expected to be widely used in edge devices by reducing the area and memory access.

Handwritten character recognition is a computer-vision-system problem that is still critical and challenging in many computer-vision tasks. With the increased interest in handwriting recognition as well as the developments in machine-learning and deep-learning algorithms, researchers have made significant improvements and advances in developing English-handwriting-recognition methodologies; however, Arabic handwriting recognition has not yet received enough interest. In this work, several deep-learning and hybrid models were created. The methodology of the current study took advantage of machine learning in classification and deep learning in feature extraction to create hybrid models. Among the standalone deep-learning models trained on the two datasets used in the experiments performed, the best results were obtained with the transfer-learning model on the MNIST dataset, with 0.9967 accuracy achieved. The results for the hybrid models using the MNIST dataset were good, with accuracy measures exceeding 0.9 for all the hybrid models; however, the results for the hybrid models using the Arabic character dataset were inferior. [ABSTRACT FROM AUTHOR]

The MNIST dataset is considered a challenging problem for machine learning algorithms. The present paper introduces a novel approach based on a truncated SVD and kernel density estimation that outputs an error rate on MNIST comparable to classical machine learning approaches. In addition to that, a hierarchical classification framework is proposed, that allows to enhance the algorithm accuracy. The resulting algorithm outperforms the reproducible SVM accuracy, which is regarded as a state-of-the-art machine learning algorithm for MNIST. The key advantage of the proposed framework consists in a lower computational cost: training on MNIST takes 142 times less and prediction takes 4 times less than for SVM on a single run. Thus, the research results state that TSVD-KDE algorithm has the potential for being an efficient classification algorithm. The source code for the experiment is released on GitHub: https://github.com/ekplesovskaya/MNIST-Classification-Using-TSVD-and-KDE. [ABSTRACT FROM AUTHOR]

This work reports 2-bits/cell hafnium oxide-based stacked resistive random access memory devices fabricated on flexible polyimide substrates for neuromorphic applications considering the high thermal budget. The ratio of low-resistance state current (${I}_{ \mathrm{\scriptscriptstyle ON}}$) to high-resistance state current (${I}_{ \mathrm{\scriptscriptstyle OFF}}$) or ${I}_{ \mathrm{\scriptscriptstyle ON}}/{I}_{ \mathrm{\scriptscriptstyle OFF}}$ for the fabricated devices was above $1.4\times10$ 3 with a low device-to-device variation at $100 \boldsymbol {\mu }\text{A}$ current compliance. The mechanical stability over 104 bending cycles at a 5 mm bending radius and endurance over 106 WRITE cycles makes these devices suitable for online neural network training. The data retention capability over 104s at 125°C also infuses these devices’ long-term inference capability. Furthermore, the performance of the devices has been verified for neuromorphic applications by system-level simulations with experimentally calibrated data. The system-level simulation reveals only a 2% loss in inference accuracy over ten years from the baseline. [ABSTRACT FROM AUTHOR]

The paper considers the problem of modelling the distribution of data with noise in the input data. In this paper, we consider encoders and decoders, which solve the problem of modelling data distribution. The improvement of variational autoencoders (VAEs) is discussed. Practical implementation is performed using the Python programming language and the Keras framework. Generative adversarial networks (GANs) and VAEs with noisy data are demonstrated.

Advanced artificial intelligence (AI) techniques have led to significant developments in optical character recognition (OCR) technologies. OCR applications, using AI techniques for transforming images of typed text, handwritten text, or other forms of text into machine-encoded text, provide a fair degree of accuracy for general text. However, even after decades of intensive research, creating OCR with human-like abilities has remained evasive. One of the challenges has been that OCR models trained on general text do not perform well on localized or personalized handwritten text due to differences in the writing style of alphabets and digits. This study aims to discuss the steps needed to create an adaptive framework for OCR models, with the intent of exploring a reasonable method to customize an OCR solution for a unique dataset of English language numerical digits were developed for this study. We develop a digit recognizer by training our model on the MNIST dataset with a convolutional neural network and contrast it with multiple models trained on combinations of the MNIST and custom digits. Using our methods, we observed results comparable with the baseline and provided recommendations for improving OCR accuracy for localized or personalized handwritten text. This study also provides an alternative perspective to generating data using conventional methods, which can serve as a gold standard for custom data augmentation to help address the challenges of scarce data and data imbalance.

To improve the accuracy of the parallax maps obtained by the binocular stereo matching network for better 3D reconstruction, this thesis is based on the AnyNet network, with the corresponding optimization to get MyNet. The Hourglass structure allows for the acquisition of multiscale image information. The original 3D Conv is replaced by a 3D residual block in the parallax network to avoid the gradient disappearance or explosion and model degradation problems of traditional convolutional neural network models, thus further improving the cost volume obtained. The experiments were tested on the KITTI2012 and KITTI2015 datasets and showed some improvement compared to the original algorithm. The experimental results show that the accuracy of the parallax map has improved significantly with almost the same time spent. [ABSTRACT FROM AUTHOR]

With the development of deep learning technology, handwritten character recognition has become the basic hotspot of research. At the same time, the recognition of handwriting in minority languages is gradually worthy of attention. In this paper, a robust and novel improved deep convolutional neural network method is proposed to recognize normal digital handwriting images and minority digital handwritten images (Yi language). To improve the recognition accuracy of handwritten fonts, there are four improvements in our work. Firstly, separable convolution is adopted in the network to cope with the problem of large parameters. Secondly, data augmentation techniques (Gaussian noise, color perturbation, and random rotation) are utilized to improve the expression ability of the dataset and the generalization capabilities of the model. Finally, the shortcut block and the weight initialization are introduced in our network for the problems of network degradation and gradient saturation. Based on deep learning technology, this paper studies the classification and recognition of handwritten digits, and improves the accuracy of the recognition in numbers such as invoices in financial bills. The recognition accuracy of 99.82% has been achieved on MNIST dataset, and 98.79% on the self-building dataset of Yi handwritten digits (YHDD). [ABSTRACT FROM AUTHOR]

Neural network classifiers (NNCs) are known to be vulnerable to malicious adversarial perturbations of inputs including those modifying a small fraction of the input features named sparse or L0 attacks. Effective and fast L0 attacks, such as the widely used Jacobian-based Saliency Map Attack (JSMA) are practical to fool NNCs but also to improve their robustness. In this paper, we show that penalising saliency maps of JSMA by the output probabilities and the input features of the NNC leads to more powerful attack algorithms that better take into account each input’s characteristics. This leads us to introduce improved versions of JSMA, named Weighted JSMA (WJSMA) and Taylor JSMA (TJSMA), and demonstrate through a variety of white-box and black-box experiments on three different datasets (MNIST, CIFAR-10 and GTSRB), that they are both significantly faster and more efficient than the original targeted and non-targeted versions of JSMA. Experiments also demonstrate, in some cases, very competitive results of our attacks in comparison with the Carlini-Wagner (CW) L0 attack, while remaining, like JSMA, significantly faster (WJSMA and TJSMA are more than 50 times faster than CW L0 on CIFAR-10). Therefore, our new attacks provide good trade-offs between JSMA and CW for L0 real-time adversarial testing on datasets such as the ones previously cited.

Nowadays, the scope of machine learning and deep learning studies is increasing day by day. Handwriting recognition is one of the examples in daily life for this field of work. Data storage in digital media is a method that almost everyone is using nowadays. At the same time, it has become a necessity for people to store their notes in digital media and even take notes directly in the digital environment. As a solution to this need, applications have been developed that can recognize numbers, characters, and even text from handwriting using machine learning and deep learning algorithms. Moreover, these applications can recognize numbers, characters, and text from handwriting and convert them into visual characters. This project, investigated the performance comparison of machine learning algorithms commonly used in handwriting recognition applications and which of them are more efficient. As a result of the study, the accuracy was 98.66% with artificial neural network, 99.45% with convolutional neural network, 97.05% with K-NN, 83.57% with Naive Bayes, 97.71% with support vector machine and 88.34% with decision tree. This study also developed a handwriting recognition system for numbers similar to these mentioned applications. A desktop application interface was developed for end users to show the instant performance of some of these algorithms and allow them to experience the handwriting recognition system. [ABSTRACT FROM AUTHOR]

An approach is presented that generalize OCR task including polish letters using deep learning technique. The paper extends EMNIST dataset so that two new classes of polish diacritics "Ą" and "Ć" are attached. Using this new dataset and deep learning technique one can analyze sensitives of standard as well as presented extended approach with different parameters of convolutional neural network. The analysis of the results shows that the shadows and noises that Polish characters, leave with their hooks leads, can be properly recognized and two similar letters "A" and "Ą" can be distinguished by convolutional neural network. On the other hand a neural network trained on dataset without Polish characters do not treat letters "Ą" and "Ć" properly. [ABSTRACT FROM AUTHOR]

This work addresses how to naturally adopt the l²-norm cosine similarity in the neuromemristive system and studies the unsupervised learning performance on handwritten digit image recognition. Proposed architecture is a two-layer fully connected neural network with a hard winner-take-all (WTA) learning module. For input layer, we propose single-spike temporal code that transforms input stimuli into the set of single spikes with different latencies and voltage levels. For a synapse model, we employ a compound memristor where stochastically switching binary-state memristors connected in parallel, which offers a reliable and scalable multi-state solution for synaptic weight storage. Hardware-friendly synaptic adaptation mechanism is proposed to realize spike-timing-dependent plasticity learning. Input spikes are sent out through those memristive synapses to each and every integrate-and-fire neuron in the fully connected output layer, where the hard WTA network motif introduces the competition based on cosine similarity for the given input stimuli. Finally, we present 92.64% accuracy performance on unsupervised digit recognition with only single-epoch MNIST dataset training via high-level simulations, including extensive analysis on the impact of system parameters. [ABSTRACT FROM AUTHOR]

Most capsule network designs rely on traditional matrix multiplication between capsule layers and computationally expensive routing mechanisms to deal with the capsule dimensional entanglement that the matrix multiplication introduces. By using Homogeneous Vector Capsules (HVCs), which use element-wise multiplication rather than matrix multiplication, the dimensions of the capsules remain unentangled. In this work, we study HVCs as applied to the highly structured MNIST dataset in order to produce a direct comparison to the capsule research direction of Geoffrey Hinton, et al. In our study, we show that a simple convolutional neural network using HVCs performs as well as the prior best performing capsule network on MNIST using 5.5× fewer parameters, 4× fewer training epochs, no reconstruction sub-network, and requiring no routing mechanism. The addition of multiple classification branches to the network establishes a new state of the art for the MNIST dataset with an accuracy of 99.87% for an ensemble of these models, as well as establishing a new state of the art for a single model (99.83% accurate). [ABSTRACT FROM AUTHOR]

Deep Neural Networks have gained popularity due to their superiority in performance in various applications. Hence, domain-specific architectures have been introduced to speed up the model training process. Since the launch of TPUs by Google, there were few literatures that examines the performance of TPUs compared to existing processing units such as the CPU and GPU in deep learning applications. The recent inclusion of the PyTorch framework to use Google Colaboratory's resources also provided an opportunity for us to build a PyTorch-based Colab notebook that can be used for further study in this area. In this study, we evaluated the performance of the Intel Xeon CPU, Nvidia T4 GPU and TPU v2-8 by using the Convolutional Neural Network (CNN) model on the CIFAR-10 and MNIST datasets. Our study also considers the accuracy and convergence of the model while testing, which were implemented by data augmentation and batch normalization techniques in the VGG16 model trained from scratch. We achieved a model accuracy of 86% and 96% for the CIFAR-10 and MNIST datasets respectively. During the experiments, we observed that the GPU performed 25x faster than the CPU; meanwhile the TPU only achieved a 12x speedup compared to the CPU as there were some issues faced. We proposed some issues and possible solutions in this paper for further study. [ABSTRACT FROM AUTHOR]

Las redes neuronales convolucionales cada vez son más populares en aplicaciones de aprendizaje profundo, como por ejemplo en clasificación de imágenes, reconocimiento de voz, medicina, entre otras. Sin embargo, estas redes son computacionalmente costosas y requieren altos recursos de memoria. En este trabajo se propone un acelerador en hardware para el proceso de inferencia de la red Lenet-5, un esquema de coprocesamiento hardware/software. El objetivo de la implementación es reducir el uso de recursos de hardware y obtener el mejor rendimiento computacional posible durante el proceso de inferencia. El diseño fue implementado en la tarjeta de desarrollo Digilent Arty Z7-20, la cual está basada en el System on Chip (SoC) Zynq-7000 de Xilinx. Nuestra implementación logró una precisión del 97,59 % para la base de datos MNIST utilizando tan solo 12 bits en el formato de punto fijo. Los resultados muestran que el esquema de co-procesamiento, el cual opera a una velocidad de 100 MHz, puede identificar aproximadamente 441 imágenes por segundo, que equivale aproximadamente a un 17% más rápido que una implementación de software a 650 MHz. Es difícil comparar nuestra implementación con otras implementaciones similares, porque las implementaciones encontradas en la literatura no son exactamente como la que realizó en este trabajo. Sin embargo, algunas comparaciones, en relación con el uso de recursos lógicos y la precisión, sugieren que nuestro trabajo supera a trabajos previos.

The article is an overview. We carry out the comparison of actual machine learning libraries that can be used the neural networks development. The first part of the article gives a brief description of TensorFlow, PyTorch, Theano, Keras, SciKit Learn libraries, SciPy library stack. An overview of the scope of these libraries and the main technical characteristics, such as performance, supported programming languages, the current state of development is given. In the second part of the article, a comparison of five libraries is carried out on the example of a multilayer perceptron, which is applied to the problem of handwritten digits recognizing. This problem is well known and well suited for testing different types of neural networks. The study time is compared depending on the number of epochs and the accuracy of the classifier. The results of the comparison are presented in the form of graphs of training time and accuracy depending on the number of epochs and in tabular form.

The purpose of this paper is to compare classical machine learning algorithms for handwritten number classification. The following algorithms were chosen for comparison: Logistic Regression, SVM, Decision Tree, Random Forest and k-NN. MNIST handwritten digit database is used in the task of training and testing the above algorithms. The dataset consists of 70,000 images of numbers from 0 to 9. The algorithms are compared considering such criteria as the learning speed, prediction construction speed, host machine load, and classification accuracy. Each algorithm went through the training and testing phases 100 times, with the desired KPIs retained at each iteration. The results were averaged to reach reliable outcomes.

Existing techniques for hand-written digit recognition (HDR) rely heavily on the hand-coded key points and requires prior knowledge. Training an efficient HDR network with these preconditions is a complicated task. Recently, work on HDR is mainly focused on deep learning (DL) approaches and has exhibited remarkable results. However, effective detection and classification of numerals is still a challenging task due to people's varying writing styles and the presence of blurring, distortion, light and size variations in the input sample. To cope with these limitations, we present an effective and efficient HDR system, introducing a customized faster regional convolutional neural network (Faster-RCNN). This approach comprises three main steps. Initially, we develop annotations to obtain the region of interest. Then, an improved Faster-RCNN is employed in which DenseNet-41 is introduced to compute the deep features. Finally, the regressor and classification layer is used to localize and classify the digits into ten classes. The performance of the proposed method is analyzed on the standard MNIST database, which is diverse in terms of changes in lighting conditions, chrominance, shape and size of digits, and the occurrence of blurring and noise effects, etc. Additionally, we have also evaluated our technique over a cross-dataset scenario to prove its efficacy. Experimental evaluations demonstrate that the approach is more competent and able to accurately detect and classify numerals than other recent methods. [ABSTRACT FROM AUTHOR]

Modern day lightweight block ciphers provide powerful encryption methods for securing IoT communication data. Tiny digital devices exchange private data which the individual users might not be willing to get disclosed. On the other hand, the adversaries try their level best to capture this private data. The first step towards this is to identify the encryption scheme. This work is an effort to construct a distinguisher to identify the cipher used in encrypting the traffic data. We try to establish a deep learning based method to identify the encryption scheme used from a set of three lightweight block ciphers viz. LBlock, PRESENT and SPECK. We make use of images from MNIST and fashion MNIST data sets for establishing the cryptographic distinguisher. Our results show that the overall classification accuracy depends firstly on the type of key used in encryption and secondly on how frequently the pixel values change in original input image. [ABSTRACT FROM AUTHOR]

Spiking neural network (SNN), known as the third generation of neural networks, is attracting more and more researchers’ attention because of its high energy efficiency. However, due to the spatiotemporal characteristics, SNN involved a few complicated computations such as exponential function and logarithmic function, making it hard to implement on hardware. In this brief, we propose a SNN processor, which balances the trade-offs between cost and performance. In terms of cost, through software and hardware co-design, a high-robustness SNN model with 2-bit weights and a novel synapse delay management mechanism are adopted to reduce memory utilization. In terms of performance, a spike encoder and a VFA (Vote-For-All) decoder are used to reduce latency and improve inference accuracy respectively. We implement the design on the Xilinx ZCU102 FPGA board and apply it to the MNIST handwritten digit classification, which achieves 90.53% classification accuracy. Compared to a previous proposed SNN processor of a similar neuron scale, our design achieves a 156 × inference speed-up and consumes 0,32 × hardware resources. [ABSTRACT FROM AUTHOR]