Search

The recognition of handwritten digits has aroused the interest of the scientific community and is the subject of a large number of research works thanks to its various applications. The objective of this paper is to develop a system capable of recognizing handwritten digits using a convolutional neural network (CNN) combined with machine learning approaches to ensure diversity in automatic classification tools. In this work, we propose a classification method based on deep learning, in particular the convolutional neural network for feature extraction, it is a powerful tool that has had great success in image classification, followed by the support vector machine (SVM) for higher performance. We used the dataset (MNIST), and the results obtained showed that the combination of CNN with SVM improves the performance of the model as well as the classification accuracy with a rate of 99.12%.

Neuralne mreže univerzalni su aproksimatori funkcija te stoga primjenjivi na rješavanja gotovo svakog oblika problema. U ovom završnom radu obraditi će se način na koji neuralne mreže uče, uvesti konvoluciju u pristup učenju i pokazati rezultate implementiranih algoritama na nekoliko različitih skupova podataka. Neural networks are universal approximators and as such applicable on almost any problem. This bachelor thesis will process how neural networks learn, introduce convolution in its learning process and show results achieved by implemented algorithms on a few different datasets.

Our project is meant to implement recognition to text and digit of any input image. We separate our project into two parts ie, recognition of digit and character by using both CNN and LSTM algorithms. Our approach builds upon yan Lecun datasets in digit recognition, and applies the techniques to characters’ recognition. Yan lecun datasets include MNIST which is for recognition of digits and EMNIST for recognition of alphabets. Handwritten digit recognition is the ability of a computing system to acknowledge the written inputs like characters etc from a large sort of sources like emails, papers, images, letters etc. This has been a subject of analysis for many years. A number of the analysis areas embrace bank check processing, postal address interpretation from envelopes etc. Handwritten digit recognition plays a significant role in many user authentication applications in the modern world. Character Recognition has been an active area of research in the past and due to its diverse applications it continues to be a challenging topic. The idea is to provide efficient algorithms which takes input as character and predict the output in digital format. After that it processes the image for better comparison. Then after the processed image is compared with already available set of font images. The last step gives a prediction of the character in percentage accuracy.

In this paper, we present a case study on approximate multipliers for MNIST Convolutional Neural Network (CNN). We apply approximate multipliers with different bit-width to the convolution layer in MNIST CNN, evaluate the accuracy of MNIST classification, and analyze the trade-off between approximate multiplier’s area, critical path delay and the accuracy. Based on the results of the evaluation and analysis, we propose a design methodology for approximate multipliers. The approximate multipliers consist of some partial products, which are carefully selected according to the CNN input. With this methodology, we further reduce the area and the delay of the multipliers with keeping high accuracy of the MNIST classification.

Deep Neural Networks (DNNs) have become the de-facto standard in computer vision, as well as in many other pattern recognition tasks. A key drawback of DNNs is that the training phase can be very computationally expensive. Organizations or individuals that cannot afford purchasing state-of-the-art hardware or tapping into cloud hosted infrastructures may face a long waiting time before the training completes or might not be able to train a model at all. Investigating novel ways to reduce the training time could be a potential solution to alleviate this drawback, and thus enabling more rapid development of new algorithms and models. In this paper, we propose LightLayers, a method for reducing the number of trainable parameters in deep neural networks (DNN). The proposed LightLayers consists of LightDense and LightConv2D layer that are as efficient as regular Conv2D and Dense layers, but uses less parameters. We resort to Matrix Factorization to reduce the complexity of the DNN models resulting into lightweight DNN models that require less computational power, without much loss in the accuracy. We have tested LightLayers on MNIST, Fashion MNIST, CIFAR 10, and CIFAR 100 datasets. Promising results are obtained for MNIST, Fashion MNIST, CIFAR-10 datasets whereas CIFAR 100 shows acceptable performance by using fewer parameters.

Convolutional Neural Networks (CNNs) are becoming increasingly popular in deep learning applications, e.g. image classification, speech recognition, medicine, to name a few. However, CNN inference is computationally intensive and demands a large number of memory resources. This work proposes a CNN inference hardware accelerator, which was implemented in a co-processing scheme. The aim is to reduce hardware resources and achieve the best possible throughput. The design is implemented in the Digilent Arty Z7-20 development board, which is based on the Xilinx Zynq-7000 System on Chip (SoC). Our implementation achieved a of accuracy for the MNIST database using only a 12-bits fixed-point format. Results show that the co-processing scheme operating at a conservative speed of 100 MHz can identify around 441 images per second, which is about 17% times faster than a 650 MHz - software implementation. It is difficult to compare our results against other Field-Programmable Gate Array (FPGA)-based implementations because they are not exactly like ours. However, some comparisons, regarding logical resources used and accuracy, suggest that our work could be better than previous ones. Besides, the proposed scheme is compared with a hardware implementation in terms of power consumption and throughput. Resumen: Las redes neuronales convolucionales cada vez son más populares en aplicaciones de aprendizaje profundo, como por ejemplo en clasificación de imágenes, reconocimiento de voz, medicina, entre otras. Sin embargo, estas redes son computacionalmente costosas y requieren altos recursos de memoria. En este trabajo se propone un acelerador en hardware para el proceso de inferencia de la red Lenet-5, un esquema de coprocesamiento hardware/software. El objetivo de la implementación es reducir el uso de recursos de hardware y obtener el mejor rendimiento computacional posible durante el proceso de inferencia. El diseño fue implementado en la tarjeta de desarrollo Digilent Arty Z7-20, la cual está basada en el System on Chip (SoC) Zynq-7000 de Xilinx. Nuestra implementación logró una precisión del 97,59 % para la base de datos MNIST utilizando tan solo 12 bits en el formato de punto fijo. Los resultados muestran que el esquema de co-procesamiento, el cual opera a una velocidad de 100 MHz, puede identificar aproximadamente 441 imágenes por segundo, que equivale aproximadamente a un 17% más rápido que una implementación de software a 650 MHz. Es difícil comparar nuestra implementación con otras implementaciones similares, porque las implementaciones encontradas en la literatura no son exactamente como la que realizó en este trabajo. Sin embargo, algunas comparaciones, en relación con el uso de recursos lógicos y la precisión, sugieren que nuestro trabajo supera a trabajos previos.

Rad daje uvod u umjetne neuronske mreže s naglaskom na konvolucijskim modelima. Ukratko je opisano učenje neuronske mreže. U radu uvodimo dodatni modul za prostornu transformaciju u konvolucijski model, kako bi poboljšali prostornu invarijantnost modela. Opisujemo arhitekturu dodatnog modula te dvije implementirane deformacije - afinu transformaciju i thin-plate spline transformaciju. Dana je programska implementacija modela te metode za učenje i evaluaciju istih. Modeli su trenirani na skupu MNIST. Prikazani su usporedni rezultati dvaju modela s modulom za prostornu transformaciju i modela bez dodatnog modula. This paper gives an introduction to artificial neural networks with emphasis on convolutional models. We described briefly learning process of neural network. In this work we introduce additional spatial transformer module in convolutional model to increase spatial invariance. We describe the architecture of additional module and two implemented deformations -- affine transformation and thin-plate spline transformation. This paper gives code of model in PyTorch and methods for training and testing models. We trained models on MNIST dataset. We gave comparative results of two models with spatial transformations and third model without STN.

The back-propagation (BP) algorithm is usually used to train convolutional neural networks (CNNs) and has made greater progress in image classification. It updates weights with the gradient descent, and the farther the sample is from the target, the greater the contribution of it to the weight change. However, the influence of samples classified correctly but that are close to the classification boundary is diminished. This paper defines the classification confidence as the degree to which a sample belongs to its correct category, and divides samples of each category into dangerous and safe according to a dynamic classification confidence threshold. Then a new learning algorithm is presented to penalize the loss function with danger samples but not all samples to enable CNN to pay more attention to danger samples and to learn effective information more accurately. The experiment results, carried out on the MNIST dataset and three sub-datasets of CIFAR-10, showed that for the MNIST dataset, the accuracy of Non-improve CNN reached 99.246%, while that of PCNN reached 99.3%, for three sub-datasets of CIFAR-10, the accuracies of Non-improve CNN are 96.15%, 88.93%, and 94.92%, respectively, while those of PCNN are 96.44%, 89.37%, and 95.22%, respectively.

Neural networks have enabled great advances in recent times due mainly to improved parallel computing capabilities in accordance to Moore’s Law, which allowed reducing the time needed for the parameter learning of complex, multi-layered neural architectures. However, with silicon technology reaching its physical limits, new types of computing paradigms are needed to increase the power efficiency of learning algorithms, especially for dealing with deep spatio-temporal knowledge on embedded applications. With the goal of mimicking the brain’s power efficiency, new hardware architectures such as the SpiNNaker board have been built. Furthermore, recent works have shown that networks using spiking neurons as learning units can match classical neural networks in supervised tasks. In this paper, we show that the implementation of state-of-the-art models on both the MNIST and the event-based NMNIST digit recognition datasets is possible on neuromorphic hardware. We use two approaches, by directly converting a classical neural network to its spiking version and by training a spiking network from scratch. For both cases, software simulations and implementations into a SpiNNaker 103 machine were performed. Numerical results approaching the state of the art on digit recognition are presented, and a new method to decrease the spike rate needed for the task is proposed, which allows a significant reduction of the spikes (up to 34 times for a fully connected architecture) while preserving the accuracy of the system. With this method, we provide new insights on the capabilities offered by networks of spiking neurons to efficiently encode spatio-temporal information. Consejo Nacional de Ciencia Y Tecnología (México) FC2016-1961 European Union's Horizon 2020 No 824164 HERMES Ministerio de Ciencia, Innovación y Universidades TEC2015-63884-C2-1-P

Most capsule network designs rely on traditional matrix multiplication between capsule layers and computationally expensive routing mechanisms to deal with the capsule dimensional entanglement that the matrix multiplication introduces. By using Homogeneous Vector Capsules (HVCs), which use element-wise multiplication rather than matrix multiplication, the dimensions of the capsules remain unentangled. In this work, we study HVCs as applied to the highly structured MNIST dataset in order to produce a direct comparison to the capsule research direction of Geoffrey Hinton, et al. In our study, we show that a simple convolutional neural network using HVCs performs as well as the prior best performing capsule network on MNIST using 5.5x fewer parameters, 4x fewer training epochs, no reconstruction sub-network, and requiring no routing mechanism. The addition of multiple classification branches to the network establishes a new state of the art for the MNIST dataset with an accuracy of 99.87% for an ensemble of these models, as well as establishing a new state of the art for a single model (99.83% accurate)., Comment: 13 pages, 7 figures, 9 tables

Проблематика. Оскільки попередні підходи до покращення рівня помилок банку зображень MNIST не мають чіткої структури, яка могла би бути чітко відтворена, розглядається формалізація покращення продуктивності. Мета дослідження. Метою роботи є строга формалізація стратегії зниження рівня помилок банку даних MNIST. Методика реалізації. Пропонується алгоритм досягнення кращої продуктивності за допомогою збільшення обсягу даних навчання та бустингу. Алгоритм використовує розроблену концепцію збільшення обсягу даних навчання. Координація цієї концепції та алгоритму визначає стратегію зниження рівня помилок. Результати дослідження. У відносному порівнянні продуктивність однієї згорткової нейронної мережі на банку даних MNIST покращена майже на 30 %. За допомогою бустингу продуктивність становить 0,21 % – не розпізнається лише 21 рукописна цифра з 10000. Висновки. Збільшення обсягу даних навчання є визначальним для зниження рівня помилок банку даних MNIST. Без цього бустинг неефективний. Застосування викладеного підходу має виразний вплив на зниження рівня помилок банку даних MNIST при використанні лише 5 або 6 згорткових нейронних мереж проти 35 в еталонній роботі. Background. Due to that the preceding approaches for improving the MNIST image dataset error rate do not have a clear structure which could let repeat it in a strengthened manner, the formalization of the performance improvement is considered. Objective. The goal is to strictly formalize a strategy of reducing the MNIST dataset error rate. Methods. An algorithm for achieving the better performance by expanding the training data and boosting with ensembles is suggested. The algorithm uses the designed concept of the training data expansion. Coordination of the concept and the algorithm defines a strategy of the error rate reduction. Results. In relative comparison, the single convolutional neural network performance on the MNIST dataset has been bettered almost by 30 %. With boosting, the performance is 0.21 % error rate meaning that only 21 handwritten digits from 10,000 are not recognized. Conclusions. The training data expansion is crucial for reducing the MNIST dataset error rate. The boosting is ineffective without it. Application of the stated approach has an impressive impact for reducing the MNIST dataset error rate, using only 5 or 6 convolutional neural networks against those 35 ones in the benchmark work. Проблематика. Поскольку предыдущие подходы к улучшению уровня ошибок набора данных MNIST не имеют четкой структуры, которая могла бы быть четко воссоздана, рассматривается формализация улучшения производительности. Цель исследования. Целью работы является строгая формализация стратегии снижения уровня ошибок набора данных MNIST. Методика реализации. Предлагается алгоритм достижения лучшей производительности с помощью увеличения объема данных обучения и бустинга. Алгоритм использует разработанную концепцию увеличения объема данных обучения. Координация этой концепции и алгоритма определяет стратегию снижения уровня ошибок. Результаты исследования. В относительном сравнении производительность одной сверточной нейронной сети на наборе данных MNIST улучшена почти на 30 %. С помощью бустинга производительность составляет 0,21 % – не распознается лишь 21 рукописная цифра из 10000. Выводы. Увеличение объема данных обучения является определяющим для снижения уровня ошибок. Без этого бустинг неэффективен. Применение изложенного подхода оказывает выразительное влияние на снижение уровня ошибок набора данных MNIST при использовании лишь 5 или 6 сверточных нейронных сетей против 35 в эталонной работе.

This paper summarizes the top state-of-the-art contributions reported on the MNIST dataset for handwritten digit recognition. This dataset has been extensively used to validate novel techniques in computer vision, and in recent years, many authors have explored the performance of convolutional neural networks (CNNs) and other deep learning techniques over this dataset. To the best of our knowledge, this paper is the first exhaustive and updated review of this dataset, there are some online rankings, but they are outdated, and most published papers survey only closely related works, omitting most of the literature. This paper makes a distinction between those works using some kind of data augmentation and works using the original dataset out-of-the-box. Also, works using CNNs are reported separately, as they are becoming the state-of-the-art approach for solving this problem. Nowadays, a significant amount of works have attained a test error rate smaller than 1% on this dataset, which is becoming non-challenging. By mid-2017, a new dataset was introduced: EMNIST, which involves both digits and letters, with a larger amount of data acquired from a database different than MNIST&rsquo, s. In this paper, EMNIST is explained and some results are surveyed.

In a broad range of computer vision tasks, convolutional neural networks (CNNs) are one of the most prominent techniques due to their outstanding performance. Yet it is not trivial to find the best performing network structure for a specific application because it is often unclear how the network structure relates to the network accuracy. We propose an evolutionary algorithm-based framework to automatically optimize the CNN structure by means of hyper-parameters. Further, we extend our framework towards a joint optimization of a committee of CNNs to leverage specialization and cooperation among the individual networks. Experimental results show a significant improvement over the state-of-the-art on the well-established MNIST dataset for hand-written digits recognition.

Background. Due to that the preceding approaches for improving the MNIST image dataset error rate do not have a clear structure which could let repeat it in a strengthened manner, the formalization of the performance improvement is considered.Objective. The goal is to strictly formalize a strategy of reducing the MNIST dataset error rate.Methods. An algorithm for achieving the better performance by expanding the training data and boosting with ensembles is suggested. The algorithm uses the designed concept of the training data expansion. Coordination of the concept and the algorithm defines a strategy of the error rate reduction.Results. In relative comparison, the single convolutional neural network performance on the MNIST dataset has been bettered almost by 30 %. With boosting, the performance is 0.21 % error rate meaning that only 21 handwritten digits from 10,000 are not recognized.Conclusions. The training data expansion is crucial for reducing the MNIST dataset error rate. The boosting is ineffective without it. Application of the stated approach has an impressive impact for reducing the MNIST dataset error rate, using only 5 or 6 convolutional neural networks against those 35 ones in the benchmark work., Проблематика. Оскільки попередні підходи до покращення рівня помилок банку зображень MNIST не мають чіткої структури, яка могла би бути чітко відтворена, розглядається формалізація покращення продуктивності.Мета дослідження. Метою роботи є строга формалізація стратегії зниження рівня помилок банку даних MNIST.Методика реалізації. Пропонується алгоритм досягнення кращої продуктивності за допомогою збільшення обсягу даних навчання та бустингу. Алгоритм використовує розроблену концепцію збільшення обсягу даних навчання. Координація цієї концепції та алгоритму визначає стратегію зниження рівня помилок.Результати дослідження. У відносному порівнянні продуктивність однієї згорткової нейронної мережі на банку даних MNIST покращена майже на 30 %. За допомогою бустингу продуктивність становить 0,21 % – не розпізнається лише 21 рукопис­на цифра з 10000.Висновки. Збільшення обсягу даних навчання є визначальним для зниження рівня помилок банку даних MNIST. Без цього бустинг неефективний. Застосування викладеного підходу має виразний вплив на зниження рівня помилок банку даних MNIST при використанні лише 5 або 6 згорткових нейронних мереж проти 35 в еталонній роботі., Проблематика. Поскольку предыдущие подходы к улучшению уровня ошибок набора данных MNIST не имеют четкой структуры, которая могла бы быть четко воссоздана, рассматривается формализация улучшения производительности.Цель исследования. Целью работы является строгая формализация стратегии снижения уровня ошибок набора данных MNIST.Методика реализации. Предлагается алгоритм достижения лучшей производительности с помощью увеличения объема данных обучения и бустинга. Алгоритм использует разработанную концепцию увеличения объема данных обучения. Координация этой концепции и алгоритма определяет стратегию снижения уровня ошибок.Результаты исследования. В относительном сравнении производительность одной сверточной нейронной сети на наборе данных MNIST улучшена почти на 30 %. С помощью бустинга производительность составляет 0,21 % – не распознается лишь 21 рукописная цифра из 10000.Выводы. Увеличение объема данных обучения является определяющим для снижения уровня ошибок. Без этого бустинг неэффективен. Применение изложенного подхода оказывает выразительное влияние на снижение уровня ошибок набора данных MNIST при использовании лишь 5 или 6 сверточных нейронных сетей против 35 в эталонной работе.

Deep spiking neural networks (SNNs) hold the potential for improving the latency and energy efficiency of deep neural networks through data-driven event-based computation. However, training such networks is difficult due to the non-differentiable nature of spike events. In this paper, we introduce a novel technique, which treats the membrane potentials of spiking neurons as differentiable signals, where discontinuities at spike times are considered as noise. This enables an error backpropagation mechanism for deep SNNs that follows the same principles as in conventional deep networks, but works directly on spike signals and membrane potentials. Compared with previous methods relying on indirect training and conversion, our technique has the potential to capture the statistics of spikes more precisely. We evaluate the proposed framework on artificially generated events from the original MNIST handwritten digit benchmark, and also on the N-MNIST benchmark recorded with an event-based dynamic vision sensor, in which the proposed method reduces the error rate by a factor of more than three compared to the best previous SNN, and also achieves a higher accuracy than a conventional convolutional neural network (CNN) trained and tested on the same data. We demonstrate in the context of the MNIST task that thanks to their event-driven operation, deep SNNs (both fully connected and convolutional) trained with our method achieve accuracy equivalent with conventional neural networks. In the N-MNIST example, equivalent accuracy is achieved with about five times fewer computational operations., Frontiers in Neuroscience, 10, ISSN:1662-453X, ISSN:1662-4548

U ovom radu napravljen je kratak pregled neuronskih mreža s naglaskom na konvolucijske neuronske mreže. Cilj rada bio je napraviti sustav koji iz datog skupa slika osoba koje gledaju u ekran nauči u koju točku na ekranu osoba gleda koristeći konvolucijske neuronske mreže. Korištena je gotova biblioteka za treniranje konvolucijskih neuronskih mreža Caffe, opisan je način njenog korištenja, te su implementirani vlastiti alati za prikupljanje vlastitog skupa za učenje. Na kraju su opisani dobiveni rezultati. This paper made a brief overview of neural networks with a focus on convolutional neural networks. The aim was to create a system that, from a given set of images of people who look at the screen, learns where on the screen people are looking at. The system is based on convolutional neural networks. Framework for training and deploying convolutional neural networks Caffe was used. This paper describes how it is used. In order to obtain training and testing data additional tools were implemented. Finally, the obtained results from experiments were discussed.