Search

Advanced artificial intelligence (AI) techniques have led to significant developments in optical character recognition (OCR) technologies. OCR applications, using AI techniques for transforming images of typed text, handwritten text, or other forms of text into machine-encoded text, provide a fair degree of accuracy for general text. However, even after decades of intensive research, creating OCR with human-like abilities has remained evasive. One of the challenges has been that OCR models trained on general text do not perform well on localized or personalized handwritten text due to differences in the writing style of alphabets and digits. This study aims to discuss the steps needed to create an adaptive framework for OCR models, with the intent of exploring a reasonable method to customize an OCR solution for a unique dataset of English language numerical digits were developed for this study. We develop a digit recognizer by training our model on the MNIST dataset with a convolutional neural network and contrast it with multiple models trained on combinations of the MNIST and custom digits. Using our methods, we observed results comparable with the baseline and provided recommendations for improving OCR accuracy for localized or personalized handwritten text. This study also provides an alternative perspective to generating data using conventional methods, which can serve as a gold standard for custom data augmentation to help address the challenges of scarce data and data imbalance.

In the era of noisy intermediate scaled quantum computers, one of the possible applications to search for an advantage of quantum computing is machine learning. Here, we report about an analysis, where a hybrid quantum‐classical network is applied to classify non‐trivial datasets (finance and MNIST data). In comparison to a pure classical network, we find an advantage when looking at several performance measures. As in classical machine learning, problems around overfitting the dataset arise. Therefore, we explore different possibilities to regularise the network.

Slides presented at the NANMATH 2022 conference, Cluj. Presentation abstract: motivated by statistical learning applications, the stochastic descent optimization algorithms are widely used today to tackle difficult numerical problems. One of the most known among them, the Stochastic Gradient Descent (SGD), has been extended in various ways resulting in Adam, Nesterov, momentum, etc. After a brief introduction to this framework, we introduce in this talk a new approach, called SGD-G2, which is a high order Runge-Kutta stochastic descent algorithm; the procedure allows for step adaptation in order to strike a optimal balance between convergence speed and stability. Numerical tests on standard datasets in machine learning are also presented together with further theoretical extensions.

This study aims to modify Vision Transformer (ViT) to achieve higher accuracy. ViT is a model used in computer vision to, among other things, classify images. By applying ViT to the MNIST data set, an accuracy of approximately 98% is achieved. ViT is modified by implementing a method called Histogram of Oriented Gradients (HOG) in two different ways. The results show that the first approach with HOG gives an accuracy of 98,74% (setup 1) and the second approach gives an accuracy of 96,87% (patch size 4x4 pixels). The study shows that when HOG is applied on the entire image, a better accuracy is obtained. However, no systematic optimization has taken place, which makes it difficult to draw conclusions with certainty.

Artificial Neural Networks make up the core of mostMachine Learning algorithms. In the past decade Machine learninghave successfully taken on fields such as image recognition,Data analytics and medical technologies. As the area of usebecome less prone to mistakes, it raises the responsibility lookinto the black box of code and understand it to a deeper level. Inthis project, I built a Deep Neural Network from scratch, withouthigh level libraries, and trained it for a supervised classificationtask. The finished algorithm is flexible and can be adapted toany classification problem. The training method is based onBackpropagation and Gradient Descent. At last, the algorithmwas trained on the Modified National Institute of Standardsand Technology (MNIST) database, and performed with a 77%prediction acccuracy. There are a few optimization methods yetto be tested to further increase the performance. Artificiella neurala nätverk utgör kärnan i de flesta maskininlärningsalgoritmer idag. Under det senaste decenniet har maskininlärning framgångsrikt tagit an områden som bildigenkänning, dataanalys och medicinsk teknik. När användningsområdena blir mindre benägna till misstag, ökar ansvaret av att titta under huven och förstå den djupare nivåkoden. I denna studie var syftet att bygga ett djupt neuralt nätverk från grunden, utan högnivåbibliotek, och träna det för en övervakad klassificeringsuppgift. Den färdiga algoritmen är flexibel och kan designas för flera klassificeringsproblem. Nätverkets träningsmetod är baserad på Backpropagation och Gradient Descent. Valideringsdatan kunde till slut köras med 77% korrekt noggrannhet, och det finns finns ytterligare optimeringsmetoder att testa för att höja prestationen. Kandidatexjobb i elektroteknik 2022, KTH, Stockholm

The article is an overview. We carry out the comparison of actual machine learning libraries that can be used the neural networks development. The first part of the article gives a brief description of TensorFlow, PyTorch, Theano, Keras, SciKit Learn libraries, SciPy library stack. An overview of the scope of these libraries and the main technical characteristics, such as performance, supported programming languages, the current state of development is given. In the second part of the article, a comparison of five libraries is carried out on the example of a multilayer perceptron, which is applied to the problem of handwritten digits recognizing. This problem is well known and well suited for testing different types of neural networks. The study time is compared depending on the number of epochs and the accuracy of the classifier. The results of the comparison are presented in the form of graphs of training time and accuracy depending on the number of epochs and in tabular form.

application/pdf, 論文(Article), ニューラルネットワークの研究には20年に渡ってベンチマークの役割を担ってきたデータセットが存在し、世界中でニューラルネットワークの研究に使用されている。近年、第二のベンチマークを目指したデータセットが公開され人工知能研究に寄与しているが、両者の違いや学習の困難度等に関する評価が定まっていない。本稿では第二のデータセットと第一のデータセットの違いを検証し、将来の第三のデータセット構築のための手掛かりを模索する。, In the research of neural networks, one data set has played the benchmark role for 20 years, and it is currently used worldwide for such research. In recent years, a second data set has been published with the aim to create a second benchmark and contribute to artificial intelligence. However, the differences between the two data sets and difficulty in learning have not been established. In this paper, we examine the differences between the two data sets and explore the possibilities of constructing a third data set in the future.

Understanding why deep nets can classify data in large dimensions remains a challenge. It has been proposed that they do so by becoming stable to diffeomorphisms, yet existing empirical measurements support that it is often not the case. We revisit this question by defining a maximum-entropy distribution on diffeomorphisms, that allows to study typical diffeomorphisms of a given norm. We confirm that stability toward diffeomorphisms does not strongly correlate to performance on benchmark data sets of images. By contrast, we find that the stability toward diffeomorphisms relative to that of generic transformations $R_f$ correlates remarkably with the test error $\epsilon_t$. It is of order unity at initialization but decreases by several decades during training for state-of-the-art architectures. For CIFAR10 and 15 known architectures, we find $\epsilon_t\approx 0.2\sqrt{R_f}$, suggesting that obtaining a small $R_f$ is important to achieve good performance. We study how $R_f$ depends on the size of the training set and compare it to a simple model of invariant learning., Comment: NeurIPS 2021 Conference

In this chapter, we consider realizing a reservoir computer on an electronic chip that allows for many tens of network nodes whose connection topology can be quickly reconfigured. The reservoir computer displays analog-like behavior and has the potential to perform computations beyond that of a classic Turning machine. In detail, we present our preliminary results of using a physical reservoir computer for performing the task of identifying written digits. The reservoir is realized on a commercially available electronic device known as a field-programmable gate array on which we create an autonomous Boolean network for information processing. Even though the network nodes are Boolean logic elements, they display analog behavior because there is no master clock that controls the nodes. In addition, the electronic signals related to the written-digit images are injected into the reservoir at high speed, leading to the possibility of full-image classification on the nanosecond time scale. We explore the dynamics of the autonomous Boolean networks in response to injected signals and, based on these results, investigate the performance of the reservoir computer on the written-digit task. For a wide range of reservoir structures, we obtain a typical performance of \(\sim \)90% for correctly identifying a written digit, which exceeds that obtained by a linear classifier. This work paves the way for achieving low-power, high-speed reservoir computing on readily available field-programmable gate arrays, which are well matched to existing computing infrastructure.

Ovaj rad analizira potencijalne mogućnosti tehnike federativnog učenja, FedAvg, na uređajima ograničenih performansi. Pojašnjava se koncept federativnog učenja te se ukratko opisuju temeljne kategorije. Navode se mogući slučajevi uporabe federativnog učenja u razvoju stvarnih rješenja te problemi koje metoda rješava. Implementacija se odnosi na kategoriju horizontalnog federativnog učenja u kojoj je korišten ESP32 mikrokontroler za treniranje modela neuronske mreže nad skupom podataka MNIST. Razmatraju se rubni slučajevi te mogući problemi kod korištenja metode federativnog učenja. Rezultati eksperimenata su komentirani i navode se sljedeći koraci koji bi poboljšali postojeću implementaciju. This thesis analyses the potential capabilities and the use cases of federated learning method called federated averaging on the devices of extremely limited memory size and processing power. Federated learning concept is explained and its basic categories are briefly summarized. Possible use cases of federated learning are mentioned, as are the issues that could be mitigated with the use of its methods. ESP32 microcontroller was used in order to perform the training of the neural network model on the MNIST dataset. Edge cases of the generated federated averaging model were studied and possible issues were raised. Results of the experiments have been commented and next increments in the implementation of training algorithm have been proposed.

In recent years, neural networks have grown in popularity, mostly thanks to the advances in the field of high performance computing. Nevertheless, some factors are still limiting the usage of neural networks. In particular, two limiting factors are storage requirements and computational cost. The aim of this project is to radically improve storage demand and provide direction for accelerating the execution of neural networks. In the scope of this thesis two compression algorithms have been developed. These algorithms share a common basis, both exploit error-tolerance is a property, because of this property the weight matrix can be divided into blocks simplifying the problem while merely impacting the accuracy. The first algorithm, groups the weights inside every block using different clustering techniques: Arithmetic mean and K-Means. To decide which clustering method to apply to which block standard deviation is employed among others. The user can specify a trade-off between accuracy and compression. This method has underperformed, obtaining a compression rate of 10,57 for AlexNet, which is not nearly state-of-the-art. The main issue is that meaningless weights are being merged with significant ones, causing a significant drop in the accuracy. The second algorithm, takes on the problem of accuracy loss by pruning all the unimportant weights. After pruning, quantization is applied. For both steps, pruning and quantization, two options have been explored which are effective for different kinds of neural networks. Of the possible combinations between pruning and quantization, one is selected by trial-and-error. The first pruning technique focuses on removing as many weights as possible, while the second pruning method considers blocks to a greater extend. The two types of quantization allow three values per block and five values per block respectively. This algorithm performed very well, obtaining a compression rate of 57,15 for AlexNet with minimal accuracy loss.

Deep Neural Networks (DNNs) have become the de-facto standard in computer vision, as well as in many other pattern recognition tasks. A key drawback of DNNs is that the training phase can be very computationally expensive. Organizations or individuals that cannot afford purchasing state-of-the-art hardware or tapping into cloud hosted infrastructures may face a long waiting time before the training completes or might not be able to train a model at all. Investigating novel ways to reduce the training time could be a potential solution to alleviate this drawback, and thus enabling more rapid development of new algorithms and models. In this paper, we propose LightLayers, a method for reducing the number of trainable parameters in deep neural networks (DNN). The proposed LightLayers consists of LightDense and LightConv2D layer that are as efficient as regular Conv2D and Dense layers, but uses less parameters. We resort to Matrix Factorization to reduce the complexity of the DNN models resulting into lightweight DNN models that require less computational power, without much loss in the accuracy. We have tested LightLayers on MNIST, Fashion MNIST, CIFAR 10, and CIFAR 100 datasets. Promising results are obtained for MNIST, Fashion MNIST, CIFAR-10 datasets whereas CIFAR 100 shows acceptable performance by using fewer parameters.

Rad daje uvod u umjetne neuronske mreže s naglaskom na konvolucijskim modelima. Ukratko je opisano učenje neuronske mreže. U radu uvodimo dodatni modul za prostornu transformaciju u konvolucijski model, kako bi poboljšali prostornu invarijantnost modela. Opisujemo arhitekturu dodatnog modula te dvije implementirane deformacije - afinu transformaciju i thin-plate spline transformaciju. Dana je programska implementacija modela te metode za učenje i evaluaciju istih. Modeli su trenirani na skupu MNIST. Prikazani su usporedni rezultati dvaju modela s modulom za prostornu transformaciju i modela bez dodatnog modula. This paper gives an introduction to artificial neural networks with emphasis on convolutional models. We described briefly learning process of neural network. In this work we introduce additional spatial transformer module in convolutional model to increase spatial invariance. We describe the architecture of additional module and two implemented deformations -- affine transformation and thin-plate spline transformation. This paper gives code of model in PyTorch and methods for training and testing models. We trained models on MNIST dataset. We gave comparative results of two models with spatial transformations and third model without STN.

Проблематика. Оскільки попередні підходи до покращення рівня помилок банку зображень MNIST не мають чіткої структури, яка могла би бути чітко відтворена, розглядається формалізація покращення продуктивності. Мета дослідження. Метою роботи є строга формалізація стратегії зниження рівня помилок банку даних MNIST. Методика реалізації. Пропонується алгоритм досягнення кращої продуктивності за допомогою збільшення обсягу даних навчання та бустингу. Алгоритм використовує розроблену концепцію збільшення обсягу даних навчання. Координація цієї концепції та алгоритму визначає стратегію зниження рівня помилок. Результати дослідження. У відносному порівнянні продуктивність однієї згорткової нейронної мережі на банку даних MNIST покращена майже на 30 %. За допомогою бустингу продуктивність становить 0,21 % – не розпізнається лише 21 рукописна цифра з 10000. Висновки. Збільшення обсягу даних навчання є визначальним для зниження рівня помилок банку даних MNIST. Без цього бустинг неефективний. Застосування викладеного підходу має виразний вплив на зниження рівня помилок банку даних MNIST при використанні лише 5 або 6 згорткових нейронних мереж проти 35 в еталонній роботі. Background. Due to that the preceding approaches for improving the MNIST image dataset error rate do not have a clear structure which could let repeat it in a strengthened manner, the formalization of the performance improvement is considered. Objective. The goal is to strictly formalize a strategy of reducing the MNIST dataset error rate. Methods. An algorithm for achieving the better performance by expanding the training data and boosting with ensembles is suggested. The algorithm uses the designed concept of the training data expansion. Coordination of the concept and the algorithm defines a strategy of the error rate reduction. Results. In relative comparison, the single convolutional neural network performance on the MNIST dataset has been bettered almost by 30 %. With boosting, the performance is 0.21 % error rate meaning that only 21 handwritten digits from 10,000 are not recognized. Conclusions. The training data expansion is crucial for reducing the MNIST dataset error rate. The boosting is ineffective without it. Application of the stated approach has an impressive impact for reducing the MNIST dataset error rate, using only 5 or 6 convolutional neural networks against those 35 ones in the benchmark work. Проблематика. Поскольку предыдущие подходы к улучшению уровня ошибок набора данных MNIST не имеют четкой структуры, которая могла бы быть четко воссоздана, рассматривается формализация улучшения производительности. Цель исследования. Целью работы является строгая формализация стратегии снижения уровня ошибок набора данных MNIST. Методика реализации. Предлагается алгоритм достижения лучшей производительности с помощью увеличения объема данных обучения и бустинга. Алгоритм использует разработанную концепцию увеличения объема данных обучения. Координация этой концепции и алгоритма определяет стратегию снижения уровня ошибок. Результаты исследования. В относительном сравнении производительность одной сверточной нейронной сети на наборе данных MNIST улучшена почти на 30 %. С помощью бустинга производительность составляет 0,21 % – не распознается лишь 21 рукописная цифра из 10000. Выводы. Увеличение объема данных обучения является определяющим для снижения уровня ошибок. Без этого бустинг неэффективен. Применение изложенного подхода оказывает выразительное влияние на снижение уровня ошибок набора данных MNIST при использовании лишь 5 или 6 сверточных нейронных сетей против 35 в эталонной работе.

Digit classifier yang dikembangkan, digunakan untuk mengklasifikasikan angka dari tulisantangan. Sudah banyak sekali digit classifier yang dikembangkan dengan berbagai algoritmamachine learning, salah satu yang populer dan terus berkembang adalah jaringan saraf tiruan.Dengan motivasi tersebut, penulis juga membangun digit classifier menggunakan jaringan saraftiruan yang dipadukan dengan teknik optimisasi. Jenis arsitektur jaringan saraf tiruan yang akandigunakan adalah feedforward dan back propagation. Teknik optimisasi feature normalization danprincipal component analysis (PCA) juga akan digunakan untuk meningkatkan performa model.Set data yang digunakan untuk melatih model merupakan data Mixed National Institute ofStandards and Technology (MNIST). Dengan memadukan jaringan saraf tiruan dan teknik-teknikoptimisasi tersebut diharapkan dapat meningkatkan performa dan akurasi untukmengklasifikasikan angka serta memberikan pemahaman baru bagi penulis.

We present a method for synthesising deep neural networks using Extreme Learning Machines (ELMs) as a stack of supervised autoencoders. We test the method using standard benchmark datasets for multi-class image classification (MNIST, CIFAR-10 and Google Streetview House Numbers (SVHN)), and show that the classification error rate can progressively improve with the inclusion of additional autoencoding ELM modules in a stack. Moreover, we found that the method can correctly classify up to 99.19% of MNIST test images, which surpasses the best error rates reported for standard 3-layer ELMs or previous deep ELM approaches when applied to MNIST. The approach simultaneously offers a significantly faster training algorithm to achieve its best performance (in the order of 5. min on a four-core CPU for MNIST) relative to a single ELM with the same total number of hidden units as the deep ELM, hence offering the best of both worlds: lower error rates and fast implementation. Refereed/Peer-reviewed

Postoje 3 vrste strojnog učenja: nadzirano, nenadzirano i podržano učenje. U ovom radu su opisane 3 metode nadziranog učenja: logistička regresija, softmax regresija i umjetne neuronske mreže. Logistička regresija služi za binarnu klasifikaciju, dok softmax regresija služi za višeklasnu klasifikaciju. S obzirom da ne daju dobre rezultate za linearno neodvojive probleme, takvi problemi se mogu rješavati umjetnim neuronskim mrežama zahvaljujući njenim skrivenim slojevima. Optičko prepoznavanje znakova implementirano je umjetnom neuronskom mrežom te se pri tome korisi biblotekom TensorFlow i podatcima iz baze MNIST. Veliku ulogu u treniranju umjetne neuronske mreže imaju takozvani hiperparametri: stopa učenja, veličina uzorka i broj epoha. Rezultati točnosti uvelike ovise o hiperparametrima mreže. There are 3 types of machine learning: supervised, unsupervised and reinforcement learning. In this thesis are explained 3 methods of supervised learning: logistic regression, softmax regression and artificial neural networks. Logistic regression is used for binary classification while softmax is used for multiclass classification. These methods don't offer good results for linearly unseparable problems where artificial neural networks can be used thanks to their hidden-layers. Optical character recognition is implemented with neural networks and uses TensorFlow library and MNIST database. Hyperparameters play huge role in training and accuracy of a neural networks. Parameters that have been studied in this thesis are learning rate, batch size and number of epochs.

Klasifikacija slika prirodnih scena je neriješen problem računalnog vida s mnogim zanimljivim primjenama. U posljednje vrijeme najbolji rezultati u tom području postižu se pristupima utemeljenima na dubokim konvolucijskim modelima. Za ovaj rad posebno su zanimljivi nadzirani pristupi gdje je svaka slika skupa za učenje označena semantičkim razredima objekata koje slika sadrži. U okviru rada, bilo je potrebno proučiti dokumentacije programskih okvira Tensorflow i PyTorch te biblioteke programskog jezika Python za rukovanje matricama i slikama. Izrađena je izvedba programskog sustava za učenje i primjenu klasifikacijskog modela. Evaluiran je utjecaj rezidualnih veza na točnost modela. Detaljno su analizirane konvolucijske mreže u okviru dubokog učenja. Prikazani su i ocijenjeni ostvareni rezultati i predložene izmjene za poboljšanje rezultata. Image classification of natural scenes is an unsolved problem of computer vision with many interesting applications. Recently, approaches with deep convolutional models achieve best results in that area. For this paper were specially interesting supervised approaches where every image from training set is labeled with semantic classes of objects which image contains. In this paper documentations of programming frameworks Tensorflow and PyTorch for matrix and image handling were studied. Implementation of programming system for learning and application of classification model have been done. Implact of residual connections on model accuracy have been evaluated. Convolutional networks were deeply analized in context of deep learning. Achieved results have been shown and evaluated and suggestions for classification accuracy improvment have been provided.

In a broad range of computer vision tasks, convolutional neural networks (CNNs) are one of the most prominent techniques due to their outstanding performance. Yet it is not trivial to find the best performing network structure for a specific application because it is often unclear how the network structure relates to the network accuracy. We propose an evolutionary algorithm-based framework to automatically optimize the CNN structure by means of hyper-parameters. Further, we extend our framework towards a joint optimization of a committee of CNNs to leverage specialization and cooperation among the individual networks. Experimental results show a significant improvement over the state-of-the-art on the well-established MNIST dataset for hand-written digits recognition.

Date of Conference: 15-18 May 2017 Conference Name: IEEE 25th Signal Processing and Communications Applications Conference, SIU 2017 In this paper, we propose a new energy efficient neural network with the universal approximation property over space of Lebesgue integrable functions. This network, called additive neural network, is very suitable for mobile computing. The neural structure is based on a novel vector product definition, called ef-operator, that permits a multiplier-free implementation. In ef-operation, the 'product' of two real numbers is defined as the sum of their absolute values, with the sign determined by the sign of the product of the numbers. This 'product' is used to construct a vector product in n-dimensional Euclidean space. The vector product induces the lasso norm. The proposed additive neural network successfully solves the XOR problem. The experiments on MNIST dataset show that the classification performances of the proposed additive neural networks are very similar to the corresponding multi-layer perceptron.