Search

In statistical learning theory (regression, classification, etc.) there are many regression models, such as algebraic polynomials

The gas hydrate formation in pipelines of industries and chemical plants can cause various operational damages and can increase economic risks. Hence, the knowledge of hydrate formation conditions has become a critical research study to overcome the problems arising from the formation of hydrates. In this study, we applied an algorithm to develop an LSSVM model to predict the formation temperature of natural gas hydrate for a comprehensive range of data points. Total 188 experimental data points were applied from the literature for the development of the LSSVM model. The input parameter was finalized based on the structure of hydrates by each gas species. The results obtained by the LSSVM model have good accuracy as compared with empirical correlations available in the literature. This model gave the squared correlation coefficient (R2), and root mean square error of 0.9901 and 0.59974, respectively. The composition of gases may affect the phase equilibrium condition of gas hydrates. The applied algorithm revealed that the developed LSSVM model could become a good alternative for calculating the formation temperature of hydrate for the range of all data sets. The results showed that the proposed LSSVM model could be applicable for the prediction of hydrate formation temperature for all data points.

In the paper, to improve the performance of discriminative information-based nonparallel support vector machine (DINPSVM), we propose a novel algorithm called reductive and effective discriminative information-based nonparallel support vector machine (REDINPSVM). First, we introduce the regularization term to achieve the structural risk minimization principle. This embodies the marrow of statistical learning theory, so this modification can enhance the generalization ability of classification algorithms. Second, we apply the k-nearest neighbor method to eliminate some redundant constraints that would cut down on time complexity. Finally, to accelerate the computation, we introduce the least squares technique to solve two systems of linear equations. Comprehensive experimental results on twenty-three UCI benchmark datasets and six Image datasets demonstrate the validity of the proposed method.

The performance of the organizations or companiesare based on the qualities possessed by their employee. Both of good or bad employee performance will have an impact on productivity and the impact of profits obtained by the company. Support Vector Machine (SVM) is a machine learning method based on statistical learning theory and can solve high non-linearity, regression, etc. In machine learning, the optimization model is a part for improving the accuracy of the model for data learning. Several techniques are used, one of which is feature selection, namely reducing data dimensions so that it can reduce computation in data modeling. This study aims to apply the method of machine learning to the employee data of the Bank Rakyat Indonesia (BRI) company. The method used is SVM method by increasing the accuracy of learning data by using a feature selection technique using a wrapper algorithm. From the results of the classification test, the average accuracy obtained is 72 percent with a precision value of 71 and the recall value is rounded off to 72 percent, with a combination of SVM and cross-validation. Data obtained from Kaggle data, which consists of training data and testing data. each consisting of 30 columns and 22005 rows in the training data and testing data consisting of 29 col-umns and 6000 rows. The results of this study get a classification score of 82 percent. The precision value obtained is rounded off to 82 percent, a recall of 86 percent and an f1-score of 81 percent.

Human detection and activity recognition is very important research area in the healthcare, video surveillance, pedestrian detection, intelligent vehicle system and home care center. Among the various human activity detection frameworks, the statistical based approach were most intensively studied and used in practice in which pattern recognition was traditionally formulated. More recently, supervised learning based techniques and methods imported from statistical learning theory have deserved increasing attention. Many new supervised learning methods such as transfer learning, multi-instance learning, and the new trends in deep learning techniques have used for the formulation of solutions to the human activity detection. This paper reviews the automatic human detection and their activity recognition in the video sequences and static images. We explain several problems of human detection and activity recognition in different steps such as processing, segmentation of human features extraction and classification. Moreover, discuss the problems in each step and provide the recent state- of-the-art methods, gaps between recent methods, technical difficulties, applications and their challenges. Several features extraction techniques and corresponding problems for human classification have been discussed in details. Special emphasis have been given on convolution neural network that solves the problem of human segmentation, efficient classification and activity recognition. The objective of this review paper is to summarize and review related of the established and recent methods used in various stages of a human detection and activity classification system and identify research topics and applications that are at the forefront of this exciting and challenging field. Further, the evaluation protocols (i.e. datasets and simulation tools) and possible solution of current limitation have been discussed briefly in this survey.

Support Vector Regression (SVR) is gaining in popularity in the detection of outliers and classification problems in high-dimensional data (HDD) as this technique does not require the data to be of...

This paper uses statistical learning theory and big data analysis to study the energy consumption structure of China from qualitative and quantitative aspects during COVID-19 According to the domestic and foreign scholars' research on the optimization of energy consumption structure, the carbon emission factor is considered in the optimization of energy consumption structure Taking the minimum energy consumption cost and carbon dioxide emission as the objective function, the carbon dioxide emission is taken as the objective function, and the total energy consumption and various energy consumption proportions as the constraint conditions, the multi-objective planning method is used to evaluate the energy consumption structure of China The optimization model of source consumption structure is analyzed, and the medium and long-term energy transformation path and optimization model under low-carbon constraints are studied Combined with the experimental algorithms related to big data, it is concluded that China's economic development mainly depends on a large amount of energy consumption during the COVID-19 period On this basis, some suggestions are put forward to realize the sustainable development of China's economy and energy © 2020 - IOS Press and the authors All rights reserved

Support vector machine (SVM) is a theoretically well motivated algorithm developed from statistical learning theory which has shown impressive performance in many fields. In spite of its success, it still suffers from the noise sensitivity problem originating from the assumption that each training point has equal importance or weight in the training process. To relax this problem, the SVM was extended to the fuzzy SVM (FSVM) by applying a fuzzy membership to each training point such that different training points can make different contributions to the learning of the decision surface. Although well-determined fuzzy memberships can improve classification performance, there are no general guidelines for their construction. In this paper, inspired by the centered kernel alignment (CKA), which measures the degree of similarity between two kernels (or kernel matrices), we propose a new fuzzy membership function calculation method in which a heuristic function derived from the CKA is used to calculate the dependence between a data point and its associated label. Although the CKA induced FSVM is similar to the kernel target alignment (KTA) induced FSVM, there is actually a critical difference. Without that centering, the definition of alignment does not correlate well with the performance of learning machines. Extensive experiments are performed on real-world data sets from the UCI benchmark repository and the application domain of computational biology which validate the superiority of the proposed FSVM model in terms of several classification performance measures.

The phenomenon of benign overfitting is one of the key mysteries uncovered by deep learning methodology: deep neural networks seem to predict well, even with a perfect fit to noisy training data. Motivated by this phenomenon, we consider when a perfect fit to training data in linear regression is compatible with accurate prediction. We give a characterization of linear regression problems for which the minimum norm interpolating prediction rule has near-optimal prediction accuracy. The characterization is in terms of two notions of the effective rank of the data covariance. It shows that overparameterization is essential for benign overfitting in this setting: the number of directions in parameter space that are unimportant for prediction must significantly exceed the sample size. By studying examples of data covariance properties that this characterization shows are required for benign overfitting, we find an important role for finite-dimensional data: the accuracy of the minimum norm interpolating prediction rule approaches the best possible accuracy for a much narrower range of properties of the data distribution when the data lie in an infinite-dimensional space vs. when the data lie in a finite-dimensional space with dimension that grows faster than the sample size.

Statistical Machine Learning (SML) refers to a body of algorithms and methods by which computers are allowed to discover important features of input data sets which are often very large in size. The very task of feature discovery from data is essentially the meaning of the keyword ‘learning’ in SML. Theoretical justifications for the effectiveness of the SML algorithms are underpinned by sound principles from different disciplines, such as Computer Science and Statistics. The theoretical underpinnings particularly justified by statistical inference methods are together termed as statistical learning theory. This paper provides a review of SML from a Bayesian decision theoretic point of view—where we argue that many SML techniques are closely connected to making inference by using the so called Bayesian paradigm. We discuss many important SML techniques such as supervised and unsupervised learning, deep learning, online learning and Gaussian processes especially in the context of very large data sets where these are often employed. We present a dictionary which maps the key concepts of SML from Computer Science and Statistics. We illustrate the SML techniques with three moderately large data sets where we also discuss many practical implementation issues. Thus the review is especially targeted at statisticians and computer scientists who are aspiring to understand and apply SML for moderately large to big data sets.

In this research paper we present designing and evaluating the electrocardiography (ECG) and Myoelectric signal (EMG) pattern recognition methods based on the adaptive machine learning. For this theoretical model to describe how the Boundary Misclassification Risk (BMR) changes along parameters including, the adaptive learning times, the adaptive learning frequencies, the generalization ability of the predictive model, and the ratio of samples without supervised information during the adaptive learning were proposed. The models are built up based on the formulated adaptive learning process of the myoelectric signal recognition, and the classification from the measured electrocardiogram (ECG) pattern. The theoretical model can be regarded as the extensions of current statistical learning theory and domain adaption theory. In the experiment, the maximum error rate (MER), and the average error rate (AER) of the RCS is employed as the approximation of the BMR. During the experiment, MER and AER change tendency matches the theoretical BMR change tendency. For different learning time interval AER is presented, from the result tendency match with the experimental and theoretical evaluated value is confirmed. Hence, the proposed theoretical model can be used for ECG and EMG pattern matching.

This article is an overview of supervised machine learning problems for regression and classification. Topics include: kernel methods, training by stochastic gradient descent, deep learning architecture, losses for classification, statistical learning theory, and dimension independent generalization bounds. Implicit regularization in deep learning examples are presented, including data augmentation, adversarial training, and additive noise. These methods are reframed as explicit gradient regularization., Comment: 16 pages, 5 figures

Data Analytics on Graphs Part III: Machine Learning on Graphs, from Graph Topology to Applications

In recent years, neural network based image priors have been shown to be highly effective for linear inverse problems, often significantly outperforming conventional methods that are based on sparsity and related notions. While pre-trained generative models are perhaps the most common, it has additionally been shown that even untrained neural networks can serve as excellent priors in various imaging applications. In this paper, we seek to broaden the applicability and understanding of untrained neural network priors by investigating the interaction between architecture selection, measurement models (e.g., inpainting vs. denoising vs. compressive sensing), and signal types (e.g., smooth vs. erratic). We motivate the problem via statistical learning theory, and provide two practical algorithms for tuning architectural hyperparameters. Using experimental evaluations, we demonstrate that the optimal hyperparameters may vary significantly between tasks and can exhibit large performance gaps when tuned for the wrong task. In addition, we investigate which hyperparameters tend to be more important, and which are robust to deviations from the optimum.

Because the actor is rational, and everyone's response to information is the same, investors can only get the average income of the market. The traditional risk measurement model has not achieved ideal results in practical application, and can not explain a large number of anomalies in the financial market, especially the abnormal fluctuations in the market. As a developing country, which is a member of global financial integration, China cannot be exposed to international financial risks. The financial crisis always reflects the lack of financial supervision and the loopholes of economic policies in a country or region in some aspects. If we can do a good job of active prevention and early control, we can avoid the influence and destruction of financial risks. Support vector machine is a classification method based on statistical learning theory, which can be used to analyze financial ratio and non-financial ratio and estimate default probability. In this paper, based on support vector machine algorithm, useful information is extracted or mined from financial data, and a financial risk measurement model is constructed to achieve the purpose of forecasting financial risks.

Support Vector Machine (SVM) is a machine learning method based on statistical learning theory and structural risk minimization principle. The selection of many parameters directly affects the performance of SVM. SVM can be used as a classifier for estimating sound source position, and the anti-noise ability of the algorithm can be improved by selecting appropriate parameters. Convolutional neural network (CNN) can directly obtain effective information from the original image, omitting the processes of preprocessing, feature extraction and data reconstruction of the original image, and is highly invariant to displacement, scaling and other forms of distortion. By setting different solver parameters, network structure and the number of training samples, the results of defect recognition are compared and analyzed, and it is found that the improved Alexnet network has strong adaptive learning ability, which provides a new idea for pattern recognition in DC cable fault diagnosis.

Η παρούσα διατριβή πραγματεύεται το πρόβλημα της Αναγνώρισης Προτύπων, στο πλαίσιο της Μηχανικής Μάθησης (ML) και, ειδικότερα, του πεδίου της Θεωρίας Στατιστικής Μάθησης (STL), μέσω των Μηχανών Διανυσμάτων Στήριξης (SVM). Η εργασία αυτή εστιάζει στην γεωμετρική ερμηνεία των SVM, που πραγματοποιείται μέσω της έννοιας των Συρρικνωμένων Κυρτών Περιβλημάτων (RCH), και της επίπτωσης που έχει στην ανάπτυξη νέων, αποδοτικών αλγορίθμων για την επίλυση του γενικού προβλήματος βελτιστοποίησης των SVM. Η συνεισφορά της παρούσης εργασίας συνίσταται στην επέκταση του μαθηματικού πλαισίου των RCH, στην ανάπτυξη νέων γεωμετρικών αλγορίθμων για τα SVM και, τέλος, στην εφαρμογή των SVM στο πεδίο της Ανάλυσης Ιατρικής Εικόνας και Διάγνωσης (Μαστογραφία). Επέκταση πλαισίου γεωμετρικών SVM. Η γεωμετρική ερμηνεία των SVM στηρίζεται στην έννοια των Συρρικνωμένων Κυρτών Περιβλημάτων. Αν και η γεωμετρική προσέγγιση των SVM παρέχει διαισθητική κατανόηση, η χρησιμότητά της ήταν περιορισμένη λόγω του γεγονότος ότι τα RCH ορίζονται μέσω συρρικνωμένων κυρτών συνδυασμών των σημείων εκπαίδευσης και, συνεπώς, το αντίστοιχο πρόβλημα βελτιστοποίησης παρουσιάζεται να έχει συνδυαστική πολυπλοκότητα. Επεκτείναμε το πλαίσιο των RCH με έναν αριθμό από θεωρητικά αποτελέσματα, που περιορίζουν την έκφραση των ακρότατων σημείων των RCH και παρέχουν αναλυτικό τύπο για την προβολή τους σε συγκεκριμένη κατεύθυνση. Τα αποτελέσματα αυτά οδήγησαν στην ανάπτυξη νέων, πολύ αποδοτικών αλγορίθμων για τα SVM. Νέοι SVM αλγόριθμοι. Οι γνωστοί (και επαρκώς μελετηθέντες ως προς τη σύγκλιση) γεωμετρικοί αλγόριθμοι πλησιέστερου σημείου, i) του Gilbert και ii) των Schlesinger-Kozinec, τροποποιήθηκαν (με βάση τα παραπάνω θεωρητικά αποτελέσματα) για την επίλυση του γενικού, δηλ., του μη-γραμμικού, μη-διαχωρίσιμου προβλήματος βελτιστοποίησης των SVM. Οι νέοι αυτοί γεωμετρικοί αλγόριθμοι για SVM εφαρμόστηκαν και ελέγχθηκαν πάνω σε δημοσίως διαθέσιμα σύνολα δεδομένων ελέγχου και παρουσίασαν σημαντικό πλεονέκτημα ως προς την απόδοση, σε σύγκριση με τους ταχύτερους αντίστοιχους αλγεβρικούς αλγορίθμους. Εφαρμογές - Μαστογραφία. Το πεδίο της Ανάλυσης Ιατρικής Εικόνας και Διάγνωσης (και ειδικότερα της Μαστογραφίας που εξετάζεται στην παρούσα εργασία) είναι ιδιαίτερα κρίσιμο για την κοινωνία, αλλά συνάμα πολύ απαιτητικό από την σκοπιά της Πληροφορικής. Στην παρούσα Διατριβή, εξετάστηκε και αποτιμήθηκε η αξία ενός συνόλου από ποιοτικά και ποσοτικά χαρακτηριστικά υφής και μορφολογίας (χρησιμοποιώντας μεθόδους στατιστικής και μορφοκλασματικής ανάλυσης)• παράλληλα, χρησιμοποιήθηκαν αρκετές μεθοδολογίες μηχανικής μάθησης, π.χ., Τεχνητά Νευρωνικά Δίκτυα (ΑΝΝ) και SVM, για να διαχωρίσουν τους κακοήθεις από τους καλοήθεις μαστογραφικούς όγκους. Τα SVM υπερείχαν σε απόδοση έναντι των υπολοίπων ταξινομητών.

Learning from fuzzy labels (LFL) refers to a generalization of supervised learning in which supervision is represented as a (epistemic) fuzzy set over the collection of possible classification labels: this represents the uncertainty of the annotating agent with respect to the true class label to be assigned to the data instances, using a possibility distribution. The two most popular LFL algorithmic approaches are either based on generalized risk minimization (GRM) or nearest-neighbor (NN) methods: while both methods have been applied successfully with promising empirical results, theoretical characterizations of these approaches, in the framework of learning theory, have been lacking. In this article we address this gap and study the LFL problem from the perspective of statistical learning theory, providing a theoretical analysis of both GRM and NN, in terms of sample complexity and risk bounds.

Statistical learning theory provides the foundation to applied machine learning, and its various successful applications in computer vision, natural language processing and other scientific domains. The theory, however, does not take into account the unique challenges of performing statistical learning in geospatial settings. For instance, it is well known that model errors cannot be assumed to be independent and identically distributed in geospatial (a.k.a. regionalized) variables due to spatial correlation; and trends caused by geophysical processes lead to covariate shifts between the domain where the model was trained and the domain where it will be applied, which in turn harm the use of classical learning methodologies that rely on random samples of the data. In this work, we introduce the geostatistical (transfer) learning problem, and illustrate the challenges of learning from geospatial data by assessing widely-used methods for estimating generalization error of learning models, under covariate shift and spatial correlation. Experiments with synthetic Gaussian process data as well as with real data from geophysical surveys in New Zealand indicate that none of the methods are adequate for model selection in a geospatial context. We provide general guidelines regarding the choice of these methods in practice while new methods are being actively researched.

This paper focuses on understanding how the generalization error scales with the amount of the training data for deep neural networks (DNNs). Existing techniques in statistical learning theory require a computation of capacity measures, such as VC dimension, to provably bound this error. It is however unclear how to extend these measures to DNNs and therefore the existing analyses are applicable to simple neural networks, which are not used in practice, e.g., linear or shallow (at most two-layer) ones or otherwise multi-layer perceptrons. Moreover many theoretical error bounds are not empirically verifiable. In this paper we derive estimates of the generalization error that hold for deep networks and do not rely on unattainable capacity measures. The enabling technique in our approach hinges on two major assumptions: i) the network achieves zero training error, ii) the probability of making an error on a test point is proportional to the distance between this point and its nearest training point in the feature space and at certain maximal distance (that we call radius) it saturates. Based on these assumptions we estimate the generalization error of DNNs. The obtained estimate scales as $\mathcal{O}\left( {\frac{1}{{\delta {N^{1/d}}}}} \right)$, where N is the size of the training data, and is parameterized by two quantities, the effective dimensionality of the data as perceived by the network (d) and the aforementioned radius (δ), both of which we find empirically. We show that our estimates match with the experimentally-obtained behavior of the error on multiple learning tasks using benchmark data-sets and realistic models. Estimating training data requirements is essential for deployment of safety critical applications such as autonomous driving, medical diagnostics etc. Furthermore, collecting and annotating training data requires a huge amount of financial, computational and human resources. Our empirical estimates will help to efficiently allocate resources.

Positive unlabeled (PU) learning is useful in various practical situations, where there is a need to learn a classifier for a class of interest from an unlabeled data set, which may contain anomalies as well as samples from unknown classes. The learning task can be formulated as an optimization problem under the framework of statistical learning theory. Recent studies have theoretically analyzed its properties and generalization performance, nevertheless, little effort has been made to consider the problem of scalability, especially when large sets of unlabeled data are available. In this work we propose a novel scalable PU learning algorithm that is theoretically proven to provide the optimal solution, while showing superior computational and memory performance. Experimental evaluation confirms the theoretical evidence and shows that the proposed method can be successfully applied to a large variety of real-world problems involving PU learning., Comment: Submitted to IEEE TPAMI

Accurate forecasting of peak electricity load has long been an active area of research in electricity markets, and power systems planning and operation. Unanticipated climate-induced surges in peak load can lead to supply shortages causing frequent brownouts and blackouts, and large-scale socioeconomic impacts. In this paper, the climate sensitivity of daily peak load is characterized by leveraging advanced statistical machine learning algorithms. More specifically, a rigorously tested and validated predictive model based on the Bayesian additive regression trees algorithm is proposed. Results from this study revealed that maximum daily temperature followed by mean dew point temperature are the most important predictors of the climate-sensitive portion of daily peak load. Among the non-climatic predictors, electricity price was found to have a strong positive association with the daily peak load. Economic growth was observed to have an inverse association with the daily peak load. While the proposed framework is established for the state of Texas, one of the most energy-intensive states with geographic and demographic susceptibility to climatic change, the methodology can be extended to other states/regions. The model can also be used to make short-term predictions of the climate-sensitive portion of daily peak load.

In this paper we introduce the current thrust of development in machine learning and artificial intelligence, fueled by advances in statistical learning theory over the last 20 years and commercial successes by leading big data companies. Then we discuss the characteristics of process manufacturing systems and briefly review the data analytics research and development in the last three decades. We give three attributes for process data analytics to make machine learning techniques applicable in the process industries. Next we provide a perspective on the currently active topics in machine learning that could be opportunities for process data analytics research and development. Finally we address the importance of a data analytics culture. Issues discussed range from technology development to workforce education and from government initiatives to curriculum enhancement.

A common assumption of statistical learning theory is that the training and testing data are drawn from the same distribution. However, in many real-world applications, this assumption does not hold true. Hence, a realistic strategy, Cross Domain Adaptation (DA) or Transfer Learning (TA), can be used to employ previously labelled source domain data to boost the task in the new target domain. Previously, Cross Domain Adaptation methods have been focused on re-weighting the instances or aligning the cross-domain distributions. However, these methods have two significant challenges: (1) There is no proper consideration of the unlabelled data of target task as in the real-world, an abundant amount of unlabelled data is available, (2) The use of normal Euclidean distance function fails to capture the appropriate similarity or dissimilarity between samples. To deal with this issue, we have proposed a Semi-Supervised Metric Transfer Learning framework called SSMT that reduces the distribution between domains both statistically and geometrically by learning the instance weights, while a regularized distance metric is learned to minimize the within-class co-variance and maximize the between-class co-variance simultaneously for the target domain. Compared with the previous works where Mahalanobis distance metric and instance weights are learned by using the labelled data or in a pipelined framework that leads to a decrease in the performance, our proposed SSMT attempts to learn a regularized distance metric and instance weights by considering unlabelled data in a parallel framework. Experimental evaluation on three cross-domain visual data sets, e.g., PIE Face, Handwriting Digit Recognition on MNIST–USPS and Object Recognition, demonstrates the effectiveness of our designed approach on facilitating the unlabelled target task learning, compared to current state-of-the-art domain adaptation approaches.

Laser-induced breakdown spectroscopy (LIBS) is a multi-elemental and real-time analytical technique with simultaneous detection of all the elements in any type of sample matrix including solid, liquid, gas, and aerosol. LIBS produces vast amount of data which contains information on elemental composition of the material among others. Classification and discrimination of spectra produced during the LIBS process are crucial to analyze the elements for both qualitative and quantitative analysis. This work reports the design and modeling of optimal classifier for LIBS data classification and discrimination using the apparatus of statistical theory of detection. We analyzed the noise sources associated during the LIBS process and created a linear model of an echelle spectrograph system. We validated our model based on assumptions through statistical analysis of “dark signal” and laser-induced breakdown spectra from the database of National Institute of Science and Technology. The results obtained from our model suggested that the quadratic classifier provides optimal performance if the spectroscopy signal and noise can be considered Gaussian.

In this paper, we propose a maximum margin classifier that deals with uncertainty in data input. More specifically, we reformulate the SVM framework such that each training example can be modeled by a multi-dimensional Gaussian distribution described by its mean vector and its covariance matrix -- the latter modeling the uncertainty. We address the classification problem and define a cost function that is the expected value of the classical SVM cost when data samples are drawn from the multi-dimensional Gaussian distributions that form the set of the training examples. Our formulation approximates the classical SVM formulation when the training examples are isotropic Gaussians with variance tending to zero. We arrive at a convex optimization problem, which we solve efficiently in the primal form using a stochastic gradient descent approach. The resulting classifier, which we name SVM with Gaussian Sample Uncertainty (SVM-GSU), is tested on synthetic data and five publicly available and popular datasets; namely, the MNIST, WDBC, DEAP, TV News Channel Commercial Detection, and TRECVID MED datasets. Experimental results verify the effectiveness of the proposed method., Comment: IEEE Transactions on Pattern Analysis and Machine Intelligence. (c) 2017 IEEE. DOI: 10.1109/TPAMI.2017.2772235 Author's accepted version. The final publication is available at http://ieeexplore.ieee.org/document/8103808/

As a result of statistical learning theory, support vector machines (SVMs)[23] are effective classifiers for the classification problems. SVMs have been successfully applied to various pattern classification problems, such as handwritten digit recognition, text categorization and face detection, due to their powerful learning ability and good generalization ability. However, SVMs require to solve a quadratic optimization problem and need training time that are at least quadratic to the number of training samples. Therefore, many large-scale problems by using traditional SVMs are too hard to be solved. To overcome this difficulty, Lu and colleagues have proposed a mi n-max modular support vector machine (M3-SVM) and part-versus-part task decomposition method [16]. A very important advantage of M 3SVMs over traditional SVMs is that a two-class problem can be further decomposed into a series of two-class subproblems. The M3-network model [15] has been applied successfully to many real-world applications such as part-of-speec h tagging [17], single-trial EEG signal classification [18], prediction of protein subcellular multi-locations [26], face recognition [2, 13] and text categorization [14]. The basic idea behind M3-network is the “divide and conquer” strategy. The task decomposition scheme of M 3-network is based on class relations, and the instances in the same class can be further decomposed randomly [15], according to parallel hyperplanes [24], or prior knowledge [13]. The learning procedure of each subproblems is independent, and therefore parallel learning can be implemented easily. The combination strategy follows two principles, the minimization principle and the maximization principle [15]. We explore the use of M3-SVMs in multi-view face recognition. Multi-view face recognition is a more challenging task than frontal view face recognition. Face recognition techniques have been developed over the past few decades. But many of those existing face recognition techniques, such as Eigenfaces and Fisher-faces [22, 1], are only effective for frontal view faces. The difficulties of multi-view face recognition is obvious because of the complicated nonlinear manifolds existing in the data space. Using M3-SVMs, we can decompose the

Deep learning model is popular and widely used. Its generalization performance is superior to traditional machine learning methods. Traditional statistical learning theory cannot explain its success. Most of network architectures are designed artificially and empirically. There are few methods to design new structures and mathematical tools to evaluate feature representation capabilities of networks. To address this issue, based on information theory, we propose an interpretable regularization method named Minimize Mutual Information Method (MMIM), which reduce the generalization error by minimizing the mutual information between hidden layer neurons. We derive a tighter generalization error boundary and propose a two-step method to reduce the neural network generalization error boundary. The experimental results verify the effectiveness of our proposed MMIM on Fashion-MNIST dataset and CIFAR datasets. Inspired by ensemble learning, we discuss why our method is effective and suggest how to design new networks.

Most pattern recognition applications within the Geoscience field involve the clustering and classification of remote sensed multispectral data, which basically aims to allocate the right class of ground category to a reflectance or radiance signal. Generally, the complexity of this problem is related to the incorporation of spatial characteristics that are complementary to the nonlinearities of land surface heterogeneity, remote sensing effects and multispectral features. The present chapter describes recent developments in the performance of a kernel method applied to the representation and classification of agricultural land use systems described by multispectral responses. In particular, we focus on the practical applicability of learning machine methods to the task of inducting a relationship between the spectral response of farms land cover to their informational typology from a representative set of instances. Such methodologies are not traditionally used in agricultural studies. Nevertheless, the list of references reviewed here show that its applications have emerged very fast and are leading to simple and theoretically robust classification models. This chapter will cover the following phases: a)learning from instances in agriculture; b)feature extraction of both multispectral and attributive data and; c) kernel supervised classification. The first provides the conceptual foundations and a historical perspective of the field. The second belongs to the unsupervised learning field, which mainly involves the appropriate description of input data in a lower dimensional space. The last is a method based on statistical learning theory, which has been successfully applied to supervised classification problems and to generate models described by implicit functions.

The use of machine-learning techniques is becoming more and more frequent in solving all those problems where it is difficult to rationally interpret the process of interest. Intrusion detection in networked systems is a problem in which, although it is not fundamental to interpret the measures that one is able to obtain from a process, it is important to obtain an answer from a classification algorithm if the network traffic is characterized by anomalies (and hence, there is a high probability of an intrusion) or not. Due to the increased adoption of SW-defined autonomous systems that are distributed and interconnected, the probability of a cyber attack is increased, as well as its consequence in terms of system reliability, availability, and even safety. In this work, we present the application of different machine-learning models to the problem of anomaly classification in the context of local area network (LAN) traffic analysis. In particular, we present the application of a K-nearest neighbors (KNN) and of an artificial neural network (ANN) to realize an algorithm for intrusion detection systems (IDS). The dataset used in this work is representative of the communication traffic in common LAN networks in military application in particular typical US Air Force LAN. This work presents a training phase of the different models based on a multidimensional-scaling preprocessing procedure, based on different metrics, to provide higher performance and generalization with respect to model prediction capability. The obtained results of KNN and ANN classifiers are compared with respect to a commonly used index of performance for classifiers evaluation.

Transformer is an important device in the power system. Dissolve gas analysis is an oil testing method for distribution transformer fault diagnosis. In this paper, support vector machine with binary decision tree has been presented for the fault prediction using the DGA gases and verified with graphical representation of different outputs. SVM is an artificial intelligence technique based on statistical learning theory. The SVM classification method is compared with IEC three ratio method, key gas method, and ANN method, and then, the test results show that SVM has better accuracy in fault detection of transformer than other conventional methods.

The remarkable practical success of deep learning has revealed some major surprises from a theoretical perspective. In particular, simple gradient methods easily find near-optimal solutions to non-convex optimization problems, and despite giving a near-perfect fit to training data without any explicit effort to control model complexity, these methods exhibit excellent predictive accuracy. We conjecture that specific principles underlie these phenomena: that overparametrization allows gradient methods to find interpolating solutions, that these methods implicitly impose regularization, and that overparametrization leads to benign overfitting, that is, accurate predictions despite overfitting training data. In this article, we survey recent progress in statistical learning theory that provides examples illustrating these principles in simpler settings. We first review classical uniform convergence results and why they fall short of explaining aspects of the behaviour of deep learning methods. We give examples of implicit regularization in simple settings, where gradient methods lead to minimal norm functions that perfectly fit the training data. Then we review prediction methods that exhibit benign overfitting, focusing on regression problems with quadratic loss. For these methods, we can decompose the prediction rule into a simple component that is useful for prediction and a spiky component that is useful for overfitting but, in a favourable setting, does not harm prediction accuracy. We focus specifically on the linear regime for neural networks, where the network can be approximated by a linear model. In this regime, we demonstrate the success of gradient flow, and we consider benign overfitting with two-layer networks, giving an exact asymptotic analysis that precisely demonstrates the impact of overparametrization. We conclude by highlighting the key challenges that arise in extending these insights to realistic deep learning settings.

Neural networks have been achieving high generalization performance on many tasks despite being highly over-parameterized. Since classical statistical learning theory struggles to explain this behavior, much effort has recently been focused on uncovering the mechanisms behind it, in the hope of developing a more adequate theoretical framework and having a better control over the trained models. In this work, we adopt an alternate perspective, viewing the neural network as a dynamical system displacing input particles over time. We conduct a series of experiments and, by analyzing the network's behavior through its displacements, we show the presence of a low kinetic energy displacement bias in the transport map of the network, and link this bias with generalization performance. From this observation, we reformulate the learning problem as follows: finding neural networks which solve the task while transporting the data as efficiently as possible. This offers a novel formulation of the learning problem which allows us to provide regularity results for the solution network, based on Optimal Transport theory. From a practical viewpoint, this allows us to propose a new learning algorithm, which automatically adapts to the complexity of the given task, and leads to networks with a high generalization ability even in low data regimes., Comment: ECML PKDD 2020

Federated learning (FL) is a machine learning setting where many clients (e.g. mobile devices or whole organizations) collaboratively train a model under the orchestration of a central server (e.g. service provider), while keeping the training data decentralized. FL embodies the principles of focused data collection and minimization, and can mitigate many of the systemic privacy risks and costs resulting from traditional, centralized machine learning and data science approaches. Motivated by the explosive growth in FL research, this paper discusses recent advances and presents an extensive collection of open problems and challenges., Published in Foundations and Trends in Machine Learning Vol 4 Issue 1. See: https://www.nowpublishers.com/article/Details/MAL-083

In this paper we survey the most recent advances in supervised machine learning and high-dimensional models for time series forecasting. We consider both linear and nonlinear alternatives. Among the linear methods we pay special attention to penalized regressions and ensemble of models. The nonlinear methods considered in the paper include shallow and deep neural networks, in their feed-forward and recurrent versions, and tree-based methods, such as random forests and boosted trees. We also consider ensemble and hybrid models by combining ingredients from different alternatives. Tests for superior predictive ability are briefly reviewed. Finally, we discuss application of machine learning in economics and finance and provide an illustration with high-frequency financial data., 42 pages, 7 figures, 1 table

Design-consistent model-assisted estimation has become the standard practice in survey sampling. However, design consistency remains to be established for many machine-learning techniques that can potentially be very powerful assisting models. We propose a subsampling Rao-Blackwell method, and develop a statistical learning theory for exactly design-unbiased estimation with the help of linear or non-linear prediction models. Our approach makes use of classic ideas from Statistical Science as well as the rapidly growing field of Machine Learning. Provided rich auxiliary information, it can yield considerable efficiency gains over standard linear model-assisted methods, while en- suring valid estimation for the given target population, which is robust against potential mis-specifications of the assisting model, even if the design consistency of following the standard recipe for plug-in model-assisted estimator cannot be established.

Hu, Z.Y., 2020. Economic forecast of coastal regions based on support vector machine model. In: Al-Tarawneh, O. and Megahed, A. (eds.), Recent Developments of Port, Marine, and Ocean Engineering. Journal of Coastal Research, Special Issue No. 110, pp. 211–214. Coconut Creek (Florida), ISSN 0749-0208.Support vector machine (SVM) is a new technology in data mining. It is a new tool to solve machine learning problems with the help of optimization. Support vector machines belong to a new machine learning that extends from statistical learning theory. Its structure is relatively simple, with good generalization ability and global optimality. Support vector machine has provided a unified framework for solving finite sample learning problems, and there are many solutions proposed. It can deal with those more complex problems and introduces the characteristics of the support vector machine model. Aiming at the application of the model in economic forecasting of coastal regions, a method to improve the prediction accuracy of the model is proposed. The theoretical analysis and practical application verification are performed, which shows that this method can obtain more accurate prediction results.

The traditional approach of statistical physics to supervised learning routinely assumes unrealistic generative models for the data: usually inputs are independent random variables, uncorrelated with their labels. Only recently, statistical physicists started to explore more complex forms of data, such as equally-labelled points lying on (possibly low dimensional) object manifolds. Here we provide a bridge between this recently-established research area and the framework of statistical learning theory, a branch of mathematics devoted to inference in machine learning. The overarching motivation is the inadequacy of the classic rigorous results in explaining the remarkable generalization properties of deep learning. We propose a way to integrate physical models of data into statistical learning theory, and address, with both combinatorial and statistical mechanics methods, the computation of the Vapnik-Chervonenkis entropy, which counts the number of different binary classifications compatible with the loss class. As a proof of concept, we focus on kernel machines and on two simple realizations of data structure introduced in recent physics literature: $k$-dimensional simplexes with prescribed geometric relations and spherical manifolds (equivalent to margin classification). Entropy, contrary to what happens for unstructured data, is nonmonotonic in the sample size, in contrast with the rigorous bounds. Moreover, data structure induces a novel transition beyond the storage capacity, which we advocate as a proxy of the nonmonotonicity, and ultimately a cue of low generalization error. The identification of a synaptic volume vanishing at the transition allows a quantification of the impact of data structure within replica theory, applicable in cases where combinatorial methods are not available, as we demonstrate for margin learning., Comment: 19 pages, 3 figures

Model complexity is one of the most important criteria for the success of models. In this study, theprominent approaches to controlling model complexity have been examined under headings. These areOccam’s razor, Popper’s falsifiability, and the statistical learning theory. Occam’s razor and Popper’sfalsifiability in the control of model complexity, yes they provide a philosophical approach and theyare also accepted. However, they do not provide a mathematical formulation on how to control modelcomplexity. However, the statistical learning theory (aka VC theory) approach to the subject is notonly at a philosophical level, but also introduces a new principle of risk minimization (structural riskminimization, YRM) by adding the VC coefficient to the empirical risk minimization (ARM) principleused in the models developed so far. . As a result, the VC theory developed by Vapnik and Chervonenkisas a control model, with its proven mathematical background and highly successful results, can be agood source of inspiration for model developers as the most consistent and reliable approach to thecontrol of model complexity in today’s framework., Model karmaşıklığı, modellerin başarısındaki en önemli ölçütlerden birisidir. Bu çalışmada bugüne dek model karmaşıklığının kontrolünde öne çıkan yaklaşımlar başlıklar halinde incelenmiştir. Bunlar Occam’ın usturası, Popper’ın yanlışlanabilirliği ve istatiksel öğrenme teorisidir. Occam’ın usturası ve Popper’ın yanlışlanabilirliği model karmaşıklığının kontrolünde, evet bir felsefi yaklaşım sağlamaktadırlar ve kabul de görmektedirler. Fakat model karmaşıklığının nasıl kontrol edileceği konusunda matematiksel bir formülasyon sağlamamaktadırlar. Fakat istatiksel öğrenme teorisi (diğer adıyla, VC teorisi) konuya yaklaşımı yalnızca felsefi bir düzeyde kalmamakta, aynı zamanda şimdiye dek geliştirilen modellerde kullanılan ampirik risk minimizasyonu (ARM) ilkesine VC katsayısını ilave ederek yeni bir risk minimizasyonu (yapısal risk minimizasyonu, YRM) ilkesi getirmektedir. Sonuç olarak Vapnik ve Chervonenkis tarafından geliştirilen VC teorisi bir kontrol modeli olarak, ispatlanmış matematiksel arka planı ve oldukça başarılı olan sonuçları itibariyle, model karmaşıklığının kontrolü konusunda, günümüz çerçevesinde, en tutarlı ve güvenilir bir yaklaşım olarak, model geliştiriciler için iyi bir ilham kaynağı olabilir.

This paper introduces the principle of SVM, kernel function selection and multi-class classification problem. at the same time, which expounds and summarizes the SVM algorithm. Secondly, it summarizes the application research of support vector machine from the aspects of handwritten digit recognition, text classification and image recognition. Finally, we discuss the advantages and disadvantages of SVM, and prospect the prospects of its application research.

A machine learning system, in general, learns from the environment, but statistical machine learning programs (systems) learn from the data. This chapter presents techniques for statistical machine learning using Support Vector Machines (SVM) to recognize the patterns and classify them, predicting structured objects using SVM, k-nearest neighbor method for classification, and Naive Bayes classifiers. The artificial neural networks are presented with brief introduction to error-correction rules, Boltzmann learning, Hebbian rule, competitive learning rule, and deep learning. The instance-based learning is treated in details with its algorithm and learning task. The chapter concludes with a summary, and a set of practice exercises.

In this short paper, a theoretical analysis of Occam’s razor formulation through statistical learning theory is presented, showing that pathological situations exist for which regularization may slow down supervised learning instead of making it faster.

The engineering classification of rock masses is the basis of rock engineering design and construction. We propose and apply a quick basic quality (BQ) classification method based on the standard BQ method of China to classify the quality grade of the rock mass around tunnels along the Ningguo-Huangshan Expressway during the construction period. Moreover, the joint continuity and surface roughness of the controlled key joint are added to the classification indices of the quick BQ method to address shortcomings of the standard BQ classification method. Therefore, an improved BQ classification method for rock mass is proposed. According to the BQ method, different personnel might select different values of correction coefficient that result in divergences in the result of rock mass classification. In order to solve this problem, the Genetic algorithm (GA) and support vector classification (SVC) coupling algorithm is introduced into the field of engineering rock mass classification. GA is used to automatically search for the optimal SVC parameters during the training process of samples. By training the classification samples of rock mass around a tunnel using the improved BQ method during the tunnel construction period, an intelligent SVC classification model is constructed with inputs based on eight classification indices and an output of the BQ quality grade. To verify the reliability and accuracy of the model, the SVC model is used to evaluate the quality grade of the rock mass around tunnel in other cross sections of the tunnels along the Ningguo-Huangshan Expressway. Only one section classification result differed from those of the improved BQ method in a total of 20 sections. In contrast, three section classification results based on the BP neural network (BPNN) model were inconsistent with those of the improved BQ method. Therefore, the proposed SVC model displays a higher rate of correct classification relative to that of the BPNN model. Meanwhile, the use of this SVC model can avoid the divergence among different people on the classification result of rock mass around a tunnel, which provides an effective new method for the rapid classification of rock mass around a tunnel during tunnel construction.

Renewable fuels such as biodiesel are introduced as promising environmental friendly fuels and they can be applied as alternative fuels instead of fossil fuels. In the present study, a modeling study based on statistical learning theory was investigated by the least square support vector machine (LSSVM) approach for non-catalytic biodiesel production in supercritical methanol. This model can estimate the biodiesel yield as a function of temperature, pressure, reaction time, and Methanol/oil ratio. The results indicated that the suggested LSSVM model was a satisfactory model to predict biodiesel yield that was confirmed by a high value of R2 (0.9961) and low value of absolute deviation (1.17%). In addition, our model has been compared with another previous Artificial neural network (ANN)-based model and great estimations of both models were proved.

In spite of the relevance of Decision Trees (DTs), there is still a disconnection between their theoretical and practical results while selecting models to address specific learning tasks. A particular criterion is provided by the Shattering coefficient, a growth function formulated in the context of the Statistical Learning Theory (SLT), which measures the complexity of the algorithm bias as sample sizes increase. In attempt to establish the basis for a relative theoretical complexity analysis, this paper introduces a method to compute the Shattering coefficient of DT models using recurrence equations. Next, we assess the bias of models provided by DT algorithms while solving practical problems as well as their overall learning bounds in light of the SLT. As the main contribution, our results support other researchers to decide on the most adequate DT models to tackle specific supervised learning tasks.

The present paper shows how statistical learning theory and machine learning models can be used to enhance understanding of AI-related epistemological issues regarding inductive reasoning and reliability of generalisations. Towards this aim, the paper proceeds as follows. First, it expounds Price’s dual image of representation in terms of the notions of e-representations and i-representations that constitute subject naturalism. For Price, this is not a strictly anti-representationalist position but rather a dualist one (e- and i-representations). Second, the paper links this debate with machine learning in terms of statistical learning theory becoming more viable epistemological tool when it abandons the perspective of object naturalism. The paper then argues that machine learning grounds a form of knowing that can be understood in terms of e- and i-representation learning. Third, this synthesis shows a way of analysing inductive reasoning in terms of reliability of generalisations stemming from a structure of e- and i-representations. In the age of Artificial Intelligence, connecting Price’s dual view of representation with Deep Learning provides an epistemological way forward and even perhaps an approach to how knowing is possible.

In many real applications of statistical learning, a decision made from misclassification can be too costly to afford; in this case, a reject option, which defers the decision until further investigation is conducted, is often preferred. In recent years, there has been much development for binary classification with a reject option. Yet, little progress has been made for the multicategory case. In this article, we propose margin-based multicategory classification methods with a reject option. In addition, and more importantly, we introduce a new and unique refine option for the multicategory problem, where the class of an observation is predicted to be from a set of class labels, whose cardinality is not necessarily one. The main advantage of both options lies in their capacity of identifying error-prone observations. Moreover, the refine option can provide more constructive information for classification by effectively ruling out implausible classes. Efficient implementations have been developed for the proposed methods. On the theoretical side, we offer a novel statistical learning theory and show a fast convergence rate of the excess $\ell$-risk of our methods with emphasis on diverging dimensionality and number of classes. The results can be further improved under a low noise assumption. A set of comprehensive simulation and real data studies has shown the usefulness of the new learning tools compared to regular multicategory classifiers. Detailed proofs of theorems and extended numerical results are included in the supplemental materials available online., A revised version of this paper was accepted for publication in the Journal of the American Statistical Association Theory and Methods Section. 52 pages, 6 figures

Wireless sensor networks (WSNs) are prone to many failures such as hardware failures, software failures, and communication failures. The fault detection in WSNs is a challenging problem due to sensor resources limitation and the variety of deployment field. Furthermore, the detection has to be precise to avoid negative alerts, and rapid to limit loss. The use of machine learning seems to be one of the most convenient solutions for detecting failure in WSNs. In this paper, support vector machines (SVMs) classification method is used for this purpose. Based on statistical learning theory, SVM is used in our context to define a decision function. As a light process in term of required resources, this decision function can be easily executed at cluster heads to detect anomalous sensor. The effectiveness of SVM for fault detection in WSNs is shown through an experimental study, comparing it to latest for the same application.

In this work, we conducted a visualized bibliometric analysis to map the research trends of machine learning in engineering (MLE) based on articles indexed in the Web of Science Core Collection published between 2000 and 2019. The research distributions, knowledge bases, research hotspots, and research frontiers for MLE studies are revealed by using VOSviewer software and visualization technology. The growth of the literature related to MLE averaged 24.3% in the past two decades. A total of 3057 peer-reviewed papers from 96 countries published in 1299 different journals were identified. The USA was the most productive country, with 23.73% of the overall articles and 32.25% of the overall citations. The most active research organization was MIT, with 41 publications and 1079 citations, and the Journal of Machine Learning Research had the largest number of citations in the field of MLE. In particular, our findings indicate that the research issues of “random forests”, “support vector machine”, “extreme learning machine”, “deep learning”, “statistical learning theory”, and “Python machine learning” formed the knowledge bases of MLE from 2000 to 2019, while the research hotspots focused on applications of machine learning benchmark algorithms. Burst detection analysis results showed that more burst keywords emerged and had a higher frequency of change after 2010. This study provides an insight view of the overall research trends of MLE and may help researchers better understand this research field and predict its dynamic directions.