Search

Adaptive gradient algorithms have recently become extremely popular because they have been applied successfully in training deep neural networks, such as Adam, AMSGrad, and AdaBound. Despite their success, however, the distributed variant of the adaptive method, which is expected to possess a rapid training speed at the beginning and a good generalization capacity at the end, is rarely studied. To fill the gap, a distributed adaptive subgradient algorithm is presented, called D-AdaBound, where the learning rates are dynamically bounded by clipping the learning rates. Moreover, we obtain the regret bound of D-AdaBound, in which the objective functions are convex. Finally, we confirm the effectiveness of D-AdaBound by simulation experiments on different datasets. The results show the performance improvement of D-AdaBound relative to existing distributed online learning algorithms. [ABSTRACT FROM AUTHOR]

Real-time forecasting of the financial time-series data is challenging for many machine learning (ML) algorithms. First, many ML models operate offline, where they need a batch of data, which may not be available during training. Besides, due to a fixed architecture of the majority of the offline-based ML models, they suffer to deal with the uncertain nature of financial time-series data. In contrast, online learning mode evolving-structured ML models could be promising for financial time-series forecasting. For real-time deployment of such models, low memory demand is a must. Besides, the model’s explainability plays a crucial role in forecasting financial time-series. Considering all the requirements, a rule-based autonomous neuro-fuzzy learning algorithm called the parsimonious learning machine (PALM) is proposed here to forecast time-varying stock indices. To provide efficient automation of the proposed algorithm by maintaining the model explainability in terms of limited number linguistic IF-THEN rules, two popular multiobjective evolutionary algorithms (MEAs), such as a real-coded genetic algorithm (GA) and a self-adaptive differential evolution (DE) algorithm are utilized here. In addition, fuzzy type-2 variants of PALMs’ are considered here due to better uncertainty handling capacity than their type-1 counterparts. To evaluate the proposed algorithm’s performance, the closing stock price of fifteen (15) different stock market indices are predicted here. From the results, it is observed that the MEA-based PALMs are performing better than the state-of-the-art benchmark online ML models and providing a rule-based explainable model to the end-user. [ABSTRACT FROM AUTHOR]

Machine learning models have gained popularity nowadays for their potential to solve real-life issues when trained on pertinent data. In many cases, the real-life data are class imbalanced and hence the corresponding machine learning models trained on the data tend to perform poorly on metrics like precision, recall, AUC, F1, and G-mean score. Since class imbalance issue poses serious challenges to the performance of trained models, a multitude of research works have addressed this issue. Two common data-based sampling techniques have mostly been proposed-undersampling the data of the majority class and oversampling the data of the minority class. In this article, we focus on the former approach. We propose two novel algorithms that employ neural network-based approaches to remove majority samples that are found to reside in the vicinity of the minority samples, thereby undersampling the former to remove (or alleviate) the imbalance issue. We delineate the proposed algorithms and then test the proposed algorithms on some publicly available imbalanced datasets. We then compare the performance of our proposed algorithms to other popular undersampling algorithms. Finally, we conclude that our proposed algorithms outperform most of the existing undersampling approaches on most performance metrics. [ABSTRACT FROM AUTHOR]

A Dandelion algorithm (DA) inspired by the seed dispersal process of dandelions has been proposed as a newly intelligent optimization algorithm. For improving its exploration ability as well as reducing the probability of its falling into a local optimum, this work proposes to add a novel competition mechanism with historical information feedback to current DA. Specifically, the fitness value of each dandelion in the next generation, which is calculated by linear prediction, is compared with the current best dandelion, and the loser is replaced by a new offspring. Current DA generates new offsprings without considering historical information. This work improves its offspring generation process by exploiting historical information with an estimation-of-distribution algorithm. Three historical information models are designed. They are best, worst, and hybrid historical information feedback models. The experimental results show that the proposed algorithms outperform DA and its variants, and the proposed algorithms are superior or competitive to nine participating algorithms benchmarked on 28 functions from CEC2013. Finally, the proposed algorithms demonstrate the effectiveness on four real-world problems, and the results indicate that the proposed algorithms have better performance than its peers. [ABSTRACT FROM AUTHOR]

Dimension reduction (DR)-based on extreme learning machine auto-encoder (ELM-AE) has achieved many successes in recent years. By minimizing the self-reconstruction error, the ELM-AE-based DR algorithms learn the compressed representations which facilitate the subsequent classification. However, the existing ELM-AEs only consider the DR problem in an unsupervised manner and ignore the valuable supervised information when these information is available. To find discriminative features of the original data, in this paper, we propose a graph embedding-based DR framework with ELM (GDR-ELM) for DR problems. Instead of self-reconstruction, the proposed GDR-ELM reconstructs all samples according to the weights in a graph matrix containing the supervised information. Furthermore, GDR-ELM can be stacked as building blocks to construct a multilayer framework like other ELM-AEs for more complicated representation learning tasks. Experiments on various datasets demonstrate the effectiveness of the proposed GDR-ELM and its multilayer framework. [ABSTRACT FROM AUTHOR]

The broad learning system (BLS) has been proved to be effective and efficient lately. In this article, several deep variants of BLS are reviewed, and a new adaptive incremental structure, Stacked BLS, is proposed. The proposed model is a novel incremental stacking of BLS. This invariant inherits the efficiency and effectiveness of BLS that the structure and weights of lower layers of BLS are fixed when the new blocks are added. The incremental stacking algorithm computes not only the connection weights between the newly stacking blocks but also the connection weights of the enhancement nodes within the BLS block. The Stacked BLS is considered as the increment of “layers” and “neurons” dynamically during the training for multilayer neural networks. The proposed architecture along with the training algorithms that utilizes the residual characteristic is very versatile in comparison with traditional fixed architecture. Finally, experimental results on UCI datasets, MNIST dataset, NORB dataset, CIFAR-10 dataset, SVHN dataset, and CIFAR-100 dataset indicate that the proposed method outperforms the selected state-of-the-art methods on both accuracy and training speed, such as deep residual networks. The results also imply that the proposed structure could highly reduce the number of nodes and the training time of the original BLS in the classification task of some datasets. [ABSTRACT FROM AUTHOR]

Big data classification problems have drawn great attention from diverse fields, and many classifiers have been developed. Among those classifiers, the extended belief rule-based system (EBRBS) has shown its potential in both big data and multiclass situations, while the time complexity and computing efficiency are two challenging issues to be handled in EBRBS. As such, three improvements of EBRBS are proposed first in this paper to decrease the time complexity and computing efficiency of EBRBS for multiclass classification under the assumption of large amount of data, including the strategy to skip rule weight calculation, a simplified evidential reasoning algorithm, and the domain division-based rule reduction method. This turns out to be a micro version of the EBRBS, called Micro-EBRBS. Moreover, one of commonly used cluster computing, named Apache Spark, is then applied to implement the parallel rule generation and inference schemes of the Micro-EBRBS for big data multiclass classification problems. The comparative analyses of experimental studies demonstrate that the Micro-EBRBS not only can obtain a desired accuracy but also has the comparatively better time complexity and computing efficiency than some popular classifiers, especially for multiclass classification problems. [ABSTRACT FROM AUTHOR]

We propose a weighted outlier mining method called WATCH to identify outliers in high-dimensional categorical datasets. WATCH is composed of two distinctive modules: 1) feature grouping by the virtue of correlation measurement among features and 2) outlier mining by assigning scores to objects in each feature groups. At the heart of WATCH is the feature grouping module, which groups an array of features into multiple groups to discover various aspects of feature patterns in each group. The outlier mining module detects outliers from high-dimensional categorical datasets. Except for the number of outliers specified by users, WATCH is conducive to bypassing the optimization of any user-given parameter. We implement and evaluate WATCH using synthetic and real-world datasets. Our experimental results show that WATCH is a promising and practical algorithm to detect outliers in high-dimensional categorical datasets, because WATCH achieves high performance in terms of precision, efficiency, and interpretability. [ABSTRACT FROM AUTHOR]

Graph matching is a fundamental problem in theoretical computer science and artificial intelligence, and lays the foundation for many computer vision and machine learning tasks. Approximate algorithms are necessary for graph matching due to its NP-complete nature. Inspired by the usage in network-related tasks, random walk is generalized to graph matching as a type of approximate algorithm. However, it may be inappropriate for the previous random walk-based graph matching algorithms to utilize continuous techniques without considering the discrete property. In this paper, we propose a novel random walk-based graph matching algorithm by incorporating both continuous and discrete constraints in the optimization process. Specifically, after interpreting graph matching by random walk, the continuous constraints are directly embedded in the random walk constraint in each iteration. Further, both the assignment matrix (vector) and the pairwise similarity measure between graphs are iteratively updated according the discrete constraints, which automatically leads the continuous solution to the discrete domain. Comparisons on both synthetic and real-world data demonstrate the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

The recent advances of compressive sensing (CS) have witnessed a great potential of efficient compressive data gathering (CDG) in wireless sensor network systems (WSNSs). However, most existing work on CDG mainly focuses on multihop relaying strategies to improve the performance of data gathering. In this paper, we propose a mobile CDG scheme including a random walk-based algorithm and a kernel-based method for sparsifying sensory data from irregular deployments. The proposed scheme allows a mobile collector to harvest data by sequentially visiting a number of nodes along a random path. More importantly, toward building the gap between CS and machine learning theories, we explore a theoretical foundation for understanding the feasibility of the proposed scheme. We prove that the CS matrices, constructed from the proposed random walk algorithm combined with a kernel-based sparsity basis, satisfy the restricted isometry property. Particularly, we also show that ${m=O}({k\log } ({n/k}))$ measurements collected by a mobile collector are sufficient to recover a ${k}$ -sparse signal and ${t=O(k\log (n/k))}$ steps are required to collect these measurements in a network with ${n}$ nodes. Finally, we also present extensive numerical results to validate the effectiveness of the proposed scheme by evaluating the performance in terms of energy consumption and the impact of packet losses. The numerical results demonstrate that the proposed scheme is able to not only significantly reduce communication cost but also combat unreliable wireless links under various packet losses compared to the state-of-the-art schemes, which provides an efficient alternative to data relaying approaches for CDG in WSNS. [ABSTRACT FROM AUTHOR]

How to tackle high dimensionality of data effectively and efficiently is still a challenging issue in machine learning. Identifying anomalous objects from given data has a broad range of real-world applications. Although many classical outlier detection or ranking algorithms have been witnessed during the past years, the high-dimensional problem, as well as the size of neighborhood, in outlier detection have not yet attracted sufficient attention. The former may trigger the distance concentration problem that the distances of observations in high-dimensional space tend to be indiscernible, whereas the latter requires appropriate values for parameters, making models high complex and more sensitive. To partially circumvent these problems, especially the high dimensionality, we introduce a concept called local projection score (LPS) to represent deviation degree of an observation to its neighbors. The LPS is obtained from the neighborhood information by the technique of low-rank approximation. The observation with high LPS is a promising candidate of outlier in high probability. Based on this notion, we propose an efficient and effective outlier detection algorithm, which is also robust to the parameter ${k}$ of ${k}$ nearest neighbors. Extensive evaluation experiments conducted on twelve public real-world data sets with five popular outlier detection algorithms show that the performance of the proposed method is competitive and promising. [ABSTRACT FROM AUTHOR]

High dimensionality is quite commonly encountered in data mining problems, and hence dimensionality reduction becomes an important task in order to improve the efficiency of learning algorithms. As a widely used technique of dimensionality reduction, feature selection is about selecting a feature subset being guided by certain criterion. In this paper, three unsupervised feature selection algorithms are proposed and addressed from the viewpoint of sparse graph embedding learning. First, using the self-characterization of the given data, we view the data themselves as a dictionary, conduct sparse coding and propose the sparsity preserving feature selection (SPFS) algorithm. Second, considering the locality preservation of neighborhoods for the data, we study a special case of the SPFS problem, namely, neighborhood preserving feature selection problem, and come up with a suitable algorithm. Third, we incorporate sparse coding and feature selection into one unified framework, and propose a neighborhood embedding feature selection (NEFS) criterion. Drawing support from nonnegative matrix factorization, the corresponding algorithm for NEFS is presented and its convergence is proved. Finally, the three proposed algorithms are validated with the use of eight publicly available real-world datasets from machine learning repository. Extensive experimental results demonstrate the superiority of the proposed algorithms over four compared state-of-the-art unsupervised feature selection methods. [ABSTRACT FROM PUBLISHER]

Physical activity recognition of everyday activities such as sitting, standing, laying, walking, and jogging was performed, through the use of smartphone accelerometer data. Activity classification was done on a remote server through the use of machine learning algorithms, data was received from the smartphone wirelessly. The smartphone was placed in the subject’s trouser pocket while data was gathered. A large sample set was used to train the classifiers and then a test set was used to verify the algorithm accuracies. Ten different classifier algorithm configurations were evaluated to determine which performed best overall, as well as, which algorithms performed best for specific activity classes. Based on the results obtained, very accurate predictions could be made for offline activity recognition. The kNN and kStar algorithms both obtained an overall accuracy of 99.01%. [ABSTRACT FROM AUTHOR]

Mining massive and high-speed data streams among the main contemporary challenges in machine learning. This calls for methods displaying a high computational efficacy, with ability to continuously update their structure and handle ever-arriving big number of instances. In this paper, we present a new incremental and distributed classifier based on the popular nearest neighbor algorithm, adapted to such a demanding scenario. This method, implemented in Apache Spark, includes a distributed metric-space ordering to perform faster searches. Additionally, we propose an efficient incremental instance selection method for massive data streams that continuously update and remove outdated examples from the case-base. This alleviates the high computational requirements of the original classifier, thus making it suitable for the considered problem. Experimental study conducted on a set of real-life massive data streams proves the usefulness of the proposed solution and shows that we are able to provide the first efficient nearest neighbor solution for high-speed big and streaming data. [ABSTRACT FROM AUTHOR]

The extreme learning machine (ELM) has become one of the most important and popular algorithms of machine learning, because of its extremely fast training speed, good generalization, and universal approximation/classification capability. The proposal of hierarchical ELM (H-ELM) extends ELM from single hidden layer feedforward networks to multilayer perceptron, greatly strengthening the applicability of ELM. Generally speaking, during training H-ELM, large-scale datasets (DSTs) are needed. Therefore, how to make use of H-ELM framework in processing big data is worth further exploration. This paper proposes a parallel H-ELM algorithm based on Flink, which is one of the in-memory cluster computing platforms, and graphics processing units (GPUs). Several optimizations are adopted to improve the performance, such as cache-based scheme, reasonable partitioning strategy, memory mapping scheme for mapping specific Java virtual machine objects to buffers. Most importantly, our proposed framework for utilizing GPUs to accelerate Flink for big data is general. This framework can be utilized to accelerate many other variants of ELM and other machine learning algorithms. To the best of our knowledge, it is the first kind of library, which combines in-memory cluster computing with GPUs to parallelize H-ELM. The experimental results have demonstrated that our proposed GPU-accelerated parallel H-ELM named as GPH-ELM can efficiently process large-scale DSTs with good performance of speedup and scalability, leveraging the computing power of both CPUs and GPUs in the cluster. [ABSTRACT FROM AUTHOR]

Multivariate time series consist of sequential vector-valued observations of some phenomenon over time. Time series forecasting (for both univariate and multivariate case) is a well-known, high-value machine learning problem, in which the goal is to predict future observations of the time series based on prior ones. Several learning algorithms based on complex fuzzy logic have recently been shown to be very accurate and compact forecasting models. However, these models have only been tested on univariate and bivariate datasets. There has as yet been no investigation of more general multivariate datasets. We report on the extension of the adaptive neuro-complex-fuzzy inferential system learning architecture to the multivariate case. We investigate single-input-single-output, multiple-input-single-output, and multiple-input-multiple-output variations of the architecture, exploring their performance on four multivariate time series. We also explore modifications to the forward- and backward-pass computations in the architecture. We find that our best designs are superior to the published results on these datasets, and at least as accurate as kernel-based prediction algorithms. [ABSTRACT FROM PUBLISHER]

The traditional k -nearest neighbor (KNN) method classifies an object by the majority vote of its neighbors, and only one parameter k is to be optimized. We propose a decomposed version of the KKN (DKNN) method, which classifies an unseen object based on the distances to the centroids of the KNNs. DKNN has a training process to learn the local-optima-free distance metric by solving a convex optimization problem. The optimization problem not only learns a metric that minimizes classification errors and maximizes the margin between intraclass and interclass distances. In addition, it also selects important features and removes irrelevant features when L1 regularization is incorporated in the optimization model. The feature selection component is extremely useful for high-dimensional classification problems. Tested on public datasets, the proposed DKNN is competitive and often outperforms traditional machine learning algorithms. [ABSTRACT FROM PUBLISHER]