Search

Liver diseases represent a significant healthcare challenge, impacting millions globally and posing complexities in diagnosis. To address this global health concern, this paper introduces a groundbreaking enhancement to the Kepler Optimization Algorithm, termed I-KOA, designed specifically for feature selection in high-dimensional datasets. By harnessing the synergies of Opposition-Based Learning and a Local Escaping Operator grounded in the k-nearest Neighbor (kNN) classifier, I-KOA asserts itself as a potent tool for local exploitation, balanced exploration, and evasion of local optima. To our knowledge, this is the first work to exploit KOA as a feature selection method. Pioneering the utilization of KOA as a feature selection method, the paper rigorously tests I-KOA in two extensive experiments, tackling the complex CEC'22 benchmark suite functions and the intricate landscape of five liver disease datasets. Results underscore I-KOA's unparalleled performance, validated through the Friedman test, where it surpasses seven rival optimization algorithms. Achieving an outstanding overall classification accuracy of 93.46%, Feature selection size of 0.1042, sensitivity of 97.46%, precision of 94.37%, and F1-score of 90.35% across the liver disease datasets, I-KOA's randomized algorithm ensures robust feature selection, striking a compelling balance between subset size and classification efficacy. Acknowledging computational demands and generalization nuances, I-KOA is a formidable tool ready to revolutionize medical diagnosis and decision support systems. The open source codes of the proposed I-KOA are available at https://www.mathworks.com/matlabcentral/fileexchange/161376-improved-kepler-optimization-algorithm. • This paper encompasses a proficient I-KOA algorithm based on OBL and LEO methods. • A new optimized feature selection model for liver disease classification using five datasets. • We employ comprehensive analysis metrics to thoroughly assess the I-KOA algorithm's efficacy. • I-KOA stands out by surpassing its competitors, attesting to its remarkable performance. [ABSTRACT FROM AUTHOR]

An Electrical Submersible Pump (ESP) is an important equipment used in the industry for lifting liquids in various types of wells. An ESP is widely used in the oil industry for offshore exploration. Detecting a faulty ESP before installation is a predictive maintenance measure in order to extend its operational time. Machine learning fault diagnosis is an effective way for performing this task. Machine learning fault diagnosis algorithms are highly dependent of the availability of an appropriate problem dataset. This paper describes in detail the problem of ESP fault diagnosis and the ESPset dataset, a real-world and public dataset for vibration-based fault diagnosis of electrical submersible pumps used on offshore oil exploration. In addition, the paper also proposes an experimental framework for adequately comparing research works based on the ESPset dataset and defines benchmark classifiers and respective results as referential to the fault diagnosis research community. The framework considers the fact that some subset of samples are not drawn independently, and therefore, proposes a cross-validation sampling strategy that mitigates the similarity bias among samples. Indeed, this work shows that a conventional k-fold cross-validation may lead to a clear overestimation of the average performance. This fact is supported by results which show that the best classification model drops from a mean F-measure of 0.887 to 0.733 when removing the similarity bias from the data. • The Electrical Submersible Pumps (ESP) fault diagnosis problem description: a broader and detailed specification. • The ESPset vibration dataset is published: 6032 instances of ESP vibration tests. • An experimental framework to avoid dependence between the training and test phases of the learning task. • The ESPset benchmark classifiers are presented. • The results of the experimental study using the proposed framework and the benchmark classifiers are shown. [ABSTRACT FROM AUTHOR]

Abstract This paper addresses the problem of a finite set of entities which are required to achieve a goal within a predefined deadline. For example, a group of students is supposed to submit a homework by a specified cutoff. Further, we are interested in predicting which entities will achieve the goal within the deadline. The predictive models are built based only on the data from that population. The predictions are computed at various time instants by taking into account updated data about the entities. The first contribution of the paper is a formal description of the problem. The important characteristic of the proposed method for model building is the use of the properties of entities that have already achieved the goal. We call such an approach “Self-Learning”. Since typically only a few entities have achieved the goal at the beginning and their number gradually grows, the problem is inherently imbalanced. To mitigate the curse of imbalance, we improved the Self-Learning method by tackling information loss and by several sampling techniques. The original Self-Learning and the modifications have been evaluated in a case study for predicting submission of the first assessment in distance higher education courses. The results show that the proposed improvements outperform the specified two base-line models and the original Self-Learner, and also that the best results are achieved if domain-driven techniques are utilised to tackle the imbalance problem. We also showed that these improvements are statistically significant using Wilcoxon signed rank test. [ABSTRACT FROM AUTHOR]

Artificial neural networks (ANNs), inspired by the interconnection of real neurons, have achieved unprecedented success in various fields such as computer vision and natural language processing. Recently, a novel mathematical ANN model, known as the dendritic neuron model (DNM), has been proposed to address nonlinear problems by more accurately reflecting the structure of real neurons. However, the single-output design limits its capability to handle multi-output tasks, significantly lowering its applications. In this paper, we propose a novel multi-in and multi-out dendritic neuron model (MODN) to tackle multi-output tasks. Our core idea is to introduce a filtering matrix to the soma layer to adaptively select the desired dendrites to regress each output. Because such a matrix is designed to be learnable, MODN can explore the relationship between each dendrite and output to provide a better solution to downstream tasks. We also model a telodendron layer into MODN to simulate better the real neuron behavior. Importantly, MODN is a more general and unified framework that can be naturally specialized as the DNM by customizing the filtering matrix. To explore the optimization of MODN, we investigate both heuristic and gradient-based optimizers and introduce a two-step training method for MODN. Extensive experimental results performed on 11 datasets on both binary and multi-class classification tasks demonstrate the effectiveness of MODN, with respect to accuracy, convergence, and generality. [ABSTRACT FROM AUTHOR]

A critical aspect of dimensionality reduction is to assess the quality of selected (or produced) feature subsets properly. Feature subset assessment in machine learning refers to split a given feature subset into a training set, which is used to estimate the parameters of a classification model, and a test set used to estimate the predictive performance of the model. Then, averaging the results of multiple splitting (i.e., Cross-Validation, CV) is commonly used to decrease the variance of the estimator. But in practice, CV scheme is very computationally expensive. In this paper, we propose a new statistics index method called LW-index for evaluation of feature subset and dimensionality reduction algorithms in general. The proposed method is a type of “classical statistics” approach that uses the feature subset to compute an empirical estimate of the quality of feature subset. A large number of performance comparisons with the machine learning approach conducted on fourteen benchmark collections show that the proposed LW-index is highly correlated with the external indices (i.e., MacroF 1 , MicroF 1 ) of SVM and Centroid-Based Classifier (CBC) trained by five-fold CV scheme. Furthermore, the experimental results indicate that LW-index has the same performance as the traditional CV scheme for evaluating the dimensionality reduction algorithms and it is more efficient than the traditional methodology. Therefore, one contribution of this paper is to present an alternative methodology, based on an internal index typically used in the unsupervised learning context, that is computationally cheaper than the traditional CV methodology. Another contribution is to propose a new internal index that behaves better than other similar indices widely used in clustering and shows high correlation with the results obtained by the traditional methodology. [ABSTRACT FROM AUTHOR]

Differently from Vector-pattern-oriented Classifier Design (VecCD), Matrix-pattern-oriented Classifier Design (MatCD) is expected to manipulate matrix-oriented patterns directly rather than turning them into a vector, and further demonstrated its effectiveness. However, some prior information, such as the local sensitive discriminant information among matrix-oriented patterns, might be neglected by MatCD. To overcome such flaw, a new regularization term named R LSD is adopted into MatCD by taking advantage of Locality Sensitive Discriminant Analysis (LSDA) in this paper. In detail, the objective function of LSDA is modified and transformed into the regularization term R LSD to explore the local sensitive discriminant information among matrix-oriented patterns. In the implementation, R LSD is collaborated with one typical MatCD, whose name is Matrix-pattern-oriented Ho-Kashyap Classifier (MatMHKS), so as to create a new classifier based on local sensitive discriminant information named LSDMatMHKS for short. Finally, comprehensive experiments are designed to validate the effectiveness of LSDMatMHKS. The major contributions of this paper can be concluded as (1) improving the classification performance and the learning ability of MatCD, (2) introducing local sensitive discriminant information into MatCD and extending the application scenario of LSDA, and (3) validating and analyzing the feasibility and effectiveness of R LSD . [ABSTRACT FROM AUTHOR]

Abstract Multi-label classification problem is a key learning task where each instance may belong to multiple class labels simultaneously. However, there exists four main challenges: (a) designing an effective multi-label classifier, (b) learning the high-order asymmetric label correlations automatically, (c) reducing the dimensionality of feature space, (d) dealing with both the full labels and missing labels cases. In this paper, we directly address the above four problems in a unified learning framework, and propose a novel Multi-Label classification approach joint with label correlations, Missing labels and Feature selection, named MLMF. The proposed MLMF not only makes the joint learning of independent binary classifiers, but also allows the joint learning of multi-label classification and label correlations. Meanwhile, the shared sparse feature structure among labels are selected by l 2 , 1 -norm. Furthermore, MLMF can also handle missing labels. Experimental results on sixteen multi-label data sets in terms of six evaluation criteria demonstrate that MLMF outperforms the state-of-the-art multi-label classification algorithms. Highlights • Joint multi-label classification with label correlations and feature selection. • Obtaining the high-order asymmetric label correlations automatically by learning. • Selecting the shared sparse feature structure among labels by l 2 , 1 -norm. • Dealing with both full labels and missing labels cases. [ABSTRACT FROM AUTHOR]

Searching for the (near) optimal subset of features is a challenging problem in the process of feature selection (FS). In the literature, Swarm Intelligence (SI) algorithms show superior performance in solving this problem. This motivated our attempts to test the performance of the newly proposed Salp Swarm Algorithm (SSA) in this area. As such, two new wrapper FS approaches that use SSA as the search strategy are proposed. In the first approach, eight transfer functions are employed to convert the continuous version of SSA to binary. In the second approach, the crossover operator is used in addition to the transfer functions to replace the average operator and enhance the exploratory behavior of the algorithm. The proposed approaches are benchmarked on 22 well-known UCI datasets and the results are compared with 5 FS methods: Binary Grey Wolf Optimizer (BGWO), Binary Gravitational Search Algorithms (BGSA), Binary Bat Algorithm (BBA), Binary Particle Swarm Optimization (BPSO), and Genetic Algorithm (GA). The paper also considers an extensive study of the parameter setting for the proposed technique. From the results, it is observed that the proposed approach significantly outperforms others on around 90% of the datasets. [ABSTRACT FROM AUTHOR]

A Siamese Deep Forest (SDF) is proposed in the paper. It is based on the Deep Forest or gcForest proposed by Zhou and Feng and can be viewed as a gcForest modification. It can be also regarded as an alternative to the well-known Siamese neural networks. The SDF uses a modified training set consisting of concatenated pairs of vectors. Moreover, it defines the class distributions in the deep forest as the weighted sum of the tree class probabilities such that the weights are determined in order to reduce distances between similar pairs and to increase them between dissimilar points. We show that the weights can be obtained by solving a quadratic optimization problem. The SDF aims to prevent overfitting which takes place in neural networks when only limited training data are available. The numerical experiments illustrate the proposed distance metric method. [ABSTRACT FROM AUTHOR]

Automated machine learning has the ability to improve the efficiency of time series classification due to the ability to combine multiple feature extraction methods and classification models. In the paper, we propose a flexible AutoML approach that combines multiple feature generation strategies (spectral, wavelet, topological, quantile) and classifiers as parts of the modeling pipeline. It allows obtaining a more robust and lightweight solution for the time series classification problem. Generation of the pipeline is based on an evolutionary algorithm. Comparison with approaches with the highest results was conducted on the UEA/UCR archive for modeling quality analysis. The proposed approach allows, on the one hand, to solve the task of time series classification automatically and could be used as part of an industrial data processing pipeline. On the other hand, it could be used as a tool for modeling and studying natural process properties described as time series. [ABSTRACT FROM AUTHOR]

In this paper, a novel decremental subspace-based learning method called Decremental Generalized Discriminative Common Vectors method (DGDCV) is presented. The method makes use of the concept of decremental learning, which we introduce in the field of supervised feature extraction and classification. By efficiently removing unnecessary data and/or classes for a knowledge base, our methodology is able to update the model without recalculating the full projection or accessing to the previously processed training data, while retaining the previously acquired knowledge. The proposed method has been validated in 6 standard face recognition datasets, showing a considerable computational gain without compromising the accuracy of the model. [ABSTRACT FROM AUTHOR]

There are situations in which lexicon-based methods for Sentiment Analysis (SA) are not able to generate a classification output for specific instances of a given dataset. Most often, the reason for this situation is the absence of specific terms in the sentiment lexicon required in the classification effort. In such cases, there were only two possible paths to follow: (1) add terms to the lexicon (off-line process) by human intervention to guarantee no noise is introduced into the lexicon, which prevents the classification system to provide an immediate answer; or (2) use the services of a word-frequency dictionary (on-line process), which is computationally costly to build. This paper investigates an alternative approach to compensate for the lack of ability of a lexicon-based method to produce a classification output. The method is based on the combination of the classification outputs of non lexicon-based tools. Specifically, firstly the outcome values of applying two or more non-lexicon classification methods are obtained. Secondly, these non-lexicon outcomes are fused using a uninorm based approach, which has been proved to have desirable compensation properties as required in the SA context, to generate the classification output the lexicon based approach is unable to achieve. Experimental results based on the execution of two well-known supervised machine learning algorithms, namely Naïve Bayes and Maximum Entropy, and the application of a cross-ratio uninorm operator are presented. Performance indices associated to options (1) and (2) above are compared against the results obtained using the proposed approach for two different datasets. Additionally, the performance of the proposed cross-ratio uninorm operator based approach is also compared when the aggregation operator used is the arithmetic mean instead. It is shown that the combination of non lexicon-based classification methods with specific uninorm operators improves the classification performance of lexicon-based methods, and it enables the offering of an alternative solution to the SA classification problem when needed. The proposed aggregation method could be used as well as a replacement of ensemble averaging techniques commonly applied when combining the results of several machine learning classifiers’ outputs. [ABSTRACT FROM AUTHOR]

In reality, the number of labeled time series data is often small and there is a huge number of unlabeled data. Manually labeling these unlabeled examples is time-consuming and expensive, and sometimes it is even impossible. In this paper, we combine active learning and semi-supervised learning to obtain a confident and sufficient labeled training data for multivariate time series classification. We first propose a sampling strategy by ranking the informativeness of unlabeled examples based on its uncertainty and its local data density. Next, an active semi-supervised learning framework is introduced to make best use of the advantage of active learning and semi-supervised learning for data annotation. Finally, we advance a valid stopping criterion of active learning to provide a sufficient and reliable labeled training dataset by costing human resources as less as possible. Our experimental results show that our approach can manually annotate examples as small as possible and simultaneously obtain a confident and informative labeled dataset, which is sufficient to learn an efficient classification. [ABSTRACT FROM AUTHOR]

There have been increasing interests in natural language processing to explore effective methods in learning better representations of text for sentiment classification in product reviews. However, most existing methods do not consider subtle interplays among words appeared in review text, authors of reviews and products the reviews are associated with. In this paper, we make use of a heterogeneous network to model the shared polarity in product reviews and learn representations of users, products they commented on and words they used simultaneously. The basic idea is to first construct a heterogeneous network which links users, products, words appeared in product reviews, as well as the polarities of the words. Based on the constructed network, representations of nodes are learned using a network embedding method, which are subsequently incorporated into a convolutional neural network for sentiment analysis. Evaluations on the product reviews, including IMDB, Yelp 2013 and Yelp 2014 datasets, show that the proposed approach achieves the state-of-the-art performance. [ABSTRACT FROM AUTHOR]

The task of document categorization is to classify documents from a stream as relevant or non-relevant to a particular user interest so as to reduce information overload. Existing solutions typically perform classification at the document level, i.e., a document is returned as relevant if at least a part of the document is of interest of the user. In this paper, we propose a novel multiple-instance stream learning framework for adaptive document categorization, named MIS-DC. Our proposed approach has the ability of making accuracy prediction at both the document level and the block level, while only requires labeling the training documents at the document level. In addition, our proposed approach can also provide adaptive document categorization by detecting and handling concept drift at a finer granularity when data streams evolve over time, thereby yielding higher prediction accuracy than existing data stream algorithms. Experiments on benchmark and real-world datasets have demonstrated the effectiveness of our proposed approach. [ABSTRACT FROM AUTHOR]

Uncertainty measures are critical evaluating tools in machine learning fields, which can measure the dependence and similarity between two feature subsets and can be used to judge the significance of features in classifying and clustering algorithms. In the classical rough sets, there are some uncertainty tools to measure a feature subset, including accuracy, roughness, information entropy, rough entropy, etc. These measures are applicable to discrete-valued information systems, but not suitable to real-valued data sets. In this paper, by introducing the neighborhood rough set model, each object is associated with a neighborhood subset, named a neighborhood granule. Several uncertainty measures of neighborhood granules are proposed, which are neighborhood accuracy, information quantity, neighborhood entropy and information granularity in the neighborhood systems. Furthermore, we prove that these uncertainty measures satisfy non-negativity, invariance and monotonicity. The maximum and minimum of these measures are also given. Theoretical analysis and experimental results show that information quantity, neighborhood entropy and information granularity measures are better than the neighborhood accuracy measure in the neighborhood systems. [ABSTRACT FROM AUTHOR]

Distance metric plays a significant role in machine learning methods(classification, clustering, etc.), especially in k -nearest neighbor classification( k NN), where the Euclidean distances are computed to decide the labels of unknown points. But Euclidean distance ignores the statistical structure which may help to measure the similarity of different inputs better. In this paper, we construct an unified framework, including two eigenvalue related methods, to learn data-dependent metric. Both methods aim to maximize the difference of intra-class distance and inter-class distance, but the optimization is considered in global view and local view respectively. Different from previous work in metric learning, our methods straight seek for equilibrium between inter-class distance and intra-class distance, and the linear transformation decomposed from the metric is to be optimized directly instead of the metric. Then we can effectively adjust the data distribution in transformed space and construct favorable regions for k NN classification. The problems can be solved simply by eigenvalue-decomposition, much faster than semi-definite programming. After selecting the top eigenvalues, the original data can be projected into low dimensional space, and then insignificant information will be mitigated or eliminated to make the classification more efficiently. This makes it possible that our novel methods make metric learning and dimension reduction simultaneously. The numerical experiments from different points of view verify that our methods can improve the accuracy of k NN classification and make dimension reduction with competitive performance. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel machine learning classifier by deriving a new adaptive momentum back-propagation (BP) artificial neural networks algorithm. The proposed algorithm is a modified version of the BP algorithm to improve its convergence behavior in both sides, accelerate the convergence process for accessing the optimum steady-state and minimizing the error misadjustment to improve the recognized patterns superiorly. This algorithm is controlled by the learning rate parameter which is dependent on the eigenvalues of the autocorrelation matrix of the input. It provides low error performance for the weights update. To discuss the performance measures of this proposed algorithm and the other supervised learning algorithms such as k-nearest neighbours (k-NN), Naive Bayes (NB), linear discriminant analysis (LDA), support vector machines (SVM), BP, and BP with adaptive momentum (PBPAM) have been compared in term of speed of convergence, Sum of Squared Error (SSE), and accuracy by implementing benchmark problem - XOR and seven datasets from UCI repository. [ABSTRACT FROM AUTHOR]

A decision forest is an ensemble of decision trees, and it is often built to discover more patterns (i.e. logic rules) and predict/classify class values more accurately than a single decision tree. Existing decision forest algorithms are typically used for building huge numbers of decision trees, involving large memory and computational overhead, in order to achieve high accuracy. Generally, many of the trees do not contribute to improving the ensemble accuracy of a forest. As a result, ensemble pruning algorithms aim to get rid of those trees while generating a subforest in order to achieve higher (or comparable) ensemble accuracy than the original forest. The objectives are two fold: select as small number of trees as possible, and maintain the ensemble accuracy of the subforest as high as possible. An optimal subforest can be found by exhaustive search; however it is not practical for any standard-sized forest as the number of candidate subforests grows exponentially. In order to avoid the computational burden of an exhaustive search, many greedy and genetic algorithm-based subforest selection techniques have been proposed in literature. In this paper, we propose a subforest selection technique that achieves small size as well as high accuracy. We use a genetic algorithm where we carefully select high quality individual trees for the initial population of the genetic algorithm in order to improve the final output of the algorithm. Experiments are conducted on 20 data sets from the UCI Machine Learning Repository to compare the proposed technique with several existing state-of-the-art techniques. The results indicate that the proposed technique can select effective subforests which are significantly smaller than original forests while achieving better (or comparable) accuracy than the original forests. [ABSTRACT FROM AUTHOR]

Feature subset selection (FSS) has been an active area of research in machine learning. A number of techniques have been developed for selecting an optimal or sub-optimal subset of features, because it is a major factor to determine the performance of a machine-learning technique. In this paper, we propose and develop a novel optimization technique, namely, a binary coordinate ascent (BCA) algorithm that is an iterative deterministic local optimization that can be coupled with wrapper or filter FSS. The algorithm searches throughout the space of binary coded input variables by iteratively optimizing the objective function in each dimension at a time. We investigated our BCA approach in wrapper-based FSS under area under the receiver-operating-characteristic (ROC) curve (AUC) criterion for the best subset of features in classification. We evaluated our BCA-based FSS in optimization of features for support vector machine, multilayer perceptron, and Naïve Bayes classifiers with 12 datasets. Our experimental datasets are distinct in terms of the number of attributes (ranging from 18 to 11,340), and the number of classes (binary or multi-class classification). The efficiency in terms of the number of subset evaluations was improved substantially (by factors of 5–37) compared with two popular FSS meta-heuristics, i.e., sequential forward selection (SFS) and sequential floating forward selection (SFFS), while the classification performance for unseen data was maintained. [ABSTRACT FROM AUTHOR]

Abstract: This paper presents a new approach for inducing decision trees based on Variable Precision Rough Set Model. The presented approach is aimed at handling uncertain information during the process of inducing decision trees and generalizes the rough set based approach to decision tree construction by allowing some extent misclassification when classifying objects. In the paper, two concepts, i.e. variable precision explicit region, variable precision implicit region, and the process for inducing decision trees are introduced. The authors discuss the differences between the rough set based approaches and the fundamental entropy based method. The comparison between the presented approach and the rough set based approach and the fundamental entropy based method on some data sets from the UCI Machine Learning Repository is also reported. [Copyright &y& Elsevier]

Multi-label text classification is an important yet challenging task in natural language processing. It is more complex than single-label text classification in that the labels tend to be correlated. To capture this complex correlations, sequence to sequence model has been widely applied, and achieved impressing performance for multi-label text classification. It encodes each document as contextual representations, and then decodes them to generate labels one by one. At each time step, the decoder usually adopts the attention mechanism to highlight important contextual representations to predict a related label, which has been proved to be effective. Nevertheless, the traditional attention approaches only utilize a hidden state to explore such contextual representations, which may result in prediction errors, or omit several trivial labels. To tackle this problem, in this paper, we propose "history-based attention", which takes history information into consideration, to effectively explore informative representations for labels' predictions in multi-label text classification. Our approach consists of two parts: history-based context attention and history-based label attention. History-based context attention considers historical weight trends to highlight important context words, which is helpful to predict trivial labels. History-based label attention explores historical labels to alleviate the error propagation problem. We conduct experiments on two popular text datasets (i.e., Arxiv Academic Paper Dataset and Reuters Corpus Volume I), it is demonstrated that the history-based attention mechanism could boost the performance to a certain extent, and the proposed method consistently outperforms highly competitive approaches. [ABSTRACT FROM AUTHOR]

Noise filters are preprocessing techniques designed to improve data quality in classification tasks by detecting and eliminating examples that contain errors or noise. However, filtering can also remove correct examples and examples containing valuable information, which could be useful for learning. This fact usually implies a margin of improvement on the noise detection accuracy for almost any noise filter. This paper proposes a scheme to improve the performance of noise filters in multi-class classification problems, based on decomposing the dataset into multiple binary subproblems. Decomposition strategies have proven to be successful in improving classification performance in multi-class problems by generating simpler binary subproblems. Similarly, we adapt the principles of the One-vs-One decomposition strategy to noise filtering, making the noise identification process simpler. In order to integrate the filtering results achieved in the binary subproblems, our proposal uses a soft voting approach considering a reliability level based on the aggregation of the noise degree prediction calculated for each binary classifier. The experimental results show that the One-vs-One decomposition strategy usually increases the performance of the noise filters studied, which can detect more accurately the noisy examples. [ABSTRACT FROM AUTHOR]

In machine learning, the presumably best model is selected from a variety of model candidates generated by testing different model types, hyperparameters, or feature subsets. The advent of deep learning has made model selection even more challenging due to the huge parameter search space. Relying on a single metric to select the best model does not consider class imbalances or the different costs of misclassifications. We argue that incorporating human knowledge to interactively analyse the per-class errors and class confusions over all model candidates enables a more efficient training process and yields better models for given applications. This paper proposes the model-agnostic approach ConfusionVis which allows to comparatively evaluate and select multi-class classifiers based on their confusion matrices. This contributes to making the models' results understandable, while treating the models as black boxes. Therefore, we propose a novel method to measure and visualise distances between confusion matrices and an interactive query interface to incorporate all composition levels of class errors. The approach is evaluated in a user study and the applicability is shown by a case study where marine biologists investigate the conservation efforts of baleen whales by classifying whale species in acoustic recordings. ConfusionVis is available online: https://www.ml-and-vis.org/confusionvis. [ABSTRACT FROM AUTHOR]

Feature weighting has been an important topic in classification learning algorithms. In this paper, we propose a new paradigm of assigning weights in classification learning, called value weighting method. While the current weighting methods assign a weight to each feature, we assign a different weight to the values of each feature. The proposed method is implemented in the context of naive Bayesian learning, and optimal weights of feature values are calculated using a gradient approach. The performance of naive Bayes learning with value weighting method is compared with that of other state-of-the-art methods for a number of datasets. The experimental results show that the value weighting method could improve the performance of naive Bayes significantly. [ABSTRACT FROM AUTHOR]

The insufficiency of labeled training data for representing the distribution of entire dataset is a major obstacle in various practical data mining applications. Semi-supervised learning algorithms, which attempt to learn from both labeled and unlabeled data, provide possibilities to solve this problem. Graph-based semi-supervised learning has recently become one of the most active research areas. In this paper, a novel graph-based semi-supervised learning approach entitled Class Dissimilarity based Linear Neighborhood Propagation (CD-LNP) is proposed, which assumes that each data point can be linearly reconstructed from its neighborhood. The neighborhood graph of the input data is constructed according to a certain kind of dissimilarity between data points, which is specially designed to integrate the class information. Our algorithm can propagate the labels from the labeled points to entire data set using these linear neighborhoods with sufficient smoothness. Experiment results demonstrate that our approach outperforms other popular graph-based semi-supervised learning methods. [ABSTRACT FROM AUTHOR]

Class imbalance is an active research area in the machine learning community. However, existing and recent literature showed that class overlap had a higher negative impact on the performance of learning algorithms. This paper provides detailed critical discussion and objective evaluation of class overlap in the context of imbalanced data and its impact on classification accuracy. First, we present a thorough experimental comparison of class overlap and class imbalance. Unlike previous work, our experiment was carried out on the full scale of class overlap and an extreme range of class imbalance degrees. Second, we provide an in-depth critical technical review of existing approaches to handle imbalanced datasets. Existing solutions from selective literature are critically reviewed and categorised as class distribution-based and class overlap-based methods. Emerging techniques and the latest development in this area are also discussed in detail. Experimental results in this paper are consistent with existing literature and show clearly that the performance of the learning algorithm deteriorates across varying degrees of class overlap whereas class imbalance does not always have an effect. The review emphasises the need for further research towards handling class overlap in imbalanced datasets to effectively improve learning algorithms' performance. [ABSTRACT FROM AUTHOR]

The poor explainability of deep learning models has hindered their adoption in safety and quality-critical applications. This paper focuses on image classification models and aims to enhance the explainability of deep learning models through the development of an uncertainty quantification-based framework. The proposed methodology consists of three major steps. In the first step, we adopt dropout-based Bayesian neural network to characterize the structure and parameter uncertainty inherent in deep learning models, propagate and represent such uncertainties to the model prediction as a distribution. Next, we employ entropy as a quantitative indicator to measure the uncertainty in model prediction, and develop an Empirical Cumulative Distribution Function (ECDF)-based approach to determine an appropriate threshold value for the purpose of deciding when to accept or reject the model prediction. Secondly, in the cases with high model prediction uncertainty, we combine the prediction difference analysis (PDA) approach with dropout-based Bayesian neural network to quantify the uncertainty in pixel-wise feature importance, and identify the locations in the input image that highly correlate with the model prediction uncertainty. In the third step, we develop a robustness-based design optimization formulation to enhance the relevance between input features and model prediction, and leverage a differential evolution approach to optimize the pixels in the input image with high uncertainty in feature importance. Experimental studies in MNIST and CIFAR-10 image classifications are included to demonstrate the effectiveness of the proposed approach in increasing the explainability of deep learning models. [ABSTRACT FROM AUTHOR]

Abstract: Robust classification models based on the ensemble methodology are proposed in the paper. The main feature of the models is that the precise vector of weights assigned for examples in the training set at each iteration of boosting is replaced by a local convex set of weight vectors. The minimax strategy is used for building weak classifiers at each iteration. The local sets of weights are constructed by means of imprecise statistical models. The proposed models are called RILBoost (Robust Imprecise Local Boost). Numerical experiments with real data show that the proposed models outperform the standard AdaBoost algorithm for several well-known data sets. [Copyright &y& Elsevier]

Abstract: Side information, like must-link (ML) and cannot-link (CL), has been widely used in single-view classification tasks. However, so far such information has never been applied in multi-view classification tasks. In many real world situations, data with multiple representations or views are frequently encountered, and most proposed algorithms for such learning situations require that all the multi-view data should be paired. Yet this requirement is difficult to satisfy in some settings and the multi-view data could be totally unpaired. In this paper, we propose an learning framework to design the multi-view classifiers by only employing the weak side information of cross-view must-links (CvML) and cross-view cannot-links (CvCL). The CvML and the CvCL generalize the traditional single-view must-link (SvML) and single-view cannot-link (SvCL), and to the best of our knowledge, are first definitely introduced and applied into the multi-view classification situations. Finally, we demonstrate the effectiveness of our method in our experiments. [Copyright &y& Elsevier]

Abstract: Most machine learning tasks involve learning from high-dimensional data, which is often quite difficult to handle. Hubness is an aspect of the curse of dimensionality that was shown to be highly detrimental to k-nearest neighbor methods in high-dimensional feature spaces. Hubs, very frequent nearest neighbors, emerge as centers of influence within the data and often act as semantic singularities. This paper deals with evaluating the impact of hubness on learning under class imbalance with k-nearest neighbor methods. Our results suggest that, contrary to the common belief, minority class hubs might be responsible for most misclassification in many high-dimensional datasets. The standard approaches to learning under class imbalance usually clearly favor the instances of the minority class and are not well suited for handling such highly detrimental minority points. In our experiments, we have evaluated several state-of-the-art hubness-aware kNN classifiers that are based on learning from the neighbor occurrence models calculated from the training data. The experiments included learning under severe class imbalance, class overlap and mislabeling and the results suggest that the hubness-aware methods usually achieve promising results on the examined high-dimensional datasets. The improvements seem to be most pronounced when handling the difficult point types: borderline points, rare points and outliers. On most examined datasets, the hubness-aware approaches improve the classification precision of the minority classes and the recall of the majority class, which helps with reducing the negative impact of minority hubs. We argue that it might prove beneficial to combine the extensible hubness-aware voting frameworks with the existing class imbalanced kNN classifiers, in order to properly handle class imbalanced data in high-dimensional feature spaces. [Copyright &y& Elsevier]

Abstract: The Support Vector Machine (SVM) is one of the most popular machine learning algorithms for classification and regression. SVM displays outstanding performance when utilized in many applications. However, different approaches have been proposed in order to improve its performance in general and in special cases. This paper proposes a new method of improving SVM performance in general. This method can be applied to all the types of SVMs that have differing kernel types. The key aim of the proposed approach is to transfer more information from the training phase to the testing phase. This information is obtained from the misclassified data of the training phase, which is ignored in the classic SVM. Experimental results from eleven datasets from the real online resources show that this approach can improve the classification performances of C-SVM and ν-SVM without adding any parameters to the learner algorithm. Statistical tests show significance in this improvement. The proposed method has been found to improve accuracies of C-SVM and ν-SVM in more than 67% of the experiments in which 11% of these improvements are more than 5%. It must be noted that the highest improvement found with this method was 25%. [Copyright &y& Elsevier]

In the age of big data, lots of data obtained is low-quality data characterized by heterogeneousness and incompleteness, referred to as heterogeneous incomplete decision systems (HIDSs) in this paper. Data classification is an important task in machine learning, with the ability to discover valuable knowledge hidden in HIDSs. However, systematic studies on data classification in HIDSs are rarely reported. Especially, there is a lack of adaptive classification methods for HIDSs, which can deal directly with heterogeneous incomplete data and do not require prior discretization of numerical attributes or filling in missing values. In this paper, a unified representation model, called parameterized tolerance granulation model (PTGM), is proposed to deal with heterogeneous incomplete data. And the principle of an adaptive granulation method of constructing appropriate PTGMs is also described using difference-based collaborative optimization. Based on PTGMs, decision logic language is used to describe classifiers consisting of decision rules satisfying given conditions. Then, a discernibility function-based and a heuristic function-based classification methods are proposed to obtain all optimized rule sets (classifiers) and to generate a particular optimized rule set, respectively. The heuristic function-based method is actually an adaptive classification method, which can deal directly with heterogeneous incomplete data. Furthermore, detailed theoretical analyses are given to illustrate the correctness and effectiveness of the proposed methods. The experimental results show that the proposed methods are effective and have obvious advantages in directly handling heterogeneous incomplete data. [ABSTRACT FROM AUTHOR]

Abstract: There are a lot of systems that make decisions or classifications on the basis of a number of rules. This set of rules that govern such a system is called the rule base. When a new system of this kind is being developed, setting up its rule base is a time-consuming and expensive process because the rule base contains the knowledge of the outside world, which could be acquired from experts or produced from previous experiences. In this latter case, machine-learning algorithms can help. In fact, many methods have been proposed to generate rules from training instances. The aim of this paper is to present a new fuzzy learning algorithm to generate IF-THEN rules, for classifying instances in one application domain. This algorithm can improve the results offered by a previously presented algorithm. In addition, the more common classification problems of the original algorithm are presented and a measure to determine the conflicts among generated rules is introduced. Moreover, we study the classification stage of that inductive fuzzy learning algorithm and an improvement is suggested to obtain better classification results. [Copyright &y& Elsevier]

Abstract: It has been shown that the structural information of data may contain useful prior domain knowledge for training a classifier. How to apply the structural information of data to build a good classifier is a new research focus recently. As we all know, the all existing structural large margin methods are the common in considering all structural information within classes into one model. In fact, these methods do not balance all structural information’s relationships both infra-class and inter-class, which directly results in these prior information not being exploited sufficiently. In this paper, we design a new Structural Twin Support Vector Machine (called -TWSVM). Unlike existing methods based on structural information, -TWSVM uses two hyperplanes to decide the category of new data, of which each model only considers one class’s structural information and closer to the class at the same time far away from the other class. This makes -TWSVM fully exploit these prior knowledge to directly improve the algorithm’s the capacity of generalization. All experiments show that our proposed method is rigidly superior to the state-of-the-art algorithms based on structural information of data in both computation time and classification accuracy. [Copyright &y& Elsevier]

Abstract: This paper presents a Crotch Ensemble classification model for high dimensional data. A Crotch Ensemble is obtained from a decision cluster tree built by calling a clustering algorithm recursively. A crotch is an inner node of the tree together with its direct children. If the children of a crotch have more than one dominant class, the crotch is defined as a crotch predictor. Each crotch predictor constructs a classifier by itself. A Crotch Ensemble consists of a set of crotch predictors. When classifying a new object, a subset of crotch predictors is selected according to the distances between the object and the crotch predictors. A classification is made on the object as the class predicted by the crotch predictors with the maximum accumulative weights. The experimental results on both synthetic and real data have shown that the Crotch Ensemble model can get better classification results on high dimensional data than other classification methods. [Copyright &y& Elsevier]

Abstract: Feature selection aims at finding a feature subset that has the most discriminative information from the original feature set. In this paper, we firstly present a new scheme for feature relevance, interdependence and redundancy analysis using information theoretic criteria. Then, a dynamic weighting-based feature selection algorithm is proposed, which not only selects the most relevant features and eliminates redundant features, but also tries to retain useful intrinsic groups of interdependent features. The primary characteristic of the method is that the feature is weighted according to its interaction with the selected features. And the weight of features will be dynamically updated after each candidate feature has been selected. To verify the effectiveness of our method, experimental comparisons on six UCI data sets and four gene microarray datasets are carried out using three typical classifiers. The results indicate that our proposed method achieves promising improvement on feature selection and classification accuracy. [Copyright &y& Elsevier]

Abstract: Translation initiation site (TIS) recognition is one of the first steps in gene structure prediction, and one of the common components in any gene recognition system. Many methods have been described in the literature to identify TIS in transcribed sequences such as mRNA, EST and cDNA sequences. However, the recognition of TIS in DNA sequences is a far more challenging task, and the methods described so far for transcripts achieve poor results in DNA sequences. Most methods approach this problem taking into account its biological characteristics. In this work we try a different view, considering this classification problem from a purely machine learning perspective. From the point of view of machine learning, TIS recognition is a class imbalance problem. Thus, in this paper we approach TIS recognition from this angle, and apply the different methods that have been developed to deal with imbalanced datasets. The proposed approach has two advantages. Firstly, it improves the results using standard classification methods. Secondly, it broadens the set of classification algorithms that can be used, as some of the class-imbalance methods, such as undersampling, are also useful as methods for scaling up data mining algorithms as they reduce the size of the dataset. In this way, classifiers that cannot be applied to the whole dataset, due to long training time or large memory requirements, can be used when undersampling method is applied. Results show an advantage of class imbalance methods with respect to the same methods applied without considering the class imbalance nature of the problem. The applied methods are also able to improve the results obtained with the best method in the literature, which is based on looking for the next in-frame stop codon from the putative TIS that must be predicted. [Copyright &y& Elsevier]

Abstract: Semi-supervised dimensional reduction methods play an important role in pattern recognition, which are likely to be more suitable for plant leaf and palmprint classification, since labeling plant leaf and palmprint often requires expensive human labor, whereas unlabeled plant leaf and palmprint is far easier to obtain at very low cost. In this paper, we attempt to utilize the unlabeled data to aid plant leaf and palmprint classification task with the limited number of the labeled plant leaf or palmprint data, and propose a semi-supervised locally discriminant projection (SSLDP) algorithm for plant leaf and palmprint classification. By making use of both labeled and unlabeled data in learning a transformation for dimensionality reduction, the proposed method can overcome the small-sample-size (SSS) problem under the situation where labeled data are scant. In SSLDP, the labeled data points, combined with the unlabeled data ones, are used to construct the within-class and between-class weight matrices incorporating the neighborhood information of the data set. The experiments on plant leaf and palmprint databases demonstrate that SSLDP is effective and feasible for plant leaf and palmprint classification. [ABSTRACT FROM AUTHOR]

Abstract: This paper presents Feature Interval Learning algorithms (FIL) which represent multi-concept descriptions in the form of disjoint feature intervals. The FIL algorithms are batch supervised inductive learning algorithms and use feature projections of the training instances to represent induced classification knowledge. The concept description is learned separately for each feature and is in the form of a set of disjoint intervals. The class of an unseen instance is determined by the weighted-majority voting of the feature predictions. The basic FIL algorithm is enhanced with adaptive interval and feature weight schemes in order to handle noisy and irrelevant features. The algorithms are empirically evaluated on twelve data sets from the UCI repository and are compared with k-NN, k-NNFP, and NBC classification algorithms. The experiments demonstrate that the FIL algorithms are robust to irrelevant features and missing feature values, achieve accuracy comparable to the best of the existing algorithms with significantly less average running times. [Copyright &y& Elsevier]

Abstract: This paper introduces a new learning technique for the multicriteria classification method PROAFTN. This new technique, called DEPRO, utilizes a Differential Evolution (DE) algorithm for learning and optimizing the output of the classification method PROAFTN. The limitation of the PROAFTN method is largely due to the set of parameters (e.g., intervals and weights) required to be obtained to perform the classification procedure. Therefore, a learning method is needed to induce and extract these parameters from data. DE is an efficient metaheuristic optimization algorithm based on a simple mathematical structure to mimic a complex process of evolution. Some of the advantages of DE over other global optimization methods are that it often converges faster and with more certainty than many other methods and it uses fewer control parameters. In this work, the DE algorithm is proposed to inductively obtain PROAFTN’s parameters from data to achieve a high classification accuracy. Based on results generated from 12 public datasets, DEPRO provides excellent results, outperforming the most common classification algorithms. [Copyright &y& Elsevier]

Abstract: Extant multiple-cost-sensitive learning algorithms are usually designed for dealing with misclassification cost (MC) and test cost (TC) together. This paper outlines a new learning algorithm, called cost-time sensitive classification, designed for minimizing tangible costs (which includes TC and waiting cost (WC)) as well as maximizing the decrease of the intangible costs (also called MC). The proposed algorithm induces decision trees from training datasets with missing data, in which the costs are measured in different units. Firstly, a split criterion is proposed for building cost-time sensitive decision trees, aiming at possibly reducing the intangible cost. Then a hybrid test strategy, which can handle missing values in test datasets, is designed for combining the sequential test with the batch test strategy. To evaluate the efficiency of the proposed method, extensive experiments were conducted on the UCI datasets at different missing rates. The experimental results show that the proposed algorithm achieves better than the existing ones in terms of reducing the intangible costs when taking into account waiting costs. [Copyright &y& Elsevier]

Imbalanced datasets classification remains an important domain in machine learning. Conventional supervised learning algorithms tend to be biased towards the majority class when addressing imbalanced datasets, thus providing poor classification results on the minority class. The learning task would become more challenging when there are overlapping and within-imbalance issues in imbalanced datasets, which are often the case and have been proven to play more negative impact on the classification performance than between-class imbalance. In this paper, we propose a novel SVDD (Support Vector Data Description) boundary and DPC (Density Peaks Clustering) clustering technique-based oversampling approach (SVDDDPCO) for handling imbalanced and overlapped data. The proposed approach first utilizes support vector data description (SVDD) model with greater penalty constant for the minority class than the majority class to generate the class boundary, and then identifies those misclassified majority or few minority instances by the boundary as potential overlapped or noisy ones and eliminates them. To address the within-balance issues, density peaks clustering (DPC) is used to cluster the minority instances due to its advantage of accurately identifying sub-clusters with different sizes and densities, which facilitates simultaneously combating between-class and within-class imbalance issues caused by various reasons. The minority instances are assigned with different weights inversely proportional to their local densities and their distances to the boundary and then the size for each identified sub-cluster to be oversampled is adaptively determined according to its own size and the weights of its elements. Such strategy aims to generate more synthetic minority instances for borderline and sparser small sub-clusters which are usually informative to the later learning tasks. Finally, oversampling is performed within each sub-cluster with different sizes to not only counteract the within imbalance but also avoid the generation of any noisy or overlapped synthetic instance. Extensive comparison results on various datasets showed the proposed method achieves statistically significant improvements in classification performance relative to state-of-the-art ones. [ABSTRACT FROM AUTHOR]

Intention recognition is based on a dialog between users to identify their real intentions, which plays a key role in the question answering system. However, the content of a dialog is usually in the form of short text. Due to data sparsity, many current classification models show poor performance on short text. To address this issue, we propose AMFF, an attention-based multi-feature fusion method for intention recognition. In this paper, we enrich short text features by fusing features extracted from frequency-inverse document frequency (TF-IDF), convolutional neural networks (CNNs) and long short-term memory (LSTM). For the purpose of measuring the important features, we utilize the attention mechanisms to assign weights for the fusion features. Experimental results on the TREC, SST1 and SST2 datasets demonstrate that the proposed AMFF model outperforms traditional machine learning models and typical deep learning models on short text classification. [ABSTRACT FROM AUTHOR]

Active learning (AL) can significantly reduce the cost of labeling instances. Extreme learning machine (ELM) has low computational cost, extremely fast training speed and strong generalization ability. Previous studies have shown that the combination of them can generate efficient learning models. Nevertheless, these researches did not focus on multiclass imbalanced data. Cold start may occur and the performance of classifier is also reduced due to the imbalanced distribution of categories. Moreover, there is no framework for processing stream-based data. To address these problems, an improved framework called AL for class incremental and weighted sequential ELM (AI-WSELM) is proposed in this paper, and its advantages are as follows: (1) similarity query and margin sampling were used to alleviate cold start and select uncertain instances, respectively, (2) an improved weighting strategy was used to tackle stream-based multiclass imbalanced distribution, (3) a class incremental mechanism was added to deal with new categories appeared in the subsequent batches, and (4) AI-WSELM greatly reduced the cost of labeling samples when ensuring classification performance. The simulation results show that the proposed model has satisfactory performance compared to the existing ELMs and the other related algorithms, which indicates the feasibility and effectiveness of AI-WSELM. [ABSTRACT FROM AUTHOR]

A class of machine learning problem where each instance may either belong to one or more than one class simultaneously is known as Multi-label classification problem. Unlike other classification problems there exist several challenges and the important among them is learning the label correlation when labels are missing. In this paper, we present a unified learning system that addresses the aforementioned issue, and suggest a novel multi-label classifier termed as Multi-label classification with Missing Labels using Label Correlation and Robust Structural Learning (MLLCRS-ML). The propose classifier considers label-specific features along with utilizing the structural property of data and pairwise label correlation (both positive and negative label correlations) for recovering the missing labels. We have use hinge loss function which ensures less sensitivity towards outliers and the accelerated proximal gradient method (APG) to efficiently solve the underlying optimization problem. Experimental results on several benchmark data sets show that our propose approach MLLCRS-ML is as competitive as other state-of-the-art approaches. • Unified framework for feature selection and classification with missing label. • Learning missing label using label correlation and structure of data. • Learning label specific features. • Experimental evaluations on ten multi-label datasets proves effectiveness of model. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, we have introduced a new method in which every training point learns what is happening in its neighborhood. So, a hyperplane is learned and associated to each point. With this hyperplane we can define the bands distance, a distance measure that bring closer or move away points depending on its classes. We have used this new distance in classification tasks and have performed tests over 68 datasets: 18 well-known UCI-Repository datasets, one private dataset, and 49 ad hoc synthetic datasets. We have used 10-fold cross-validation and, in order to compare the results of the classifiers, we have considered the mean accuracy and have also performed a paired two-tailored t-Student’s test with a significance level of 95%. The results are encouraging and confirm the good behavior of the new proposed classification method. The bands distance has obtained the best overall results with 1-NN and k-NN classifiers when compared with other distances. Finally, we extract conclusions and outline some lines of future work. [Copyright &y& Elsevier]

Many classification algorithms require that training examples contain only discrete values. In order to use these algorithms when some attributes have continuous numeric values, the numeric attributes must be converted into discrete ones. This paper describes a new way of discretizing numeric values using information theory. Our method is context-sensitive in the sense that it takes into account the value of the target attribute. The amount of information each interval gives to the target attribute is measured using Hellinger divergence, and the interval boundaries are decided so that each interval contains as equal amount of information as possible. In order to compare our discretization method with some current discretization methods, several popular classification data sets are selected for discretization. We use naive Bayesian classifier and C4.5 as classification tools to compare the accuracy of our discretization method with that of other methods. [Copyright &y& Elsevier]

In many applications, an enormous amount of unlabeled data is available with little cost. Therefore, it is natural to ask whether we can take advantage of these unlabeled data in classification learning. In this paper, we analyzed the role of unlabeled data in the context of naive Bayesian learning. Experimental results show that including unlabeled data as part of training data can significantly improve the performance of classification accuracy. [Copyright &y& Elsevier]

In molecular biology, biological macromolecules, like desoxyribonucleic acids (DNA) and proteins are coded by strings, called ‘primary structures’. For a long time, biologists gathered these primary structures in large databases. Now, they focus on analyzing these primary structures in order to extract useful knowledge. Data mining approaches can be helpful to reach this goal. In this paper, we present a data mining approach based on machine learning techniques to do classification of biological sequences. By using our approach, we use four steps as follows. (1) In the first step, we construct the set of the discriminant substrings, called discriminant descriptor (DD), associated with each family of primary structures. This construction is made thanks to an adaptation of the Karp, Miller and Rosenberg (KMR) algorithm. (2) In the second step, we use the DDs constructed during the first step to code the families of primary structures by a table of examples vs attributes, called ‘context’. (3) In the third step, we extract knowledge from the context constructed during the second step and represent it by production rules. This extraction is made by using an incremental production rules approach. (4) Finally, during the last step, we use the obtained production rules to do classification of primary structures. [Copyright &y& Elsevier]