Search

Feature selection wrapper methods are powerful mechanisms for reducing the complexity of prediction models while preserving and even improving their precision. Meta-heuristic methods, such as multi-objective evolutionary algorithms, are commonly used as search strategies in feature selection wrapper methods since they allow minimizing the cardinality of the attribute subset and simultaneously maximizing the predictive capacity of the model. However, in high-dimensional problems, multi-objective evolutionary algorithms for wrapper-type feature selection may require excessive computational time, sometimes impractical, especially when the learning algorithm has a high computational cost, such as deep learning. To address this drawback, in this paper we propose a multi-surrogate assisted multi-objective evolutionary algorithm for feature selection, specially designed to improve generalization error. The proposed method has been compared with conventional feature selection wrapper methods that use random forest, support vector machine and long short-term memory learning algorithms to evaluate subsets of attributes. The experiments have been carried out with regression and classification problems with time series data for air quality forecasting in the south-east of Spain and for indoor temperature forecasting in a domotic house. The results demonstrate the superiority of the proposed multi-surrogate assisted method over conventional wrapper methods using the same run times. [ABSTRACT FROM AUTHOR]

Feature selection wrapper methods are powerful mechanisms for reducing the complexity of prediction models while preserving and even improving their precision. Meta-heuristic methods, such as multi-objective evolutionary algorithms, are commonly used as search strategies in feature selection wrapper methods since they allow minimizing the cardinality of the attribute subset and simultaneously maximizing the predictive capacity of the model. However, in high-dimensional problems, multi-objective evolutionary algorithms for wrapper-type feature selection may require excessive computational time, sometimes impractical, especially when the learning algorithm has a high computational cost, such as deep learning. To address this drawback, in this paper we propose a multi-surrogate assisted multi-objective evolutionary algorithm for feature selection, specially designed to improve generalization error. The proposed method has been compared with conventional feature selection wrapper methods that use random forest, support vector machine and long short-term memory learning algorithms to evaluate subsets of attributes. The experiments have been carried out with regression and classification problems with time series data for air quality forecasting in the south-east of Spain and for indoor temperature forecasting in a domotic house. The results demonstrate the superiority of the proposed multi-surrogate assisted method over conventional wrapper methods using the same run times. [ABSTRACT FROM AUTHOR]

Selecting discriminative features to build effective learning models is a significant research work in machine learning. In practical applications, the data distribution characteristics are diverse, and the uncertainties pose challenges for building learning models with robustness and generalization capabilities. Since one-hot encoding is good at representing independent labels, the label matrix of regression-based feature selection (FS) methods is usually encoded with one-hot encoding. However, it's not well adapted to the different data distributions. This paper proposes a sparse orthogonal supervised FS model with global redundancy minimization, label scaling, and robustness (GRMLSRSOFS) to address the above problems. This model uses the label scaling technique proposed in this paper to better adapt to different data distributions. An iterative optimization method is given, and its convergence is demonstrated theoretically and experimentally. Further, experimental results on 12 public datasets show that 1) The GRMLSRSOFS can achieve higher classification accuracy with fewer features in most cases than several state-of-the-art FS methods. For example, the GRMLSRSOFS achieves 100% classification accuracy using only 20 features on the warpPIE10P dataset and obtains nearly 6% improvement over other methods on the Yale dataset. 2) The convergence speed of the GRMLSRSOFS will be faster after label scaling. [ABSTRACT FROM AUTHOR]

As data collected from different sources have multiple representations, multi-view learning has become an important paradigm of machine learning. To exploit multi-view data, previous works either tackle each view separately or concatenate all views directly, such that the distinctions as well as correlations of different views are often ignored. Furthermore, existing models usually involve intractable parameters that need to be manually determined to balance the contributions of different views, degrading the efficiency and applicability of models. In this paper, a novel multi-view learning framework, namely Robust Multi-view learning via Adaptive Regression (RMAR), is derived to discriminate diverse views in a self-supervised weighting manner without extra parameters. Meanwhile, RMAR coalesces multiple feature projections with adaptive view-wise weights and adopts L 2 , 1 -norm regression loss to learn a joint projection subspace compatible across all views, not only increasing the robustness of model but also preserving the consistency and diversity among views. Furthermore, RMAR can be naturally extended for feature selection by imposing L 2 , 1 -norm constraint on feature projections. Additionally, an efficient convergent algorithm is developed to solve RMAR. Extensive experiments have been performed to validate the effectiveness of RMAR for classification and feature selection and show its superiority over state-of-the-arts. [ABSTRACT FROM AUTHOR]

As data collected from different sources have multiple representations, multi-view learning has become an important paradigm of machine learning. To exploit multi-view data, previous works either tackle each view separately or concatenate all views directly, such that the distinctions as well as correlations of different views are often ignored. Furthermore, existing models usually involve intractable parameters that need to be manually determined to balance the contributions of different views, degrading the efficiency and applicability of models. In this paper, a novel multi-view learning framework, namely Robust Multi-view learning via Adaptive Regression (RMAR), is derived to discriminate diverse views in a self-supervised weighting manner without extra parameters. Meanwhile, RMAR coalesces multiple feature projections with adaptive view-wise weights and adopts L 2 , 1 -norm regression loss to learn a joint projection subspace compatible across all views, not only increasing the robustness of model but also preserving the consistency and diversity among views. Furthermore, RMAR can be naturally extended for feature selection by imposing L 2 , 1 -norm constraint on feature projections. Additionally, an efficient convergent algorithm is developed to solve RMAR. Extensive experiments have been performed to validate the effectiveness of RMAR for classification and feature selection and show its superiority over state-of-the-arts. [ABSTRACT FROM AUTHOR]

Multi-label classification is a machine learning task to construct a model for assigning an entity in the dataset to two or more class labels. In order to improve the performance of multi-label classification, a multi-objective feature selection algorithm has been proposed in this paper. Feature selection as a preprocessing task for Multi-label classification problems aims to choose a subset of relevant features. Selecting a small number of high-quality features decreases the computational cost and at the same time maximizes the classification performance. However extreme decreasing the number of features causes the failure of classification. As a result, feature selection has two conflicting objectives, namely, minimizing the classification error and minimizing the number of selected features. This paper proposes a multi-objective optimization algorithm to tackle the multi-label feature selection. The task is to find a set of solutions (a subset of features) in a sophisticated large-scale search space using a reference-based multi-objective optimization method. The proposed algorithm utilizes an opposition-based binary operator to generate more diverse solutions. Injection of extreme point of the Pareto-front is another component of the algorithm which aims to find feature subsets with less classification error. The proposed method is compared with two other existing methods on eight multi-label benchmark datasets. The experimental results show that the proposed method outperforms existing algorithms in terms of various multi-objective evaluation measures, such as Hyper-volume indicator, Pure diversity, Two-set coverage, and Pareto-front proportional contribution. The proposed method leads to get a set of well-distributed trade-off solutions which reach less classification error in comparing with competitors, even with the fewer number of features. [ABSTRACT FROM AUTHOR]

• Automatic approach for Combined Algorithm Selection and Hyperparameter Optimization problem. • Efficient and high accuracy. • Reinforced policy for feature selection. With years of development, machine learning algorithms have excellent performance in some tasks of data analysis and data mining. To apply machine learning to new tasks, suitable algorithm and hyperparameters selection techniques, which is known as Combined Algorithm Selection and Hyperparameter optimization problem , are in demand. In the field of data analysis, how to automate the algorithm selection process has become a hot research topic in recent years. Most of the existing approaches are developed under the background of Automated Machine Learning with high time or space complexity. To alleviate the issue, an approach extracts and learns from prior experience based on meta-learning theory named Auto-CASH is proposed in this paper. One of the major drawbacks of existing meta-learning methods is that they rely too much on human expertise to extract and filter knowledge that guides subsequent training. Auto-CASH can automatically select features of tasks by introducing a reinforcement learning strategy. Thus Auto-CASH becomes less dependent on human expertise. Besides, two pruning strategies when processing Hyperparameter Optimization to improve efficiency are firstly proposed. Extensive experiments on classification tasks are conducted and results demonstrate that Auto-CASH outperforms state-of-the-art CASH approaches and popular AutoML systems with less time cost. [ABSTRACT FROM AUTHOR]

When dealing with large-scale datasets, high-dimensional feature selection plays a crucial role in improving the performance and interpretability of machine learning models. However, it still faces the problems of the "dimensionality curse" and high computational cost in dealing with high-dimensional datasets. In this paper, we develop a dual-module hybrid feature selection algorithm based on correlation filtering and a multi-objective evolutionary algorithm based on dynamic variation distance (HMOFS-CFDVD), which aims to reduce the computational cost of high-dimensional features and search space. Firstly, a coarse-grained feature filtering method based on correlation is proposed, enabling the algorithm to quickly identify potentially better feature subsets in the later stages. After that, a fine-grained feature selection is further implemented based on the multi-objective evolutionary algorithm with sample variation distance to obtain a high-quality feature subset. This stage incorporates a novel population initialization method with self-regulation, an adaptive evolution strategy, an optimal sample selection mechanism based on dynamic sample distance, and an improved diversity mechanism to enhance the evolutionary performance. Moreover, the feature relevance metric is introduced as a third objective to improve the overall algorithm's performance. Experimental results on 12 high-dimensional feature datasets demonstrate that the proposed HMOFS-CFDVD achieves high accuracy and produces a smaller subset of features. [ABSTRACT FROM AUTHOR]

Generally, every label has its own indispensable features, and it is reasonable to assume higher repetitiveness of indispensable features represents higher correlations among labels. Inspired by discernibility matrix and granular computing, this paper draws conclusions as follows: • Essential elements for any label are defined and characterized to reflect the internalconnection between features and label. • Label correlation and relevance judgement matrix for label set are described considering overlap of essential element sets. • Labels with strong correlation are assigned to a relevant label group. • The compact subset of indispensable features for any relevant label group is effectively captured from original feature set. • A novel multi-label learning method, called CLSF-MK, is proposed to increase effectiveness and practicability of prediction. Each example of multi-label data is represented in an object with its feature vector (i.e. an instance) while being related to multiple labels. Learning label correlation can effectively reduce the labels needed to be predicted and optimize the classification performance. For this reason, label correlation plays a crucial role in multi-label learning and has been explored by many existing algorithms. Generally, every label has its own indispensable features, and it is reasonable to assume that a higher repetitiveness of indispensable features represents higher correlations among labels. Inspired by this fact, the essential elements for each label, which are composed of indispensable features, are constructed in this paper. A method is proposed for learning label correlation and applying it to feature selection of multi-label data based on the overlap of different families of essential elements related to label. Firstly, the essential elements of each label are defined and characterized to reflect the internal connection between features and label. In addition, a process for calculating the essential elements of a single label is provided. Secondly, by considering the overlap of the essential element collections that are determined by the different labels, relevancy of label and corresponding relevance judgement matrix with the label set are described. Therefore, several labels with strong relationships are assigned to a relevant group of labels. Meanwhile, local and global label correlations can be computed. Thus a novel multi-label learning algorithm, called CLSF, is presented to select a compact subset of indispensable features for each relevant group of labels by applying local label correlation to feature selection of multi-label data. Finally, comprehensive experiments on eleven benchmark data sets clearly illustrate the effectiveness and efficiency of CLSF against five other multi-label learning algorithms. [ABSTRACT FROM AUTHOR]

Feature selection has become a mandatory task in data mining, due to the overwhelming amount of features in Big Data problems. To handle this high-dimensional data and avoid the well-known curse of dimensionality, we need to pre-select an optimal subset of features to reduce redundant computations. Federated learning is a machine learning technique based on training an algorithm over many decentralized edge devices holding local rather than global data on a centralized server. Application of this technique is extending to fields such as self-driving cars, medicine and health, and Industry 4.0, where data privacy is compulsory. Feature selection through federated learning is a complicated task since suboptimal features calculated by feature selection methods may be different in heterogeneous datasets from different nodes. In this paper, we propose a lossless federated version of the classic minimum redundancy maximum relevance (mRMR) feature selection algorithm, called federated mRMR (fed-mRMR), which, without losing any effectiveness of the original mRMR method, is applicable to federated learning approaches and capable of dealing with data that are not independent and identically distributed (non-IID data). Implementation can be found at: https://github.com/jorgehermo9/fed-mrmr • Feature Selection (FS) is a key step in machine learning. • Classical FS does not consider privacy preservation. • Federated FS allows to perform feature selection in federated environments. • fed-mRMR is a federated feature selection tool publicly available. • fed-mRMR performs better than original's mRMR proposal in terms of computation. [ABSTRACT FROM AUTHOR]

Feature selection plays a crucial role in machine learning and data mining, and improves the performance of learning models by selecting a distinguishing feature subset and eliminating irrelevant features. Existing feature selection methods are mainly used for single-label learning and multi-label learning; however, there are only a few feature selection methods for label distribution learning. Label distribution learning has the "curse of dimensionality" problem, similar to that in multi-label learning. In label distribution learning, the related labels of each sample have different levels of importance. Therefore, multi-label feature selection algorithms can not be directly applied to label distribution data, and discretizing the label distribution data into multi-label data would result in the loss of some important supervised information. To solve this problem, a novel feature selection algorithm for label distribution learning is proposed in this paper. The proposed method utilizes neighborhood granularity to explore feature similarity, and it uses a correlation coefficient to generate the label correlations. In addition, sparse learning is used to improve the robustness and control complexity. Experimental results indicate that our proposed method is more effective than five state-of-art feature selection algorithms on twelve datasets, with respect to six representative evaluation measures. [ABSTRACT FROM AUTHOR]

Multi-label learning algorithms have significant challenges due to high-dimensional feature space and noises in multi-label datasets. Feature selection methods are effective techniques to deal with these problems. ParetoCluster is an effective multi-label feature selection algorithm based on Pareto dominance and cluster analysis concepts which considers each label an objective function. This algorithm loses its effectiveness to differentiate features when dealing with high labeled datasets and makes most features incomparable (e.g., when most features fall into the first layer). Thus, a cluster analysis criterion in ParetoCluster will play a decisive role in determining the most relevant features. Bearing this in mind, in this paper, we have modeled the multi-label feature selection problem into a bi-objective optimization problem regarding the relevancy and redundancy degree of the features. We then handle it using Pareto dominance in this bi-objective domain. To illustrate the optimality and efficiency of the proposed method, we have compared our approach against some similar techniques. [ABSTRACT FROM AUTHOR]

• Shockable and non-shockable ECG signals are analyzed. • A review study for shockable ECG signal recognition is discussed. • Unique bispectrum and recurrence plots are proposed for shockable and non-shockable ECG signals. • Performance of nonlinear features for differentiating shockable and non-shockable. • ECG signals is compared. Sudden cardiac death from lethal arrhythmia is a preventable cause of death. Ventricular fibrillation and tachycardia are shockable electrocardiographic (ECG)rhythms that can respond to emergency electrical shock therapy and revert to normal sinus rhythm if diagnosed early upon cardiac arrest with the restoration of adequate cardiac pump function. However, manual inspection of ECG signals is a difficult task in the acute setting. Thus, computer-aided arrhythmia classification (CAAC) systems have been developed to detect shockable ECG rhythm. Traditional machine learning and deep learning methods are now progressively employed to enhance the diagnostic accuracy of CAAC systems. This paper reviews the state-of-the-art machine and deep learning based CAAC expert systems for shockable ECG signal recognition, discussing their strengths, advantages, and drawbacks. Moreover, unique bispectrum and recurrence plots are proposed to represent shockable and non-shockable ECG signals. Deep learning methods are usually more robust and accurate than standard machine learning methods but require big data of good quality for training. We recommend collecting large accessible ECG datasets with a meaningful proportion of abnormal cases for research and development of superior CAAC systems. [ABSTRACT FROM AUTHOR]

Unsupervised feature selection (UFS) is a fundamental task in machine learning and data analysis, aimed at identifying a subset of non-redundant and relevant features from a high-dimensional dataset. Embedded methods seamlessly integrate feature selection into model training, resulting in more efficient and interpretable models. Current embedded UFS methods primarily rely on self-representation or pseudo-supervised feature selection approaches to address redundancy and irrelevant feature issues, respectively. Nevertheless, there is currently a lack of research showcasing the fusion of these two approaches. This paper proposes the Orthogonal Encoder-Decoder factorization for unsupervised Feature Selection (OEDFS) model, combining the strengths of self-representation and pseudo-supervised approaches. This method draws inspiration from the self-representation properties of autoencoder architectures and leverages encoder and decoder factorizations to simulate a pseudo-supervised feature selection approach. To further enhance the part-based characteristics of factorization, orthogonality constraints and local structure preservation restrictions are incorporated into the objective function. The optimization process is based on the multiplicative update rule, ensuring efficient convergence. To assess the effectiveness of the proposed method, comprehensive experiments are conducted on 14 datasets and compare the results with eight state-of-the-art methods. The experimental results demonstrate the superior performance of the proposed approach in terms of UFS efficiency. [ABSTRACT FROM AUTHOR]

Feature selection has been widely used in machine learning for a long time. In this paper, we propose a supervised local sparse discriminative feature selection method named LSDFS to obtain sparse features by imposing ℓ 2 , 0 -norm constraint on transformation matrix. Differently from traditional approaches, our method does not require approximation or relaxation schemes, such as ℓ 2 , p -norm to solve long-standing challenge. Our method is based on the trace difference form of Linear Discriminant Analysis (LDA), which can efficiently obtain discriminative information in low-dimensional space. In order to explore the local structure of data which contains more discriminative information, we adopt a sparse connections graph between anchor points and data points instead of fully-connected graph with time-consuming, and add a decay parameter to avoid trivial solutions, making the model more precisely. Extensive experiments conducted on synthetic datasets and several real-world datasets have demonstrated the advantages of our method. [ABSTRACT FROM AUTHOR]

Practitioners use feature importance to rank and eliminate weak predictors during model development in an effort to simplify models and improve generality. Unfortunately, they also routinely conflate such feature importance measures with feature impact, the isolated effect of an explanatory variable on the response variable. This can lead to real-world consequences when importance is inappropriately interpreted as impact in applications like medicine and business. The dominant approach for computing feature importance is through interrogation of a fitted model, which works well for feature selection, but gives distorted measures of feature impact. For example, the same method applied to the same data set can yield different feature importances, depending on the model, leading us to conclude that impact should be computed directly from the data. While there are nonparametric feature selection algorithms, they typically provide feature rankings, rather than direct measures of impact or importance. They also often focus on single-variable associations with the response. In this paper, we provide mathematical definitions of feature impact and importance, derived from partial dependence curves, that operate directly on the data. We develop two methods, StratImpact and StratImp, that estimate feature impact and importance from partial dependence measures using stratification of the explanatory variables. We show that features ranked by these definitions are competitive with, and often better than, existing feature selection techniques. We validate our methods through a comparison with contemporary methods using three real data sets and a testbed of simulated data. [ABSTRACT FROM AUTHOR]

Current machine learning algorithms encounter challenges such as missing labels and high dimensionality. Feature selection serves as an effective dimensionality reduction technique, enhancing the efficiency and accuracy of subsequent machine learning tasks by eliminating irrelevant and redundant features. Given the difficulty in obtaining fully labeled data, partially labeled data has become a crucial target for machine learning models to address. The related family is an efficient, rough set-based feature selection approach; however, it cannot be applied to semi-supervised learning tasks. Consequently, this paper introduces a semi-supervised feature selection method based on a fuzzy related family for partially labeled data. At first, the fuzzy label values of unlabeled samples are calculated based on fuzzy similarity relationships by establishing a novel fuzzy covering system. Subsequently, a fuzzy related family is constructed by a consistent fuzzy set. Then a semi-supervised feature selection algorithm, referred to as the Semi-supervised Fuzzy Related Family (SFRF), is developed using the established feature significance measurement. Compared to existing semi-supervised feature selection algorithms, SFRF considerably enhances feature selection efficiency while preserving classification accuracy. Specifically, the average reduction efficiency across twelve datasets increased by up to 109 times. [ABSTRACT FROM AUTHOR]

Discrimination of mirco-earthquake on source depth plays an important role in the field of micro-seismic monitoring. Conventional machine learning methods for data classification rely on carefully hand-engineered features that are vulnerable to low signal-to-noise ratio. Convolutional neural networks (CNNs) demonstrate some merits in dealing with structured data modelling where a set of meaningful features can be automatically extracted from sample learning. This paper explores the use of machine learning techniques for discrimination between deep and shallow mirco-seismic events. A benchmarked dataset including 444 micro-earthquakes from an underground cavern collapse in South Louisiana is employed for performance evaluation in this study, where several feature-based classifiers are compared against the CNN classifier. Empirical results show that the deep learning method outperforms the conventional classification techniques in discriminating the source depth. [ABSTRACT FROM AUTHOR]

• A simple filter method to quantify the similarity between the observed and expected probabilities and generate scores for features. • Reduction of data dimensionality and features overlapping. • Results shows that the proposed method significantly reduces the numbers of selected features on 27 datasets. • Results analysis showed competitive performance of the proposed method over common Filter methods in terms of evaluation measures. Identifying the relevant set of features in a dataset is an important part of data analytics. Discarding significant variables or keeping irrelevant variables has significant effects on the performance of the learning algorithm during knowledge discovery. In this paper, a feature selection method called Least Loss (L2) is proposed that significantly reduces the dimensionality of data by disposing weakly correlated variables in a robust manner without diminishing the predictive performance of classifiers. The proposed method is based on quantifying the similarity between the observed and expected probabilities and generating scores for each independent variable, which makes it simple and intuitive. The evaluation of the proposed method was done by comparing its performance against Information Gain (IG) and Chi Square (CHI) feature selection methods on 27 different datasets modeled using a probabilistic classifier. The results reveal that L2 is highly competitive with respect to error rate, precision, and recall measures while substantially reducing the number of selected variables in the datasets. Our study would be of high interest to data analysts, scholars and domain experts who deal with applications that include large numbers of features using statistical analysis methods. [ABSTRACT FROM AUTHOR]

The Markov blanket (MB), a key concept in a Bayesian network (BN), is essential for large-scale BN structure learning and optimal feature selection. Many MB discovery algorithms that are either efficient or effective have been proposed for addressing high-dimensional data. In this paper, we propose a new algorithm for E fficient and E ffective MB discovery, called EEMB. Specifically, given a target feature, the EEMB algorithm discovers the PC (i.e., parents and children) and spouses of the target simultaneously and can distinguish PC from spouses during MB discovery. We compare EEMB with the state-of-the-art MB discovery algorithms using a series of benchmark BNs and real-world datasets. The experiments demonstrate that EEMB is competitive with the fastest MB discovery algorithm in terms of computational efficiency and achieves almost the same MB discovery accuracy as the most accurate of the compared algorithms. [ABSTRACT FROM AUTHOR]

• The best combination of feature selection and discretization is identified. • Particularly, there are two different orderings of combining the two data pre-processing steps. • Three feature selection and four discretizers are used for performance comparison. • Performing MDLP first and C4.5 second is the better choice for the SVM classifier. • Performing C4.5 first and MDLP second is recommended for the DT classifier. Feature selection and data discretization are two important data pre-processing steps in data mining, with the focus in the former being on filtering out unrepresentative features and in the latter on transferring continuous attributes into discrete ones. In the literature, these two domain problems have often been studied, individually. However, the combination of these two steps has not been fully explored, although both feature selection and discretization may be required for some real-world datasets. In this paper, two different combination orders of feature selection and discretization are examined in terms of their classification accuracies and computational times. Specifically, filter, wrapper, and embedded feature selection methods are employed, which are PCA, GA, and C4.5, respectively. For discretization, both supervised and unsupervised learning based discretizers are used, specifically MDLP, ChiMerge, equal frequency binning, and equal width binning. The experimental results, based on 10 UCI datasets, show that, for the SVM classifier performing MDLP first and C4.5 second outperforms the other combinations. Not only is less computational time required but this also provides the highest rate of classification accuracy. For the decision tree classifier, performing C4.5 first and MDLP second is recommended. [ABSTRACT FROM AUTHOR]

Multi-label classification aims to construct prediction models from input space to output space for multi-label datesets. However, the feature space of multi-label dataset and the hypothesis space of classifier are always complex. In order to obtain high performance multi-kernel learning algorithms, the core problem is to design algorithms that compress both hypothesis space and feature space. In this paper, by combining the local Rademacher complexity and Hilbert-Schmidt independence criterion (HSIC), an effective multi-kernel learning algorithm for multi-label classification is proposed to compress the feature space and hypothesis space simultaneously. Based on the tail sum of eigenvalues of the integral operator corresponding to kernels, the upper bound of local Rademacher complexity of linear functions class in the feature space is determined. The Hilbert-Schmidt independence criterion is applied to maximize the correlation between the convex combination of the input kernel matrices and the ideal kernel matrix in the label space. Finally, the Laplace regularization model is built by controlling the tail sum of eigenvalues of the integral operator corresponding to kernels of this criterion. Therefore, the above process obtains a combined kernel, simultaneously by sorting the coefficients of the combined kernel, a feature selection algorithm can be constructed. On the basis, the combined kernel can also be used to design a new binary classifier for multi-label classification. The experimental results verify that our proposed multi-kernel learning algorithms can effectively compress the hypothesis space, while the feature selection algorithms can effectively compress the feature space. [ABSTRACT FROM AUTHOR]

As the density of railway operations increases annually, train delays typically result in a greater degree of economic loss. If we can accurately assess the current evolution of delayed trains, railway authorities will be able to make timely decisions and reduce travel anxiety among passengers. In this paper, a deep learning model integrating wavelet transform (WT), fully connected neural network (FCNN), and long short-term memory (LSTM) network is proposed to address the short-term delay prediction problem for trains. Its predictive performance is enhanced by introducing two front train selection schemes, two methods for converting train operating time features into time-series data, and five commonly used wavelet functions. For validation purposes, the model is tested using real-world operational data from two distinct high-speed railway lines in China, and common machine learning algorithms are compared, including random forest (RF), gradient boosting regression tree (GBRT), XGBoosting (XGB), and support vector regression (SVR). The final experimental results demonstrate that our proposed model outperforms the benchmark models and can be effectively applied to practical prediction problems. • We investigate the spatio-temporal effects of delay propagation, categorizing delay data attributes into front trains and target trains. Two feature selection schemes are proposed to assess the impact of front train features on prediction performance. • Wavelet Transform (WT) is employed for the first time in train delay prediction to eliminate random interference. The model's prediction performance is enhanced through denoising and expanding feature dimensionality. • Real data from two Chinese high-speed railways validate the model's accuracy, providing practical guidance for its application. [ABSTRACT FROM AUTHOR]

Highlights • Exploit patterns in filter measure values to auto-identify an optimal feature subset. • Reduce 79.2% of feature dimensionality without sacrificing detection accuracy. • Boost detection accuracy by using the recommended best performing classifier. • Robust and adaptable to different datasets with significant performance gain. • Prescribe a set of baseline features as the de facto standard for phishing detection. Abstract This paper proposes a new feature selection framework for machine learning-based phishing detection system, called the Hybrid Ensemble Feature Selection (HEFS). In the first phase of HEFS, a novel Cumulative Distribution Function gradient (CDF-g) algorithm is exploited to produce primary feature subsets, which are then fed into a data perturbation ensemble to yield secondary feature subsets. The second phase derives a set of baseline features from the secondary feature subsets by using a function perturbation ensemble. The overall experimental results suggest that HEFS performs best when it is integrated with Random Forest classifier, where the baseline features correctly distinguish 94.6% of phishing and legitimate websites using only 20.8% of the original features. In another experiment, the baseline features (10 in total) utilised on Random Forest outperforms the set of all features (48 in total) used on SVM, Naive Bayes, C4.5, JRip, and PART classifiers. HEFS also shows promising results when benchmarked using another well-known phishing dataset from the University of California Irvine (UCI) repository. Hence, the HEFS is a highly desirable and practical feature selection technique for machine learning-based phishing detection systems. [ABSTRACT FROM AUTHOR]

Highlights • A novel filter type unsupervised feature selection algorithm, namely, a factor graph model for unsupervised feature selection (FGUFS) is proposed. • In FGUFS, the maximal information coefficient (MIC) is used to measure the similarities between features, and a message passing algorithm developed for the purpose is used to infer the factor graph. • Extensive experiments show the strengths of FGUFS over existing methods to achieve high clustering accuracy, RI and purity while containing few redundant features. Abstract In this paper, a factor graph model for unsupervised feature selection (FGUFS) is proposed. FGUFS explicitly measures the similarities between features; these similarities are passed to each other as messages in the graph model. The importance score of each feature is calculated using the message-passing algorithm, and then feature selection is performed based on the final importance scores. Extensive experiments were performed on several datasets, and the results demonstrate that FGUFS outperforms other state-of-art unsupervised feature selection algorithms on several performance measures. [ABSTRACT FROM AUTHOR]

Abstract As a popular method in machine learning, lasso performs regression and feature selection simultaneously. However, for large datasets, the training efficiency of lasso remains a challenge. Recently, an Enhanced screening rule via Dual Polytope Projection (EDPP) was proposed to substantially reduce the scale of lasso by deleting inactive features beforehand. However, EDPP may mistakenly discard active features in practice due to the unavailable optimal solutions. To solve this problem, a safe reinforced feature screening rule based on EDPP and feasible solutions (S-EDPP) is introduced in this paper. By utilizing feasible solutions and estimating a proper upper bound of the deviation, S-EDPP can be guaranteed to be safe both in theory and in practice. Theoretical analysis of the deviation term in S-EDPP is given to verify its efficiency. Furthermore, S-EDPP is also extended to accelerate the Elastic Net, which is a corrective method of lasso. Experiments on synthetic and real datasets verify that S-EDPP is a safe modification of EDPP and it gives superior performance than other existing safe rules. [ABSTRACT FROM AUTHOR]

Highlights • The U-curve problem can be used to model feature selection. • We pointed out an error in the first algorithm proposed to solve the U-curve problem. • We introduced UCS, an optimal algorithm for the U-curve problem. • We also provide UCSR, an algorithm for a special case of the U-curve problem. • UCS and UCSR outperformed BFS, CHCGA and SFFS in feature selection experiments. Abstract The U-curve optimization problem is characterized by a decomposable in U-shaped curves cost function over the chains of a Boolean lattice. This problem can be applied to model the classical feature selection problem in Machine Learning. In this paper, we point out that the firstly proposed algorithm to tackle the U-curve problem, the RBM algorithm, is in fact suboptimal. We also present two new algorithms: UCS, which is actually optimal to tackle this problem; and UCSR, a variation of UCS that solves a special case of the U-curve problem and relies on a reduced, ordered binary decision diagram to control the search space. We provide results of two computational assays with these new algorithms: first, W-operator design for filtering of binary images; second, linear SVM design for classification of data sets from the UCI Machine Learning Repository. We show that, in these assays, UCS and UCSR outperformed an exhaustive search and also three widely used heuristics: the SFFS sequential selection, the BFS graph-based search, and the CHCGA genetic algorithm. Finally, we analyze the obtained results and point out improvements that might enhance the performance of these two novel algorithms. [ABSTRACT FROM AUTHOR]

Highlights • Orthogonal projection is proposed to mitigate the domain shift problem. • Semantic feature representation is included to alleviate visual category ambiguity. • Deep model is applied to improve the performance. • Extensive experiments show the superiority of our algorithm. Abstract To mitigate the problems of visual ambiguity and domain shift in conventional zero-shot learning (ZSL), in this paper, we propose a novel method, namely, dual-verification network (DVN), which accepts features and attributes in a pairwise manner as input and verifies the result in both the attribute and feature spaces. First, the DVN projects a feature onto an orthogonal space, where the projected feature has maximum correlation with its corresponding attribute and is orthogonal to all the other attributes. Second, we adopt the concept of semantic feature representation, which computes the relationship between the semantic feature and class labels. Based on this concept, we project the attributes onto the feature space by extending the attributes and labels from the class level to instance level. In addition, we employ a deep architecture and utilize the cross entropy loss to train an end-to-end network for dual verification. Extensive experiments in ZSL and generalized ZSL are performed on four well-known datasets, and the results show that the proposed DVN exhibits a competitive performance relative to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Identifying Drug-Target Interactions (DTIs) is an important process in drug discovery. Traditional experimental methods are expensive and time-consuming for detecting DTIs. Therefore, computational approaches provide many effective strategies to deal with this issue. In recent years, most of computational methods only use the information of drug-drug similarity or target-target similarity, which cannot perfectly capture all characteristics to identify DTIs. In this paper, we propose a novel computational model of DTIs prediction, based on machine learning methods. To improve the performance of prediction, we further use molecular substructure fingerprints, Multivariate Mutual Information (MMI) of proteins and network topology to represent drugs, targets and relationship between them. Moreover, we employ Support Vector Machine (SVM) and Feature Selection (FS) to construct model for predicting DTIs. Experiments of evaluation show that proposed approach achieves better results than other outstanding methods for feature-based DTIs prediction. The proposed approach achieves AUPRs of 0.899, 0.929, 0.821 and 0.655 on Enzyme, Ion Channel (IC), GPCR and Nuclear Receptor datasets, respectively. Compared with existing best methods, AUPRs are increased by 0.016 on Ion Channel datasets. In addition, our method obtains the second best performance on GPCR and Enzyme datasets. The source code and all datasets are available at https://figshare.com/s/53bf5a6065f3911d46f6 . [ABSTRACT FROM AUTHOR]

In this paper, we study a novel and challenging issue, multi-label feature selection with streaming labels, in which the number of labels is unknown in advance, and the size of the feature set is constant. In this problem, we assume that the labels arrive one at a time, and the learning task is to rank features iteratively when a new label arrives. Traditional multi-label feature selection methods cannot perform well in this scenario. Therefore, we present an optimization framework where the weight of each label’s feature rank list and the final feature rank list are defined as two sets of unknown variables. The objective is to minimize the overall weighted deviation between the final feature rank list and each label’s feature rank list. Extensive experiments on benchmark data sets demonstrate that the proposed method outperforms other multi-label feature selection methods. [ABSTRACT FROM AUTHOR]

Instance selection is an important preprocessing step in machine learning. By choosing a subset of a data set, it achieves the same performance of a machine learning algorithm as if the whole data set is used, and it enables a machine learning algorithm to be feasible for and to work effectively with large data sets. Based on voting mechanism, this paper proposes a large data sets instance selection algorithm with MapReduce and random weight networks (RWNs). Firstly, the proposed algorithm employs the Map of MapReduce to partition the large data sets into some small subsets, and deploys them to different cloud computing nodes. Secondly, the informative instances are selected in parallel with an instance selection algorithm. Thirdly, the Reduce of MapReduce is used to collect the selected instances from different cloud computing nodes and a selected instance subset is obtained. The previous three processes are repeated p times ( p is a parameter defined by the user), and p instance subsets are obtained. Finally, the voting method is used to select the most informative instances from the p subsets. The random weight network classifier is trained with the selected instance subset, and the testing accuracy is verified on the testing set. The proposed algorithm is experimentally compared with three state-of-the-art approaches which are CNN, ENN and RNN. The experimental results show that the proposed algorithm is effective and efficient. [ABSTRACT FROM AUTHOR]

In multiple-instance learning (MIL), class labels are attached to bags instead of instances, and the goal is to predict the class labels of unseen bags. Existing MIL algorithms generally fall into two types: those designed at the bag level and those designed at the instance level. In this paper, we aim to employ bags directly as learning objects and design multiple-instance feature-weighting algorithms at the bag level. In particular, we initially provide a brief introduction of the bag-level large margin feature-weighting framework and then adopt the three bag-level distances minimal Hausdorff (minH), class-to-bag (C2B) and bag-to-bag (B2B) as examples to design the corresponding bag-level feature-weighting algorithms. Experiments conducted on synthetic and real-world datasets empirically demonstrate the effectiveness of our work in improving MIL performances. [ABSTRACT FROM AUTHOR]

Fuzzy rule-based models have been extensively used in regression problems. Besides high accuracy, one of the most appreciated characteristics of these models is their interpretability, which is generally measured in terms of complexity. Complexity is affected by the number of features used for generating the model: the lower the number of features, the lower the complexity. Feature selection can therefore considerably contribute not only to speed up the learning process, but also to improve the interpretability of the final model. Nevertheless, a very few methods for selecting features before learning regression models have been proposed in the literature. In this paper, we focus on these methods, which perform feature selection as pre-processing step. In particular, we have adapted two state-of-the-art feature selection algorithms, namely NMIFS and CFS, originally proposed for classification, to deal with regression. Further, we have proposed FMIFS, a novel forward sequential feature selection approach, based on the minimal-redundancy-maximal-relevance criterion, which can manage directly fuzzy partitions. The relevance and the redundancy of a feature are measured in terms of, respectively, the fuzzy mutual information between the feature and the output variable, and the average fuzzy mutual information between the feature and the just selected features. The stopping criterion for the sequential selection is based on the average values of relevance and redundancy of the just selected features. We have performed two experiments on twenty regression datasets. In the first experiment, we aimed to show the effectiveness of feature selection in fuzzy rule-based regression model generation by comparing the mean square errors achieved by the fuzzy rule-based models generated using all the features, and the features selected by FMIFS, NMIFS and CFS. In order to avoid possible biases related to the specific algorithm, we adopted the well-known Wang and Mendel algorithm for generating the fuzzy rule-based models. We present that the mean square errors obtained by models generated by using the features selected by FMIFS are on average similar to the values achieved by using all the features and lower than the ones obtained by employing the subset of features selected by NMIFS and CFS. In the second experiment, we intended to evaluate how feature selection can reduce the convergence time of the evolutionary fuzzy systems, which are probably the most effective fuzzy techniques for tackling regression problems. By using a state-of-the-art multi-objective evolutionary fuzzy system based on rule learning and membership function tuning, we show that the number of evaluations can be considerably reduced when pre-processing the dataset by feature selection. [ABSTRACT FROM AUTHOR]

The present paper describes a new feature weighting method based on a membership margin. Distinctive properties of the proposed method include its capability to process problems characterized by mixed-type data (quantitative, qualitative and interval) as well as a huge number of features. The key idea is to map simultaneously all the features of different types into a common space; the membership space. Once all features are represented in a homogeneous space, a feature weighting task can be performed in unified way. This weighting approach is integrated here within a fuzzy classifier through a fuzzy rule weighted concept in order to improve its performance. Each antecedent fuzzy set in the fuzzy if–then rule is weighted to characterize the importance of each proposition and therefore its corresponding feature. Weight estimation process is based on membership margin maximization to estimate a fuzzy weight of each feature in the membership space. Experiments on low and high dimensional real-world datasets demonstrate that the proposed approach can improve significantly the performance of the fuzzy rule-based as well as other state of the art classifiers and can even outperform classical feature weighting approaches. In particular, we show that this approach can yield meaningful results on two real-world applications for cancer prognosis and industrial process diagnosis. [ABSTRACT FROM AUTHOR]

Microarray data classification is a difficult challenge for machine learning researchers due to its high number of features and the small sample sizes. Feature selection has been soon considered a de facto standard in this field since its introduction, and a huge number of feature selection methods were utilized trying to reduce the input dimensionality while improving the classification performance. This paper is devoted to reviewing the most up-to-date feature selection methods developed in this field and the microarray databases most frequently used in the literature. We also make the interested reader aware of the problematic of data characteristics in this domain, such as the imbalance of the data, their complexity, or the so-called dataset shift. Finally, an experimental evaluation on the most representative datasets using well-known feature selection methods is presented, bearing in mind that the aim is not to provide the best feature selection method, but to facilitate their comparative study by the research community. [ABSTRACT FROM AUTHOR]

Abstract: Feature selection refers to the problem of selecting those input features that are most predictive of a given outcome; a problem encountered in many areas such as machine learning, pattern recognition and signal processing. In particular, solution to this has found successful application in tasks that involve datasets containing huge numbers of features (in the order of tens of thousands), which would otherwise be impossible to process further. Recent examples include text processing and web content classification. Rough set theory has been used as such a dataset pre-processor with much success, but current methods are inadequate at finding globally minimal reductions, the smallest sets of features possible. This paper proposes a technique that considers this problem from a propositional satisfiability perspective. In this framework, globally minimal subsets can be located and verified. [Copyright &y& Elsevier]

Abstract: Feature selection techniques have been successfully applied in many applications for making supervised learning more effective and efficient. These techniques have been widely used and studied in traditional supervised learning settings, where each instance is expected to have a label. In multiple instance learning (MIL) each example or bag consists of a variable set of instances, and the label is known for the bag as a whole, but not for the individual instances it consists of. Therefore utilizing these labels for feature selection in MIL becomes less straightforward. In this paper we study a new feature subset selection method for MIL called HyDR-MI (hybrid dimensionality reduction method for multiple instance learning). The hybrid consists of the filter component based on an extension of the ReliefF algorithm developed for working with MIL and the wrapper component based on a genetic algorithm that optimizes the search for the best feature subset from a reduced set of features, output by the filter component. We conducted an extensive experimental evaluation of our method on five benchmark datasets and 17 classification algorithms for MIL. The results of our study show the potential of the proposed hybrid with respect to the desirable effect it produces: a significant improvement of the predictive performance of many MIL classification techniques as compared to the effect of filter-based feature selection. This is achieved due to the possibility to decide how many of the top ranked features are useful for each particular algorithm and the possibility to discard redundant attributes. [Copyright &y& Elsevier]

Abstract: This paper presents a new feature discovery approach called FEADIS that strengthens learning algorithms with discovered features. The discovered features are formed by various mathematical functions including ceil, mod, sin, and similar. These features are constructed in an iterative manner to improve gradually its learning performance. We demonstrate FEADIS capabilities by testing different types of datasets including periodical datasets. From the results, we conclude that FEADIS increases the performance of learning algorithms in a wide range of datasets for nominal or numeric target feature. Furthermore, most of the well known classifiers without FEADIS strengthening have severe difficulty in handling datasets that have periodical functional relations between input features and target feature – a difficulty circumvented by their potential use of FEADIS. [Copyright &y& Elsevier]

Abstract: In multi-label learning, the training set is made up of instances each associated with a set of labels, and the task is to predict the label sets of unseen instances. In this paper, this learning problem is addressed by using a method called Mlnb which adapts the traditional naive Bayes classifiers to deal with multi-label instances. Feature selection mechanisms are incorporated into Mlnb to improve its performance. Firstly, feature extraction techniques based on principal component analysis are applied to remove irrelevant and redundant features. After that, feature subset selection techniques based on genetic algorithms are used to choose the most appropriate subset of features for prediction. Experiments on synthetic and real-world data show that Mlnb achieves comparable performance to other well-established multi-label learning algorithms. [Copyright &y& Elsevier]

Abstract: Feature subset selection is viewed as an important preprocessing step for pattern recognition, machine learning and data mining. Most of researches are focused on dealing with homogeneous feature selection, namely, numerical or categorical features. In this paper, we introduce a neighborhood rough set model to deal with the problem of heterogeneous feature subset selection. As the classical rough set model can just be used to evaluate categorical features, we generalize this model with neighborhood relations and introduce a neighborhood rough set model. The proposed model will degrade to the classical one if we specify the size of neighborhood zero. The neighborhood model is used to reduce numerical and categorical features by assigning different thresholds for different kinds of attributes. In this model the sizes of the neighborhood lower and upper approximations of decisions reflect the discriminating capability of feature subsets. The size of lower approximation is computed as the dependency between decision and condition attributes. We use the neighborhood dependency to evaluate the significance of a subset of heterogeneous features and construct forward feature subset selection algorithms. The proposed algorithms are compared with some classical techniques. Experimental results show that the neighborhood model based method is more flexible to deal with heterogeneous data. [Copyright &y& Elsevier]