Search

Highlights • We proposed a weighted heterogeneous distance metric (WHDM). • We presented WHDM and Dempster–Shafer theory based k NN algorithm. • We proposed a multimodal perturbation method (RRSB) for k NN ensemble. • The effectiveness of our algorithms was shown on multiple UCI data sets and a KDD data set. Abstract In this paper, to address the issue that ensembling k -nearest neighbor (k NN) classifiers with resampling approaches cannot generate component classifiers with a large diversity, we consider ensembling k NN through a multimodal perturbation-based method. Since k NN is sensitive to the input attributes, we propose a weighted heterogeneous distance Metric (WHDM). By using a WHDM and evidence theory, a progressive k NN classifier is developed. Based on a progressive k NN, the random subspace method, attribute reduction, and Bagging, a novel algorithm termed RRSB (reduced random subspace-based Bagging) is proposed for construct ensemble classifier, which can increase the diversity of component classifiers without damaging the accuracy of the component classifiers. In detail, RRSB adopts the perturbation on the learning parameter with a weighted heterogeneous distance metric, the perturbation on the input space with random subspace and attribute reduction, the perturbation on the training data with Bagging, and the perturbation on the output target of k neighbors with evidence theory. In the experimental stage, the value of k , the different perturbations on RRSB and the ensemble size are analyzed. In addition, RRSB is compared with other multimodal perturbation-based ensemble algorithms on multiple UCI data sets and a KDD data set. The results from the experiments demonstrate the effectiveness of RRSB for k NN ensembling. [ABSTRACT FROM AUTHOR]

In this paper, a new random subspace based ensemble sparse representation (RS_ESR) algorithm is proposed, where the random subspace is introduced into sparse representation model. For high-dimensional data, the random subspace method can not only reduce dimension of data but also make full use of effective information of data. It is not like traditional dimensionality reduction methods that may lose some information of original data. Additionally, a joint sparse representation model is emloyed to obtain the sparse representation of a sample set in the low dimensional random subspace. Then the sparse representations in multiple random subspaces are integrated as an ensemble sparse representation. Moreover, the obtained RS_ESR is applied in classical clustering and semi-supervised classification. The experimental results on different real-world data sets show the superiority of RS_ESR over traditional methods. [ABSTRACT FROM AUTHOR]

There are not many classifier ensemble approaches which investigate the data sample space and the feature space at the same time, and this multi-pronged approach will be helpful for constructing more powerful learning models. For example, the AdaBoost approach only investigates the data sample space, while the random subspace technique only focuses on the feature space. To address this limitation, we propose the progressive subspace ensemble learning approach (PSEL) which takes into account the data sample space and the feature space at the same time. Specifically, PSEL first adopts the random subspace technique to generate a set of subspaces. Then, a progressive selection process based on new cost functions that incorporate current and long-term information to select the classifiers sequentially will be introduced. Finally, a weighted voting scheme is used to summarize the predicted labels and obtain the final result. We also adopt a number of non-parametric tests to compare PSEL and its competitors over multiple datasets. The results of the experiments show that PSEL works well on most of the real datasets, and outperforms a number of state-of-the-art classifier ensemble approaches. [ABSTRACT FROM AUTHOR]

Abstract: Intuitive and efficient, the random subspace ensemble approach provides an appealing solution to the problem of the vast dimensionality of functional magnetic resonance imaging (fMRI) data for maximal-accuracy brain state decoding. Recently, efforts to generate biologically plausible and interpretable maps of brain regions which contribute information to the ensemble decoding task have been made and two approaches have been introduced: globally multivariate random subsampling and locally multivariate Monte Carlo mapping. Both types of maps reflect voxel-wise decoding accuracies averaged across repeatedly randomly sampled voxel subsets, highlighting voxels which consistently participate in high-classification subsets. We compare the mapping sensitivities of the approaches on realistic simulated data containing both locally and globally multivariate information and demonstrate that utilizing the inherent volumetric nature of fMRI through clustered Monte Carlo mapping yields dramatically improved performances in terms of voxel detection sensitivity and efficiency. These results suggest that, unless a priori information specifically dictates a global search, variants of clustered sampling should be the priority for random subspace brain mapping. [Copyright &y& Elsevier]

Abstract: Linear discriminant analysis (LDA) often suffers from the small sample size problem when dealing with high-dimensional face data. Random subspace can effectively solve this problem by random sampling on face features. However, it remains a problem how to construct an optimal random subspace for discriminant analysis and perform the most efficient discriminant analysis on the constructed random subspace. In this paper, we propose a novel framework, random discriminant analysis (RDA), to handle this problem. Under the most suitable situation of the principal subspace, the optimal reduced dimension of the face sample is discovered to construct a random subspace where all the discriminative information in the face space is distributed in the two principal subspaces of the within-class and between-class matrices. Then we apply Fisherface and direct LDA, respectively, to the two principal subspaces for simultaneous discriminant analysis. The two sets of discriminant analysis features from dual principal subspaces are first combined at the feature level, and then all the random subspaces are further integrated at the decision level. With the discriminating information fusion at the two levels, our method can take full advantage of useful discriminant information in the face space. Extensive experiments on different face databases demonstrate its performance. [Copyright &y& Elsevier]

• End-to-end trainable ensemble method in a single model. We implement the random subspace method using existing CNN layers and it achieves both higher accuracy and faster inference speed than conventional forest methods. We show that such designs are attractive for many real-world tasks dealing with vector inputs. • Efficient and effective non-linear transformation. NRS can be seamlessly installed at a CNN as a non-linear component. NRS is more efficient than previous higher-order pooling methods and achieves higher accuracy than standard GAP with negligible additional cost. We demonstrate its effectiveness on both 2D image and 3D object recognition tasks under various CNN architectures. • Good generalization ability and transferability. The proposed method achieves better performance on various tasks when compared with different algorithms. Our method is easy to implement and can be easily extended to other deep learning libraries and deployed to different devices. The random subspace method, also known as the pillar of random forests, is good at making precise and robust predictions. However, there is as yet no straightforward way to combine it with deep learning. In this paper, we therefore propose Neural Random Subspace (NRS), a novel deep learning based random subspace method. In contrast to previous forest methods, NRS enjoys the benefits of end-to-end, data-driven representation learning, as well as pervasive support from deep learning software and hardware platforms, hence achieving faster inference speed and higher accuracy. Furthermore, as a non-linear component to be encoded into Convolutional Neural Networks (CNNs), NRS learns non-linear feature representations in CNNs more efficiently than contemporary, higher-order pooling methods, producing excellent results with negligible increase in parameters, floating point operations (FLOPs) and real running time. Compared with random subspaces, random forests and gradient boosting decision trees (GBDTs), NRS demonstrates superior performance on 35 machine learning datasets. Moreover, on both 2D image and 3D point cloud recognition tasks, integration of NRS with CNN architectures achieves consistent improvements with only incremental cost. [ABSTRACT FROM AUTHOR]