Search

Linear discriminant analysis (LDA) aims to find a low-dimensional space in which data points in the same class are to be close to each other while keeping data points from different classes apart. To improve the robustness of LDA to non-Gaussian distribution data, most existing discriminant analysis methods extend LDA by approximating the underlying manifold of data. However, these methods suffer from the following problems: 1) local affinity or reconstruction coefficients are learned on the basis of the relationships of all data pairs, which would lead to a sharp increase in the amount of computation and 2) they learn the manifold information in the original space, ignoring the interference of the noise and redundant features. Motivated by these challenges, this article represents a novel discriminant analysis model, called fast and adaptive locality discriminant analysis (FALDA), to improve the efficiency and robustness. First, with the anchor-based strategy, a bipartite graph of each class is constructed to characterize the local structure of data. Since the number of anchor points is far less than that of data points, learning of fuzzy membership relationships between data points and anchor points within each class can save training time. Second, a maximum entropy regularization is introduced to control the uniformity of the weights of graphs and avoid the trivial solution. Third, the above relationships are updated adaptively in the process of dimensionality reduction, which can suppress the interference of the noise and redundant features. Fourth, the whitening constraint is imposed on the projection matrix to remove the relevance between features and restrict the total scatter of data in the subspace. Last but not the least, data with complex distribution can be explicitly divided into sub-blocks according to the learned anchor points (or subclass center points). We test our proposed method on synthetic data, benchmark datasets, and imbalanced datasets. Promising experimental results demonstrate the success of this novel model.

The recent work Unsupervised Linear Discriminant Analysis (Un-LDA) completes its clustering process during the alternating optimization by converting equivalently the objective and finally using the K-means algorithm. However, the K-means algorithm has its inherent drawbacks. It is hard for the K-means algorithm to deal well with some complex clustering cases where there are too many real clusters or non-convex clusters. In this paper, a novel clustering optimization method is presented to accomplish the clustering process in Un-LDA and the resulting method can be named Un-LDA(CD). Specifically, instead of the K-means algorithm, an elaborately designed coordinate descent algorithm is adopted to obtain the clusters after the objective function goes through a series of simple but deft equivalent conversions. Extensive experiments have demonstrated that the coordinate descent clustering solution for Un-LDA can outperform the original K-means based solution on the tested data sets especially those complex data sets with a pretty large number of real clusters.

In this article, we focus on utilizing the idea of co-clustering algorithms to address the subspace clustering problem. In recent years, co-clustering methods have been developed greatly with many important applications, such as document clustering and gene expression analysis. Different from the traditional graph-based methods, co-clustering can utilize the bipartite graph to extract the duality relationship between samples and features. It means that the bipartite graph can obtain more information than other traditional graph methods. Therefore, we proposed a novel method to handle the subspace clustering problem by combining dictionary learning with a bipartite graph under the constraint of the (normalized) Laplacian rank. Besides, to avoid the effect of redundant information hiding in the data, the original data matrix is not used as the static dictionary in our model. By updating the dictionary matrix under the sparse constraint, we can obtain a better coefficient matrix to construct the bipartite graph. Based on Theorem 2 and Lemma 1, we further speed up our algorithm. Experimental results on both synthetic and benchmark datasets demonstrate the superior effectiveness and stability of our model.

In the field of data mining, how to deal with high-dimensional data is a fundamental problem. If they are used directly, it is not only computationally expensive but also difficult to obtain satisfactory results. Unsupervised feature selection is designed to reduce the dimension of data by finding a subset of features in the absence of labels. Many unsupervised methods perform feature selection by exploring spectral analysis and manifold learning, such that the intrinsic structure of data can be preserved. However, most of these methods ignore a fact: due to the existence of noise features, the intrinsic structure directly built from original data may be unreliable. To solve this problem, a new unsupervised feature selection model is proposed. The graph structure, feature weights, and projection matrix are learned simultaneously, such that the intrinsic structure is constructed by the data that have been feature weighted and projected. For each data point, its nearest neighbors are acquired in the process of graph construction. Therefore, we call them adaptive neighbors. Besides, an additional constraint is added to the proposed model. It requires that a graph, corresponding to a similarity matrix, should contain exactly c connected components. Then, we present an optimization algorithm to solve the proposed model. Next, we discuss the method of determining the regularization parameter ɣ in our proposed method and analyze the computational complexity of the optimization algorithm. Finally, experiments are implemented on both synthetic and real-world datasets to demonstrate the effectiveness of the proposed method.

The goal of multi-view clustering is to partition samples into different subsets according to their diverse features. Previous multi-view clustering methods mainly exist two forms: multi-view spectral clustering and multi-view matrix factorization. Although they have shown excellent performance in many occasions, there are still many disadvantages. For example, multi-view spectral clustering usually needs to perform postprocessing. Multi-view matrix factorization directly decomposes the original data features. When the size of features is large, it encounters the expensive time consumption to decompose these data features thoroughly. Therefore, we proposed a novel multi-view clustering approach. The main advantages include the following three aspects: 1) it searches for a common joint graph across multiple views, which fully explores the hidden structure information by utilizing the compatibility among views; 2) the introduced nonnegative constraint manipulates that the final clustering results can be directly obtained; and 3) straightforwardly decomposing the similarity matrix can transform the eigenvalue factorization in spectral clustering with computational complexity O(n³) into the singular value decomposition (SVD) with O(nc²) time cost, where n and c, respectively, denote the numbers of samples and classes. Thus, the computational efficiency can be improved. Moreover, in order to learn a better clustering model, we set that the constructed similarity graph approximates each view affinity graph as close as possible by adding the constraint as the initial affinity matrices own. Furthermore, substantial experiments are conducted, which verifies the superiority of the proposed two clustering methods comparing with single-view clustering approaches and state-of-the-art multi-view clustering methods.

Finding a consensus embedding from multiple views is the mainstream task in multiview graph-based clustering, in which the key problem is to handle the inconsistence among multiple views. In this article, we consider clustering effectiveness and practical applicability collectively, and propose a parameter-free model to alleviate the inconsistence of multiple views cleverly. To be specific, the proposed model considers the diversities of multiple views as two-layers. The first layer considers the inconsistence among different features of each view and the second layer considers linking the preembeddings of multiple views attentively. By this way, a consensus embedding can be learned via kernel method effectively and the whole learning procedure is parameter-free. To solve the optimization problem involved in the proposed model, we propose an alternative algorithm which is efficient and easy to implement in practice. In the experiments, we evaluate the proposed model on synthetic and real datasets and the experimental results demonstrate its effectiveness.

Dimension reduction is a critical technology for high-dimensional data processing, where Linear Discriminant Analysis (LDA) and its variants are effective supervised methods. However, LDA prefers to feature with smaller variance, which causes feature with weak discriminative ability retained. In this paper, we propose a novel Ratio Sum for Linear Discriminant Analysis (RSLDA), which aims at maximizing discriminative ability of each feature in subspace. To be specific, it maximizes the sum of ratio of the between-class distance to the within-class distance in each dimension of subspace. Since the original RSLDA problem is difficult to obtain the closed solution, an equivalent problem is developed which can be solved by an alternative optimization algorithm. For solving the equivalent problem, it is transformed into two sub-problems, one of which can be solved directly, the other is changed into a convex optimization problem, where singular value decomposition is employed instead of matrix inversion. Consequently, performance of algorithm cannot be affected by the non-singularity of covariance matrix. Furthermore, Kernel RSLDA (KRSLDA) is presented to improve the robustness of RSLDA. Additionally, time complexity of RSLDA and KRSLDA are analyzed. Extensive experiments show that RSLDA and KRSLDA outperforms other comparison methods on toy datasets and multiple public datasets.

Feature selection has attracted a lot of attention in obtaining discriminative and non-redundant features from high-dimension data. Compared with traditional filter and wrapper methods, embedded methods can obtain a more informative feature subset by fully considering the importance of features in the classification tasks. However, the existing embedded methods emphasize the above importance of features and mostly ignore the correlation between the features, which leads to retain the correlated and redundant features with similar scores in the feature subset. To solve the problem, we propose a novel supervised embedded feature selection framework, called feature selection under global redundancy minimization in orthogonal regression (GRMOR). The proposed framework can effectively recognize redundant features from a global view of redundancy among the features. We also incorporate the large margin constraint into GRMOR for robust multi-class classification. Compared with the traditional embedded methods based on least square regression, the proposed framework utilizes orthogonal regression to preserve more discriminative information in the subspace, which can help accurately rank the importance of features in the classification tasks. Experimental results on twelve public datasets demonstrate that the proposed framework can obtain superior classification performance and redundancy removal performance than twelve other feature selection methods.

Matrix completion, in essence, involves recovering a low-rank matrix from a subset of its entries. Most existing methods for matrix completion neglect two significant issues. First, in several practical applications, such as collaborative filtering, some samples may be corrupted completely. However, most of the robust algorithms consider only the condition that a few components of each column have been arbitrarily contaminated. Second, many real data are not static in nature. Nevertheless, the conventional batch-based matrix completion methods cannot efficiently deal with the out-of-sample, that is, the vector completion problem. In this article, we first provide a novel robust matrix completion model and then develop an efficient optimization method that only requires conducting one time singular value decomposition for a thin matrix per iteration. Furthermore, by exploiting the essence of online matrix completion algorithms, we develop a vector completion model which can help users predict the missing values of out of sample. Numerical comparisons with traditional batch-based and online matrix completion algorithms demonstrate the benefits of the proposed method on streaming data corrupted by column outliers. Moreover, we show that our model can be used to detect outliers from incomplete information. This advantage is validated via numerous experimental results on synthetic and real-world data.

High-dimensional data are highly correlative and redundant, making it difficult to explore and analyze. Amount of unsupervised dimensionality reduction (DR) methods has been proposed, in which constructing a neighborhood graph is the primary step of DR methods. However, there exist two problems: 1) the construction of graph is usually separate from the selection of projection direction and 2) the original data are inevitably noisy. In this article, we propose an unsupervised adaptive embedding (UAE) method for DR to solve these challenges, which is a linear graph-embedding method. First, an adaptive allocation method of neighbors is proposed to construct the affinity graph. Second, the construction of affinity graph and calculation of projection matrix are integrated together. It considers the local relationship between samples and global characteristic of high-dimensional data, in which the cleaned data matrix is originally proposed to remove noise in subspace. The relationship between our method and local preserving projections (LPPs) is also explored. Finally, an alternative iteration optimization algorithm is derived to solve our model, the convergence and computational complexity of which are also analyzed. Comprehensive experiments on synthetic and benchmark datasets illustrate the superiority of our method.

Inspired by the mean calculation of RPCA_OM and inductiveness of IRPCA, we first propose an inductive robust principal component analysis method with removing the optimal mean automatically, which is shorted as IRPCA_OM. Furthermore, IRPCA_OM is extended to Schatten-p norm and a more general framework (i.e., EIRPCA_OM) is presented. The objective function of EIRPCA_OM includes two terms, the first term is a robust reconstruction error term constrained by an l2,1-norm and the second term is a regularization term constrained by a Schatten-p norm. The proposed EIRPCA_OM method is robust, inductive and accurate. However, on the high-dimensional data, it would spend a large computation cost in training stage. To this end, a fast version of EIRPCA_OM called as FEIRPCA_OM is proposed, and its basic idea is to eliminate the zero eigenvalues of data matrix. More importantly, an effective theoretical proof is presented to ensure that FEIRPCA_OM has faster processing speed than EIRPCA_OM when processing high-dimensional data, but without any performance loss. Based on it, we also can exchange the less performance loss for the higher computation efficiency by removing the small eigenvalues of data matrix. Experimental results on the public datasets demonstrate that FEIRPCA_OM works efficiently on the high-dimensional data.

Semi-supervised learning as one of most attractive problems in machine learning research field has aroused broad attentions in recent years. In this paper, we propose a novel locality preserved dimensionality reduction framework, named Semi-supervised Adaptive Local Embedding learning (SALE), which learns a local discriminative embedding by constructing a K1 Nearest Neighbors (K1NN) graph on labeled data, so as to explore the intrinsic structure, i.e., sub-manifolds from non-Gaussian labeled data. Then, mapping all samples into learned embedding and constructing another K2NN graph on all embedded data to explore the global structure of all samples. Therefore, the unlabeled data and their corresponding labeled neighbors can be clustered into same sub-manifold, such that the discriminative power of embedded data is improved. Furthermore, we propose two semi-supervised dimensionality reduction methods with orthogonal and whitening constraints based on proposed SALE framework. An efficient alternatively iterative optimization algorithm is developed to solve the NP-hard problem in our models. Extensive experiments conducted on several synthetic and real-world data sets demonstrate the superiorities of our methods on local structure exploration and classification task.

Benefit from avoiding the utilization of labeled samples, which are usually insufficient in the real world, unsupervised learning has been regarded as a speedy and powerful strategy on clustering tasks. However, clustering directly from primal data sets leads to high computational cost, which limits its application on large-scale and high-dimensional problems. Recently, anchor-based theories are proposed to partly mitigate this problem and field naturally sparse affinity matrix, while it is still a challenge to get excellent performance along with high efficiency. To dispose of this issue, we first presented a fast semisupervised framework (FSSF) combined with a balanced K -means-based hierarchical K -means (BKHK) method and the bipartite graph theory. Thereafter, we proposed a fast self-supervised clustering method involved in this crucial semisupervised framework, in which all labels are inferred from a constructed bipartite graph with exactly k connected components. The proposed method remarkably accelerates the general semisupervised learning through the anchor and consists of four significant parts: 1) obtaining the anchor set as interim through BKHK algorithm; 2) constructing the bipartite graph; 3) solving the self-supervised problem to construct a typical probability model with FSSF; and 4) selecting the most representative points regarding anchors from BKHK as an interim and conducting label propagation. The experimental results on toy examples and benchmark data sets have demonstrated that the proposed method outperforms other approaches.

Dimensionality reduction (DR) technique has been frequently used to alleviate information redundancy and reduce computational complexity. Traditional DR methods generally are inability to deal with nonlinear data and have high computational complexity. To cope with the problems, we propose a fast unsupervised projection (FUP) method. The simplified graph of FUP is constructed by samples and representative points, where the number of the representative points selected through iterative optimization is less than that of samples. By generating the presented graph, it is proved that large-scale data can be projected faster in numerous scenarios. Thereafter, the orthogonality FUP (OFUP) method is proposed to ensure the orthogonality of projection matrix. Specifically, the OFUP method is proved to be equivalent to PCA upon certain parameter setting. Experimental results on benchmark data sets show the effectiveness in retaining the essential information.

In big data time, selecting informative features has become an urgent need. However, due to the huge cost of obtaining enough labeled data for supervised tasks, researchers have turned their attention to semisupervised learning, which exploits both labeled and unlabeled data. In this article, we propose a sparse discriminative semisupervised feature selection (SDSSFS) method. In this method, the e-dragging technique for the supervised task is extended to the semisupervised task, which is used to enlarge the distance between classes in order to obtain a discriminative solution. The flexible l2,p norm is implicitly used as regularization in the new model. Therefore, we can obtain a more sparse solution by setting smaller p. An iterative method is proposed to simultaneously learn the regression coefficients and e-dragging matrix and predicting the unknown class labels. Experimental results on ten real-world datasets show the superiority of our proposed method.

Linear discriminant analysis (LDA) is a well-known technique for supervised dimensionality reduction and has been extensively applied in many real-world applications. LDA assumes that the samples are Gaussian distributed, and the local data distribution is consistent with the global distribution. However, real-world data seldom satisfy this assumption. To handle the data with complex distributions, some methods emphasize the local geometrical structure and perform discriminant analysis between neighbors. But the neighboring relationship tends to be affected by the noise in the input space. In this research, we propose a new supervised dimensionality reduction method, namely, locality adaptive discriminant analysis (LADA). In order to directly process the data with matrix representation, such as images, the 2-D LADA (2DLADA) is also developed. The proposed methods have the following salient properties: 1) they find the principle projection directions without imposing any assumption on the data distribution; 2) they explore the data relationship in the desired subspace, which contains less noise; and 3) they find the local data relationship automatically without the efforts for tuning parameters. The performance of dimensionality reduction shows the superiorities of the proposed methods over the state of the art.