On Rank Selection in Non-Negative Matrix Factorization Using Concordance.

The choice of the factorization rank of a matrix is critical, e.g., in dimensionality reduction, filtering, clustering, deconvolution, etc., because selecting a rank that is too high amounts to adjusting the noise, while selecting a rank that is too low results in the oversimplification of the signal. Numerous methods for selecting the factorization rank of a non-negative matrix have been proposed. One of them is the cophenetic correlation coefficient ( c c c ), widely used in data science to evaluate the number of clusters in a hierarchical clustering. In previous work, it was shown that c c c performs better than other methods for rank selection in non-negative matrix factorization (NMF) when the underlying structure of the matrix consists of orthogonal clusters. In this article, we show that using the ratio of c c c to the approximation error significantly improves the accuracy of the rank selection. We also propose a new criterion, c o n c o r d a n c e , which, like c c c , benefits from the stochastic nature of NMF; its accuracy is also improved by using its ratio-to-error form. Using real and simulated data, we show that c o n c o r d a n c e , with a CUSUM-based automatic detection algorithm for its original or ratio-to-error forms, significantly outperforms c c c . It is important to note that the new criterion works for a broader class of matrices, where the underlying clusters are not assumed to be orthogonal. [ABSTRACT FROM AUTHOR]