Search

Sparse and outlier-robust Principal Component Analysis (PCA) has been a very active field of research recently. Yet, most existing methods apply PCA to a single dataset whereas multi-source data-i.e. multiple related datasets requiring joint analysis-arise across many scientific areas. We introduce a novel PCA methodology that simultaneously (i) selects important features, (ii) allows for the detection of global sparse patterns across multiple data sources as well as local source-specific patterns, and (iii) is resistant to outliers. To this end, we develop a regularization problem with a penalty that accommodates global-local structured sparsity patterns, and where the ssMRCD estimator is used as plug-in to permit joint outlier-robust analysis across multiple data sources. We provide an efficient implementation of our proposal via the Alternating Direction Method of Multiplier and illustrate its practical advantages in simulation and in applications.

Principal component analysis and factor analysis are fundamental multivariate analysis methods. In this paper a unified framework to connect them is introduced. Under a general latent variable model, we present matrix optimization problems from the viewpoint of loss function minimization, and show that the two methods can be viewed as solutions to the optimization problems with specific loss functions. Specifically, principal component analysis can be derived from a broad class of loss functions including the L2 norm, while factor analysis corresponds to a modified L0 norm problem. Related problems are discussed, including algorithms, penalized maximum likelihood estimation under the latent variable model, and a principal component factor model. These results can lead to new tools of data analysis and research topics., Comment: 24 pages, 2 figures

Principal Component Analysis and biplots are so well-established and readily implemented that it is just too tempting to give for granted their internal workings. In this note I get back to basics in comparing how PCA and biplots are implemented in base-R and contributed R packages, leveraging an implementation-agnostic understanding of the computational structure of each technique. I do so with a view to illustrating discrepancies that users might find elusive, as these arise from seemingly innocuous computational choices made under the hood. The proposed evaluation grid elevates aspects that are usually disregarded, including relationships that should hold if the computational rationale underpinning each technique is followed correctly. Strikingly, what is expected from these equivalences rarely follows without caveats from the output of specific implementations alone., Comment: Preprint. 20 pages, 3 figure, 5 tables. Currently submitted to the R Journal

Varimax factor rotations, while popular among practitioners in psychology and statistics since being introduced by H. Kaiser, have historically been viewed with skepticism and suspicion by some theoreticians and mathematical statisticians. Now, work by K. Rohe and M. Zeng provides new, fundamental insight: varimax rotations provably perform statistical estimation in certain classes of latent variable models when paired with spectral-based matrix truncations for dimensionality reduction. We build on this newfound understanding of varimax rotations by developing further connections to network analysis and spectral methods rooted in entrywise matrix perturbation analysis. Concretely, this paper establishes the asymptotic multivariate normality of vectors in varimax-transformed Euclidean point clouds that represent low-dimensional node embeddings in certain latent space random graph models. We address related concepts including network sparsity, data denoising, and the role of matrix rank in latent variable parameterizations. Collectively, these findings, at the confluence of classical and contemporary multivariate analysis, reinforce methodology and inference procedures grounded in matrix factorization-based techniques. Numerical examples illustrate our findings and supplement our discussion., Comment: 21 pages with bibliography and supplement

The present study investigates to what degree the common variance of the factor score predictor with the original factor, i.e., the determinacy coefficient or the validity of the factor score predictor, depends on the mean-difference between groups. When mean-differences between groups in the factor score predictor are eliminated by means of covariance analysis, regression, or group specific norms, this may reduce the covariance of the factor score predictor with the common factor. It is shown that in a one-factor model with the same group mean-difference on all observed variables, the common factor cannot be distinguished from a common factor representing the group mean-difference. It is also shown that for common factor loadings equal or larger than .60, the elimination of a d = .50 mean-difference between two groups in the factor score predictor leads to only small decreases of the determinacy coefficient. A compensation-factor k is proposed allowing for the estimation of the number of additional observed variables necessary to recover the size of the determinacy coefficient before elimination of a group mean-difference. It turns out that for factor loadings equal or larger than .60 only a few additional items are needed in order to recover the initial determinacy coefficient after the elimination of moderate or large group mean-differences.

This paper presents a novel approach to functional principal component analysis (FPCA) in Bayes spaces in the setting where densities are the object of analysis, but only few individual samples from each density are observed. We use the observed data directly to account for all sources of uncertainty, instead of relying on prior estimation of the underlying densities in a two-step approach, which can be inaccurate if small or heterogeneous numbers of samples per density are available. To account for the constrained nature of densities, we base our approach on Bayes spaces, which extend the Aitchison geometry for compositional data to density functions. For modeling, we exploit the isometric isomorphism between the Bayes space and the $\mathbb{L}^2$ subspace $\mathbb{L}_0^2$ with integration-to-zero constraint through the centered log-ratio transformation. As only discrete draws from each density are observed, we treat the underlying functional densities as latent variables within a maximum likelihood framework and employ a Monte Carlo Expectation Maximization (MCEM) algorithm for model estimation. Resulting estimates are useful for exploratory analyses of density data, for dimension reduction in subsequent analyses, as well as for improved preprocessing of sparsely sampled density data compared to existing methods. The proposed method is applied to analyze the distribution of maximum daily temperatures in Berlin during the summer months for the last 70 years, as well as the distribution of rental prices in the districts of Munich.

Introduction: Oblique Target-rotation in the context of exploratory factor analysis is a relevant method for the investigation of the oblique independent clusters model. It was argued that minimizing single cross-loadings by means of target rotation may lead to large effects of sampling error on the target rotated factor solutions. Method: In order to minimize effects of sampling error on results of Target-rotation we propose to compute the mean cross-loadings for each block of salient loadings of the independent clusters model and to perform target rotation for the block-wise mean cross-loadings. The resulting transformation-matrix is than applied to the complete unrotated loading matrix in order to produce mean Target-rotated factors. Results: A simulation study based on correlated independent factor models revealed that mean oblique Target-rotation resulted in smaller negative bias of factor inter-correlations than conventional Target-rotation based on single loadings, especially when sample size was small and when the number of factors was large. An empirical example revealed that the similarity of Target-rotated factors computed for small subsamples with Target-rotated factors of the total sample was more pronounced for mean Target-rotation than for conventional Target-rotation. Discussion: Mean Target-rotation can be recommended in the context of oblique independent factor models, especially for small samples. An R-script and an SPSS-script for this form of Target-rotation are provided in the Appendix.

The relevance of determinacy coefficients as indicators for the validity of factor score predictors has regularly been emphasized. Previous simulation studies revealed biased determinacy coefficients for factor score predictors based on categorical variables. Therefore, and because there are different possibilities to compute determinacy coefficients, the present study compared bias of determinacy coefficients for the best linear factor score predictor and for a correlation-preserving factor score predictor based on confirmatory factor models with observed variables with 2, 4, 6, and 8 categories and maximum likelihood estimation, diagonally weighted least squares estimation, and Bayesian estimation. Positive bias was found when data were based on variables with two categories, population factors were correlated, and when there were unmodeled cross-loadings. Based on the results, the correction for sampling error, the use of maximum likelihood or Bayesian parameters, and data with at least four categories are recommended to avoid overestimation of parameter-based determinacy coefficients.

Regression factor score predictors have the maximum factor score determinacy, i.e., the maximum correlation with the corresponding factor, but they do not have the same inter-correlations as the factors. As it might be useful to compute factor score predictors that have the same inter-correlations as the factors, correlation-preserving factor score predictors have been proposed. However, correlation-preserving factor score predictors have smaller correlations with the corresponding factors (factor score determinacy) than regression factor score predictors. Thus, higher factor score determinacy goes along with bias of the inter-correlations and unbiased inter-correlations go along with lower factor score determinacy. The aim of the present study was therefore to investigate the size of the trade-off between factor score determinacy and bias of inter-correlations by means of a simulation study. It turns out that under several conditions very small gains of factor score determinacy of the regression factor score predictor go along with a large bias of inter-correlations. Instead of using the regression factor score predictor by default, it is proposed to check whether substantial bias of inter-correlations can be avoided without substantial loss of factor score determinacy by using a correlation-preserving factor score predictor. A syntax that allows to compute correlation-preserving factor score predictors from regression factor score predictors and to compare factor score determinacy and inter-correlations of the factor score predictors is given in the Appendix.

We analyze the prediction error of principal component regression (PCR) and prove high probability bounds for the corresponding squared risk conditional on the design. Our first main result shows that PCR performs comparably to the oracle method obtained by replacing empirical principal components by their population counterparts, provided that an effective rank condition holds. On the other hand, if the latter condition is violated, then empirical eigenvalues start to have a significant upward bias, resulting in a self-induced regularization of PCR. Our approach relies on the behavior of empirical eigenvalues, empirical eigenvectors and the excess risk of principal component analysis in high-dimensional regimes., Comment: 25 pages. arXiv admin note: text overlap with arXiv:1811.02998

Mean plausible values can be computed when Bayesian structural equation modeling (BSEM) is performed. As mean plausible values do not preserve the inter-factor correlations, they yield path coefficients that are different from the estimated path coefficients of the model. As it might be of interest to perform exactly the same prediction on the level of plausible values that has been estimated by BSEM, correlation-preserving mean plausible values were proposed. An example for the computation of the correlation preserving mean plausible values is given and the corresponding syntax is given in the Appendix.

The Bayesian information criterion (BIC), defined as the observed data log likelihood minus a penalty term based on the sample size $N$, is a popular model selection criterion for factor analysis with complete data. This definition has also been suggested for incomplete data. However, the penalty term based on the `complete' sample size $N$ is the same no matter whether in a complete or incomplete data case. For incomplete data, there are often only $N_i

This study aims to find out the current status and prospects of using a bicycle as a mode for commuting within Dhaka city. Bicycling is a very sustainable mode of transport but unfortunately is used very less by the commuters of Dhaka. There has been a lot of factors affecting the choice of bicycle to commute. This study was aimed to find out what factors could motivate the commuters of Dhaka to use a bicycle as a mode of transport. For determining the motivators, a survey was administered among the commuters of Dhaka city in which the respondents were asked to answer how certain factors would affect their choice to use a bicycle to commute. A Likert scale was used in the survey and the responses were analyzed, from which the top motivators were found. The Motivators were then grouped together using exploratory factor analysis to support possible policy making. Four factors were extracted using the method. The factors were named Additional Perks, General benefits, Personal Benefits, and Infrastructural benefits, Comment: 6 Pages, 2 tables, Conference

Effects of performing R-factor analysis of observed variables based on population models comprising R- and Q-factors were investigated. It was noted that estimating a model comprising R- and Q-factors has to face loading indeterminacy beyond rotational indeterminacy. Although R-factor analysis of data based on a population model comprising R- and Q-factors is nevertheless possible, this may lead to model error. Accordingly, even in the population, the resulting R-factor loadings are not necessarily close estimates of the original population R-factor loadings. It was shown in a simulation study that large Q-factor variance induces an increase of the variation of R-factor loading estimates beyond chance level. The results indicate that performing R-factor analysis with data based on a population model comprising R- and Q-factors may result in substantial loading bias. Tests of the multivariate kurtosis of observed variables are proposed as an indicator of possible Q-factor variance in observed variables as a prerequisite for R-factor analysis.

Principal component analysis (PCA) is a popular dimension reduction technique for vector data. Factored PCA (FPCA) is a probabilistic extension of PCA for matrix data, which can substantially reduce the number of parameters in PCA while yield satisfactory performance. However, FPCA is based on the Gaussian assumption and thereby susceptible to outliers. Although the multivariate $t$ distribution as a robust modeling tool for vector data has a very long history, its application to matrix data is very limited. The main reason is that the dimension of the vectorized matrix data is often very high and the higher the dimension, the lower the breakdown point that measures the robustness. To solve the robustness problem suffered by FPCA and make it applicable to matrix data, in this paper we propose a robust extension of FPCA (RFPCA), which is built upon a $t$-type distribution called matrix-variate $t$ distribution. Like the multivariate $t$ distribution, the matrix-variate $t$ distribution can adaptively down-weight outliers and yield robust estimates. We develop a fast EM-type algorithm for parameter estimation. Experiments on synthetic and real-world datasets reveal that RFPCA is compared favorably with several related methods and RFPCA is a simple but powerful tool for matrix-valued outlier detection., Comment: 37 pages

In this paper, we examine the materiality of ESG on country creditworthiness from a credit risk and fundamental analysis viewpoint. We first determine the ESG indicators that are most relevant when it comes to explaining the sovereign bond yield, after controlling the effects of traditional fundamental variables such as economic strength and credit rating. We also emphasize the major themes that are directly useful for investors when assessing the country risk premium. At the global level, we notice that these themes mainly belong to the E and G pillars. Those results confirm that extra-financial criteria are integrated into bond pricing. However, we also identify a clear difference between high-and middle-income countries. Indeed, whereas the S pillar is lagging for the highest income countries, it is nearly as important as the G pillar for the middle-income ones. Second, we determine which ESG metrics are indirectly valuable for assessing a country's solvency. More precisely, we attempt to infer credit rating solely from extra-financial criteria, that is the ESG indicators that are priced in by credit rating agencies. We find that there is no overlap between the set of indicators that predict credit ratings and those that directly explain sovereign bond yields. The results also highlight the importance of the G and S pillars when predicting credit ratings. The E pillar is lagging, suggesting that credit rating agencies are undermining the impact of climate change and environmental topics on country creditworthiness. This is consistent with the traditional view that social and governance issues are the main drivers of the sovereign risk, because they are more specific and less global than environmental issues. Finally, taking these different results together, this research shows that opposing extra-financial and fundamental analysis does not make a lot of sense., Comment: 102 pages, 22 figures, 69 tables. This version has been removed by arXiv administrators due to a copyright claim by a 3rd party

In the context of Bayesian factor analysis, it is possible to compute mean plausible values, which might be used as covariates or predictors or in order to provide individual scores for the Bayesian latent variables. Previous simulation studies ascertained the validity of the plausible values by the mean squared difference of the plausible values and the generating factor scores. However, the generating factor scores are unknown in empirical studies so that an indicator that is solely based on model parameters is needed in order to evaluate the validity of factor score estimates in empirical studies. The coefficient of determinacy is based on model parameters and can be computed whenever Bayesian factor analysis is performed in empirical settings. Therefore, the central aim of the present simulation study was to compare the coefficient of determinacy based on model parameters with the correlation of mean plausible values with the generating factors. It was found that the coefficient of determinacy yields an acceptable estimate for the validity of mean plausible values. As for small sample sizes and a small salient loading size the coefficient of determinacy overestimates the validity, it is recommended to report the coefficient of determinacy together with a bias-correction in order to estimate the validity of mean plausible values in empirical settings.

The paper is devoted to the consequences of blind random selection of items from different item populations that might be based on completely uncorrelated factors for item inter-correlations and corresponding factor loadings. Based on the model of essentially parallel measurements, we explore the consequences of presenting items from different populations across individuals and items from identical populations within each individual for the factor model and item inter-correlations in the total population of individuals. Moreover, we explore the consequences of presenting items from different as well as identical item populations across and within individuals. We show that correlations can be substantial in the total population of individuals even when -- in subpopulations of individuals -- items are drawn from populations with uncorrelated factors. In order to address this challenge for the validity of a scale, we propose a method that helps to detect whether item inter-correlations result from different item populations in different subpopulations of individuals and evaluate the method by means of a simulation study. Based on the analytical results and on results from a simulation study, we provide recommendations for the detection of subpopulations of individuals responding to items from different item populations.

The availability of multi-omics data has revolutionized the life sciences by creating avenues for integrated system-level approaches. Data integration links the information across datasets to better understand the underlying biological processes. However, high-dimensionality, correlations and heterogeneity pose statistical and computational challenges. We propose a general framework, probabilistic two-way partial least squares (PO2PLS), which addresses these challenges. PO2PLS models the relationship between two datasets using joint and data-specific latent variables. For maximum likelihood estimation of the parameters, we implement a fast EM algorithm and show that the estimator is asymptotically normally distributed. A global test for testing the relationship between two datasets is proposed, and its asymptotic distribution is derived. Notably, several existing omics integration methods are special cases of PO2PLS. Via extensive simulations, we show that PO2PLS performs better than alternatives in feature selection and prediction performance. In addition, the asymptotic distribution appears to hold when the sample size is sufficiently large. We illustrate PO2PLS with two examples from commonly used study designs: a large population cohort and a small case-control study. Besides recovering known relationships, PO2PLS also identified novel findings. The methods are implemented in our R-package PO2PLS. Supplementary materials for this article are available online., Comment: 36 pages, 4 figures, Submitted to Journal of the American Statistical Association

Multiple Indicators Multiple Causes (MIMIC) models are type of structural equation models, a theory-based approach to confirm the influence of a set of exogenous causal variables on the latent variable, and also the effect of the latent variable on observed indicator variables. In a common MIMIC model, multiple indicators reflect the underlying latent variables/factors, and the multiple causes (observed predictors) affect latent variables/factors. Basic assumptions of MIMIC are clearly violated in case of a variable being both an indicator and a cause, i.e. in the presence of reverse causality. Furthermore, the model is then unidentified. To resolve the situation, which can arise frequently, and as MIMIC estimation lacks closed form solutions for parameters we utilize a version of Bollen's (1996) 2SLS estimator for structural equation models combined with J\"oreskog (1970)'s method of the analysis of covariance structures to derive a new, 2SLS estimator for MIMIC models. Our 2SLS empirical estimation is based on static MIMIC specification but we point also to dynamic/error-correction MIMIC specification and 2SLS solution for it. We derive basic asymptotic theory for static 2SLS-MIMIC, present a simulation study and apply findings to an interesting empirical case of estimating precarious status of older workers (using dataset of Survey of Health, Ageing and Retirement in Europe) which solves an important issue of the definition of precarious work as a multidimensional concept, not modelled adequately so far., Comment: 21 pages, 3 figures

Score Predictor Factor Analysis (SPFA) was introduced as a method that allows to compute factor score predictors that are -- under some conditions -- more highly correlated with the common factors resulting from factor analysis than the factor score predictors computed from the common factor model. In the present study, we investigate SPFA as a model in its own rights. In order to provide a basis for this, the properties and the utility of SPFA factor score predictors and the possibility to identify single-item indicators in SPFA loading matrices were investigated. Regarding the factor score predictors, the main result is that the best linear predictor of the score predictor factor analysis has not only perfect determinacy but is also correlation preserving. Regarding the SPFA loadings it was found in a simulation study that five or more population factors that are represented by only one variable with a rather substantial loading can more accurately be identified by means of SPFA than with conventional factor analysis. Moreover, the percentage of correctly identified single-item indicators was substantially larger for SPFA than for the common factor model. It is therefore argued that SPFA is a tool that can be especially helpful when very short scales or single-item indicators are to be identified.

NMR spectral datasets, especially in systems with limited samples, can be difficult to interpret if they contain multiple chemical components (phases, polymorphs, molecules, crystals, glasses, etc...) and the possibility of overlapping resonances. In this paper, we benchmark several blind source separation techniques for analysis of NMR spectral datasets containing negative intensity. For benchmarking purposes, we generated a large synthetic datasbase of quadrupolar solid-state NMR-like spectra that model spin-lattice T1 relaxation or nutation tip/flip angle experiments. Our benchmarking approach focused exclusively on the ability of blind source separation techniques to reproduce the spectra of the underlying pure components. In general, we find that FastICA (Fast Independent Component Analysis), SIMPLISMA (SIMPLe-to-use-Interactive Self-modeling Mixture Analysis), and NNMF (Non-Negative Matrix Factorization) are top-performing techniques. We demonstrate that dataset normalization approaches prior to blind source separation do not considerably improve outcomes. Within the range of noise levels studied, we did not find drastic changes to the ranking of techniques. The accuracy of FastICA and SIMPLISMA degrades quickly if excess (unreal) pure components are predicted. Our results indicate poor performance of SVD (Singular Value Decomposition) methods, and we propose alternative techniques for matrix initialization. The benchmarked techniques are also applied to real solid state NMR datasets. In general, the recommendations from the synthetic datasets agree with the recommendations and results from the real data analysis. The discussion provides some additional recommendations for spectroscopists applying blind source separation to NMR datasets, and for future benchmark studies., Comment: 28 pages, 6 figures, 5 tables

Probability density functions (PDFs) can be understood as continuous compositions by the theory of Bayes spaces. The origin of a Bayes space is determined by a given reference measure. This can be easily changed through the well-known chain rule which has an impact on the geometry of the Bayes space. This work provides a mathematical framework for setting a reference measure. It is used to develop a weighting scheme on the bounded domain of distributional data. The impact on statistical analysis is shown from the perspective of simplicial functional principal component analysis. Moreover, a novel centered log-ratio transformation is proposed to map a weighted Bayes spaces into an unweighted $L^2$ space, enabling to use most tools developed in functional data analysis (e.g. clustering, regression analysis, etc.) while accounting for the weighting strategy. The potential of our proposal is shown through simulation and on a real case study using Italian income data., Comment: 28 pages, 10 figures

We derive high-probability bounds for the reconstruction error of PCA in infinite dimensions. We apply our bounds in the case that the eigenvalues of the covariance operator satisfy polynomial or exponential upper bounds., Comment: 10 pages, to appear in Statistics & Probability Letters

We study the problem of high-dimensional Principal Component Analysis (PCA) with missing observations. In simple, homogeneous missingness settings with a noise level of constant order, we show that an existing inverse-probability weighted (IPW) estimator of the leading principal components can (nearly) attain the minimax optimal rate of convergence. However, deeper investigation reveals both that, particularly in more realistic settings where the missingness mechanism is heterogeneous, the empirical performance of the IPW estimator can be unsatisfactory, and moreover that, in the noiseless case, it fails to provide exact recovery of the principal components. Our main contribution, then, is to introduce a new method for high-dimensional PCA, called `primePCA', that is designed to cope with situations where observations may be missing in a heterogeneous manner. Starting from the IPW estimator, primePCA iteratively projects the observed entries of the data matrix onto the column space of our current estimate to impute the missing entries, and then updates our estimate by computing the leading right singular space of the imputed data matrix. It turns out that the interaction between the heterogeneity of missingness and the low-dimensional structure is crucial in determining the feasibility of the problem. We therefore introduce an incoherence condition on the principal components and prove that in the noiseless case, the error of primePCA converges to zero at a geometric rate when the signal strength is not too small. An important feature of our theoretical guarantees is that they depend on average, as opposed to worst-case, properties of the missingness mechanism. Our numerical studies on both simulated and real data reveal that primePCA exhibits very encouraging performance across a wide range of scenarios., Comment: 42 pages, 4 figures

The model implied by factor score predictors does not reproduce the non-diagonal elements of the observed covariance matrix as well as the factor loadings. It is therefore investigated whether it is possible to estimate factor loadings for which the model implied by the factor score predictors optimally reproduces the non-diagonal elements of the observed covariance matrix. Accordingly, loading estimates are proposed for which the model implied by the factor score predictors allows for a least-squares approximation of the non-diagonal elements of the observed covariance matrix. This estimation method is termed Score predictor factor analysis and algebraically compared with Minres factor analysis as well as principal component analysis. A population based and a sample based simulation study was performed in order to compare Score predictor factor analysis, Minres factor analysis, and principal component analysis. It turns out that the non-diagonal elements of the observed covariance matrix can more exactly be reproduced from the factor score predictors computed from Score predictor factor analysis than from the factor score predictors computed from Minres factor analysis and from principal components. Moreover, Score predictor factor analysis can be helpful to identify factors when the factor model does not perfectly fit to the data because of model error.

This study was conducted to determine the structures of a set of n correlated variables and creates a new set of uncorrelated indices which are the underlying components of the Philippine health data.The data utilized in this study was the 2009 Philippine Health Data which was made available by National Statistical Coordination Board(NSCB) in its 2009 publication.The publication contains the health data of 81 provinces of the Philippines consisting of ten system-related determinants which was considered as the variables in this study. From the ten health system-related determinants, it was found out that there are three significant underlying components that could summarize the Philippine health data. The first component was named as importance of safe water supply and emphasis on child heat while the second and third component were named as importance of Barangay Health Stations, government health workers and emphasis on pregnant women's health and emphasis on women's health, respectively. These three components jointly account for a total of 73.01% of the total variance explained by the component.

The MIXANDMIX (mixtures by Anderson mixing) tool for the computation of the empirical spectral distribution of random matrices generated by mixtures of populations is described. Within the population mixture model the mapping between the population distributions and the limiting spectral distribution can be obtained by solving a set of systems of non-linear equations, for which an efficient implementation is provided. The contributions include a method for accelerated fixed point convergence, a homotopy continuation strategy to prevent convergence to non-admissible solutions, a blind non-uniform grid construction for effective distribution support detection and approximation, and a parallel computing architecture. Comparisons are performed with available packages for the single population case and with results obtained by simulation for the more general model implemented here. Results show competitive performance and improved flexibility., Comment: 17 pages, 6 figures

We analyse the prediction error of principal component regression (PCR) and prove non-asymptotic upper bounds for the corresponding squared risk. Under mild assumptions, we show that PCR performs as well as the oracle method obtained by replacing empirical principal components by their population counterparts. Our approach relies on upper bounds for the excess risk of principal component analysis., Comment: 24 pages

Modern data are increasingly both high-dimensional and heteroscedastic. This paper considers the challenge of estimating underlying principal components from high-dimensional data with noise that is heteroscedastic across samples, i.e., some samples are noisier than others. Such heteroscedasticity naturally arises, e.g., when combining data from diverse sources or sensors. A natural way to account for this heteroscedasticity is to give noisier blocks of samples less weight in PCA by using the leading eigenvectors of a weighted sample covariance matrix. We consider the problem of choosing weights to optimally recover the underlying components. In general, one cannot know these optimal weights since they depend on the underlying components we seek to estimate. However, we show that under some natural statistical assumptions the optimal weights converge to a simple function of the signal and noise variances for high-dimensional data. Surprisingly, the optimal weights are not the inverse noise variance weights commonly used in practice. We demonstrate the theoretical results through numerical simulations and comparisons with existing weighting schemes. Finally, we briefly discuss how estimated signal and noise variances can be used when the true variances are unknown, and we illustrate the optimal weights on real data from astronomy., Comment: 39 pages, 9 figures

This article proposes a new approach to modeling high-dimensional time series by treating a $p$-dimensional time series as a nonsingular linear transformation of certain common factors and idiosyncratic components. Unlike the approximate factor models, we assume that the factors capture all the non-trivial dynamics of the data, but the cross-sectional dependence may be explained by both the factors and the idiosyncratic components. Under the proposed model, (a) the factor process is dynamically dependent and the idiosyncratic component is a white noise process, and (b) the largest eigenvalues of the covariance matrix of the idiosyncratic components may diverge to infinity as the dimension $p$ increases. We propose a white noise testing procedure for high-dimensional time series to determine the number of white noise components and, hence, the number of common factors, and introduce a projected Principal Component Analysis (PCA) to eliminate the diverging effect of the idiosyncratic noises. Asymptotic properties of the proposed method are established for both fixed $p$ and diverging $p$ as the sample size $n$ increases to infinity. We use both simulated data and real examples to assess the performance of the proposed method. We also compare our method with two commonly used methods in the literature concerning the forecastability of the extracted factors and find that the proposed approach not only provides interpretable results, but also performs well in out-of-sample forecasting. Supplementary materials of the article are available online., Comment: 37 pages, 3 figures

This paper considers a structural-factor approach to modeling high-dimensional time series and space-time data by decomposing individual series into trend, seasonal, and irregular components. For ease in analyzing many time series, we employ a time polynomial for the trend, a linear combination of trigonometric series for the seasonal component, and a new factor model for the irregular components. The new factor model can simplify the modeling process and achieve parsimony in parameterization. We propose a Bayesian Information Criterion (BIC) to consistently determine the order of the polynomial trend and the number of trigonometric functions. A test statistic is used to determine the number of common factors. The convergence rates for the estimators of the trend and seasonal components and the limiting distribution of the test statistic are established under the setting that the number of time series tends to infinity with the sample size, but at a slower rate. We use simulation to study the performance of the proposed analysis in finite samples and apply the proposed approach to two real examples. The first example considers modeling weekly PM$_{2.5}$ data of 15 monitoring stations in the southern region of Taiwan and the second example consists of monthly value-weighted returns of 12 industrial portfolios., Comment: 29 pages, 10 figures. arXiv admin note: text overlap with arXiv:1310.1990 by other authors

The allocation of a (treatment) condition-effect on the wrong principal component (misallocation of variance) in principal component analysis (PCA) has been addressed in research on event-related potentials of the electroencephalogram. However, the correct allocation of condition-effects on PCA components might be relevant in several domains of research. The present paper investigates whether different loading patterns at each condition-level are a basis for an optimal allocation of between-condition variance on principal components. It turns out that a similar loading shape at each condition-level is a necessary condition for an optimal allocation of between-condition variance, whereas a similar loading magnitude is not necessary.

Deep latent variable models (DLVMs) combine the approximation abilities of deep neural networks and the statistical foundations of generative models. Variational methods are commonly used for inference; however, the exact likelihood of these models has been largely overlooked. The purpose of this work is to study the general properties of this quantity and to show how they can be leveraged in practice. We focus on important inferential problems that rely on the likelihood: estimation and missing data imputation. First, we investigate maximum likelihood estimation for DLVMs: in particular, we show that most unconstrained models used for continuous data have an unbounded likelihood function. This problematic behaviour is demonstrated to be a source of mode collapse. We also show how to ensure the existence of maximum likelihood estimates, and draw useful connections with nonparametric mixture models. Finally, we describe an algorithm for missing data imputation using the exact conditional likelihood of a deep latent variable model. On several data sets, our algorithm consistently and significantly outperforms the usual imputation scheme used for DLVMs.

We introduce a new method for sparse principal component analysis, based on the aggregation of eigenvector information from carefully-selected axis-aligned random projections of the sample covariance matrix. Unlike most alternative approaches, our algorithm is non-iterative, so is not vulnerable to a bad choice of initialisation. We provide theoretical guarantees under which our principal subspace estimator can attain the minimax optimal rate of convergence in polynomial time. In addition, our theory provides a more refined understanding of the statistical and computational trade-off in the problem of sparse principal component estimation, revealing a subtle interplay between the effective sample size and the number of random projections that are required to achieve the minimax optimal rate. Numerical studies provide further insight into the procedure and confirm its highly competitive finite-sample performance., Comment: 32 pages

Recruitment market analysis provides valuable understanding of industry-specific economic growth and plays an important role for both employers and job seekers. With the rapid development of online recruitment services, massive recruitment data have been accumulated and enable a new paradigm for recruitment market analysis. However, traditional methods for recruitment market analysis largely rely on the knowledge of domain experts and classic statistical models, which are usually too general to model large-scale dynamic recruitment data, and have difficulties to capture the fine-grained market trends. To this end, in this paper, we propose a new research paradigm for recruitment market analysis by leveraging unsupervised learning techniques for automatically discovering recruitment market trends based on large-scale recruitment data. Specifically, we develop a novel sequential latent variable model, named MTLVM, which is designed for capturing the sequential dependencies of corporate recruitment states and is able to automatically learn the latent recruitment topics within a Bayesian generative framework. In particular, to capture the variability of recruitment topics over time, we design hierarchical dirichlet processes for MTLVM. These processes allow to dynamically generate the evolving recruitment topics. Finally, we implement a prototype system to empirically evaluate our approach based on real-world recruitment data in China. Indeed, by visualizing the results from MTLVM, we can successfully reveal many interesting findings, such as the popularity of LBS related jobs reached the peak in the 2nd half of 2014, and decreased in 2015., Comment: 11 pages, 30 figure, SIGKDD 2016

Individual scores on common factors are required in some applied settings (e.g., business and marketing settings). Common factors are based on reflective indicators, but their scores cannot unambiguously be determined. Therefore, factor score estimates and unit-weighted scales are used in order to provide individual scores. It is shown that these scores are based on treating the reflective indicators as if they were causal-formative indicators. This modification of the causal status of the indicators should be justified. Therefore, the fit of the models implied by factor score estimates and unit-weighted scales should be investigated in order to ascertain the validity of the scores.

We discuss the problem of estimating the number of principal components in Principal Com- ponents Analysis (PCA). Despite of the importance of the problem and the multitude of solutions proposed in the literature, it comes as a surprise that there does not exist a coherent asymptotic framework which would justify different approaches depending on the actual size of the data set. In this paper we address this issue by presenting an approximate Bayesian approach based on Laplace approximation and introducing a general method for building the model selection criteria, called PEnalized SEmi-integrated Likelihood (PESEL). Our general framework encompasses a variety of existing approaches based on probabilistic models, like e.g. Bayesian Information Criterion for the Probabilistic PCA (PPCA), and allows for construction of new criteria, depending on the size of the data set at hand. Specifically, we define PESEL when the number of variables substantially exceeds the number of observations. We also report results of extensive simulation studies and real data analysis, which illustrate good properties of our proposed criteria as compared to the state-of- the-art methods and very recent proposals. Specifially, these simulations show that PESEL based criteria can be quite robust against deviations from the probabilistic model assumptions. Selected PESEL based criteria for the estimation of the number of principal components are implemented in R package varclust, which is available on github (https://github.com/psobczyk/varclust)., Comment: 31 pages, 7 figures

We consider the problem of accurately and efficiently querying a remote server to retrieve information about images captured by a mobile device. In addition to reduced transmission overhead and computational complexity, the retrieval protocol should be robust to variations in the image acquisition process, such as translation, rotation, scaling, and sensor-related differences. We propose to extract scale-invariant image features and then perform clustering to reduce the number of features needed for image matching. Principal Component Analysis (PCA) and Non-negative Matrix Factorization (NMF) are investigated as candidate clustering approaches. The image matching complexity at the database server is quadratic in the (small) number of clusters, not in the (very large) number of image features. We employ an image-dependent information content metric to approximate the model order, i.e., the number of clusters, needed for accurate matching, which is preferable to setting the model order using trial and error. We show how to combine the hypotheses provided by PCA and NMF factor loadings, thereby obtaining more accurate retrieval than using either approach alone. In experiments on a database of urban images, we obtain a top-1 retrieval accuracy of 89% and a top-3 accuracy of 92.5%., Comment: 6 Pages, 7 figures, ICMEWorkshop 2016

Monitoring multichannel profiles has important applications in manufacturing systems improvement, but it is non-trivial to develop efficient statistical methods due to two main challenges. First, profiles are high-dimensional functional data with intrinsic inner- and inter-channel correlations, and one needs to develop a dimension reduction method that can deal with such intricate correlations for the purpose of effective monitoring. The second, and probably more fundamental, challenge is that the functional structure of multi-channel profiles might change over time, and thus the dimension reduction method should be able to automatically take into account the potential unknown change. To tackle these two challenges, we propose a novel thresholded multivariate principal component analysis (PCA) method for multi-channel profile monitoring. Our proposed method consists of two steps of dimension reduction: It first applies the functional PCA to extract a reasonable large number of features under the normal operational (in-control) state, and then use the soft-thresholding techniques to further select significant features capturing profile information in the out-of-control state. The choice of tuning parameter for soft-thresholding is provided based on asymptotic analysis, and extensive simulation studies are conducted to illustrate the efficacy of our proposed thresholded PCA methodology.

Estimates for the reliability of Thurstone's regression factor score predictor, Bartlett's factor score predictor, and McDonald's factor score predictor were proposed. As in Kuder-Richardson's formula, the reliability estimates are based on a hypothetical set of equivalent items. The reliability estimates were compared by means of simulation studies. Overall, the reliability estimates were largest for the regression score predictor, so that the reliability estimates for Bartlett's and McDonald's factor score predictor should be compared with the reliability of the regression score predictor, whenever Bartlett's or McDonald's factor score predictor are to be computed. An R-script and an SPSS-script for the computation of the respective reliability estimates is presented., Comment: 2 figures

Principal component analysis (PCA) is a widely used method for dimension reduction. In high dimensional data, the "signal" eigenvalues corresponding to weak principal components (PCs) do not necessarily separate from the bulk of the "noise" eigenvalues. Therefore, popular tests based on the largest eigenvalue have little power to detect weak PCs. In the special case of the spiked model, certain tests asymptotically equivalent to linear spectral statistics (LSS)---averaging effects over all eigenvalues---were recently shown to achieve some power. We consider a nonparametric, non-Gaussian generalization of the spiked model to the setting of Marchenko and Pastur (1967). This allows a general bulk of the noise eigenvalues, accomodating correlated variables even under the null hypothesis of no significant PCs. We develop new tests based on LSS to detect weak PCs in this model. We show using the CLT for LSS that the optimal LSS satisfy a Fredholm integral equation of the first kind. We develop algorithms to solve it, building on our recent method for computing the limit empirical spectrum. In contrast to the standard spiked model, we find that under "widely spread" null eigenvalue distributions, the new tests have a lot of power., Comment: 46 pages, 9 figures

We present new findings in regard to data analysis in very high dimensional spaces. We use dimensionalities up to around one million. A particular benefit of Correspondence Analysis is its suitability for carrying out an orthonormal mapping, or scaling, of power law distributed data. Power law distributed data are found in many domains. Correspondence factor analysis provides a latent semantic or principal axes mapping. Our experiments use data from digital chemistry and finance, and other statistically generated data., Comment: 13 pages, 3 figures

The calculation of common factor means in structured means analysis (SMM) is considered. The SMM equations imply that the unique factors are defined as having zero means. It was shown within the one factor solution that this definition implies larger absolute common factor loadings to co-occur with larger absolute expectations of the observed variables in the single group case. This result was illustrated by means of a small simulation study. It is argued that the proportionality of factor loadings and observed means should be critically examined in the context of SMM. It is recommended that researchers should freely estimate the observed expectations, whenever, the proportionality of loadings and observed means does not make sense. It was also shown that for a given size of observed expectations, smaller common factor loadings result in larger common factor mean estimates. It should therefore be checked whether variables with very small factor loadings occur when SMM results in very large absolute common factor means.

Visualization and interpretation of contingency tables by correspondence analysis (CA), as developed by Benzecri, has a rich structure based on Euclidean geometry. However, it is a well established fact that, often CA is very sensitive to sparse contingency tables, where we caracterize sparsity as the existence of relatively high-valued counts, rare observations discussed by Rao (1995), and zero-block structure emphasized by Novak and Bar-Hen (2005) and Greenacre (2013). In this paper, we aim to emphasize the important roles played by L1 and L2 geometries. This will be done by comparing the maps obtained by CA with the maps obtained by taxicab correspondence analysis (TCA), where TCA is a robust L1 variant of correspondence analysis. If the projections of view of both maps are quite different, we refer to this phenomenon as parallax. In astronomy, parallax means the apparent change in the position of an object as seen from two different points. In our case the two different points correspond to the two different geometries, Euclidean and Taxicab. The existence of a parallax highlights the important, but hidden, role of the underlying geometry in the interpretation of the maps obtained in multivariate data analysis. We emphasize the following fact: Only by comparing CA and TCA graphical displays, we are able to reveal the phenomenon of parallax. Examples are provided., Comment: 23 pages, 4 figures

We decompose a matrix Y into a sum of bilinear terms in a stepwise manner, by considering Y as a mapping from a finite dimensional Banach space into another finite dimensional Banach space. We provide transition formulas, and represent them in a duality diagram, thus generalizing the well known duality diagram in the french school of data analysis. As an application, we introduce a family of Euclidean multidimensional scaling models., Comment: 20 pages

In the context of confirmatory factor analysis, the independent clusters model has been found to be overly restrictive in several research contexts. Therefore, a less restrictive criterion for parsimony of non-salient loadings in confirmatory factor analysis was proposed. The criterion is based on 'buffered scales', which have been introduced by Cattell and Tsujioka (1964) as optimal indicators of corresponding factors. Variables with positive and negative loadings on an unwanted factor are balanced out in a buffered scale, so that the variance of the unwanted factor is at minimum. It is proposed here to specify a balance of positive and negative secondary loadings by means of model constraints in order to achieve parsimony of loading patterns. The specification of buffered simple structure by means of model constraints was illustrated by means of a simulation study and an empirical example., Comment: This version contains further developments of the constraints for confirmatory factor analysis, whereas the aspects referring to exploratory factor analysis were removed. The concepts referring to buffered simple structure for exploratory factor analysis can be found in the previous version

Because the covariances of observed variables reproduced from conventional factor score predictors are generally not the same as the covariances reproduced from the common factors, it is proposed to find a factor score predictor that optimally reproduces the common part of the observed covariances. It is shown that, under some conditions, the single observed variable with highest loading on a factor perfectly reproduces the non-diagonal observed covariances. This refers to Spearman's and Wilson's 1929 debate on the use of single variables as factor score predictors. The implications of this finding were investigated in a population based and in a sample based simulation study confirming that taking a single variable outperforms conventional factor score predictors in reproducing the observed covariances when the salient loading size and the number of salient loadings per factor are small. Implications of this finding for factor score predictors are discussed.

When measuring a range of different genomic, epigenomic, transcriptomic and other variables, an integrative approach to analysis can strengthen inference and give new insights. This is also the case when clustering patient samples, and several integrative cluster procedures have been proposed. Common for these methodologies is the restriction of a joint cluster structure, which is equal for all data layers. We instead present Joint and Individual Clustering (JIC), which estimates both joint and data type-specific clusters simultaneously, as an extension of the JIVE algorithm (Lock et. al, 2013). The method is compared to iCluster, another integrative clustering method, and simulations show that JIC is clearly advantageous when both individual and joint clusters are present. The method is used to cluster patients in the Metabric study, integrating gene expression data and copy number aberrations (CNA). The analysis suggests a division into three joint clusters common for both data types and seven independent clusters specific for CNA. Both the joint and CNA-specific clusters are significantly different with respect to survival, also when adjusting for age and treatment., Comment: 20 pages, 4 figures

The Davis--Kahan theorem is used in the analysis of many statistical procedures to bound the distance between subspaces spanned by population eigenvectors and their sample versions. It relies on an eigenvalue separation condition between certain relevant population and sample eigenvalues. We present a variant of this result that depends only on a population eigenvalue separation condition, making it more natural and convenient for direct application in statistical contexts, and improving the bounds in some cases. We also provide an extension to situations where the matrices under study may be asymmetric or even non-square, and where interest is in the distance between subspaces spanned by corresponding singular vectors., Comment: 12 pages