Exploring the Replicability of a Study's Results: Bootstrap Statistics for the Multivariate Case.

Conventional statistical significance tests do not inform the researcher regarding the likelihood that results will replicate. One strategy for evaluating result replication is to use a "bootstrap" resampling of a study's data so that the stability of results across numerous configurations of the subjects can be explored. This paper illustrates the use of the bootstrap in a canonical correlation analysis. Canonical correlation analysis is the most general case of classical general linear model analyses, subsuming other univariate and multivariate parametric method (e.g., t-tests, analysis of variance, analysis of covariance, regression, multivariate analysis of variance, and discriminant analysis) as special cases. A sample of 50 out of 301 subjects from a study by K. J. Holzinger and F. Swineford (1939) is used. Since bootstrap analyses capitalize during resampling on the commonalities inherent in a given sample, they yield somewhat inflated evaluations of replicability. However, inflated empirical evaluations of replicability are often superior to a mere presumption of replicability. Ten tables and one figure present details of the analysis. A 63-item list of references and an appendix listing the 50 analysis cases are included. (Author/SLD)