Planned versus Unplanned and Orthogonal versus Nonorthogonal Contrasts: The Neo-Classical Perspective.

The use of multiple comparisons in analysis of variance (ANOVA) is discussed. It is argued that experimentwise Type I error rate inflation can be serious and that its influences are often unnoticed in ANOVA applications. Both classical balanced omnibus and orthogonal planned contrast tests inflate experimentwise error to an identifiable maximum. Significance test results are overinterpreted in contemporary analytic practice, and researchers must consider effect sizes and replicability or invariance analyses when formulating interpretations. To guide analytic practice, it is suggested that omnibus hypotheses that are not of interest or which cannot be interpreted should not be tested, since such tests can distort hypothesis tests that are of interest. Orthogonal contrasts should be preferred over non-orthogonal contrasts. The use of planned contrasts is suggested in place of omnibus or unplanned hypothesis tests. Use of planned comparisons tends to result in more thoughtful research with greater power against Type II error. Small data sets and examples support the discussion, and nine tables illustrate these examples. A 73-item list of references is included. (SLD)