Search

There have been many discussions of how Type I errors should be controlled when many hypotheses are tested (e.g., all possible comparisons of means, correlations, proportions, the coefficients in hierarchical models, etc.). By and large, researchers have adopted familywise (FWER) control, though this practice certainly is not universal. Familywise control is intended to deal with the multiplicity issue of computing many tests of significance, yet such control is conservative--that is, less powerful--compared to per test/hypothesis control. The purpose of our article is to introduce the readership, particularly those readers familiar with issues related to controlling Type I errors when many tests of significance are computed, to newer methods that provide protection from the effects of multiple testing, yet are more powerful than familywise controlling methods. Specifically, we introduce a number of procedures that control the "k"-FWER. These methods--say, 2-FWER instead of 1-FWER (i.e., FWER)--are equivalent to specifying that the probability of 2 or more false rejections is controlled at 0.05, whereas FWER controls the probability of any (i.e., 1 or more) false rejections at 0.05. 2-FWER implicitly tolerates 1 false rejection and makes no explicit attempt to control the probability of its occurrence, unlike FWER, which tolerates no false rejections at all. More generally, "k"-FWER tolerates "k" - 1 false rejections, but controls the probability of "k" or more false rejections at [alpha] = 0.05. We demonstrate with two published data sets how more hypotheses can be rejected with "k"-FWER methods compared to FWER control. (Contains 4 tables and 8 footnotes.)

Kelley compared three methods for setting a confidence interval (CI) around Cohen's standardized mean difference statistic: the noncentral-"t"-based, percentile (PERC) bootstrap, and biased-corrected and accelerated (BCA) bootstrap methods under three conditions of nonnormality, eight cases of sample size, and six cases of population effect size (ES) magnitude. Kelley recommended the BCA bootstrap method. The authors expand on his investigation by including additional cases of nonnormality. Like Kelley, they find that under many conditions, the BCA bootstrap method works best; however, they also find that in some cases of nonnormality, the method does not control probability coverage. The authors also define a robust parameter for ES and a robust sample statistic, based on trimmed means and Winsorized variances, and cite evidence that coverage probability for this parameter is good over the range of nonnormal distributions investigated when the PERC bootstrap method is used to set CIs for the robust ES. (Contains 7 tables.)

Locating pairwise differences among treatment groups is a common practice of applied researchers. Articles published in this journal have addressed the issue of statistical inference within the context of an analysis of variance (ANOVA) framework, describing procedures for comparing means, among other issues. In particular, 1 article (Jaccard & Guilamo-Ramos, 2002b) presented some new methods of performing contrasts of means whereas another presented a framework for obtaining robust tests within this same context (Jaccard & Guilamo-Ramos, 2002a). The purpose of this article is to add to these contributions by presenting some newer methods for conducting pairwise comparisons of means, that is by extending the contributions of the first article and applying the framework of the second article to pairwise multiple comparisons. The newer methods are intended to provide additional sensitivity to detect treatment group differences and provide tests that are robust to the effects of variance heterogeneity, nonnormality, or both.

The approximate degrees of freedom Welch-James (WJ) and Brown-Forsythe (BF) procedures for testing within-subjects effects in multivariate groups by trials repeated measures designs were investigated under departures from covariance homogeneity and normality. Empirical Type I error and power rates were obtained for least-squares estimators and robust estimators based on trimming/Winsorization. The BF and WJ procedures with least-squares estimators produced comparable Type I error rates for the main effect. For the interaction, the former could become conservative, while the latter was liberal when sample sizes were small. When robust estimators were adopted, the WJ test generally controlled the error rate while the BF test was conservative. Both procedures had similar power for the main effect test, while the WJ procedure was more powerful for tests of the interaction when covariances were heterogeneous.

Compared strategies for performing multiple comparisons with nonnormal data under various data conditions, including simultaneous violations of the assumptions of normality and variance homogeneity. Monte Carlo study results show the conditions under which different strategies are most appropriate. (SLD)

Compared two bootstrap methods that use trimmed means, the percentile and percentile T methods and considered how these methods might be adapted to comparing "K" measures corresponding to two independent groups. Results from simulation studies lead to an extension of the percentile bootstrap approach that gives better results. (SLD)

Comparison of six procedures to test for location equality among two or more groups when population variances are heterogeneous suggests that, when the variance homogeneity and normality assumptions are not satisfied, and the design is unbalanced, the use of any of these test statistics with the usual least squares estimators is not recommended. (SLD)

The analysis of variance, Welch, and Brown and Forsyth tests for mean equality were compared using Monte Carlo methods. The tests' rates of Type I error and power were examined when populations were nonnormal, variances were heterogeneous, and group sizes were unequal. Recommendations for use are presented. (Author/JKS)