Search

There have been many discussions of how Type I errors should be controlled when many hypotheses are tested (e.g., all possible comparisons of means, correlations, proportions, the coefficients in hierarchical models, etc.). By and large, researchers have adopted familywise (FWER) control, though this practice certainly is not universal. Familywise control is intended to deal with the multiplicity issue of computing many tests of significance, yet such control is conservative--that is, less powerful--compared to per test/hypothesis control. The purpose of our article is to introduce the readership, particularly those readers familiar with issues related to controlling Type I errors when many tests of significance are computed, to newer methods that provide protection from the effects of multiple testing, yet are more powerful than familywise controlling methods. Specifically, we introduce a number of procedures that control the "k"-FWER. These methods--say, 2-FWER instead of 1-FWER (i.e., FWER)--are equivalent to specifying that the probability of 2 or more false rejections is controlled at 0.05, whereas FWER controls the probability of any (i.e., 1 or more) false rejections at 0.05. 2-FWER implicitly tolerates 1 false rejection and makes no explicit attempt to control the probability of its occurrence, unlike FWER, which tolerates no false rejections at all. More generally, "k"-FWER tolerates "k" - 1 false rejections, but controls the probability of "k" or more false rejections at [alpha] = 0.05. We demonstrate with two published data sets how more hypotheses can be rejected with "k"-FWER methods compared to FWER control. (Contains 4 tables and 8 footnotes.)

The increase in the squared multiple correlation coefficient ([delta]R[superscript 2]) associated with a variable in a regression equation is a commonly used measure of importance in regression analysis. Algina, Keselman, and Penfield found that intervals based on asymptotic principles were typically very inaccurate, even though the sample size was quite large (i.e., larger than 200). However, they also reported that probability coverage for the confidence intervals based on a bootstrap method was typically quite accurate, and moreover, this accuracy was obtained with relatively small sample sizes with six or fewer predictors. They further speculated that nonnormality would likely affect the accuracy of interval coverage. In the present study, the authors investigated the accuracy of coverage probability for confidence intervals obtained by using asymptotic and percentile bootstrap methodology when either predictors, residuals, or both are nonnormal. Coverage probability for asymptotic confidence intervals is poor, but adequate coverage probability can be obtained with reasonable sample sizes by using percentile bootstrap methodology. As well, the authors found that the width of these intervals was relatively precise (i.e., narrow) for the larger cases of sample size investigated. (Contains 7 tables.)

Standard least squares analysis of variance methods suffer from poor power under arbitrarily small departures from normality and fail to control the probability of a Type I error when standard assumptions are violated. This article describes a framework for robust estimation and testing that uses trimmed means with an approximate degrees of freedom heteroscedastic statistic for independent and correlated groups designs in order to achieve robustness to the biasing effects of nonnormality and variance heterogeneity. The authors describe a nonparametric bootstrap methodology that can provide improved Type I error control. In addition, the authors indicate how researchers can set robust confidence intervals around a robust effect size parameter estimate. In an online supplement, the authors use several examples to illustrate the application of an SAS program to implement these statistical methods. (Contains 4 tables, 1 figure, and 24 footnotes.)

Applications of distribution theory for the squared multiple correlation coefficient and the squared cross-validation coefficient are reviewed, and computer programs for these applications are made available. The applications include confidence intervals, hypothesis testing, and sample size selection. (Contains 2 tables.)

The increase in the squared multiple correlation coefficient ([Delta]R[squared]) associated with a variable in a regression equation is a commonly used measure of importance in regression analysis. The coverage probability that an asymptotic and percentile bootstrap confidence interval includes [Delta][rho][squared] was investigated. As expected, coverage probability for the asymptotic confidence interval was often inadequate (outside the interval 0.925 to 0.975 for a 95% confidence interval), even when sample size was quite large (i.e., 200). However, adequate coverage probability for the confidence interval based on a bootstrap interval could typically be obtained with a sample size of 200 or less, and moreover, this accuracy was obtained with relatively small sample sizes (100 or less) with six or fewer predictors. (Contains 4 tables.)

Kelley compared three methods for setting a confidence interval (CI) around Cohen's standardized mean difference statistic: the noncentral-"t"-based, percentile (PERC) bootstrap, and biased-corrected and accelerated (BCA) bootstrap methods under three conditions of nonnormality, eight cases of sample size, and six cases of population effect size (ES) magnitude. Kelley recommended the BCA bootstrap method. The authors expand on his investigation by including additional cases of nonnormality. Like Kelley, they find that under many conditions, the BCA bootstrap method works best; however, they also find that in some cases of nonnormality, the method does not control probability coverage. The authors also define a robust parameter for ES and a robust sample statistic, based on trimmed means and Winsorized variances, and cite evidence that coverage probability for this parameter is good over the range of nonnormal distributions investigated when the PERC bootstrap method is used to set CIs for the robust ES. (Contains 7 tables.)

The authors argue that a robust version of Cohen's effect size constructed by replacing population means with 20% trimmed means and the population standard deviation with the square root of a 20% Winsorized variance is a better measure of population separation than is Cohen's effect size. The authors investigated coverage probability for confidence intervals for the new effect size measure. The confidence intervals were constructed by using the noncentral t distribution and the percentile bootstrap. Over the range of distributions and effect sizes investigated in the study, coverage probability was better for the percentile bootstrap confidence interval.

Probability coverage for eight different confidence intervals (CIs) of measures of effect size (ES) in a two-level repeated measures design was investigated. The CIs and measures of ES differed with regard to whether they used least squares or robust estimates of central tendency and variability, whether the end critical points of the interval were obtained using a theoretical or an empirical sampling distribution, and whether the ESs used a pooled or nonpooled estimate of error variability. These intervals were compared when data were obtained from both normal and nonnormal distributions and when the population magnitude of effect, size of sample, and variance heterogeneity were varied. It was found that the ESs and intervals that used robust estimators and critical values were obtained through a bootstrap method better at controlling the probability coverage (i.e., within [.925, .975]).

Locating pairwise differences among treatment groups is a common practice of applied researchers. Articles published in this journal have addressed the issue of statistical inference within the context of an analysis of variance (ANOVA) framework, describing procedures for comparing means, among other issues. In particular, 1 article (Jaccard & Guilamo-Ramos, 2002b) presented some new methods of performing contrasts of means whereas another presented a framework for obtaining robust tests within this same context (Jaccard & Guilamo-Ramos, 2002a). The purpose of this article is to add to these contributions by presenting some newer methods for conducting pairwise comparisons of means, that is by extending the contributions of the first article and applying the framework of the second article to pairwise multiple comparisons. The newer methods are intended to provide additional sensitivity to detect treatment group differences and provide tests that are robust to the effects of variance heterogeneity, nonnormality, or both.

One approach to the analysis of repeated measures data allows researchers to model the covariance structure of their data rather than presume a certain structure, as is the case with conventional univariate and multivariate test statistics. This mixed-model approach, available through SAS PROC MIXED, was compared to a Welch-James type statistic. The Welch-James approach is known to provide generally robust tests of treatment effects in a repeated measures between-by within-subjects design under assumption violations given certain sample size requirements. The mixed-model "F" tests were based on Kenward-Roger's adjusted degrees of freedom solution, an approach specifically proposed for small sample settings. The authors investigated Type I error control for repeated measures main and interaction effects in unbalanced designs when normality and covariance homogeneity assumptions did not hold. The mixed-model Kenward-Roger's adjusted "F" tests showed superior Type I error control in small sample size conditions in which the Welch-James type statistic was nonrobust; power rates, however, did not favor one approach over the other.

The approximate degrees of freedom Welch-James (WJ) and Brown-Forsythe (BF) procedures for testing within-subjects effects in multivariate groups by trials repeated measures designs were investigated under departures from covariance homogeneity and normality. Empirical Type I error and power rates were obtained for least-squares estimators and robust estimators based on trimming/Winsorization. The BF and WJ procedures with least-squares estimators produced comparable Type I error rates for the main effect. For the interaction, the former could become conservative, while the latter was liberal when sample sizes were small. When robust estimators were adopted, the WJ test generally controlled the error rate while the BF test was conservative. Both procedures had similar power for the main effect test, while the WJ procedure was more powerful for tests of the interaction when covariances were heterogeneous.

The Welch-James (WJ) and the Huynh Improved General Approximation (IGA) tests for interaction were examined with respect to Type I error in a between- by within-subjects repeated measures design when data were non-normal, non-spherical and heterogeneous, particularly when group sizes were unequal. The tests were computed with aligned ranks and compared to the use of least squares and robust estimators (i.e., trimmed means and Winsorized variances/covariances). Critical values were either obtained theoretically or through a bootstrapping method. The IGA and WJ procedures based on aligned ranks always provided a valid test of a repeated measures interaction effect when group sizes were equal and covariance matrices across groups were homogeneous. On the other hand, the use of aligned ranks did not provide a valid test for a repeated measures interaction when covariance matrices were nonspherical with unequal variances across the levels of the repeated measures factor combined with unequal covariance matrices across the grouping factor. The IGA and WJ procedures based on robust estimators provided a valid test of the interaction across investigated conditions, however under a heavy-tailed distribution, the IGA and WJ procedures based on least squares estimators showed better Type I error control than when based on robust estimators.

Compared strategies for performing multiple comparisons with nonnormal data under various data conditions, including simultaneous violations of the assumptions of normality and variance homogeneity. Monte Carlo study results show the conditions under which different strategies are most appropriate. (SLD)

Investigated the approximate confidence intervals for effect sizes developed by K. Bird (2002) and proposed a more accurate method developed through simulation studies. The average coverage probability for the new method was 0.959. (SLD)

Compared two bootstrap methods that use trimmed means, the percentile and percentile T methods and considered how these methods might be adapted to comparing "K" measures corresponding to two independent groups. Results from simulation studies lead to an extension of the percentile bootstrap approach that gives better results. (SLD)

Discusses the squared cross-validity coefficient as a measure of the predictive validity of a sample linear prediction equation. Presents sample-size tables that should result in very small discrepancies between the squared multiple correlation and the squared cross-validity correction, thus facilitating the selection of sample size for predictive studies. (Author/SLD)

Three approaches to the analysis of main and interaction effect hypotheses in nonorthogonal designs were compared in a 2 x 2 design for data that was neither normal in form nor equal in variance. The Welch-James test with trimmed means and Winsorized variances provided excellent Type I error control. (SLD)

Comparison of six procedures to test for location equality among two or more groups when population variances are heterogeneous suggests that, when the variance homogeneity and normality assumptions are not satisfied, and the design is unbalanced, the use of any of these test statistics with the usual least squares estimators is not recommended. (SLD)

Nonparametric and robust statistics (those using trimmed means and Winsorized variances) were compared for their ability to detect treatment effects in the two-sample case. The use of two specialized tests, designed to be sensitive to treatment effects when data distributions are skewed to the right, is not supported by the analyses. (SLD)

Examines the recommendations of H. Keselman, K. Carriere, and L. Lix (1993) regarding choice of sample size for obtaining robust tests of the repeated measures main and interaction hypotheses in a one Between-Subjects by one Within- Subjects design with a Welch-James type multivariate test when covariance matrices are heterogeneous. (SLD)