Search

As an extension of B. Thompson's 1998 invited address to the American Educational Research Association, this paper cites two additional common faux pas in research methodology and explores some research issues for the future. These two errors in methodology are the use of univariate analyses in the presence of multiple outcome variables (with the converse use of univariate analyses in post hoc explorations of detected multivariate effects) and the conversion of intervally scaled predictor variables into nominally scaled data in the service of the "of variance" (OVA) analyses. Among the research issues to receive further attention in the future is the appropriate use of statistical significance tests. The use of the descriptive bootstrap and the various types of effect size from which the researcher should select when characterizing quantitative results are also discussed. The paper concludes with an exploration of the conditions necessary and sufficient for the realization of improved practices in educational research. Three appendixes contain the Statistical Package for the Social Sciences for Windows syntax used to analyze data for three tables. (Contains 16 tables, 15 figures, and 173 references.) (SLD)

Aptitude-treatment interaction (ATI) studies have been used with some frequency, yet many researchers do not understand fully what interaction effects are. Because the means for interactions involve fewer persons per mean, power to detect interaction effects is typically smallest for the highest-order interaction in a given design. This phenomenon has been formalized by some methodologists as the Type IV error--the failure to detect statistical significance for the interaction null hypotheses that really should be rejected. This paper reviews the concept of the interaction effect. Small heuristic data sets are used to make the discussion more concrete. The concept of interaction is important because it links the two-way factorial designs with designs involving three or more independent variables. An appendix illustrates the analysis of variance technique. (Contains 4 figures, 5 tables, and 12 references.) (Author/SLD)

This paper explains the underlying assumptions of the sampling distribution and its role in significance testing. To compute statistical significance, estimates of population parameters must be obtained so that only one sampling distribution is defined. A sampling distribution is the underlying distribution of a statistic. Sampling distributions are theoretical distributions that comprise an infinite number of sample statistics from an infinite number of randomly selected samples of a specified sample size. The influence that a large sample size has on statistical significance is demonstrated through some "what if" analyses. A "what if" analysis is simply an analysis of variance summary table in which the sample size is changed to see how statistical significance is affected. A large enough sample size invariably leads to statistical significance. Researchers with large sample sizes should look for other ways to interpret their results. One such way is effect size, which is a variance accounted for statistic that can tell how much of the variability in a dependent variable can be explained by the independent variables. (Contains 2 tables, 4 figures, and 12 references.) (SLD)

This research explored the degree to which group sizes can differ before the robustness of analysis of variance (ANOVA) and analysis of covariance (ANCOVA) are jeopardized. Monte Carlo methodology was used, allowing for the experimental investigation of potential threats to robustness under conditions common to researchers in education. The effects of unequal group sizes were explored under the following data set conditions: (1) heterogeneity of group variances; (2) skew; (3) kurtosis; and (4) in ANCOVA, heterogeneity of regression slopes. Two independent sets of simulations were conducted, one using a total group of 90 and the other a total of 60. Experimentation was limited to simulations using three groups, with the total number divided in a systematic fashion. Results of these studies produced results consistent with previous research. In the analyses having homogeneity of group variances, the only simulations that emerged as statistically significant from the theoretical F test were those that had a large degree of difference in group numbers and unequal regression slopes. No significant differences emerged in simulations with near equal numbers. Tables in this paper document the simulation results, but also offer the research practitioner some idea of the true risk of Type I error in such situations. (Contains 7 tables and 24 references.) (SLD)

Conventional statistical significance tests do not inform the researcher regarding the likelihood that results will replicate. One strategy for evaluating result replication is to use a "bootstrap" resampling of a study's data so that the stability of results across numerous configurations of the subjects can be explored. This paper illustrates the use of the bootstrap in a canonical correlation analysis. Canonical correlation analysis is the most general case of classical general linear model analyses, subsuming other univariate and multivariate parametric method (e.g., t-tests, analysis of variance, analysis of covariance, regression, multivariate analysis of variance, and discriminant analysis) as special cases. A sample of 50 out of 301 subjects from a study by K. J. Holzinger and F. Swineford (1939) is used. Since bootstrap analyses capitalize during resampling on the commonalities inherent in a given sample, they yield somewhat inflated evaluations of replicability. However, inflated empirical evaluations of replicability are often superior to a mere presumption of replicability. Ten tables and one figure present details of the analysis. A 63-item list of references and an appendix listing the 50 analysis cases are included. (Author/SLD)

Detailed methodology used to evaluate a causal model of school environment is presented in this report. The model depicts societal features that influence school district values and organizational characteristics, which in turn influence school operations and personnel attitudes and values. These school variables affect school community members' perceptions of school climate, which then influence student outcomes. Questionnaires were completed by 352 principals, 14,721 teachers, and 24,874 students from 362 junior and senior high schools in 36 states and Canada. Findings suggest that student and/or teacher climate variables played a significant role in all the analyses. Of the sociodemographic variables, the percentages of students receiving free lunches and the percent of minority students in the school had the largest and most consistently direct and indirect effects on the outcome variables. Most school input variables' effects on outcomes were mediated by school climate variables. The findings indicate reasonable support for Keefe's mediated (1985) model of school outcomes and constitute the basis for a comprehensive assessment of school environments and possible interventions. Seven figures and one table of final variables are included. (8 references) (LMI)

Numerous designs using analysis of variance (ANOVA) to test ordinal hypotheses were assessed using a Monte Carlo simulation. Each statistic was computed on each of over 10,000 random samples drawn from a variety of population conditions. The number of groups, population variance, and patterns of population means were varied. In the non-null patterns, the power of the alpha-corrected tests was extremely low. Extremely low power was also evident for both the uncorrected and the no significance test for ordering technique. The linear trend test exhibited the highest power in almost every fully monotonic circumstance. The tau and monotonic trend tests also had high power in most fully monotonic population means instances, but they varied as to which was more powerful. Of the tests for disconfirmations of monotonicity, the curvilinearity test for detecting inversions or ties was too often significant in monotonic but non-linear cases. Combined tests were preferred when they had high acceptance values (power) for the fully monotonic cases, but low acceptance values when disconfirmations were present. Ultimately, the choice of technique comes down to the balance between power when the ordinal hypothesis is correct versus spurious acceptance of the ordinal hypothesis when it is slightly in error. Sweep tests, particularly the linear trend test, have the needed power, but lack the ability to alone detect disconfirmations of monotonicity. The tests of disconfirmation that were evaluated showed far too many false positives to warrant wide acceptance. Six data tables are included, and the Statistical Analysis System program used to run the Monte Carlo study and the program used to calculate taus are outlined. (TJH)

The assumptions and consequences of applying conventional and newer statistical methods to meta-analytic data sets are reviewed. The application of the two approaches to a meta-analytic data set described by L. V. Hedges (1984) illustrates the differences. Hedges analyzed six studies of the effects of open education on student cooperation. The conventional way to test the hypothesis that treatment fidelity significantly influenced results is through a t-test for independent results. Hedges' more modern approach was to use a chi-square analog of the analysis of variance (ANOVA), a method that, in contrast to conventional statistics, found strong support for the hypothesized effect. Conventional ANOVA and newer techniques were also applied to a data set in which all studies were of the same size, with each assumed to have experimental and control groups containing 25 students each. The cell means and variances for Hedges' meta-analytic data set were reconstructed to determine the source of the difference in results between conventional and newer tests. It is concluded that conventional ANOVA is appropriate for use with meta-analytic data sets because conventional ANOVA uses the correct error term for testing the significance of effects of group factors. Newer meta-analytic methods are not recommended because of their use of an inappropriate error term. (SLD)

The use of multiple comparisons in analysis of variance (ANOVA) is discussed. It is argued that experimentwise Type I error rate inflation can be serious and that its influences are often unnoticed in ANOVA applications. Both classical balanced omnibus and orthogonal planned contrast tests inflate experimentwise error to an identifiable maximum. Significance test results are overinterpreted in contemporary analytic practice, and researchers must consider effect sizes and replicability or invariance analyses when formulating interpretations. To guide analytic practice, it is suggested that omnibus hypotheses that are not of interest or which cannot be interpreted should not be tested, since such tests can distort hypothesis tests that are of interest. Orthogonal contrasts should be preferred over non-orthogonal contrasts. The use of planned contrasts is suggested in place of omnibus or unplanned hypothesis tests. Use of planned comparisons tends to result in more thoughtful research with greater power against Type II error. Small data sets and examples support the discussion, and nine tables illustrate these examples. A 73-item list of references is included. (SLD)

This paper focuses on two methods of performing multiple comparisons (post hoc or unplanned versus a priori or planned) for determining exactly where statistically significant results arise when there are more than "k equals two" groups. Textbook views and recommendations regarding the use of these two types of multiple comparisons are summarized. Analysis of variance (ANOVA) and related methods remain popular, although other methods are sometimes more useful and although the frequency of use of these methods in published research is declining. Researchers conducting an ANOVA frequently conduct unplanned or post hoc tests to determine exactly where statistically significant group differences actually arise. Various methodologists have referred to this practice as "fishing","milking", or "data snooping". It is widely known that planned multiple comparisons have greater statistical power than do unplanned multiple comparisons. Secondly, and more importantly, planned comparisons tend to make the researcher think in advance. Graduate study in statistics should reflect the importance of planned versus post hoc statistical design. In addition, there is a need for further investigation of current statistics textbooks being used in the different schools of education, business, and the behavioral sciences to determine if the issue of planned versus post hoc tests is being adequately addressed in instructional materials. A 63-item list of references, 4 data tables, and a list of selected computer control cards are included. (TJH)

Advantages of the use of multivariate commonality analysis are discussed and a small data set is used to illustrate the analysis and as a model to enable readers to conduct such an analysis. A noteworthy advantage of commonality analysis is that commonality honors the relationships among variables by determining the degree to which predictors in a set share variance with the criterion variables. Since commonality indicates the extent of overlap of the variables, it is especially useful in the behavioral sciences where predictor variables are often correlated with each other. Commonality also reinforces the recognition that canonical analysis is the most general case of parametric significance testing. The disadvantage that there are no statistical significance tests for commonality analyses is outweighed by the advantages. The illustrative example uses a hypothetical data set of 22 observations, 3 predictor variables, and 2 criterion variables. Seven data tables are provided. (SLD)

The conceptualization of analysis of covariance (ANCOVA), as an analysis of variance (ANOVA) on the residual scores that are obtained when the dependent variable is regressed on the covariate, is mathematically incorrect. If residuals are obtained from the pooled within-groups regression coefficient, ANOVA on the residuals results in an inflated alpha-level. If the regression coefficient for the total sample combined into one group is used, ANOVA on the residuals yields an inappropriately conservative test. In either case, ANOVA of residuals fails to provide a correct test, because the significance test in ANCOVA requires consideration of both the pooled within-groups regression coefficient and the regression coefficient for the total sample combined into one group. It is recommended that the significance test of treatment effects in ANCOVA be conceptualized as a comparison of models whose parameters are estimated by the principle of least squares. (Author/RL)

A fundamental fact of the analysis of variance statistical procedure is that if the omnibus F test of an effect is significant, then there exists at least one contrast of that effect that will be significantly different from zero according to the S-method of Scheffe. The caveat to this rule is that the significant contrast(s) may not be of any interest to or interpretable by the investigator. Frustration does arise when the investigator determines the omnibus F test to be significant and then fails to find any meaningful group comparisons to be significant. The purpose of this paper is to discuss the coefficients of a contrast which will yield the largest test statistic according to the S-method. Then according to the fundamental theorem, this contrast will be significantly different from zero and, additionally, should offer the investigator some assistance in developing a parsimonious interpretation of the data. The S-method is a post-hoc method of comparing experimental groups and is often referred to as a data-snooping or sifting device. The techniques discussed here are also of the data-snooping species and may be employed to generate new hypotheses or to amend the existing theory of a practical problem. (Author/CTM)

The current state of usage of regression models in analysis of variance (ANOVA) designs is empirically examined, and examples of several statistical errors made in usage are presented. The assumptions of the general linear model are that all predictors are known without error of measurement and are fixed with no replication or sample variation; in the population, errors are normally distributed independently with variance, and errors are independent of all predictors. The rules for construction of the ANOVA allow the expected mean squares to hold just as if the levels of each factor had been randomly sampled. Analysis of Covariance (ANCOVA) combines the elements of regression analysis with design, albeit in a restricted manner. The homogeneity of regression coefficients is a parameter restriction from the design view point. The regression weight associated with a given covariate level is discussed. Most regression approaches to ANOVA and ANCOVA assume a fixed factor model under all design specifications. A major oversight of expected mean squares has contributed to the current lack of concern for the level of generalizability warranted from the design specification. Aptitude treatment interaction models are examined. (Author/CM)

This 3-year study, part of the Effective Schools Project of the Seattle School District and the University of Washington, was undertaken to investigate the relationship between teachers' perceptions of the principal as instructional leader and average gain scores of students in 67 elementary schools in Seattle. Scores were disaggregated by student ethnicity and student free lunch status as a surrogate measure of socioeconomic status (SES). Data pertaining to teacher perceptions of the principal as instructional leader were obtained through the Staff Assessment Questionnaire, and specific attention was given to four general aspects of principal behavior: (1) mobilizing resources, (2) communicating, (3) serving as instructional resource, and (4) being a visible presence. These data were then correlated with school means for aggregated student improvement on Total Reading and Total Mathematics of the California Achievement Test (CAT). Individual gains for students were computed and aggregated for four groups: white, black, free lunch, and non-free lunch. Regression analyses were used to assess the relationships between the leadership of the principal and other effective school characteristics, while analysis of variance was used to assess the effects of leadership on student gain scores. The findings of this study suggest that teachers' perceptions of the principal as instructional leader are critical to the reading and mathematics achievement of students, particularly among historically low-achieving groups of students. (TE)

This paper evaluates the logic underlying various criticisms of statistical significance testing and makes specific recommendations for scientific and editorial practice that might better increase the knowledge base. Reliance on the traditional hypothesis testing model has led to a major bias against nonsignificant results and to misinterpretation of significant results. A finding of statistical significance does not mean that the null hypothesis is false, since there are many factors affecting statistical significance such as sample size and the measurement reliability of the data. Furthermore, statistical significance alone does not permit evaluation of the importance of a finding. An effect size statistic, such as eta-squared, is more appropriate for this purpose, and editors of scholarly publications should encourage routine reporting of effect sizes. Greater reporting of nonsignificant results should also be encouraged, accompanied by power analyses to estimate Type II error. Nonsignificant results can be meaningful if the study's power to detect an effect was high. Finally, the paper emphasizes the crucial role of replication in separating true effects from Type I errors. The paper integrates analyses and criticisms of statistical practice from a variety of sources--77 references are included. (LPG)

Although methodological integrity is not the sole determinant of the value of a program evaluation, decision-makers do have a right, at a minimum, to be able to expect competent work from evaluators. This paper explores five areas where evaluators might improve methodological practices. First, evaluation reports should reflect the limited contribution that statistical significance testing can make to the interpretation of results. Second, evaluation reports should also reflect the fact that multivariate statistics are often vital in educational research because multivariate methods best honor the reality to which the researcher is trying to generalize. Third, evaluation reports should reflect the recognition that discarding variance to conduct chi-square or analyses of variance and covariance and multivariate analyses of variance and covariance (OVA methods) can lead to serious distortions in interpretations. Even when OVA methods are appropriate, the methods should usually be implemented using regression approaches. Fourth, evaluation reports should reflect the recognition that covariance statistical correlations are usually least helpful when corrections are most needed. Finally, evaluation reports should reflect the recognition that stepwise analytic methods can lead to seriously distorted interpretations. Fourteen tables and three figures illustrate these contentions. A 102-item list of references is provided. (SLD)

The use of planned, or "a priori," and unplanned, or "post hoc," comparisons to isolate differences among means in analysis of variance research is discussed. Planned comparisons typically involve weighting data by sets of "contrasts." Planned comparison offer more power against Type II errors. In addition, they force the researcher to be more thoughtful in conducting research. It is suggested that these two reasons make planned comparisons superior to unplanned comparisons. It is further suggested that orthogonal tests are more useful than non-orthogonal tests. Planned comparisons can be used even when omnibus tests are not statistically significant. (SLD)

This paper reports the results of a Monte Carlo investigation of Type I errors in the single group repeated measures design where multiple measures are collected from each observational unit at each measurement occasion. The Type I error of three multivariate tests were examined. These were the doubly multivariate F test, the multivariate mixed model F test, and a recently proposed multivariate extension of Mauchly's sphericity test. The study examined the efficacy of using the latter test as preliminary test to evaluate the tenability of the multivariate sphericity assumption made by the multivariate mixed model F. The study also examined the impact of violations of the multivariate sphericity assumption upon the Type I error control of the multivariate mixed model F. The results indicate an analysis strategy for the researcher confronted with the task of analyzing data gathered using this research design. Example data sets are presented to illustrate three likely patterns of behavioral data. The data analysis strategy is discussed as it relates to each data set. (Author)

Political communication researchers recently have emphasized conditional relationships as crucial to specifying and understanding the effects of the mass media. In many cases, however, researchers will not predict which variables will modify the form or existence of media effects. They therefore will not include measures of the conditional variables in research designs. The presence of heteroscedasticity, or unequal precision of prediction at different levels of an independent variable, suggests the likelihood of an overlooked conditional relationship. This paper examines the possibilities for the use of statistical tests of variation in mass communication research. Research situations that employ both discrete and continuous independent variables are covered, and the visual examination of scatterplots as an alternative to performing variability tests is examined. Problems with the variation tests available on commonly used computer packages is addressed. A number of recently developed, apparently dependable variability tests also are presented. A research example involving two-way, linear interactions is provided. Additional concerns with unequal variability are also discussed. Three figures and a four-page list of references are included. (Author/PN)

Hypothetical data sets are used to demonstrate how canonical correlation methods subsume other commonly utilized parametric methods. Analysis of variance, analysis of covariance, multiple analysis of variance, and multiple analysis of covariance are heavily used by educational researchers. It is concluded that researchers would do well to consider using general linear model (GLM) techniques as opposed to analysis of variance and its analogs. GLM and analysis of variance are equivalent when the ways in an analysis of variance design each have exactly two levels. In other cases, the GLM analytic approach yields more specific information regarding effects, and greater power against Type II error, because the comparisons are planned a priori rather than post hoc. Multivariate canonical correlation methods are superior to univariate techniques in providing important information on the interrelationships among variables. Furthermore, multiple univariate procedures tend to inflate the probability of a Type I error. (GDC)

A dichotomous dependent variable is used to determine a combination of variables that will predict group membership. Dichotomous variables are frequently encountered in multiple regression analysis. However, several textbooks question the appropriateness of using multiple regression analysis when analyzing dichotomous dependent variables. The critics state that in addition to the predictions made by the regression equation with a dichotomous dependent variable, the statistical tests are also invalid. This paper assesses the meaning of the charges against multiple regression and deals with some logical extensions of them. Data from the A3 data set in Gunst and Mason is used to examine the criticisms by comparing results of cases viewed as appropriate by critics with cases viewed as inappropriate for one predictor and two predictor examples. As the tests of significance are identical whether the dichotomous variable is independent or dependent, critics must deal with all significance testing with t-tests, analysis of variance, analysis of covariance, discriminant analysis, and any use of dummy variables in multiple regression if they are to be taken seriously. (BS)

This study investigated different loci of instructional control and prior background knowledge in terms of two aspects of metacognitive effects (utilization and correctness of metacognitive monitoring) and two aspects of cognitive effects (knowledge acquisition and application). The study also investigated the effectiveness of learner strategies used under learner control. Two parallel versions of computer assisted instruction (CAI) lessons on the computer programming language Logo were developed for the program control and learner control groups. The two versions were the same in the tutorial sessions, but differed in the presentation of the question for self-estimates of understanding and in the choices available in the practice sessions. The subjects were 62 third graders in a suburban elementary school who had no prior experience with Logo, and the study was conducted in the school computer lab over seven weeks. When supportive factors such as provision of clearly labeled options, basic requirements, and presentation of feedback and advice on on-going progress were integrated into the design of a learner control study, this condition seemed to foster students' metacognitive as well as cognitive knowledge and skills in a more effective way than did program control. The overall results of the study indicate that, as an instructional design strategy, learner control can provide learners with opportunities to improve their metacognitive skills and knowledge, while attempting to teach cognitive skills and knowledge of specific subject matter content in a more effective way than can program control. Limitations of the study are noted, and nine tables, copies of the pretest and the interview schedule for metacognitive monitoring, and formulas for the evaluation of the effectiveness of learner strategy are appended. (27 references) (BBM)