Five Steps for Improving Evaluation Reports by Using Different Data Analysis Methods.

Although methodological integrity is not the sole determinant of the value of a program evaluation, decision-makers do have a right, at a minimum, to be able to expect competent work from evaluators. This paper explores five areas where evaluators might improve methodological practices. First, evaluation reports should reflect the limited contribution that statistical significance testing can make to the interpretation of results. Second, evaluation reports should also reflect the fact that multivariate statistics are often vital in educational research because multivariate methods best honor the reality to which the researcher is trying to generalize. Third, evaluation reports should reflect the recognition that discarding variance to conduct chi-square or analyses of variance and covariance and multivariate analyses of variance and covariance (OVA methods) can lead to serious distortions in interpretations. Even when OVA methods are appropriate, the methods should usually be implemented using regression approaches. Fourth, evaluation reports should reflect the recognition that covariance statistical correlations are usually least helpful when corrections are most needed. Finally, evaluation reports should reflect the recognition that stepwise analytic methods can lead to seriously distorted interpretations. Fourteen tables and three figures illustrate these contentions. A 102-item list of references is provided. (SLD)