A Simulation Study of the Use of Change Measures to Compare Educational Programs. Report No. 183.

Recent research results strongly suggest that the theoretical problems of change measures have limited practical significance for measuring individual growth, and it is important to determine whether this is also the case for measuring school impact. Accordingly, in this study artifical data were used to assess the correlation between several estimates of average student change in various schools and the "true" impact of the same schools. Because it seems desirable for artificial data to resemble real data, the computer procedure was designed to reproduce selected aspects of the Educational Testing Service Growth Study and of the Project TALENT study of high schools in the U.S. Results indicate that all estimates involving pretest-posttest differences measure school impact with reasonable accuracy. It is important to measure change over the entire course of learning, however, and not just over the later stages of learning. The correlations between change scores and other school characteristics reflect with reasonable accuracy the relationships between those characteristics and impact, but will be large only when the underlying relationships are substantial. Simple gain scores measure the true situation about as accurately as other change estimates, are easier to compute, and probably are more meaningful to nonresearchers. (Author/JM)