Search

Teacher-made mastery tests were administered in a classroom-sized sample to study their decision consistency. Decision-consistency of criterion-referenced tests is usually defined in terms of the proportion of examinees who are classified in the same way after two test administrations. Single-administration estimates of decision consistency were used in this study. First, it was shown that the Model 2 estimate of Marshall and Serlin's mean split-half coefficient of agreement was equivalent to the Subkoviak estimate of agreement. Ten 20-item criterion-referenced tests were constructed by 10 teachers, following strict criterion-referenced test specifications. Another 10 tests were constructed by another set of teachers, based on loosely-specified sets of objectives. These were called unit tests. It was found that the two estimation models examined produced estimates which were generally acceptable with teacher-constructed tests and classroom-sized samples, but the Model 2 estimate was marginally preferable in that it was generally less biased than Model 1 estimates. There was no systematic difference between the criterion referenced tests and the unit tests with respect to decision-consistency accuracy. (GDC)