Search

Test publishers have promoted their commercially available, norm-referenced achievement tests as viable solutions to assessment challenges faced by states. They argue that their tests are developed professionally, and, therefore, possess sound psychometric properties not often found in state-specific efforts. This study compared judgments from two sources, test publishers from two companies and teachers, on the alignment of test items from two commercially available norm-referenced achievement tests in a Midwestern state's content standards at three grade levels in reading/writing and mathematics. Teacher panels consisted of 10 teachers for each of language and mathematics panels at grade 8 and high school levels and 20 teachers for each panel for grade 4. Analyses in this study focus on the level of agreement between the state's teachers' perceptions of how well the tests aligned to the standards and the alignment reported by the two test publishers. Results indicate that there may be some inconsistency between teachers' and publishers' perceptions. (Author/SLD)

Judgmental standard setting methods, such as the W. H. Angoff (1971) method, use item performance estimates as the basis for determining the minimum passing score (MPS). Therefore, the accuracy of these item performance estimates is crucial to the validity of the resulting MPS. Recent researchers, (L. A. Shephard 1994; J. Impara, 1997) have called into question the ability of the judges to make accurate item performance estimates for the target subgroups of candidates, such as minimally competent candidates (MCCs). The purpose of this study was to examine the validity of the judges' estimates of MCC performance that are used to determine a minimum passing score in a standard setting study context. Results from the operational standard setting workshops for a certification program in financial management that used 29 judges in 1996 and 30 in 1997 provide evidence that item performance estimates were valid. Factors that might have influenced this high degree of validity in the item performance estimates in the standard setting study are discussed. (Contains 10 references.) (SLD)