Field Trials To Determine Which Rating Method(s) To Use in the 1998 NAEP Achievement Levels-Setting Process for Civics and Writing.

Field trials were conducted to test rating methods and the impact of feedback about consequences for the 1998 achievement levels-setting (ALS) process for the National Assessment of Educational Progress (NAEP). The field trials provided the opportunity to try out different methods similar to those used successfully by others, as well as to try out some new methods. The American College Testing program (ACT) had proposed a new method to be tested in the field trials. Although successful implementation of the method had been reported, the method was found to be biased, and the ACT stopped tests with the method after the first field trials. Reservations about item maps were not overcome in the field trial process, and item maps were eliminated as a choice. Concerns about computational procedures and the logistic demands of the Booklet Classification Method eliminated this approach. The Technical Advisory Committee on Standard Setting recommended a new combination method based on the method developed by M. Reckase in conjunction with the strong research base and extensive experience by ACT associated with the Mean Estimation method. Procedures based on this approach were used to set achievement levels for the 1998 NAEP in civics and writing. Appendixes contain examples of charts used in the rating method study. (Contains 18 tables, 33 figures, and 25 references.) (Author/SLD)