The Effect of Restricting Ability Distributions in the Estimation of Item Difficulties: Implications for a CAT Implementation.

Responses to previously calibrated items administered in a computerized adaptive testing (CAT) mode may be used to recalibrate the items. This live-data simulation study investigated the possibility, and limitations, of on-line adaptive recalibration of precalibrated items. Responses to items of a Rasch-based paper-and-pencil licensure examination were used to simulate CAT and paper-and-pencil administrations, defining CAT forms of varying difficulty levels and samples of various sizes. Forms were calibrated and new b-values were compared with the bank b-values obtained from responses of the reference group taking the paper-and-pencil examination. Results indicate that bank b-values were not well replicated when difficult items were calibrated using responses from able examinees and easy items were calibrated using responses from less able examinees. On-line adaptive recalibration as simulated in this study has limitations. However, a "modified" on-line adaptive recalibration may still be a possibility as long as: (1) a reasonably large CAT recalibration sample is predefined from the reference group; (2) the sample has a mean ability similar to that of the reference group; and (3) items to be recalibrated together are relatively heterogeneous in Rasch difficulty. Eight tables are included. (Contains 3 references.) (Author/SLD)