The Effects of Sample Size and Matching Strategy on Mantel-Haenszel and Logit DIF Procedures.

Research efforts have established that the Mantel-Haenszel procedure (MHP) is an effective method for detecting the presence of test items exhibiting differential item functioning (DIF). While the MHP has been advocated for situations where item response theory based methods may not be usable, recent findings have suggested that the performance of the MHP and Logit needs to be examined in detail. The present research examined the impact of manipulating sample size, the ratio of focal to reference group members, and the number of levels of the matching criterion on the performance of the MHP and Logit. Reference and focal groups consisted of 7,320 majority group Scholastic Aptitude Examination (SAT) takers and 791 minority group SAT takers. Other samples of 1,000 reference and 500 focal group members were obtained through a random number generating program. Four, 8, and 12 levels of matching on the SAT score were used. Results suggest that the MHP and Logit are sensitive to these manipulations. As sample sizes decreased, mean values for the MHP and Logit decreased, and agreement between them declined. As the number of levels of matching increased in the full sample condition, agreement between Logit and the MHP also dropped. Four tables present study findings, and there is a 23-item list of references. (Author/SLD)