An Analysis of TOEFL CBT Writing Prompt Difficulty and Comparability for Different Gender Groups. Research Reports. Report 76. RR-04-05

This investigation of the comparability of writing assessment prompts was conducted in two phases. In an exploratory Phase I, 47 writing prompts administered in the computer-based Test of English as a Foreign Language[TM] (TOEFL[R] CBT) from July through December 1998 were examined. Logistic regression procedures were used to estimate prompt difficulty and gender effects. A panel of experts reviewed selected prompts, and a taxonomy of prompt characteristics was developed and related to prompt difficulty and gender differences. In Phase II, 87 prompts administered from July 1998 through March 2000 were analyzed. All of the prompts used in Phase I, together with 40 new prompts, were analyzed using the larger Phase II database. Recommendations are made for statistical quality control procedures to identify less comparable prompts. Appended are: (1) Logistic Regression Model for Polytomous Items: The Proportional Odds-Ratio Model; (2) Number of Essays, Means, and Standard Deviations of English Language Ability and Essay Scores for Male and Female Examinee Groups; (3) Mean Expected Essay Scores, Residuals, and Standardized Mean Group Differences; (4) Uniform and Nonuniform Effect Sizes; (5) Prompt Difficulty Indices; (6) Taxonomy of TOEFL CBT Writing Prompts (Phase I); and (7) Scoring Rubrics for TOEFL CBT Writing Prompts. (Contains 13 tables, 6 figures, and 1 note.)