Understanding Learner Strengths and Weaknesses: Assessing Performance on an Integrated Writing Task

The present study examined the factor structures across features of 446 examinees' responses to a writing task that integrates reading and listening modalities as well as reading and listening comprehension items of the TOEFL iBT[R] (Internet-based test). Both human and automated scores obtained for the integrated essays were utilized. Based on a series of preliminary factor analyses, a confirmatory factor analysis (CFA) identified a model that specified a higher order factor for comprehension. In the model, the Comprehension factor underlay factors representing content of the written essay as well as reading and listening comprehension. The Comprehension factor correlated with two writing factors--Productive Vocabulary and Sentence Conventions. Furthermore, follow-up CFA models with covariates (multiple indicators multiple causes models, or MIMIC models) were tested to compare performance between a group of 190 examinees scoring above a frequently used TOEFL iBT Total score requirement for international student admission and the other group of 128 examinees scoring below the requirement. The higher ability group performed significantly better than the lower ability group on all three constructs: Comprehension, Productive Vocabulary, and Sentence Conventions. The identification of the multiple distinct factors in this study may hold promise for obtaining writing profiles that inform instruction in contexts such as test preparation. (Contains 9 tables, 4 figure and 1 footnote.)