Using the Think Aloud Method (Cognitive Labs) to Evaluate Test Design for Students with Disabilities and English Language Learners. Technical Report 44

Finding ways to improve the design of large-scale tests is a timely issue. Recent changes in Federal legislation (including the No Child Left Behind Act of 2001) have placed greater emphasis on accountability via large-scale tests. Students who have previously been exempted from large-scale assessments, including students with disabilities and English language learners, are now expected to reach achievement levels comparable to their non-disabled or English proficient peers. Schools are held accountable for their performance, and their scores are reported publicly. With such high stakes placed on large-scale assessment, there is a critical need for states to have valid information about how the design of assessments affects student test performance. This report provides information on the use of "think aloud methods" to detect design issues in large-scale assessments. Various design problems may introduce construct-irrelevant variance or hinder students from showing what they know on assessments. Research included a variety of students, including students with learning disabilities, students with hearing impairments, students with cognitive disabilities, English language learners, and students without disabilities who were proficient in English. In this project, the authors asked students to "think out loud" when solving mathematics large-scale assessment items. The sentences that students uttered produced data that led the authors to believe that think aloud methods, under certain circumstances, can successfully detect design issues. Specifically, they found issues related to unclearly defined constructs, inaccessibility of items, unclear instructions, incomprehensible language, and illegible text and graphics. To this end, think aloud methods appear to be a useful strategy in the design and refinement of large-scale assessments. Think aloud methods, as the authors designed them, were not effective for students with cognitive disabilities. This population had great difficulty in producing the language needed to explain problem-solving processes and may require additional research accommodations to help them participate in think aloud research. All other groups sufficiently participated in research activities. Think aloud methods also did not produce informative data for very difficult mathematics items because students had difficulty verbalizing their thoughts while solving problems. Despite shortcomings found in this study, the think aloud method appears to be an effective way to determine the effects of item design for a wide variety of students (with the exception of students with cognitive disabilities) and for items with low to moderate difficulty levels. (Contains 6 tables and 2 figures.)