Investigating Test-Taking Strategies in Listening Assessment: A Comparative Study of Eye-Tracking and Self-Report Questionnaires

This study aimed to investigate the test-taking strategies needed for successful completion of a lecture-based listening test by employing self-reported test-taking strategy use, actual strategy use measured via eye-tracking, and test scores. In this study, participants' gaze behavior (measured by fixation and visit duration and frequency) were recorded while they completed two listening tests of three stages each: pre-listening, in which participants (n = 66) previewed question stems; while-listening, in which participants simultaneously listened to the recording and filled in their answers; and post-listening, in which they had time to review their answers and make necessary amendments. Following the listening tests, participants filled up a posttest questionnaire that asked about their strategy use in each of the three stages. Rasch measurement, "t"-test, and path analysis were performed on test scores, questionnaire results, and gaze patterns. Results suggest that gaze measures (visit duration and fixation frequency) predicted participants' final test performance, while self-reports had moderate predicting power. The findings of this study have implications for the cognitive validity of listening tests, listening test design and pedagogical approaches in building listening competence.