Test-Taking Engagement in PIAAC. OECD Education Working Papers, No. 133

In this study, we investigated how empirical indicators of test-taking engagement can be defined, empirically validated, and used to describe group differences in the context of the Programme of International Assessment of Adult Competences (PIAAC). The approach was to distinguish between disengaged and engaged response behavior by means of response time thresholds. Constant thresholds of 3000 ms and 5000 ms were considered, as well as item-specific thresholds based on the visual inspection of (bimodal) response time distributions (VI method) and the proportion correct conditional on response time (P+>0% method). Overall, the validity checks comparing the proportion correct of engaged and disengaged response behavior by domain and by item showed that the P+>0% method performed slightly better than the VI method and the methods assuming constant thresholds. The results for Literacy and Numeracy by module revealed that there was an increase from Module 1 to Module 2 in the proportion of disengaged responses, suggesting a drop in test-taking engagement. The investigation of country differences in test-taking engagement by domain using the P+>0% method showed that the proportion of responses classified as disengaged was quite low. For Literacy, the proportion was well below 5% for the majority of countries; in Numeracy, the proportion was even smaller than 1% for almost all countries; while for Problem solving, the proportion of disengaged responses was more than 5% but usually well below 10%. There were significant differences in test-taking engagement between countries; the obtained effect sizes were small to medium. Population differences in test-taking engagement were highly correlated between the three domains, suggesting that test-taking engagement can be conceived as a consistent characteristic. Furthermore, there was a clear negative association between test-taking disengagement and proficiency in Literacy, Numeracy and Problem solving, respectively. Finally, subgroup differences for gender, age, educational attainment, and language proved to be insignificant or very small. Results suggest that males tend to be more disengaged, that disengagement increases with age in Problem solving, with lower educational attainment and when the test language is not the same as a testee's native language. Appended are: (1) Country differences in test-taking engagement; and (2) Subgroup differences in test-taking engagement.