Back to Search Start Over

Detecting and Mitigating Encoded Bias in Deep Learning-Based Stealth Assessment Models for Reflection-Enriched Game-Based Learning Environments

Authors :
Anisha Gupta
Dan Carpenter
Wookhee Min
Jonathan Rowe
Roger Azevedo
James Lester
Source :
International Journal of Artificial Intelligence in Education. 2024 34(3):1138-1165.
Publication Year :
2024

Abstract

Reflection plays a critical role in learning. Game-based learning environments have significant potential to elicit and support student reflection by prompting learners to think critically about their own learning processes and performance. Stealth assessment models, used for unobtrusively assessing student competencies from evidence of game interaction data and facilitating learning through adaptive feedback, can be enhanced by incorporating evidence from students' written reflections. We present a deep learning-based stealth assessment framework that predicts depth of student reflections and science content post-test scores during game-based learning. With the increasing adoption of AI techniques in decision-making processes, it is important to evaluate the fairness of these models. To address this concern, we investigate encoded bias in our stealth assessment model with respect to student gender and prior game-playing experience in deep learning-based stealth assessment models and examine the impact of debiasing on the models' predictive performance. We evaluate the predictive performance of the deep learning-based stealth assessment models and measure encoded bias with the Absolute Between-ROC Area (ABROCA) statistic using gameplay data from 119 students collected in a series of classroom studies with a reflection-enriched game-based learning environment for middle school microbiology, Crystal Island. The results demonstrate the effectiveness of deep learning-based stealth assessment models and multiple debiasing techniques for deriving algorithmically fair stealth assessment models.

Details

Language :
English
ISSN :
1560-4292 and 1560-4306
Volume :
34
Issue :
3
Database :
ERIC
Journal :
International Journal of Artificial Intelligence in Education
Publication Type :
Academic Journal
Accession number :
EJ1446562
Document Type :
Journal Articles<br />Reports - Evaluative
Full Text :
https://doi.org/10.1007/s40593-023-00379-6