Early Detection of Wheel Spinning: Comparison across Tutors, Models, Features, and Operationalizations

"Wheel spinning" is the phenomenon in which a student fails to master a Knowledge Component (KC), despite significant practice. Ideally, an intelligent tutoring system would detect this phenomenon early, so that the system or a teacher could try alternative instructional strategies. Prior work has put forward several criteria for wheel spinning and has demonstrated that wheel spinning can be detected reasonably early. Yet the literature lacks systematic comparisons among the multiple wheel spinning criteria, features, and models that have been proposed, across multiple evaluation criteria (e.g., earliness, precision, and generalizability) and datasets. In our experiments, we constructed six wheel spinning detectors and compared their performance under two different wheel spinning criteria with three datasets. The results show that two prominent criteria for wheel spinning diverge substantially, and that a Random Forest model has the most consistent performance in early detection of wheel spinning across datasets and wheel spinning criteria. In addition, we found that a simple model overlooked by previous research (Logistic Regression trained on a single feature) is able to detect wheel spinning at an early stage with decent performance. This work brings us closer to unifying strands of prior work on wheel spinning (e.g., understanding how different criteria compare) and to early detection of wheel spinning in educational practice. [This paper was published in: Proceedings of the 12th International Conference on Educational Data Mining Montreal, QC, Canada, July 2-5, 2019.]