Search

Recent work describes methods for systematic, data-driven improvement to instructional content and calls for diverse teams of learning engineers to implement and evaluate such improvements. Focusing on an approach called "design-loop adaptivity," we consider the problem of how developers might use data to target or prioritize particular instructional content for improvement processes when faced with large portfolios of content and limited engineering resources to implement improvements. To do so, we consider two data-driven metrics that may capture different facets of how instructional content is "working." The first is a measure of the extent to which learners struggle to master target skills, and the second is a metric based on the difference in prediction performance between deep learning and more "traditional" approaches to knowledge tracing. This second metric may point learning engineers to workspaces that are, effectively, "too easy." We illustrate aspects of the diversity of learning content and variability in learner performance often represented by large educational datasets. We suggest that "monolithic" treatment of such datasets in prediction tasks and other research endeavors may be missing out on important opportunities to drive improved learning within target systems. [For the full proceedings, see ED615472.]

Extensive literature in artificial intelligence in education focuses on developing automated methods for detecting cases in which students struggle to master content while working with educational software. Such cases have often been called "wheel-spinning," "unproductive persistence," or "unproductive struggle." We argue that most existing efforts rely on operationalizations and prediction targets that are misaligned to the approaches of real-world instructional systems. We illustrate facets of misalignment using Carnegie Learning's "MATHia" as a case study, raising important questions being addressed by on-going efforts and for future work. [This paper was published in: I. Bitencourt, M. Cukurova, K. Muldner, R. Luckin, & E. Millan (Eds.), "Proceedings of the 21st International Conference on Artificial Intelligence in Education" (AIED 2020). Lecture Notes in Computer Science (LNCS, Vol. 12164 pp.92-97). Springer, Cham.]