Detecting Off-Task Behavior from Student Dialogue in Game-Based Collaborative Learning

Collaborative game-based learning environments integrate game-based learning and collaborative learning. These environments present students with a shared objective and provide them with a means to communicate, which allows them to share information, ask questions, construct explanations, and work together toward their shared goal. A key challenge in collaborative learning is that students may engage in unproductive discourse, which may affect learning activities and outcomes. Collaborative game-based learning environments that can detect this off-task behavior in real-time have the potential to enhance collaboration between students by redirecting the conversation back to more productive topics. This paper investigates the use of dialogue analysis to classify student conversational utterances as either off-task or on-task. Using classroom data collected from 13 groups of four students, we trained off-task dialogue models for text messages from a group chat feature integrated into Crystal Island: EcoJourneys, a collaborative game-based learning environment for middle school ecosystem science. We evaluate the effectiveness of the off-task dialogue models, which use different word embeddings (i.e., word2vec, ELMo, and BERT), as well as predictive off-task dialogue models that capture varying amounts of contextual information from the chat log. Results indicate that predictive off-task dialogue models that incorporate a window of recent context and represent the sequential nature of the chat messages achieve higher predictive performance compared to models that do not leverage this information. These findings suggest that off-task dialogue models for collaborative game-based learning environments can reliably recognize and predict students’ off-task behavior, which introduces the opportunity to adaptively scaffold collaborative dialogue.