Unraveling Students' Interaction around a Tangible Interface Using Multimodal Learning Analytics

In this paper, we describe multimodal learning analytics (MMLA) techniques to analyze data collected around an interactive learning environment. In a previous study (Schneider & Blikstein, submitted), we designed and evaluated a Tangible User Interface (TUI) where dyads of students were asked to learn about the human hearing system by reconstructing it. In the current study, we present the analysis of the data collected in the form of logs, both from students' interaction with the tangible interface and as well as from their gestures, and we describe how we extracted meaningful predictors for student learning from these two datasets. First we show how Natural Language Processing (NLP) techniques can be used on the tangible interface logs to predict learning gains. Second, we explored how Kinect™ data can inform "in-situ" interactions around a tabletop by using clustering algorithms to find prototypical body positions. Finally, we fed those features to a machine-learning classifier (Support Vector Machine) and divided students in two groups after performing a median split on their learning scores. We found that we were able to predict students' learning gains (i.e. being above or belong the median split) with very high accuracy. We discuss the implications of these results for analyzing rich data from multimodal learning environments.