Search

From a theoretical viewpoint, educational interfaces that facilitate communicative actions involving representations central to a domain can maximize students’ effort associated with constructing new schemas. In addition, interfaces that minimize working memory demands due to the interface per se, for example by mimicking existing non-digital work practice, can preserve students’ attentional focus on their learning task. In this research, we asked the question: What type of interface input capabilities provide best support for science problem solving in both low- and high-performing students? High school students’ ability to solve a diverse range of biology problems was compared over longitudinal sessions while they used: (1) hardcopy paper and pencil (2) a digital paper and pen interface (3) pen tablet interface, and (4) graphical tablet interface. Post-test evaluations revealed that time to solve problems, meta-cognitive control, solution correctness, and memory all were significantly enhanced when using the digital pen and paper interface, compared with tablet interfaces. The tangible pen and paper interface also was the only alternative that significantly facilitated skill acquisition in low-performing students. Paradoxically, all students nonetheless believed that the tablet interfaces provided best support for their performance, revealing a lack of self-awareness about how to use computational tools to best advantage. Implications are discussed for how pen interfaces can be optimized for future educational purposes, and for establishing technology fluency curricula to improve students’ awareness of the impact of digital tools on their performance.

This paper provides documentation on dataset resources for establishing a new research area called multimodal learning analytics (MMLA). Research on this topic has the potential to transform the future of educational practice and technology, as well as computational techniques for advancing data analytics. The Math Data Corpus includes high-fidelity time-synchronized multimodal data recordings (speech, digital pen, images) on collaborating groups of students as they work together to solve mathematics problems that vary in difficulty level. The Math Data Corpus resources include initial coding of problem segmentation, problem-solving correctness, and representational content of students' writing. These resources are made available to participants in the data-driven grand challenge for the Second International Workshop on Multimodal Learning Analytics. The primary goal of this event is to analyze coherent signal, activity, and lexical patterns that can identify domain expertise and change in domain expertise early, reliably, and objectively, as well as learning-oriented precursors. An additional aim is to build an international research community in the emerging area of multimodal learning analytics by organizing a series of workshops that bring together multidisciplinary scientists to work on MMLA topics.

One aim of multimodal learning analytics is to analyze rich natural communication modalities to identify domain expertise and learning rapidly and reliably. In this research, written and multimodal representations are analyzed from the Math Data Corpus, which involves multimodal data (digital pen, speech, images) on collaborating students as they solve math problems. Findings reveal that in 96-97% of cases the correctness of a group's solution was predictable in advance based on students' written work content. In addition, a linear regression revealed that 65% of the variance in individual students' domain expertise rankings could be accounted for based on their written work content. A multimodal content analysis based on both written and spoken input correctly predicted the dominant domain expert in a group 100% of the time, exceeding unimodal prediction rates. Further analysis revealed a reversal between experts and non-experts in the percentage of time that a match versus mismatch was present between their oral and written answer contributions, with non-experts demonstrating higher mismatches. Implications are discussed for developing reliable multimodal learning analytics systems that incorporate digital pen input to automatically track consolidation of domain expertise.