Search

Multimodal data have the potential to explore emerging learning practices that extend human cognitive capacities. A critical issue stretching in many multimodal learning analytics (MLA) systems and studies is the current focus aimed at supporting researchers to model learner behaviours, rather than directly supporting learners. Moreover, many MLA systems are designed and deployed without learners' involvement. We argue that in order to create MLA interfaces that directly support learning, we need to gain an expanded understanding of how multimodal data can support learners' authentic needs. We present a qualitative study in which 40 computer science students were tracked in an authentic learning activity using wearable and static sensors. Our findings outline learners' curated representations about multimodal data and the non‐technical challenges in using these data in their learning practice. The paper discusses 10 dimensions that can serve as guidelines for researchers and designers to create effective and ethically aware student‐facing MLA innovations. Lay Description: What is already known about this topic: Many MLA systems are designed without learners' involvement.Understanding of how multimodal data can support learners' authentic needs is needed.MLA have the potential to enable the automated generation of models to provide real‐time feedback. What this paper adds: Understanding how learners perceive multimodal data and the learning context.The findings outline the soft challenges in using multimodal data in the learning practice.Ten dimensions were identified to serve as guidelines for researchers and designers.The paper reports learners' perspectives describing new, intriguing, and under‐developed ideas about potential uses of multimodal data in educational context.The discussion on the need for human‐centred design approaches for educational technologies has been presented. Implications for practice and/or policy: Relevant pointers for what is appropriate and ethically necessary when designing learning technologies.Change in perspective among educators, on how they address progress and engagement in learning. [ABSTRACT FROM AUTHOR]

Background: Problem‐solving is a multidimensional and dynamic process that requires and interlinks cognitive, metacognitive, and affective dimensions of learning. However, current approaches practiced in computing education research (CER) are not sufficient to capture information beyond the basic programming process data (i.e., IDE‐log data). Therefore, how cognition and affect intertwine and unfold over time in programming problem‐solving activities are rarely investigated. Objectives: In this study, we examined how the theory‐informed measures from multimodal data that we have selected as proxies for cognitive and affective dimensions of learning, are associated with student performance, and in comparison, to prior‐knowledge. Methods: A high‐frequency temporal data was collected with a camera, an electroencephalogram, and an eye‐tracker from 40 computer science students (bachelor and master studies) in the context of a code‐debugging activity. To study the cognitive processes associated with learning we focused on cognitive load theory (CLT) and the human information processing model. In addition, we complemented CLT with the model of affective dynamics in learning to avoid the machine reductionism perspective. Results: Our findings demonstrated that attention, convergent thinking, and frustration were positively correlated with students' successful code‐debugging (i.e., performance), and frequently manifested by high performing participants. Cognitive load, memory load, and boredom were negatively correlated with students' performance, and typically manifested by low performing participants. Implications: Extending the context of analysis in reference to student cognitive processes and affective states, affords educators not just to identify lower performers, but also to understand the potential reasons behind their performance, making our method an important contribution in the confluence of CER and the learning technology communities. In addition, the insights extracted from our analyses allow us to discuss potential avenues for improving learning design and the design of interactive learning systems to support the development of students' problem‐solving skills. Lay Description: What is currently known about the subject matter: ● Problem‐solving interlinks cognitive, affective and metacognitive dimensions of learning. ● In digital settings instructors struggle to distinguish cognitive‐affective states of students. ● Learning technologies require interventions based on cognitive‐affective states of students. What the paper adds: ● When and what cognitive‐affective states are triggered during solving a programming problem. ● What and how measures as proxies for cognitive‐affective states are associated with performance. ● Emphasizes the benefits of augmenting IDE‐log data with sensor data in the context of learning design. Implications of study findings for practitioners: ● Methodology guidelines to identify and model variability in humans' natural behaviours. ● Importance of content and instructions personalization for student engagement and learning. ● Identification of multimodal data proxies for flexible and adaptive multimodal interfaces. [ABSTRACT FROM AUTHOR]