Recognitions of Image and Speech to Improve Learning Diagnosis on STEM Collaborative Activity for Precision Education

The rise of precision education has encouraged teachers to use intelligent diagnostic systems to understand students' learning processes and provide immediate guidance to prevent students from giving up when facing learning difficulties. However, current research on precision education rarely employs multimodal learning analytics approaches to understand students' learning behaviors. Therefore, this study aims to investigate the impact of teachers intervene based on different modalities of learning analytics diagnosing systems on students' learning behaviors, learning performance, and motivation in STEM collaborative learning activities. We conducted a quasi-experiment with three groups: a control group without any learning analytics system assistance, experimental group 1 with a unimodal learning analytics approach based on image data, and experimental group 2 with a multimodal learning analytics approach based on both image and voice data. We collected students' image or voice data according to the experimental design and employed artificial intelligence techniques for facial expression recognition, eye gaze tracking, and speech recognition to identify students' learning behaviors. The results of this research indicate that teacher interventions, augmented by learning analytics systems, have a significant positive impact on student learning outcomes and motivation. In experimental group 2, the acquisition of multimodal data facilitated a more precise identification and addressing of student learning challenges. Relative to the control group, students in the experimental groups exhibited heightened self-efficacy and were more motivated learners. Moreover, students in experimental group 2 demonstrated a deeper level of engagement in collaborative processes and the behavior associated with constructing knowledge.