Human Expert Labeling Process (HELP): Towards a Reliable Higher-Order User State Labeling Process and Tool to Assess Student Engagement

In a series of longitudinal research studies, researchers at Intel Corporation in Turkey have been working towards an adaptive learning system automatically detecting student engagement as a higher-order user state in real-time. The labeled data necessary for supervised learning can be obtained through labeling conducted by human experts. Using multiple labelers to label collected data and obtaining agreement among different labelers on the same samples of data, it is critical to train all to use the engagement model accurately. Addressing these challenges, the researchers developed a rigorous human expert labeling process (HELP) specific to the educational context, with multi-faceted labels and multiple expert labelers. HELP has three distinct stages: (1) "Pre-Labeling", including planning, labeler recruitment, training, and evaluation steps; (2) "Labeling", involving actual labeling conducted by multiple labelers, and related steps for formative assessment of their performance; and (3) "Post-Labeling", generating final labels and agreement measures through processing multiple decisions. In this article, the researchers outline proposed methods in HELP and describe the developed labeling tool.