Search

Abstract Comprehending digital content written in natural language online is vital for many aspects of life, including learning, professional tasks, and decision-making. However, facing comprehension difficulties can have negative consequences for learning outcomes, critical thinking skills, decision-making, error rate, and productivity. This paper introduces an innovative approach to predict comprehension difficulties at the local content level (e.g., paragraphs). Using affordable wearable devices, we acquire physiological responses non-intrusively from the autonomous nervous system, specifically pulse rate variability, and electrodermal activity. Additionally, we integrate data from a cost-effective eye-tracker. Our machine learning algorithms identify ’hotspots’ within the content and regions corresponding to a high cognitive load. These hotspots represent real-time predictors of comprehension difficulties. By integrating physiological data with contextual information (such as the levels of experience of individuals), our approach achieves an accuracy of 72.11% ± 2.21, a precision of 0.77, a recall of 0.70, and an f1 score of 0.73. This study opens possibilities for developing intelligent, cognitive-aware interfaces. Such interfaces can provide immediate contextual support, mitigating comprehension challenges within content. Whether through translation, content generation, or content summarization using available Large Language Models, this approach has the potential to enhance language comprehension.

Cognitive human error and recent cognitive taxonomy on human error causes of software defects support the intuitive idea that, for instance, mental overload, attention slips, and working memory overload are important human causes for software bugs. In this paper, we approach the EEG as a reliable surrogate to MRI-based reference of the programmer's cognitive state to be used in situations where heavy imaging techniques are infeasible. The idea is to use EEG biomarkers to validate other less intrusive physiological measures, that can be easily recorded by wearable devices and useful in the assessment of the developer's cognitive state during software development tasks. Herein, our EEG study, with the support of fMRI, presents an extensive and systematic analysis by inspecting metrics and extracting relevant information about the most robust features, best EEG channels and the best hemodynamic time delay in the context of software development tasks. From the EEG-fMRI similarity analysis performed, we found significant correlations between a subset of EEG features and the Insula region of the brain, which has been reported as a region highly related to high cognitive tasks, such as software development tasks. We concluded that despite a clear inter-subject variability of the best EEG features and hemodynamic time delay used, the most robust and predominant EEG features, across all the subjects, are related to the Hjorth parameter Activity and Total Power features, from the EEG channels F4, FC4 and C4, and considering in most of the cases a hemodynamic time delay of 4 seconds used on the hemodynamic response function. These findings should be taken into account in future EEG-fMRI studies in the context of software debugging.