Improved Multimodal Data Acquisition and Synchronization through NLP Enabled Event Detection in Simulation-Based Medical Education

Significant advancements have been made in the field of education due to the introduction of innovative technologies and methodologies. Notably, simulation-based learning has had a profound impact on various learning domains, including Healthcare, Aviation and Aerospace, Military, and Emergency Services, among others. The adoption of Simulation-Based Medical Education (SBME) in healthcare has proven effective for training and evaluating Healthcare Professionals (HCPs). Multimodal data from various levels such as the instructor, learner, and training environment is crucial for a comprehensive assessment of learners within SBME. Currently, these assessments are conducted using either paper-based scales or standard checklists. A platform that provides multimodal assessment capabilities at each of these levels is necessary. This research aims to enhance the data fidelity and availability of a novel multimodal assessment platform (PREPARE) that is used for learner assessment and performance monitoring during training and real-world events. Currently, the platform provides multimodal data acquisition; however, data collected at the instructor and training environment levels is not always synchronized with learner-level data. This research aims to address some of these limitations by incorporating Natural Language Processing (NLP). The goal is to detect the occurrence of key events occurring during training (via processing audio data collected at the training environment level) and to synchronize instructor (observer-based) assessment with learner-level performance data. We also introduce a foundation for automated performance assessment which is intended to measure learner performance that includes derivation of objective performance measures such as time to diagnosis, time to treatment/intervention, etc. The NLP-based module added to this existing platform has the potential to revolutionize the assessment process in SBME, providing more accurate and timely feedback for learners and instructors alike. With this enhanced assessment capability, we aim to improve the effectiveness of SBME and thereby elevate the skills and competencies of Healthcare Professionals. The multimodal dataset made possible with the NLP-based module will provide increased objective performance data which can be used by future machine learning-based models for the platform to personalize training rather than implement the one-size-fits-all training that is currently the standard practice.