Performance of machine learning versus the national early warning score for predicting patient deterioration risk: a single-site study of emergency admissions.

Objectives: Increasing operational pressures on emergency departments (ED) make it imperative to quickly and accurately identify patients requiring urgent clinical intervention. The widespread adoption of electronic health records (EHR) makes rich feature patient data sets more readily available. These large data stores lend themselves to use in modern machine learning (ML) models. This paper investigates the use of transformer-based models to identify critical deterioration in unplanned ED admissions, using free-text fields, such as triage notes, and tabular data, including early warning scores (EWS).
Design: A retrospective ML study.
Setting: A large ED in a UK university teaching hospital.
Methods: We extracted rich feature sets of routine clinical data from the EHR and systematically measured the performance of tree- and transformer-based models for predicting patient mortality or admission to critical care within 24 hours of presentation to ED. We compared our proposed models to the National EWS (NEWS).
Results: Models were trained on 174 393 admission records. We found that models including free-text triage notes outperform structured tabular data models, achieving an average precision of 0.92, compared with 0.75 for tree-based models and 0.12 for NEWS.
Conclusions: Our findings suggests that machine learning models using free-text data have the potential to improve clinical decision-making in the ED; our techniques significantly reduce alert rate while detecting most high-risk patients missed by NEWS.
Competing Interests: Competing interests: All authors have completed the Unified Competing Interest form and declare: MW, SBL, DG, MH and NAM report support from an HDR UK and NIHR Winter Pressures grant for this project. MW, PC and NAM also report support from an Innovate UK grant. PC reports research funding from Gilead and Pfizer that is unrelated to this research. MH reports three separate honoraria/payments from the Society for Acute Medicine, Welsh Acute Physicians Society and Doctors.NET for invited talks and/or educational packages on the National Early Warning Score. PC reports honoraria from GSK, unrelated to this research. PC reports support for attending educational meetings from Gilead, unrelated to this research. NAM is employed by Evergreen Life Ltd. All authors report no other financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.
(© Author(s) (or their employer(s)) 2024. Re-use permitted under CC BY. Published by BMJ.)