Search

Objectives: Natural language processing (NLP) represents one of the adjunct technologies within artificial intelligence and machine learning, creating structure out of unstructured data. This study aims to assess the performance of employing NLP to identify and categorize unstructured data within the emergency medicine (EM) setting. Methods: We systematically searched publications related to EM research and NLP across databases including MEDLINE, Embase, Scopus, CENTRAL, and ProQuest Dissertations & Theses Global. Independent reviewers screened, reviewed, and evaluated article quality and bias. NLP usage was categorized into syndromic surveillance, radiologic interpretation, and identification of specific diseases/events/syndromes, with respective sensitivity analysis reported. Performance metrics for NLP usage were calculated and the overall area under the summary of receiver operating characteristic curve (SROC) was determined. Results: A total of 27 studies underwent meta‐analysis. Findings indicated an overall mean sensitivity (recall) of 82%–87%, specificity of 95%, with the area under the SROC at 0.96 (95% CI 0.94–0.98). Optimal performance using NLP was observed in radiologic interpretation, demonstrating an overall mean sensitivity of 93% and specificity of 96%. Conclusions: Our analysis revealed a generally favorable performance accuracy in using NLP within EM research, particularly in the realm of radiologic interpretation. Consequently, we advocate for the adoption of NLP‐based research to augment EM health care management. [ABSTRACT FROM AUTHOR]

Introduction: Emergency medicine (EM) milestones are used to assess residents’ progress. While some milestone validity evidence exists, there is a lack of standardized tools available to reliably assess residents. Inherent to this is a concern that we may not be truly measuring what we intend to assess. The purpose of this study was to design a direct observation milestone assessment instrument supported by validity and reliability evidence. In addition, such a tool would further lend validity evidence to the EM milestones by demonstrating their accurate measurement.Methods: This was a multi-center, prospective, observational validity study conducted at eight institutions. The Critical Care Direct Observation Tool (CDOT) was created to assess EM residents during resuscitations. This tool was designed using a modified Delphi method focused on content, response process, and internal structure validity. Paying special attention to content validity, the CDOT was developed by an expert panel, maintaining the use of the EM milestone wording. We built response process and internal consistency by piloting and revising the instrument. Raters were faculty who routinely assess residents on the milestones. A brief training video on utilization of the instrument was completed by all. Raters used the CDOT to assess simulated videos of three residents at different stages of training in a critical care scenario. We measured reliability using Fleiss’ kappa and interclass correlations.Results: Two versions of the CDOT were used: one used the milestone levels as global rating scales with anchors, and the second reflected a current trend of a checklist response system. Although the raters who used the CDOT routinely rate residents in their practice, they did not score the residents’ performances in the videos comparably, which led to poor reliability. The Fleiss’ kappa of each of the items measured on both versions of the CDOT was near zero.Conclusion: The validity and reliability of the current EM milestone assessment tools have yet to be determined. This study is a rigorous attempt to collect validity evidence in the development of a direct observation assessment instrument. However, despite strict attention to validity evidence, inter-rater reliability was low. The potential sources of reducible variance include rater- and instrument-based error. Based on this study, there may be concerns for the reliability of other EM milestone assessment tools that are currently in use.

Background: Recognition of the provider's name, provider empathy, and the patient's satisfaction with their care are patient-provider rapport measures. This study aimed to determine: 1) resident physicians' name recognition by patients in the emergency department; and 2) name recognition in association with patient perception of the resident's empathy and their satisfaction with the resident's care., Methods: This was a prospective observational study. A patient recognizing a resident physician was defined as the patient remembering a resident's name, understanding the level of training, and understanding a resident's role in patient care. A patient's perception of resident physician empathy was measured by the Jefferson Scale of Patient Perception of Physician Empathy (JSPPPE). Patient satisfaction of the resident was measured utilizing a real-time satisfaction survey. Multivariate logistic regressions were performed to determine the association amongst patient recognition of resident physicians, JSPPPE, and patient satisfaction after adjustments were made for demographics and resident training level., Results: We enrolled 30 emergency medicine resident physicians and 191 patients. Only 26% of studied patients recognized resident physicians. High JSPPPE scores were given by 39% of patients recognizing resident physicians compared to 5% of those who were not recognized (P = 0.013). High patient satisfaction scores were recorded in 31% of patients who recognized resident physicians compared to 7% who did not (P = 0.008). The adjusted odds ratios of patient recognition of resident physicians to high JSPPPE and high satisfaction scores were 5.29 (95% confidence interval (CI): 1.33 - 21.02, P = 0.018) and 6.12 (1.84 - 20.38, P = 0.003) respectively., Conclusions: Patient recognition of resident physicians is low in our study. However, patient recognition of resident physicians is associated with a higher patient perception of physician empathy and higher patient satisfaction. Our study suggests that resident education advocating for patient recognition of their healthcare provider's status needs to be emphasized as part of patient-centered health care., Competing Interests: All authors report no conflict of interest., (Copyright 2023, Holmes et al.)