Search

Purpose: Patients with pneumonia often present to the emergency department (ED) and require prompt diagnosis and treatment. Clinical decision support systems for the diagnosis and management of pneumonia are commonly utilized in EDs to improve patient care. The purpose of this study is to investigate whether a deep learning model for detecting radiographic pneumonia and pleural effusions can improve functionality of a clinical decision support system (CDSS) for pneumonia management (ePNa) operating in 20 EDs., Materials and Methods: In this retrospective cohort study, a dataset of 7434 prior chest radiographic studies from 6551 ED patients was used to develop and validate a deep learning model to identify radiographic pneumonia, pleural effusions, and evidence of multilobar pneumonia. Model performance was evaluated against 3 radiologists' adjudicated interpretation and compared with performance of the natural language processing of radiology reports used by ePNa., Results: The deep learning model achieved an area under the receiver operating characteristic curve of 0.833 (95% confidence interval [CI]: 0.795, 0.868) for detecting radiographic pneumonia, 0.939 (95% CI: 0.911, 0.962) for detecting pleural effusions and 0.847 (95% CI: 0.800, 0.890) for identifying multilobar pneumonia. On all 3 tasks, the model achieved higher agreement with the adjudicated radiologist interpretation compared with ePNa., Conclusions: A deep learning model demonstrated higher agreement with radiologists than the ePNa CDSS in detecting radiographic pneumonia and related findings. Incorporating deep learning models into pneumonia CDSS could enhance diagnostic performance and improve pneumonia management., Competing Interests: The authors declare no conflicts of interest., (Copyright © 2021 Wolters Kluwer Health, Inc. All rights reserved.)

A real-time electronic CDS for pneumonia (ePNa) identifies possible pneumonia patients, measures severity and antimicrobial resistance risk, and then recommends disposition, antibiotics, and microbiology studies. Use is voluntary, and clinicians may modify treatment recommendations. ePNa was associated with lower mortality in emergency department (ED) patients versus usual care (Annals EM 66:511). We adapted ePNa for the Cerner EHR, and implemented it across Intermountain Healthcare EDs (Utah, USA) throughout 2018. We introduced ePNa through didactic, interactive presentations to ED clinicians; follow-up visits identified barriers and facilitators to use. Email reminded clinicians and answered questions. Hospital admitting clinicians encouraged ePNa use to smooth care transitions. Audit-and-feedback measured utilization, showing variations from best practice when ePNa and associated electronic order sets were not used. Use was initially low, but gradually increased especially at larger hospitals. A user-friendly interface, frequent reminders, audit-and- feedback, a user survey, a nurse educator, and local physician champions are additive towards implementation success., (©2019 AMIA - All rights reserved.)

Study Objective: Despite evidence that guideline adherence improves clinical outcomes, management of pneumonia patients varies in emergency departments (EDs). We study the effect of a real-time, ED, electronic clinical decision support tool that provides clinicians with guideline-recommended decision support for diagnosis, severity assessment, disposition, and antibiotic selection., Methods: This was a prospective, controlled, quasi-experimental trial in 7 Intermountain Healthcare hospital EDs in Utah's urban corridor. We studied adults with International Classification of Diseases, Ninth Revision codes and radiographic evidence for pneumonia during 2 periods: baseline (December 2009 through November 2010) and post-tool deployment (December 2011 through November 2012). The tool was deployed at 4 intervention EDs in May 2011, leaving 3 as usual care controls. We compared 30-day, all-cause mortality adjusted for illness severity, using a mixed-effect, logistic regression model., Results: The study population comprised 4,758 ED pneumonia patients; 14% had health care-associated pneumonia. Median age was 58 years, 53% were female patients, and 59% were admitted to the hospital. Physicians applied the tool for 62.6% of intervention ED study patients. There was no difference overall in severity-adjusted mortality between intervention and usual care EDs post-tool deployment (odds ratio [OR]=0.69; 95% confidence interval [CI] 0.41 to 1.16). Post hoc analysis showed that patients with community-acquired pneumonia experienced significantly lower mortality (OR=0.53; 95% CI 0.28 to 0.99), whereas mortality was unchanged among patients with health care-associated pneumonia (OR=1.12; 95% CI 0.45 to 2.8). Patient disposition from the ED postdeployment adhered more to tool recommendations., Conclusion: This study demonstrates the feasibility and potential benefit of real-time electronic clinical decision support for ED pneumonia patients., (Copyright © 2015 American College of Emergency Physicians. Published by Elsevier Inc. All rights reserved.)

Purpose: This study aimed to develop Natural Language Processing (NLP) approaches to supplement manual outcome validation, specifically to validate pneumonia cases from chest radiograph reports., Methods: We trained one NLP system, ONYX, using radiograph reports from children and adults that were previously manually reviewed. We then assessed its validity on a test set of 5000 reports. We aimed to substantially decrease manual review, not replace it entirely, and so, we classified reports as follows: (1) consistent with pneumonia; (2) inconsistent with pneumonia; or (3) requiring manual review because of complex features. We developed processes tailored either to optimize accuracy or to minimize manual review. Using logistic regression, we jointly modeled sensitivity and specificity of ONYX in relation to patient age, comorbidity, and care setting. We estimated positive and negative predictive value (PPV and NPV) assuming pneumonia prevalence in the source data., Results: Tailored for accuracy, ONYX identified 25% of reports as requiring manual review (34% of true pneumonias and 18% of non-pneumonias). For the remainder, ONYX's sensitivity was 92% (95% CI 90-93%), specificity 87% (86-88%), PPV 74% (72-76%), and NPV 96% (96-97%). Tailored to minimize manual review, ONYX classified 12% as needing manual review. For the remainder, ONYX had sensitivity 75% (72-77%), specificity 95% (94-96%), PPV 86% (83-88%), and NPV 91% (90-91%)., Conclusions: For pneumonia validation, ONYX can replace almost 90% of manual review while maintaining low to moderate misclassification rates. It can be tailored for different outcomes and study needs and thus warrants exploration in other settings., (Copyright © 2013 John Wiley & Sons, Ltd.)

Objectives: To present a system that uses knowledge stored in a medical ontology to automate the development of diagnostic decision support systems. To illustrate its function through an example focused on the development of a tool for diagnosing pneumonia., Materials and Methods: We developed a system that automates the creation of diagnostic decision-support applications. It relies on a medical ontology to direct the acquisition of clinic data from a clinical data warehouse and uses an automated analytic system to apply a sequence of machine learning algorithms that create applications for diagnostic screening. We refer to this system as the ontology-driven diagnostic modeling system (ODMS). We tested this system using samples of patient data collected in Salt Lake City emergency rooms and stored in Intermountain Healthcare's enterprise data warehouse., Results: The system was used in the preliminary development steps of a tool to identify patients with pneumonia in the emergency department. This tool was compared with a manually created diagnostic tool derived from a curated dataset. The manually created tool is currently in clinical use. The automatically created tool had an area under the receiver operating characteristic curve of 0.920 (95% CI 0.916 to 0.924), compared with 0.944 (95% CI 0.942 to 0.947) for the manually created tool., Discussion: Initial testing of the ODMS demonstrates promising accuracy for the highly automated results and illustrates the route to model improvement., Conclusions: The use of medical knowledge, embedded in ontologies, to direct the initial development of diagnostic computing systems appears feasible.

The goal of this study was to determine the accuracy and the impact of 5 different claims-based pneumonia definitions. Three International Classification of Diseases, Version 9, (ICD-9), and 2 diagnosis-related group (DRG)-based case identification algorithms were compared against an independent, clinical pneumonia reference standard. Among 10748 patients, 272 (2.5%) had pneumonia verified by the reference standard. The sensitivity of claims-based algorithms ranged from 47.8% to 66.2%. The positive predictive values ranged from 72.6% to 80.8%. Patient-related variables were not significantly different from the reference standard among the 3 ICD-9-based algorithms. DRG-based algorithms had significantly lower hospital admission rates (57% and 65% vs 73.2%), lower 30-day mortality (5.0% and 5.8% vs 10.7%), shorter length of stay (3.9 and 4.1 days vs 5.6 days), and lower costs (USD $4543 and USD $5159 vs USD $8585). Claims-based identification algorithms for defining pneumonia in administrative databases are imprecise. ICD-9-based algorithms did not influence patient variables in our population. Identifying pneumonia patients with DRG codes is significantly less precise.

Planning the clinical evaluation of a computerized decision support system requires a strategy that encompasses the different aspects of the clinical problem, the technical difficulties of software and hardware integration and implementation, the behavioral aspects of the targeted users, and the discipline of study design. Although clinical information systems are becoming more widely available, only a few decision support systems have been formally evaluated in clinical environments. Published accounts of difficulties associated with the clinical evaluation of decision support systems remain scarce. The authors report on a variety of behavioral, logistical, technical, clinical, cost, and work flow issues that they had to address when choosing a study design for a clinical trial for the evaluation of an integrated, real-time decision support system for the automatic identification of patients likely to have pneumonia in an emergency department. In the absence of a true gold standard, they show how they created a credible, clinically acceptable, and economical reference standard for the diagnosis of pneumonia, to determine the overall accuracy of the system. For the creation of a reference standard, they describe the importance of recognizing verification bias and avoiding it. Finally, advantages and disadvantages of different study designs are explored with respect to the targeted users and the clinical setting.

Objective: To evaluate the performance of a computerized decision support system that combines two different decision support methodologies (a Bayesian network and a natural language understanding system) for the diagnosis of patients with pneumonia., Design: Evaluation study using data from a prospective, clinical study., Patients: All patients 18 years and older who presented to the emergency department of a tertiary care setting and whose chest x-ray report was available during the encounter., Methods: The computerized decision support system calculated a probability of pneumonia using information provided by the two systems. Outcome measures were the area under the receiver operating characteristic curve, sensitivity, specificity, predictive values, likelihood ratios, and test effectiveness., Results: During the 3-month study period there were 742 patients (45 with pneumonia). The area under the receiver operating characteristic curve was 0.881 (95% CI: 0.822, 0.925) for the Bayesian network alone and 0.916 (95% CI: 0.869, 0.949) for the Bayesian network combined with the natural language understanding system (p=0.01)., Conclusion: Combining decision support methodologies that process information stored in different data formats can increase the performance of a computerized decision support system.

Background: In busy clinical settings, physicians often do not have enough time to identify patients for specific therapeutic guidelines. As a solution, decision support systems could automatically identify eligible patients and trigger computerized guidelines for specific diseases. Applying this idea to community-acquired pneumonia (CAP), we developed a Bayesian network (BN) and an artificial neural network (ANN) for identifying patients who have CAP and are eligible for a pneumonia guideline., Objective: The aim of this study was to determine whether the diagnostic accuracy of these two decision support models differs in terms of identifying CAP patients., Methods: We trained and tested the networks with a data set of 32,662 adult patients. For each network, we (1) calculated the specificity, the positive predictive value (PPV), and the negative predictive value (NPV) at a sensitivity of 95%, and (2) determined the area under the receiver operating characteristic curve (AUC) as a measure of overall accuracy. We tested for statistical difference between the AUCs using the correlated area z statistic., Results: At a sensitivity of 95%, the respective values for specificity, PPV, and NPV were: 92.3%, 15.1%, and 99.9% for the BN, and 94.0%, 18.6%, and 99.9% for the ANN. The BN had an AUC of 0.9795 (95% CI: 0.9736, 0.9843), and the ANN had an AUC of 0.9855 (95% CI: 0.9805, 0.9894). The difference between the AUCs was statistically significant (p=0.0044)., Conclusions: The networks achieved high overall accuracies on the testing data set. Because the difference in accuracies is statistically significant but not clinically significant, both networks are equally suited to drive a guideline.

Objective: To evaluate if a medical language processing (MLP) system is able to support real-time computerization of community-acquired pneumonia (CAP) guidelines., Methods: Prospective validation study in the emergency department of a tertiary care facility. All the chest x-ray reports available in real-time for an admission decision during a five-week period were included. The MLP system was compared to a physician for the automatic selection of eligible patients and on the extraction of radiographic findings required by five different CAP guidelines. The gold standard comprised of three independent physicians and reliability measures were calculated. The outcome measures were the area under the receiver operated characteristic curve (AUC) for selecting eligible patients, sensitivity, positive predictive value (PPV), and specificity for the extraction of radiographic findings., Results: During the five-week period, 243 reports were available in real-time. The AUCs on selecting eligible CAP patients were 89.7% (CI: 84.2%, 93.7%) for the MLP system, and 93.3% (CI: 83.9%, 97.8%) for the physician. The average sensitivity, PPV, and specificity for radiographic findings that assessed localization and extension of CAP were respectively: 94%, 87%, 96% (physician); and 34%, 90%, 95% (MLP system). Both, the MLP system and the physician had average sensitivity, PPV, and specificity of 97%, 97%, and 99%, respectively, when localization was not an issue. Reliability measures for the gold standard were above 70%., Conclusion: The MLP system was able to support real-time computerization of guidelines by selecting eligible patients and extracting radiographic findings that do not assess localization and extension of CAP.

Objective: To assess the ability of an integrated, real-time diagnostic system (Bayesian network) to identify patients with community-acquired pneumonia who are eligible for a computerized pneumonia guideline without requiring clinicians to enter additional data., Design: Prospective validation study., Patients: All patients 18 years and older who presented to the emergency department of a tertiary care hospital., Methods: The diagnostic system computed a probability of pneumonia for every patient. The final diagnosis was established using ICD-9 discharge diagnoses. Outcome measures were sensitivity, specificity, predictive values, likelihood ratios, area under the receiver operating characteristic curve, and test effectiveness., Results: During the 9-week study period there were 4,361 patients (112 pneumonia patients). The area under the receiver operating characteristic curve was 0.930 (CI: 0.907, 0.948). At a fixed sensitivity of 95%, the specificity was 68.5%, the positive predictive value 7.3%, the negative predictive value 99.8%, the positive likelihood ratio 3.0, the negative likelihood ratio 0.08, and the test effectiveness 2.05., Conclusion: The diagnostic system was able to detect patients who are eligible for a pneumonia guideline. The detection of eligible patients can be applied to automatically initiate and evaluate computerized guidelines.

Objective: This study examined whether clinical data routinely available in a computerized patient record (CPR) can be used to drive a complex guideline that supports physicians in real time and at the point of care in assessing the risk of mortality for patients with community-acquired pneumonia., Setting: Emergency department of a tertiary-care hospital., Design: Retrospective analysis with medical chart review., Patients: All 241 inpatients during a 17-month period (Jun 1995 to Nov 1996) who presented to the emergency department and had a primary discharge diagnosis of community-acquired pneumonia. METHODS/MAIN OUTCOME MEASURES: The 20 guideline variables were extracted from the CPR (HELP System) and the paper chart. The risk score and the risk class of the Pneumonia Severity Index were computed using data from the CPR alone and from a reference standard of all data available in the paper chart and the CPR at the time of the emergency department encounters. Availability and concordance were quantified to determine data quality. The type and cause of errors were analyzed depending on the source and format of the clinical variables., Results: Of the 20 guideline variables, 12 variables were required to be present for every computer-charted emergency department patient, seven variables were required for selected patients only, and one variable was not typically available in the HELP System during a patient's encounter. The risk class was identical for 86.7 percent of the patients. The majority of patients with different risk classes were assigned too low a risk class. The risk scores were identical for 72.1 percent of the patients. The average availability was 0.99 for the data elements that were required to be present and 0.79 for the data elements that were not required to be present. The average concordance was 0.98 when all a patient's variables were taken into account. The cause of error was attributed to the nurse charting in 77 percent of the cases and to the computerized evaluation in 23 percent. The type of error originated from the free-text fields in 64 percent, from coded fields in 21 percent, from vital signs in 14 percent, and from laboratory results in 1 percent., Conclusion: From a clinical perspective, the current level of data quality in the HELP System supports the automation and the prospective evaluation of the Pneumonia Severity Index as a computerized decision support tool.

Objective: Evaluate the effect of a radiology speech recognition system on a real-time computerized guideline in the emergency department., Methods: We collected all chest x-ray reports (n = 727) generated for patients in the emergency department during a six-week period. We divided the concurrently generated reports into those generated with speech recognition and those generated by traditional dictation. We compared the two sets of reports for availability during the patient's emergency department encounter and for readability., Results: Reports generated by speech recognition were available seven times more often during the patients' encounters than reports generated by traditional dictation. Using speech recognition reduced the turnover time of reports from 12 hours 33 minutes to 2 hours 13 minutes. Readability scores were identical for both kinds of reports., Conclusion: Using speech recognition to generate chest x-ray reports reduces turnover time so reports are available while patients are in the emergency department.

A medical language processing system called SymText, two other automated methods, and a lay person were compared against an internal medicine resident for their ability to identify pneumonia related concepts on chest x-ray reports. Sensitivity (recall), specificity, and positive predictive value (precision) are reported with respect to an independent panel of physicians. Overall the performance of SymText was similar to the physician and superior to the other methods. The automatic encoding of pneumonia concepts will support clinical research, decision making, computerized clinical protocols, and quality assurance in a radiology department.

Decision support systems that integrate guidelines have become popular applications to reduce variation and deliver cost-effective care. However, adverse characteristics of decision support systems, such as additional and time-consuming data entry or manually identifying eligible patients, result in a "behavioral bottleneck" that prevents decision support systems to become part of the clinical routine. This paper describes the design and the implementation of an integrated decision support system that explores a novel approach for bypassing the behavioral bottleneck. The real-time decision support system does not require health care providers to enter additional data and consists of a diagnostic and a management component.

We present the development and the evaluation of a Bayesian network for the diagnosis of community-acquired pneumonia. The Bayesian network is intended to be part of a larger decision support system which assists emergency room physicians in the management of pneumonia patients. Minimal data entry from the nurse or the physician, timely availability of clinical parameters, and high accuracy were requirements we tried to meet. Data from more than 32,000 emergency room patients over a period of 2 years (June 1995-June 1997) were extracted from the clinical information system to train and test the Bayesian network. The network performed well in discriminating patients with pneumonia from patients with other diseases. The Bayesian network achieved a sensitivity of 95%, a specificity of 96.5%, an area under the receiver operating characteristic of 0.98, and a predictive value positive of 26.8%. Our feasibility study demonstrates that the proposed Bayesian network is an appropriate method to detect pneumonia patients with high accuracy. The study suggests that the proposed Bayesian network may represent a successful component within a larger decision support system for the management of community-acquired pneumonia.