1. Multi-center validation of an artificial intelligence system for detection of COVID-19 on chest radiographs in symptomatic patients
- Author
-
Michael D. Kuo, Keith W. H. Chiu, David S. Wang, Anna Rita Larici, Dmytro Poplavskiy, Adele Valentini, Alessandro Napoli, Andrea Borghesi, Guido Ligabue, Xin Hao B. Fang, Hing Ki C. Wong, Sailong Zhang, John R. Hunter, Abeer Mousa, Amato Infante, Lorenzo Elia, Salvatore Golemi, Leung Ho P. Yu, Christopher K. M. Hui, and Bradley J. Erickson
- Subjects
Artificial intelligence ,Public health ,Deep Learning ,Thoracic ,Humans ,COVID-19 ,Radiology ,Radiography, Thoracic ,Radiology, Nuclear Medicine and imaging ,General Medicine ,Tomography, X-Ray Computed ,Retrospective Studies - Abstract
While chest radiograph (CXR) is the first-line imaging investigation in patients with respiratory symptoms, differentiating COVID-19 from other respiratory infections on CXR remains challenging. We developed and validated an AI system for COVID-19 detection on presenting CXR.A deep learning model (RadGenX), trained on 168,850 CXRs, was validated on a large international test set of presenting CXRs of symptomatic patients from 9 study sites (US, Italy, and Hong Kong SAR) and 2 public datasets from the US and Europe. Performance was measured by area under the receiver operator characteristic curve (AUC). Bootstrapped simulations were performed to assess performance across a range of potential COVID-19 disease prevalence values (3.33 to 33.3%). Comparison against international radiologists was performed on an independent test set of 852 cases.RadGenX achieved an AUC of 0.89 on 4-fold cross-validation and an AUC of 0.79 (95%CI 0.78-0.80) on an independent test cohort of 5,894 patients. Delong's test showed statistical differences in model performance across patients from different regions (p0.01), disease severity (p0.001), gender (p0.001), and age (p = 0.03). Prevalence simulations showed the negative predictive value increases from 86.1% at 33.3% prevalence, to greater than 98.5% at any prevalence below 4.5%. Compared with radiologists, McNemar's test showed the model has higher sensitivity (p0.001) but lower specificity (p0.001).An AI model that predicts COVID-19 infection on CXR in symptomatic patients was validated on a large international cohort providing valuable context on testing and performance expectations for AI systems that perform COVID-19 prediction on CXR.• An AI model developed using CXRs to detect COVID-19 was validated in a large multi-center cohort of 5,894 patients from 9 prospectively recruited sites and 2 public datasets. • Differences in AI model performance were seen across region, disease severity, gender, and age. • Prevalence simulations on the international test set demonstrate the model's NPV is greater than 98.5% at any prevalence below 4.5%.
- Published
- 2023