Gibson E, Georgescu B, Ceccaldi P, Trigan PH, Yoo Y, Das J, Re TJ, Rs V, Balachandran A, Eibenberger E, Chekkoury A, Brehm B, Bodanapally UK, Nicolaou S, Sanelli PC, Schroeppel TJ, Flohr T, Comaniciu D, and Lui YW
Purpose: To present a method that automatically detects, subtypes, and locates acute or subacute intracranial hemorrhage (ICH) on noncontrast CT (NCCT) head scans; generates detection confidence scores to identify high-confidence data subsets with higher accuracy; and improves radiology worklist prioritization. Such scores may enable clinicians to better use artificial intelligence (AI) tools., Materials and Methods: This retrospective study included 46 057 studies from seven "internal" centers for development (training, architecture selection, hyperparameter tuning, and operating-point calibration; n = 25 946) and evaluation ( n = 2947) and three "external" centers for calibration ( n = 400) and evaluation ( n = 16 764). Internal centers contributed developmental data, whereas external centers did not. Deep neural networks predicted the presence of ICH and subtypes (intraparenchymal, intraventricular, subarachnoid, subdural, and/or epidural hemorrhage) and segmentations per case. Two ICH confidence scores are discussed: a calibrated classifier entropy score and a Dempster-Shafer score. Evaluation was completed by using receiver operating characteristic curve analysis and report turnaround time (RTAT) modeling on the evaluation set and on confidence score-defined subsets using bootstrapping., Results: The areas under the receiver operating characteristic curve for ICH were 0.97 (0.97, 0.98) and 0.95 (0.94, 0.95) on internal and external center data, respectively. On 80% of the data stratified by calibrated classifier and Dempster-Shafer scores, the system improved the Youden indexes, increasing them from 0.84 to 0.93 (calibrated classifier) and from 0.84 to 0.92 (Dempster-Shafer) for internal centers and increasing them from 0.78 to 0.88 (calibrated classifier) and from 0.78 to 0.89 (Dempster-Shafer) for external centers ( P < .001). Models estimated shorter RTAT for AI-prioritized worklists with confidence measures than for AI-prioritized worklists without confidence measures, shortening RTAT by 27% (calibrated classifier) and 27% (Dempster-Shafer) for internal centers and shortening RTAT by 25% (calibrated classifier) and 27% (Dempster-Shafer) for external centers ( P < .001)., Conclusion: AI that provided statistical confidence measures for ICH detection on NCCT scans reliably detected and subtyped hemorrhages, identified high-confidence predictions, and improved worklist prioritization in simulation. Keywords: CT, Head/Neck, Hemorrhage, Convolutional Neural Network (CNN) Supplemental material is available for this article . © RSNA, 2022., Competing Interests: Disclosures of conflicts of interest: E.G. Employed by Siemens Medical Solutions USA; inventor on related patent filing (to be held by Siemens Medical Solutions USA if granted); stock held in Siemens Healthineers. B.G. Employed by Siemens Healthineers. P.C. Employment at Siemens Healthineers, stock or stock options in Siemens Healthineers. P.H.T. Previously employed by Siemens Healthineers. Y.Y. Relevant patent filed December 3, 2021 (filing no. 2020E25048 US). J.D. Employed by Siemens Healthineers. T.J.R. Consultant fees from Siemens Healthineers, via intermediary Randstad North America (consulting payroll intermediary); support from Siemens Healthineers, Princeton, NJ for attending Radiological Society of North America Annual Meeting December 2018; patents planned, issued, or pending as follows: 16/270,918 (filed November 24, 2020), 21168065.7 (pending), 202110399157.6 (pending), 21180032.1 (pending), 16/946,435 (pending), 16/837,979 (pending), 17/249,783 (pending), 201910491911.1 (pending), 202110346295.8 (pending), 21165531.1 (pending), 16/356,086 (pending), 17/211,927 (pending), 19163699.2, (pending), 16/865,266 (pending), 19177891.9 (pending). V.R.S. Employed by Siemens Healthineers. A.B. Employed by Siemens Healthcare, India as Research & Technology Manager (radiologist by profession). E.E. Employed by Siemens Healthcare; patents planned, issued, or pending. A.C. Employed by Siemens Healthcare; patents planned, issued, or pending; stock/stock options. B.B. Previously employed by Siemens Healthineers. U.K.B. Support from Siemens; has grant. S.N. No relevant relationships. P.C.S. Grants from National Institutes of Health National Institute of Neurological Disorders and Stroke, Siemens Healthineers, and Neiman Health Policy Institute to author’s institution; leadership or fiduciary role in American Society of Neuroradiology. T.J.S. No relevant relationships. T.F. Employed by Siemens Healthcare. D.C. Employed by Siemens Healthineers; stock/stock options in Siemens Healthineers. Y.W.L. Support from Siemens Healthineers (institution received reimbursement for costs from institutional review board and IT related to data collection, clinical research support unit, and a finance and administration surcharge relating to this work)., (© 2022 by the Radiological Society of North America, Inc.)