Back to Search
Start Over
Validation of natural language processing to determine the presence and size of abdominal aortic aneurysms in a large integrated health system
- Source :
- Journal of vascular surgery. 74(2)
- Publication Year :
- 2020
-
Abstract
- Objective Previous studies of the natural history of abdominal aortic aneurysms (AAAs) have been limited by small cohort sizes or heterogeneous analyses of pooled data. By quickly and efficiently extracting imaging data from the health records, natural language processing (NLP) has the potential to substantially improve how we study and care for patients with AAAs. The aim of the present study was to test the ability of an NLP tool to accurately identify the presence or absence of AAAs and detect the maximal abdominal aortic diameter in a large dataset of imaging study reports. Methods Relevant imaging study reports (n = 230,660) from 2003 to 2017 were obtained for 32,778 patients followed up in a prospective aneurysm surveillance registry within a large, diverse, integrated healthcare system. A commercially available NLP algorithm was used to assess the presence of AAAs, confirm the absence of AAAs, and extract the maximal diameter of the abdominal aorta, if stated. A blinded expert manual review of 18,000 randomly selected imaging reports was used as the reference standard. The positive predictive value (PPV or precision), sensitivity (recall), and the kappa statistics were calculated. Results Of the randomly selected 18,000 studies that underwent expert manual review, 48.7% were positive for AAAs. In confirming the presence of an AAA, the interrater reliability of the NLP compared with the expert review showed a kappa value of 0.84 (95% confidence interval [CI], 0.83-0.85), with a PPV of 95% and sensitivity of 88.5%. The NLP algorithm showed similar results for confirming the absence of an AAA, with a kappa of 0.79 (95% CI, 0.799-0.80), PPV of 77.7%, and sensitivity of 91.9%. The kappa, PPV, and sensitivity of the NLP for correctly identifying the maximal aortic diameter was 0.88 (95% CI, 0.87-0.89), 88.8%, and 88.2% respectively. Conclusions The use of NLP software can accurately analyze large volumes of radiology report data to detect AAA disease and assemble a contemporary aortic diameter-based cohort of patients for longitudinal analysis to guide surveillance, medical management, and operative decision making. It can also potentially be used to identify from the electronic medical records pre- and postoperative AAA patients “lost to follow-up,” leverage human resources engaged in the ongoing surveillance of patients with AAAs, and facilitate the construction and implementation of AAA screening programs.
- Subjects :
- Male
Clinical Decision-Making
030204 cardiovascular system & hematology
computer.software_genre
03 medical and health sciences
0302 clinical medicine
Cohen's kappa
Aneurysm
Predictive Value of Tests
Image Interpretation, Computer-Assisted
Medicine
Humans
030212 general & internal medicine
Diagnosis, Computer-Assisted
Registries
Aged
Natural Language Processing
Aged, 80 and over
business.industry
Delivery of Health Care, Integrated
Medical record
Reproducibility of Results
medicine.disease
Prognosis
Confidence interval
Abdominal aortic aneurysm
United States
Inter-rater reliability
Cohort
cardiovascular system
Surgery
Female
Artificial intelligence
Cardiology and Cardiovascular Medicine
business
computer
Natural language processing
Kappa
Aortic Aneurysm, Abdominal
Subjects
Details
- ISSN :
- 10976809
- Volume :
- 74
- Issue :
- 2
- Database :
- OpenAIRE
- Journal :
- Journal of vascular surgery
- Accession number :
- edsair.doi.dedup.....4477e25fedf9429bc44832949d57df42