Unstructured Data Are Superior to Structured Data for Eliciting Quantitative Smoking History From the Electronic Health Record

PURPOSE Develop a method for extracting smoking status and quantitative smoking history from clinician notes to facilitate cohort identification for low-dose computed tomography (LDCT) scanning for early detection of lung cancer. MATERIALS AND METHODS A sample of 4,615 adult patients were randomly selected from the Multiparameter Intelligent Monitoring in Critical Care (MIMIC-III) database. The structured data were obtained by queries of the diagnosis tables using the International Classification of Diseases codes in use at that time. Unstructured data were drawn from clinician notes via natural language processing (NLP) using named entity recognition and our clinical data processing and extraction algorithms to identify two main clinical criteria for each smoking patient: (1) pack years smoked and (2) time from quit date (if applicable). A subset of 10% of the patient charts were manually reviewed for accuracy and precision. RESULTS The structured data revealed 575 (12.5%) ever smokers (current plus past use). None of these patients had quantification of their smoking history, and 4,040 (87.5%) had no smoking information in the diagnosis tables; consequently, a cohort of patients eligible for LDCT could not be determined. Review of the physician notes by NLP disclosed 1,930 (41.8%) ever smokers of whom 537 were active smokers and 1,299 former smokers, and in 94 cases, it could not be determined if they were active or former smokers. A total of 1365 patients (29.6%) had no smoking data recorded. When the smoking and the age criteria for LDCT were applied to this group, 276 were found to be eligible for LDCT using the USPSTF criteria. As determined by clinician review, our F-score for identifying patients eligible for LDCT was 0.88. CONCLUSION Unstructured data, obtained by NLP, can accurately identify a precise cohort that meets the USPSTF guidelines for LDCT.