1. Standardized patient profile review using large language models for case adjudication in observational research.
- Author
-
Schuemie, Martijn J., Ostropolets, Anna, Zhuk, Aleh, Korsik, Uladzislau, Seo, Seung In, Suchard, Marc A., Hripcsak, George, and Ryan, Patrick B.
- Subjects
DATA security ,TASK performance ,SCIENTIFIC observation ,MEDICAL case management ,BENCHMARKING (Management) ,NATURAL language processing ,WORKFLOW ,MEDICAL records ,ACQUISITION of data ,ELECTRONIC health records ,ALGORITHMS ,COMORBIDITY ,SENSITIVITY & specificity (Statistics) - Abstract
Using administrative claims and electronic health records for observational studies is common but challenging due to data limitations. Researchers rely on phenotype algorithms, requiring labor-intensive chart reviews for validation. This study investigates whether case adjudication using the previously introduced Knowledge-Enhanced Electronic Profile Review (KEEPER) system with large language models (LLMs) is feasible and could serve as a viable alternative to manual chart review. The task involves adjudicating cases identified by a phenotype algorithm, with KEEPER extracting predefined findings such as symptoms, comorbidities, and treatments from structured data. LLMs then evaluate KEEPER outputs to determine whether a patient truly qualifies as a case. We tested four LLMs including GPT-4, hosted locally to ensure privacy. Using zero-shot prompting and iterative prompt optimization, we found LLM performance, across ten diseases, varied by prompt and model, with sensitivities from 78 to 98% and specificities from 48 to 98%, indicating promise for automating phenotype evaluation. [ABSTRACT FROM AUTHOR]
- Published
- 2025
- Full Text
- View/download PDF