Search

Objectives: This study evaluates the accuracy and portability of a natural language processing (NLP) tool for extracting clinical findings of influenza from clinical notes across two large healthcare systems. Effectiveness is evaluated on how well NLP supports downstream influenza case-detection for disease surveillance., Methods: We independently developed two NLP parsers, one at Intermountain Healthcare (IH) in Utah and the other at University of Pittsburgh Medical Center (UPMC) using local clinical notes from emergency department (ED) encounters of influenza. We measured NLP parser performance for the presence and absence of 70 clinical findings indicative of influenza. We then developed Bayesian network models from NLP processed reports and tested their ability to discriminate among cases of (1) influenza, (2) non-influenza influenza-like illness (NI-ILI), and (3) 'other' diagnosis., Results: On Intermountain Healthcare reports, recall and precision of the IH NLP parser were 0.71 and 0.75, respectively, and UPMC NLP parser, 0.67 and 0.79. On University of Pittsburgh Medical Center reports, recall and precision of the UPMC NLP parser were 0.73 and 0.80, respectively, and IH NLP parser, 0.53 and 0.80. Bayesian case-detection performance measured by AUROC for influenza versus non-influenza on Intermountain Healthcare cases was 0.93 (using IH NLP parser) and 0.93 (using UPMC NLP parser). Case-detection on University of Pittsburgh Medical Center cases was 0.95 (using UPMC NLP parser) and 0.83 (using IH NLP parser). For influenza versus NI-ILI on Intermountain Healthcare cases performance was 0.70 (using IH NLP parser) and 0.76 (using UPMC NLP parser). On University of Pisstburgh Medical Center cases, 0.76 (using UPMC NLP parser) and 0.65 (using IH NLP parser)., Conclusion: In all but one instance (influenza versus NI-ILI using IH cases), local parsers were more effective at supporting case-detection although performances of non-local parsers were reasonable.

Natural language processing (NLP) technologies provide an opportunity to extract key patient data from free text documents within the electronic health record (EHR). We are developing a series of components from which to construct NLP pipelines. These pipelines typically begin with a component whose goal is to label sections within medical documents with codes indicating the anticipated semantics of their content. This Clinical Section Labeler prepares the document for further, focused information extraction. Below we describe the evaluation of six algorithms designed for use in a Clinical Section Labeler. These algorithms are trained with N-gram-based feature sets extracted from document sections and the document types. In the evaluation, 6 different Bayesian models were trained and used to assign one of 27 different topics to each section. A tree-augmented Bayesian network using the document type and N-grams derived from section headers proved most accurate in assigning individual sections appropriate section topics.

Research Objective: To develop scalable informatics infrastructure for normalization of both structured and unstructured electronic health record (EHR) data into a unified, concept-based model for high-throughput phenotype extraction., Materials and Methods: Software tools and applications were developed to extract information from EHRs. Representative and convenience samples of both structured and unstructured data from two EHR systems-Mayo Clinic and Intermountain Healthcare-were used for development and validation. Extracted information was standardized and normalized to meaningful use (MU) conformant terminology and value set standards using Clinical Element Models (CEMs). These resources were used to demonstrate semi-automatic execution of MU clinical-quality measures modeled using the Quality Data Model (QDM) and an open-source rules engine., Results: Using CEMs and open-source natural language processing and terminology services engines-namely, Apache clinical Text Analysis and Knowledge Extraction System (cTAKES) and Common Terminology Services (CTS2)-we developed a data-normalization platform that ensures data security, end-to-end connectivity, and reliable data flow within and across institutions. We demonstrated the applicability of this platform by executing a QDM-based MU quality measure that determines the percentage of patients between 18 and 75 years with diabetes whose most recent low-density lipoprotein cholesterol test result during the measurement year was <100 mg/dL on a randomly selected cohort of 273 Mayo Clinic patients. The platform identified 21 and 18 patients for the denominator and numerator of the quality measure, respectively. Validation results indicate that all identified patients meet the QDM-based criteria., Conclusions: End-to-end automated systems for extracting clinical information from diverse EHR systems require extensive use of standardized vocabularies and terminologies, as well as robust information models for storing, discovering, and processing that information. This study demonstrates the application of modular and open-source resources for enabling secondary use of EHR data through normalization into standards-based, comparable, and consistent format for high-throughput phenotyping to identify patient cohorts.

Objective: Natural language processing (NLP) tasks are commonly decomposed into subtasks, chained together to form processing pipelines. The residual error produced in these subtasks propagates, adversely affecting the end objectives. Limited availability of annotated clinical data remains a barrier to reaching state-of-the-art operating characteristics using statistically based NLP tools in the clinical domain. Here we explore the unique linguistic constructions of clinical texts and demonstrate the loss in operating characteristics when out-of-the-box part-of-speech (POS) tagging tools are applied to the clinical domain. We test a domain adaptation approach integrating a novel lexical-generation probability rule used in a transformation-based learner to boost POS performance on clinical narratives., Methods: Two target corpora from independent healthcare institutions were constructed from high frequency clinical narratives. Four leading POS taggers with their out-of-the-box models trained from general English and biomedical abstracts were evaluated against these clinical corpora. A high performing domain adaptation method, Easy Adapt, was compared to our newly proposed method ClinAdapt., Results: The evaluated POS taggers drop in accuracy by 8.5-15% when tested on clinical narratives. The highest performing tagger reports an accuracy of 88.6%. Domain adaptation with Easy Adapt reports accuracies of 88.3-91.0% on clinical texts. ClinAdapt reports 93.2-93.9%., Conclusions: ClinAdapt successfully boosts POS tagging performance through domain adaptation requiring a modest amount of annotated clinical data. Improving the performance of critical NLP subtasks is expected to reduce pipeline error propagation leading to better overall results on complex processing tasks.

Purpose: This study aimed to develop Natural Language Processing (NLP) approaches to supplement manual outcome validation, specifically to validate pneumonia cases from chest radiograph reports., Methods: We trained one NLP system, ONYX, using radiograph reports from children and adults that were previously manually reviewed. We then assessed its validity on a test set of 5000 reports. We aimed to substantially decrease manual review, not replace it entirely, and so, we classified reports as follows: (1) consistent with pneumonia; (2) inconsistent with pneumonia; or (3) requiring manual review because of complex features. We developed processes tailored either to optimize accuracy or to minimize manual review. Using logistic regression, we jointly modeled sensitivity and specificity of ONYX in relation to patient age, comorbidity, and care setting. We estimated positive and negative predictive value (PPV and NPV) assuming pneumonia prevalence in the source data., Results: Tailored for accuracy, ONYX identified 25% of reports as requiring manual review (34% of true pneumonias and 18% of non-pneumonias). For the remainder, ONYX's sensitivity was 92% (95% CI 90-93%), specificity 87% (86-88%), PPV 74% (72-76%), and NPV 96% (96-97%). Tailored to minimize manual review, ONYX classified 12% as needing manual review. For the remainder, ONYX had sensitivity 75% (72-77%), specificity 95% (94-96%), PPV 86% (83-88%), and NPV 91% (90-91%)., Conclusions: For pneumonia validation, ONYX can replace almost 90% of manual review while maintaining low to moderate misclassification rates. It can be tailored for different outcomes and study needs and thus warrants exploration in other settings., (Copyright © 2013 John Wiley & Sons, Ltd.)

A semantic lexicon which associates words and phrases in text to concepts is critical for extracting and encoding clinical information in free text and therefore achieving semantic interoperability between structured and unstructured data in Electronic Health Records (EHRs). Directly using existing standard terminologies may have limited coverage with respect to concepts and their corresponding mentions in text. In this paper, we analyze how tokens and phrases in a large corpus distribute and how well the UMLS captures the semantics. A corpus-driven semantic lexicon, MedLex, has been constructed where the semantics is based on the UMLS assisted with variants mined and usage information gathered from clinical text. The detailed corpus analysis of tokens, chunks, and concept mentions shows the UMLS is an invaluable source for natural language processing. Increasing the semantic coverage of tokens provides a good foundation in capturing clinical information comprehensively. The study also yields some insights in developing practical NLP systems.

Objective: To understand how the source of information affects different adverse event (AE) surveillance methods., Design: Retrospective analysis of inpatient adverse drug events (ADEs) and hospital-associated infections (HAIs) detected by either a computerized surveillance system (CSS) or manual chart review (MCR)., Measurement: Descriptive analysis of events detected using the two methods by type of AE, type of information about the AE, and sources of the information., Results: CSS detected more HAIs than MCR (92% vs 34%); however, a similar number of ADEs was detected by both systems (52% vs 51%). The agreement between systems was greater for HAIs than ADEs (26% vs 3%). The CSS missed events that did not have information in coded format or that were described only in physician narratives. The MCR detected events missed by CSS using information in physician narratives. Discharge summaries were more likely to contain information about AEs than any other type of physician narrative, followed by emergency department reports for HAIs and general consult notes for ADEs. Some ADEs found by MCR were detected by CSS but not verified by a clinician., Limitations: Inability to distinguish between CSS false positives and suspected AEs for cases in which the clinician did not document their assessment in the CSS., Conclusion: The effect that information source has on different surveillance methods depends on the type of AE. Integrating information from physician narratives with CSS using natural language processing would improve the detection of ADEs more than HAIs.

Natural language processing applications that extract information from text rely on semantic representations. The objective of this paper is to describe a methodology for creating a semantic representation for information that will be automatically extracted from textual clinical records. We illustrate two of the four steps of the methodology in this paper using the case study of encoding information from dictated dental exams: (1) develop an initial representation from a set of training documents and (2) iteratively evaluate and evolve the representation while developing annotation guidelines. Our approach for developing and evaluating a semantic representation is based on standard principles and approaches that are not dependent on any particular domain or type of semantic representation.

In this study, we evaluate the performance of a Natural Language Processing (NLP) application designed to extract medical problems from narrative text clinical documents. The documents come from a patient's electronic medical record and medical problems are proposed for inclusion in the patient's electronic problem list. This application has been developed to help maintain the problem list and make it more accurate, complete, and up-to-date. The NLP part of this system-analyzed in this study-uses the UMLS MetaMap Transfer (MMTx) application and a negation detection algorithm called NegEx to extract 80 different medical problems selected for their frequency of use in our institution. When using MMTx with its default data set, we measured a recall of 0.74 and a precision of 0.756. A custom data subset for MMTx was created, making it faster and significantly improving the recall to 0.896 with a non-significant reduction in precision.

Background: The medical problem list is an important part of the electronic medical record in development in our institution. To serve the functions it is designed for, the problem list has to be as accurate and timely as possible. However, the current problem list is usually incomplete and inaccurate, and is often totally unused. To alleviate this issue, we are building an environment where the problem list can be easily and effectively maintained., Methods: For this project, 80 medical problems were selected for their frequency of use in our future clinical field of evaluation (cardiovascular). We have developed an Automated Problem List system composed of two main components: a background and a foreground application. The background application uses Natural Language Processing (NLP) to harvest potential problem list entries from the list of 80 targeted problems detected in the multiple free-text electronic documents available in our electronic medical record. These proposed medical problems drive the foreground application designed for management of the problem list. Within this application, the extracted problems are proposed to the physicians for addition to the official problem list., Results: The set of 80 targeted medical problems selected for this project covered about 5% of all possible diagnoses coded in ICD-9-CM in our study population (cardiovascular adult inpatients), but about 64% of all instances of these coded diagnoses. The system contains algorithms to detect first document sections, then sentences within these sections, and finally potential problems within the sentences. The initial evaluation of the section and sentence detection algorithms demonstrated a sensitivity and positive predictive value of 100% when detecting sections, and a sensitivity of 89% and a positive predictive value of 94% when detecting sentences., Conclusion: The global aim of our project is to automate the process of creating and maintaining a problem list for hospitalized patients and thereby help to guarantee the timeliness, accuracy and completeness of this information.

To improve the use and quality of the electronic Problem List, which is at the heart of the problem-oriented medical record in development in our institution (Intermountain Health Care, Utah, U.S.), we developed an Automated Problem List system using Natural Language Processing (NLP) technologies. A key part of this system is a module that automatically extracts potential medical problems from free-text clinical documents. The NLP module uses MMTx, developed at the U.S. National Library of Medicine. Negation detection was added to this application by adapting a negation detection algorithm called NegEx. To evaluate the adequacy of the performance of the NLP module for our Automated Problem List system, we evaluated it with 160 electronic clinical documents of different types. Two different data sets for MMTx were used: the default full UMLS data set and a customised subset adapted to detect the set of 80 medical problems we are interested in. With the default data set, we measured a recall of 0.74 (95% CI 0.68-0.8) and a precision of 0.76 (0.69-0.82). The customised subset had a significantly better recall of 0.9 (0.85-0.94), and a non-significantly different precision of 0.69 (0.63-0.75).

Objective: Develop and evaluate a natural language processing application for classifying chief complaints into syndromic categories for syndromic surveillance., Introduction: Much of the input data for artificial intelligence applications in the medical field are free-text patient medical records, including dictated medical reports and triage chief complaints. To be useful for automated systems, the free-text must be translated into encoded form., Methods: We implemented a biosurveillance detection system from Pennsylvania to monitor the 2002 Winter Olympic Games. Because input data was in free-text format, we used a natural language processing text classifier to automatically classify free-text triage chief complaints into syndromic categories used by the biosurveillance system. The classifier was trained on 4700 chief complaints from Pennsylvania. We evaluated the ability of the classifier to classify free-text chief complaints into syndromic categories with a test set of 800 chief complaints from Utah., Results: The classifier produced the following areas under the ROC curve: Constitutional = 0.95; Gastrointestinal = 0.97; Hemorrhagic = 0.99; Neurological = 0.96; Rash = 1.0; Respiratory = 0.99; Other = 0.96. Using information stored in the system's semantic model, we extracted from the Respiratory classifications lower respiratory complaints and lower respiratory complaints with fever with a precision of 0.97 and 0.96, respectively., Conclusion: Results suggest that a trainable natural language processing text classifier can accurately extract data from free-text chief complaints for biosurveillance.

To help maintain a complete, accurate and timely Problem List, we are developing a system to automatically retrieve medical problems from free-text documents. This system uses Natural Language Processing to analyze all electronic narrative text documents in a patient's record. Here we evaluate and compare 3 different applications of NLP technology in our system: the first using MMTx (MetaMap Transfer) with a negation detection algorithm (NegEx), the second using an alpha version of a locally developed NLP application called MPLUS2, and the third using keyword searching. They were adapted and trained to extract medical problems from a set of 80 problems of diagnosis type. The version using MMTx and NegEx was improved by adding some disambiguation and modifying the negation detection algorithm, and these modifications significantly improved recall and precision. The different versions of the NLP module were compared, and showed the following recall / precision results: standard MMTx with NegEx version 0.775 / 0.398; improved MMTx with NegEx version 0.892 / 0.753; MPLUS2 version 0.693 / 0.402; and keyword searching version 0.575 / 0.807. Average results for the reviewers were a recall of 0.788 and a precision of 0.912.

We are developing tools to help maintain a complete, accurate and timely problem list within a general purpose Electronic Medical Record system. As a part of this project, we have designed a system to automatically retrieve medical problems from free-text documents. Here we describe an information model based on XML (eXtensible Markup Language) and compliant with the CDA (Clinical Document Architecture). This model is used to ease the exchange of clinical data between the Natural Language Understanding application that retrieves potential problems from narrative document, and the problem list management application.

Objective: To evaluate the performance of a natural language processing system in extracting pneumonia-related concepts from chest x-ray reports., Design: Four physicians, three lay persons, a natural language processing system, and two keyword searches (designated AAKS and KS) detected the presence or absence of three pneumonia-related concepts and inferred the presence or absence of acute bacterial pneumonia from 292 chest x-ray reports. Gold standard: Majority vote of three independent physicians. Reliability of the gold standard was measured., Outcome Measures: Recall, precision, specificity, and agreement (using Finn's R: statistic) with respect to the gold standard. Differences between the physicians and the other subjects were tested using the McNemar test for each pneumonia concept and for the disease inference of acute bacterial pneumonia., Results: Reliability of the reference standard ranged from 0.86 to 0.96. Recall, precision, specificity, and agreement (Finn R:) for the inference on acute bacterial pneumonia were, respectively, 0.94, 0.87, 0.91, and 0.84 for physicians; 0.95, 0.78, 0.85, and 0.75 for natural language processing system; 0.46, 0.89, 0.95, and 0.54 for lay persons; 0.79, 0.63, 0.71, and 0.49 for AAKS; and 0.87, 0.70, 0.77, and 0.62 for KS. The McNemar pairwise comparisons showed differences between one physician and the natural language processing system for the infiltrate concept and between another physician and the natural language processing system for the inference on acute bacterial pneumonia. The comparisons also showed that most physicians were significantly different from the other subjects in all pneumonia concepts and the disease inference., Conclusion: In extracting pneumonia related concepts from chest x-ray reports, the performance of the natural language processing system was similar to that of physicians and better than that of lay persons and keyword searches. The encoded pneumonia information has the potential to support several pneumonia-related applications used in our institution. The applications include a decision support system called the antibiotic assistant, a computerized clinical protocol for pneumonia, and a quality assurance application in the radiology department.

Objective: To evaluate if a medical language processing (MLP) system is able to support real-time computerization of community-acquired pneumonia (CAP) guidelines., Methods: Prospective validation study in the emergency department of a tertiary care facility. All the chest x-ray reports available in real-time for an admission decision during a five-week period were included. The MLP system was compared to a physician for the automatic selection of eligible patients and on the extraction of radiographic findings required by five different CAP guidelines. The gold standard comprised of three independent physicians and reliability measures were calculated. The outcome measures were the area under the receiver operated characteristic curve (AUC) for selecting eligible patients, sensitivity, positive predictive value (PPV), and specificity for the extraction of radiographic findings., Results: During the five-week period, 243 reports were available in real-time. The AUCs on selecting eligible CAP patients were 89.7% (CI: 84.2%, 93.7%) for the MLP system, and 93.3% (CI: 83.9%, 97.8%) for the physician. The average sensitivity, PPV, and specificity for radiographic findings that assessed localization and extension of CAP were respectively: 94%, 87%, 96% (physician); and 34%, 90%, 95% (MLP system). Both, the MLP system and the physician had average sensitivity, PPV, and specificity of 97%, 97%, and 99%, respectively, when localization was not an issue. Reliability measures for the gold standard were above 70%., Conclusion: The MLP system was able to support real-time computerization of guidelines by selecting eligible patients and extracting radiographic findings that do not assess localization and extension of CAP.

A medical language processing system called SymText, two other automated methods, and a lay person were compared against an internal medicine resident for their ability to identify pneumonia related concepts on chest x-ray reports. Sensitivity (recall), specificity, and positive predictive value (precision) are reported with respect to an independent panel of physicians. Overall the performance of SymText was similar to the physician and superior to the other methods. The automatic encoding of pneumonia concepts will support clinical research, decision making, computerized clinical protocols, and quality assurance in a radiology department.

Free-text documents are the main type of data produced by a radiology department in a hospital information system. While this type of data is readily accessible for clinical data review it can not be accessed by other applications to perform medical decision support, quality assurance, and outcome studies. In an attempt to solve this problem, natural language processing systems have been developed and tested against chest x-rays reports to extract relevant clinical information and make it accessible to other computer applications. We have used a natural language processing tool called SymText to extract relevant clinical information from a different type of radiology report, the Ventilation/Perfusion lung scan report. Results of this effort can be analyzed in terms of precision and recall. The overall precision was 0.88 and recall was 0.92. In addition, the natural language processing system functions differently in reports with and without an impression section. If this type of information can be successfully extracted from radiology reports, one can develop quality monitors for the diagnostic performance of the radiologist by correlating the impressions with gold standard data present in a hospital information system. Avoiding the manual effort previously necessary to create quality assurance data, can lead to a higher frequency of quality review in a radiology department.

Our natural language understanding system outputs a list of diseases, findings, and appliances found in a chest x-ray report. The system described in this paper links those diseases and findings that are causally related. Using Bayesian networks to model the conceptual and diagnostic information found in a chest x-ray we are able to infer more specific information about the findings that are linked to diseases.

Free-text or natural language documents make up an increasing part of the computerized medical record. While they do provide accessible clinical information to health care personnel, they fail to support processes that require clinical data coded according to a shared lexicon and data structure. We have developed a natural language parser that converts free-text admitting diagnoses into a coded form. This application has proven acceptably accurate in the experimental laboratory to warrant a test in the target clinical environment. Here we describe an approach to moving this research application into a production environment where it can contribute to the efforts of the Health Information Services Department. This transition is essential if the products of natural language understanding research are to contribute to patient care in a routine and sustainable way.

The value of the computerized medical record is derived in part from the availability of medical information in a coded form accessible to manipulation by processes designed for automated decision support, medical research, and computer assistance in the management of health care delivery. To meet these needs medical reports captured and stored as natural language documents must be encoded. Below we discuss an ongoing formative process aimed at developing a natural language understanding system for chest x-ray reports. Comparative data showing the progress of this process is presented.

A computerized data acquisition tool, the special purpose radiology understanding system (SPRUS), has been implemented as a module in the Health Evaluation through Logical Processing Hospital Information System. This tool uses semantic information from a diagnostic expert system to parse free-text radiology reports and to extract and encode both the findings and the radiologists' interpretations. These coded findings and interpretations are then stored in a clinical data base. The system recognizes both radiologic findings and diagnostic interpretations. Initial tests showed a true-positive rate of 87% for radiographic findings and a bad data rate of 5%. Diagnostic interpretations are recognized at a rate of 95% with a bad data rate of 6%. Testing suggests that these rates can be improved through enhancements to the system's thesaurus and the computerized medical knowledge that drives it. This system holds promise as a tool to obtain coded radiologic data for research, medical audit, and patient care.