Search

When machine learning algorithms are applied to data collected during the course of clinical care, it is generally accepted that the data has not been consistently collected. The absence of expected data elements is common and the mechanism through which a data element is missing often involves the clinical relevance of that data element in a specific patient. Therefore, the absence of data may have information value of its own. In the process of designing an application intended to support a medical problem list, we have studied whether the "missingness" of clinical data can provide useful information in building prediction models. In this study, we experimented with four methods of treating missing values in a clinical data set-two of them explicitly model the absence or "missingness" of data. Each of these data sets were used to build four different kinds of Bayesian classifiers-a naive Bayes structure, a human-composed network structure, and two networks based on structural learning algorithms. We compared the performance between groups with and without explicit models of missingness using the area under the ROC curve. The results showed that in most cases the classifiers trained using the explicit missing value treatments performed better. The result suggests that information may exist in "missingness" itself. Thus, when designing a decision support system, we suggest one consider explicitly representing the presence/absence of data in the underlying logic.

The increasing use of information system has resulted in the accumulation of a large volume of nursing data in electronic medical records. These data have great potential for supporting the various clinical decisions made by physicians, nurses, and managers. However, how to re-use of nursing data remains largely an issue of informatics. The aim of this study was to demonstrate how these nursing data can be used and how much they could contribute to developing a predictive model for an expert system for early detection of acute pancreatitis. We employed a probability-based model consisting of a Bayesian network and trained this model with the patient data retrospectively retrieved from the enterprise data warehouse of a tertiary hospital. The performance of the predictive model was measured based on the error rate and the area under receiver operating characteristics curve, which were 13.89 % and 0.93, respectively. The sensitivity of the acute pancreatitis to the findings from each nursing data was measured using a test of sensitivity. The results showed that the role of nursing data is as important as laboratory data in formulating a model for an expert system.

Objective: Develop and evaluate a natural language processing application for classifying chief complaints into syndromic categories for syndromic surveillance., Introduction: Much of the input data for artificial intelligence applications in the medical field are free-text patient medical records, including dictated medical reports and triage chief complaints. To be useful for automated systems, the free-text must be translated into encoded form., Methods: We implemented a biosurveillance detection system from Pennsylvania to monitor the 2002 Winter Olympic Games. Because input data was in free-text format, we used a natural language processing text classifier to automatically classify free-text triage chief complaints into syndromic categories used by the biosurveillance system. The classifier was trained on 4700 chief complaints from Pennsylvania. We evaluated the ability of the classifier to classify free-text chief complaints into syndromic categories with a test set of 800 chief complaints from Utah., Results: The classifier produced the following areas under the ROC curve: Constitutional = 0.95; Gastrointestinal = 0.97; Hemorrhagic = 0.99; Neurological = 0.96; Rash = 1.0; Respiratory = 0.99; Other = 0.96. Using information stored in the system's semantic model, we extracted from the Respiratory classifications lower respiratory complaints and lower respiratory complaints with fever with a precision of 0.97 and 0.96, respectively., Conclusion: Results suggest that a trainable natural language processing text classifier can accurately extract data from free-text chief complaints for biosurveillance.

We compared the performance of expert-crafted rules, a Bayesian network, and a decision tree at automatically identifying chest X-ray reports that support acute bacterial pneumonia. We randomly selected 292 chest X-ray reports, 75 (25%) of which were from patients with a hospital discharge diagnosis of bacterial pneumonia. The reports were encoded by our natural language processor and then manually corrected for mistakes. The encoded observations were analyzed by three expert systems to determine whether the reports supported pneumonia. The reference standard for radiologic support of pneumonia was the majority vote of three physicians. We compared (a) the performance of the expert systems against each other and (b) the performance of the expert systems against that of four physicians who were not part of the gold standard. Output from the expert systems and the physicians was transformed so that comparisons could be made with both binary and probabilistic output. Metrics of comparison for binary output were sensitivity (sens), precision (prec), and specificity (spec). The metric of comparison for probabilistic output was the area under the receiver operator characteristic (ROC) curve. We used McNemar's test to determine statistical significance for binary output and univariate z-tests for probabilistic output. Measures of performance of the expert systems for binary (probabilistic) output were as follows: Rules--sens, 0.92; prec, 0.80; spec, 0.86 (Az, 0.960); Bayesian network--sens, 0.90; prec, 0.72; spec, 0.78 (Az, 0.945); decision tree--sens, 0.86; prec, 0.85; spec, 0.91 (Az, 0.940). Comparisons of the expert systems against each other using binary output showed a significant difference between the rules and the Bayesian network and between the decision tree and the Bayesian network. Comparisons of expert systems using probabilistic output showed no significant differences. Comparisons of binary output against physicians showed differences between the Bayesian network and two physicians. Comparisons of probabilistic output against physicians showed a difference between the decision tree and one physician. The expert systems performed similarly for the probabilistic output but differed in measures of sensitivity, precision, and specificity produced by the binary output. All three expert systems performed similarly to physicians.

Objective: To evaluate the performance of a computerized decision support system that combines two different decision support methodologies (a Bayesian network and a natural language understanding system) for the diagnosis of patients with pneumonia., Design: Evaluation study using data from a prospective, clinical study., Patients: All patients 18 years and older who presented to the emergency department of a tertiary care setting and whose chest x-ray report was available during the encounter., Methods: The computerized decision support system calculated a probability of pneumonia using information provided by the two systems. Outcome measures were the area under the receiver operating characteristic curve, sensitivity, specificity, predictive values, likelihood ratios, and test effectiveness., Results: During the 3-month study period there were 742 patients (45 with pneumonia). The area under the receiver operating characteristic curve was 0.881 (95% CI: 0.822, 0.925) for the Bayesian network alone and 0.916 (95% CI: 0.869, 0.949) for the Bayesian network combined with the natural language understanding system (p=0.01)., Conclusion: Combining decision support methodologies that process information stored in different data formats can increase the performance of a computerized decision support system.

Background: In busy clinical settings, physicians often do not have enough time to identify patients for specific therapeutic guidelines. As a solution, decision support systems could automatically identify eligible patients and trigger computerized guidelines for specific diseases. Applying this idea to community-acquired pneumonia (CAP), we developed a Bayesian network (BN) and an artificial neural network (ANN) for identifying patients who have CAP and are eligible for a pneumonia guideline., Objective: The aim of this study was to determine whether the diagnostic accuracy of these two decision support models differs in terms of identifying CAP patients., Methods: We trained and tested the networks with a data set of 32,662 adult patients. For each network, we (1) calculated the specificity, the positive predictive value (PPV), and the negative predictive value (NPV) at a sensitivity of 95%, and (2) determined the area under the receiver operating characteristic curve (AUC) as a measure of overall accuracy. We tested for statistical difference between the AUCs using the correlated area z statistic., Results: At a sensitivity of 95%, the respective values for specificity, PPV, and NPV were: 92.3%, 15.1%, and 99.9% for the BN, and 94.0%, 18.6%, and 99.9% for the ANN. The BN had an AUC of 0.9795 (95% CI: 0.9736, 0.9843), and the ANN had an AUC of 0.9855 (95% CI: 0.9805, 0.9894). The difference between the AUCs was statistically significant (p=0.0044)., Conclusions: The networks achieved high overall accuracies on the testing data set. Because the difference in accuracies is statistically significant but not clinically significant, both networks are equally suited to drive a guideline.

Objective: To evaluate the performance of a natural language processing system in extracting pneumonia-related concepts from chest x-ray reports., Design: Four physicians, three lay persons, a natural language processing system, and two keyword searches (designated AAKS and KS) detected the presence or absence of three pneumonia-related concepts and inferred the presence or absence of acute bacterial pneumonia from 292 chest x-ray reports. Gold standard: Majority vote of three independent physicians. Reliability of the gold standard was measured., Outcome Measures: Recall, precision, specificity, and agreement (using Finn's R: statistic) with respect to the gold standard. Differences between the physicians and the other subjects were tested using the McNemar test for each pneumonia concept and for the disease inference of acute bacterial pneumonia., Results: Reliability of the reference standard ranged from 0.86 to 0.96. Recall, precision, specificity, and agreement (Finn R:) for the inference on acute bacterial pneumonia were, respectively, 0.94, 0.87, 0.91, and 0.84 for physicians; 0.95, 0.78, 0.85, and 0.75 for natural language processing system; 0.46, 0.89, 0.95, and 0.54 for lay persons; 0.79, 0.63, 0.71, and 0.49 for AAKS; and 0.87, 0.70, 0.77, and 0.62 for KS. The McNemar pairwise comparisons showed differences between one physician and the natural language processing system for the infiltrate concept and between another physician and the natural language processing system for the inference on acute bacterial pneumonia. The comparisons also showed that most physicians were significantly different from the other subjects in all pneumonia concepts and the disease inference., Conclusion: In extracting pneumonia related concepts from chest x-ray reports, the performance of the natural language processing system was similar to that of physicians and better than that of lay persons and keyword searches. The encoded pneumonia information has the potential to support several pneumonia-related applications used in our institution. The applications include a decision support system called the antibiotic assistant, a computerized clinical protocol for pneumonia, and a quality assurance application in the radiology department.

Objective: To evaluate if a medical language processing (MLP) system is able to support real-time computerization of community-acquired pneumonia (CAP) guidelines., Methods: Prospective validation study in the emergency department of a tertiary care facility. All the chest x-ray reports available in real-time for an admission decision during a five-week period were included. The MLP system was compared to a physician for the automatic selection of eligible patients and on the extraction of radiographic findings required by five different CAP guidelines. The gold standard comprised of three independent physicians and reliability measures were calculated. The outcome measures were the area under the receiver operated characteristic curve (AUC) for selecting eligible patients, sensitivity, positive predictive value (PPV), and specificity for the extraction of radiographic findings., Results: During the five-week period, 243 reports were available in real-time. The AUCs on selecting eligible CAP patients were 89.7% (CI: 84.2%, 93.7%) for the MLP system, and 93.3% (CI: 83.9%, 97.8%) for the physician. The average sensitivity, PPV, and specificity for radiographic findings that assessed localization and extension of CAP were respectively: 94%, 87%, 96% (physician); and 34%, 90%, 95% (MLP system). Both, the MLP system and the physician had average sensitivity, PPV, and specificity of 97%, 97%, and 99%, respectively, when localization was not an issue. Reliability measures for the gold standard were above 70%., Conclusion: The MLP system was able to support real-time computerization of guidelines by selecting eligible patients and extracting radiographic findings that do not assess localization and extension of CAP.

Objective: To assess the ability of an integrated, real-time diagnostic system (Bayesian network) to identify patients with community-acquired pneumonia who are eligible for a computerized pneumonia guideline without requiring clinicians to enter additional data., Design: Prospective validation study., Patients: All patients 18 years and older who presented to the emergency department of a tertiary care hospital., Methods: The diagnostic system computed a probability of pneumonia for every patient. The final diagnosis was established using ICD-9 discharge diagnoses. Outcome measures were sensitivity, specificity, predictive values, likelihood ratios, area under the receiver operating characteristic curve, and test effectiveness., Results: During the 9-week study period there were 4,361 patients (112 pneumonia patients). The area under the receiver operating characteristic curve was 0.930 (CI: 0.907, 0.948). At a fixed sensitivity of 95%, the specificity was 68.5%, the positive predictive value 7.3%, the negative predictive value 99.8%, the positive likelihood ratio 3.0, the negative likelihood ratio 0.08, and the test effectiveness 2.05., Conclusion: The diagnostic system was able to detect patients who are eligible for a pneumonia guideline. The detection of eligible patients can be applied to automatically initiate and evaluate computerized guidelines.

We present the development and the evaluation of a Bayesian network for the diagnosis of community-acquired pneumonia. The Bayesian network is intended to be part of a larger decision support system which assists emergency room physicians in the management of pneumonia patients. Minimal data entry from the nurse or the physician, timely availability of clinical parameters, and high accuracy were requirements we tried to meet. Data from more than 32,000 emergency room patients over a period of 2 years (June 1995-June 1997) were extracted from the clinical information system to train and test the Bayesian network. The network performed well in discriminating patients with pneumonia from patients with other diseases. The Bayesian network achieved a sensitivity of 95%, a specificity of 96.5%, an area under the receiver operating characteristic of 0.98, and a predictive value positive of 26.8%. Our feasibility study demonstrates that the proposed Bayesian network is an appropriate method to detect pneumonia patients with high accuracy. The study suggests that the proposed Bayesian network may represent a successful component within a larger decision support system for the management of community-acquired pneumonia.

Our natural language understanding system outputs a list of diseases, findings, and appliances found in a chest x-ray report. The system described in this paper links those diseases and findings that are causally related. Using Bayesian networks to model the conceptual and diagnostic information found in a chest x-ray we are able to infer more specific information about the findings that are linked to diseases.

This paper describes a prototype for research to evaluate the impact of diagnostic decision support systems on the behavior of physicians. Several indices that can be used to quantify the magnitude of impact are proposed. A large medical diagnostic knowledge base in internal medicine (the Iliad knowledge base) was used in this evaluation. The impact on behavior when different inference models are run against this knowledge base is evaluated for two different case domains and physician's specialties.

In this paper, we describe our clinical evaluation of the diagnostic accuracy of the renal mass diagnostic system (RMDS) and of seven physicians. To investigate the value of intravenous urography (IVU) and/or retrograde urography (RU) in diagnosing renal parenchymal tumors and tumors of the renal pelvis, RMDS and the seven physicians were tested with and without the information regarding IVU/RU at two different times. From this study we believe that RMDS can help residents in making more accurate presurgical renal mass diagnosis, and may eliminate the need for IVU/RU in the diagnosing process for a specific group of patients.

Few diagnostic decision-support systems are in routine clinical use, mainly because these systems typically require time-consuming manual data entry. This research investigated the feasibility of reducing manual data entry by integrating a stand-alone diagnostic expert system with an existing comprehensive hospital information system (HIS). A knowledge-based intervocabulary mapping technique was developed to map disparate vocabularies. The results of a retrospective study indicate that transferring clinical data from the HIS to the diagnostic expert system at the beginning of workup significantly reduces the manual data entry required for generating the correct diagnoses for patients.

Iliad is a large medical diagnostic system that covers more than 2000 diagnoses and 9000 findings. Due to the size and the complexity of this system, a robust knowledge representation is essential to consistently and efficiently model the medical knowledge involved. In this paper, we describe the knowledge representation currently used in Iliad and a probabilistic representation based on the Bayesian network formalism which can be derived using the information that the Iliad knowledge base contains.

The development and testing of computerized systems to assist in the diagnostic process is a time honored research activity in medical information science. The focus of the majority of the applications produced is on providing accurate diagnostic suggestions when appropriate clinical information is entered. We believe that diagnostic knowledge has a much wider range of uses than that of simply assigning diagnostic labels. Below we describe three applications which illustrate alternate uses for diagnostic systems. Applications that assist in data collection, assess the quality of medical reports, and extract relevant clinical data from natural language x-ray reports are discussed. We believe that more effort should be directed toward studying the use of diagnostic knowledge bases in processes that help plan diagnostic strategies, in quality assurance applications, and in processes that facilitate all aspects of medical communication.

In this paper we describe the evaluation of a probabilistic diagnostic system for patients with renal mass. Three inference models: Multi-membership Bayesian (MB), Minimal Diagnosis (MD) and Bayesian Network (BN), and 72 patients are used to illustrate three interrelated measures of system performance: accuracy, reliability and discriminating power. The inferencing strategies we tested demonstrated the kind of trade-offs in the performance measures that can be expected from imperfect systems. Ultimately, the purpose and expected use of a system should dictate the relative importance ascribed to different aspects of system performance.

Manual graphing of the progress of labor is considered useful but is not often done. The early detection of some deviations requires special graphics aids. Our objective was to develop an easy-to-use computer program for the integrated visual presentation of information characterizing the progress of labor. Through the use of inexpensive personal computers equipped with graphics monitors, the program provides a combined graphics display of timed progressive cervical dilatation, fetal station, and stimulation of uterine activity (oxytocin infusion). For the early detection of abnormalities, phase-specific normal ranges (reference areas) are displayed. In addition, protraction/arrest as well as precipitate labor disorders are highlighted and computer messages are displayed. The program was evaluated through the assessment of 405 labors entered into a local area network of computers. On average, the program identified 1.5 abnormalities per recorded labor (2.0 for labors resulting in vaginal delivery). The graphic presentation of the labor curve, produced within 3 seconds, displayed 27% more information than the tabular format on the same screen area and provided a single-screen display of the labor curve even for patients with excessive data. The computer-generated display of labor curves facilitates visual presentation and interpretation of labor progress and can also help to translate quality assurance criteria into clinical practice.

Iliad is a computerized, expert system for internal medical diagnosis. The system is designed to teach diagnostic skills by means of simulated patient case presentations. We report the results of a controlled trial in which junior students were randomly assigned to received Iliad training on one of two different simulated case mixes. Each group was subsequently tested in both their "trained" and "untrained" case domain. The testing consisted of computerized, simulated patient cases for which no training feedback was provided. Outcome variables were designed to measure the students' performance on these test cases. The results indicate that students made fewer diagnostic errors and more conclusively confirmed their diagnostic hypotheses when they were tested in their trained domain. We conclude that expert systems such as Iliad can effectively teach diagnostic skills by supplementing trainees' actual case experience with computerized simulations.

Iliad is a diagnostic expert system for internal medicine. One important feature that Iliad offers is the ability to analyze a particular patient case and to determine the most cost-effective method for pursuing the work-up. Iliad's current "best information" algorithm has not been previously validated and compared to other potential algorithms. Therefore, this paper presents a comparison of four new algorithms to the current algorithm. The basis for this comparison was eighteen "vignette" cases derived from real patient cases from the University of Utah Medical Center. The results indicated that the current algorithm can be significantly improved. More promising algorithms are suggested for future investigation.