Search

Objective: Clinicians face numerous information needs during patient care activities and most of these needs are not met. Infobuttons are information retrieval tools that help clinicians to fulfill their information needs by providing links to on-line health information resources from within an electronic medical record (EMR) system. The aim of this study was to produce classification models based on medication infobutton usage data to predict the medication-related content topics (e.g., dose, adverse effects, drug interactions, patient education) that a clinician is most likely to choose while entering medication orders in a particular clinical context.Design: We prepared a dataset with 3078 infobutton sessions and 26 attributes describing characteristics of the user, the medication, and the patient. In these sessions, users selected one out of eight content topics. Automatic attribute selection methods were then applied to the dataset to eliminate redundant and useless attributes. The reduced dataset was used to produce nine classification models from a set of state-of-the-art machine learning algorithms. Finally, the performance of the models was measured and compared.Measurements: Area under the ROC curve (AUC) and agreement (kappa) between the content topics predicted by the models and those chosen by clinicians in each infobutton session.Results: The performance of the models ranged from 0.49 to 0.56 (kappa). The AUC of the best model ranged from 0.73 to 0.99. The best performance was achieved when predicting choice of the adult dose, pediatric dose, patient education, and pregnancy category content topics.Conclusion: The results suggest that classification models based on infobutton usage data are a promising method for the prediction of content topics that a clinician would choose to answer patient care questions while using an EMR system. [ABSTRACT FROM AUTHOR]

OBJECTIVE: Do-not-resuscitate (DNR) orders and advance directives are increasingly prevalent and may affect medical interventions and outcomes. Simple, automated techniques to identify patients with DNR orders do not currently exist but could help avoid costly and time-consuming chart review. This study hypothesized that a decision to withhold cardiopulmonary resuscitation would be included in a patient's dictated reports. The authors developed and validated a simple computerized search method, which screens dictated reports to detect patients with DNR status. METHODS: A list of concepts related to DNR order documentation was developed using emergency department, hospital admission, consult, and hospital discharge reports of 665 consecutive, hospitalized pneumonia patients during a four-year period (1995-1999). The list was validated in an independent group of 190 consecutive inpatients with pneumonia during a five-month period (1999-2000). The reference standard for the presence of DNR orders was manual chart review of all study patients. Sensitivity, specificity, predictive values, and nonerror rates were calculated for individual and combined concepts. RESULTS: The list of concepts included: DNR, Do Not Attempt to Resuscitate (DNAR), DNI, NCR, advanced directive, living will, power of attorney, Cardiopulmonary Resuscitation (CPR), defibrillation, arrest, resuscitate, code, and comfort care. As determined by manual chart review, a DNR order was written for 32.6% of patients in the derivation and for 31.6% in the validation group. Dictated reports included DNR order-related information for 74.5% of patients in the derivation and 73% in the validation group. If mentioned in the dictated report, the combined keyword search had a sensitivity of 74.2% in the derivation group (70.0% in the validation group), a specificity of 91.5% (81.5%), a positive predictive value of 80.9% (63.6%), a negative predictive value of 88.0% (85.5%), and a nonerror rate of 85.9% (77.9%). DNR and resuscitate were the most frequently used and power of attorney and advanced directives the least frequently used terms. CONCLUSION: Dictated hospital reports frequently contained DNR order-related information for patients with a written DNR order. Using an uncomplicated keyword search, electronic screening of dictated reports yielded good accuracy for identifying patients with DNR order information. [ABSTRACT FROM AUTHOR]

Planning the clinical evaluation of a computerized decision support system requires a strategy that encompasses the different aspects of the clinical problem, the technical difficulties of software and hardware integration and implementation, the behavioral aspects of the targeted users, and the discipline of study design. Although clinical information systems are becoming more widely available, only a few decision support systems have been formally evaluated in clinical environments. Published accounts of difficulties associated with the clinical evaluation of decision support systems remain scarce. The authors report on a variety of behavioral, logistical, technical, clinical, cost, and work flow issues that they had to address when choosing a study design for a clinical trial for the evaluation of an integrated, real-time decision support system for the automatic identification of patients likely to have pneumonia in an emergency department. In the absence of a true gold standard, they show how they created a credible, clinically acceptable, and economical reference standard for the diagnosis of pneumonia, to determine the overall accuracy of the system. For the creation of a reference standard, they describe the importance of recognizing verification bias and avoiding it. Finally, advantages and disadvantages of different study designs are explored with respect to the targeted users and the clinical setting. [ABSTRACT FROM AUTHOR]

Radiology Information Systems (RIS) are designed to capture and manage the data associated with ordering, executing, reporting, and billing x-ray procedures. The HELP Hospital Information System contains a radiology subsystem that supports these functions. In an effort to enhance quality assurance initiatives, we have created a supplemental data base. This data base contains not only the data traditionally generated by RISs but also data from the hospital system that is relevant to quality assurance. One of the goals associated with this data base is to use techniques from the discipline of Continuous Quality Improvement (CQI) in the radiology department. A focus of our initial efforts has been the time necessary to provide x-ray reports to ordering physicians once the imaging examination has been performed. Efforts to manage the portion of this time interval caused by transcription have resulted in a substantial decrease in the time required for this function. A second goal of this project is to evaluate the quality of x-ray ordering. This objective requires a computerized record of the outcome of the x-ray procedure. Initial analysis of data derived from this data base indicates significant differences in the ordering behavior for computed tomography (CT) examinations among a test group of physicians. A third goal is to do quality assurance on x-ray reports. Experience with pilot systems has shown promising results using a mathematical model of report quality. We hope to leverage these techniques and this quality assurance data base to define a CQI process for medical reports in general and for x-ray reports in particular. (C) 1997 by W.B. Saunders Company [ABSTRACT FROM AUTHOR]

How to deliver best care in various clinical settings remains a vexing problem. All pertinent healthcare-related questions have not, cannot, and will not be addressable with costly time- and resource-consuming controlled clinical trials. At present, evidence-based guidelines can address only a small fraction of the types of care that clinicians deliver. Furthermore, underserved areas rarely can access state-of-the-art evidence-based guidelines in real-time, and often lack the wherewithal to implement advanced guidelines. Care providers in such settings frequently do not have sufficient training to undertake advanced guideline implementation. Nevertheless, in advanced modern healthcare delivery environments, use of eActions (validated clinical decision support systems) could help overcome the cognitive limitations of overburdened clinicians. Widespread use of eActions will require surmounting current healthcare technical and cultural barriers and installing clinical evidence/data curation systems. The authors expect that increased numbers of evidence-based guidelines will result from future comparative effectiveness clinical research carried out during routine healthcare delivery within learning healthcare systems., (© The Author(s) 2022. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com.)

Objective: To facilitate the development of standards-based clinical decision support (CDS) systems, we review the current set of CDS standards that are based on Health Level Seven International Fast Healthcare Interoperability Resources (FHIR). Widespread adoption of these standards may help reduce healthcare variability, improve healthcare quality, and improve patient safety., Target Audience: This tutorial is designed for the broad informatics community, some of whom may be unfamiliar with the current, FHIR-based CDS standards., Scope: This tutorial covers the following standards: Arden Syntax (using FHIR as the data model), Clinical Quality Language, FHIR Clinical Reasoning, SMART on FHIR, and CDS Hooks. Detailed descriptions and selected examples are provided., (© The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com.)

Clinical decision-making is based on knowledge, expertise, and authority, with clinicians approving almost every intervention-the starting point for delivery of "All the right care, but only the right care," an unachieved healthcare quality improvement goal. Unaided clinicians suffer from human cognitive limitations and biases when decisions are based only on their training, expertise, and experience. Electronic health records (EHRs) could improve healthcare with robust decision-support tools that reduce unwarranted variation of clinician decisions and actions. Current EHRs, focused on results review, documentation, and accounting, are awkward, time-consuming, and contribute to clinician stress and burnout. Decision-support tools could reduce clinician burden and enable replicable clinician decisions and actions that personalize patient care. Most current clinical decision-support tools or aids lack detail and neither reduce burden nor enable replicable actions. Clinicians must provide subjective interpretation and missing logic, thus introducing personal biases and mindless, unwarranted, variation from evidence-based practice. Replicability occurs when different clinicians, with the same patient information and context, come to the same decision and action. We propose a feasible subset of therapeutic decision-support tools based on credible clinical outcome evidence: computer protocols leading to replicable clinician actions (eActions). eActions enable different clinicians to make consistent decisions and actions when faced with the same patient input data. eActions embrace good everyday decision-making informed by evidence, experience, EHR data, and individual patient status. eActions can reduce unwarranted variation, increase quality of clinical care and research, reduce EHR noise, and could enable a learning healthcare system., (© The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com.)

A real-time electronic CDS for pneumonia (ePNa) identifies possible pneumonia patients, measures severity and antimicrobial resistance risk, and then recommends disposition, antibiotics, and microbiology studies. Use is voluntary, and clinicians may modify treatment recommendations. ePNa was associated with lower mortality in emergency department (ED) patients versus usual care (Annals EM 66:511). We adapted ePNa for the Cerner EHR, and implemented it across Intermountain Healthcare EDs (Utah, USA) throughout 2018. We introduced ePNa through didactic, interactive presentations to ED clinicians; follow-up visits identified barriers and facilitators to use. Email reminded clinicians and answered questions. Hospital admitting clinicians encouraged ePNa use to smooth care transitions. Audit-and-feedback measured utilization, showing variations from best practice when ePNa and associated electronic order sets were not used. Use was initially low, but gradually increased especially at larger hospitals. A user-friendly interface, frequent reminders, audit-and- feedback, a user survey, a nurse educator, and local physician champions are additive towards implementation success., (©2019 AMIA - All rights reserved.)

The prediction and characterization of outbreaks of infectious diseases such as influenza remains an open and important problem. This paper describes a framework for detecting and characterizing outbreaks of influenza and the results of testing it on data from ten outbreaks collected from two locations over five years. We model outbreaks with compartment models and explicitly model non-influenza influenza-like illnesses., (This is an Open Access article. Authors own copyright of their articles appearing in the Journal of Public Health Informatics. Readers may copy articles without permission of the copyright owner(s), as long as the author and OJPHI are acknowledged in the copy and the copy is used for educational, not-for-profit purposes.)

Intermountain Healthcare has designed and implemented a publish-subscribe (PubSub) infrastructure to support essential event processing workflows across our organization. A recent implementation of a commercial EMR highlighted the need to provide this capability on top of the EMR to support external applications and services that require access to triggering events within the EMR. A description of the PubSub architecture is presented. Use cases for health information exchange, public health reporting, and pulmonary embolism diagnosis that utilize PubSub are described, along with benefits of using the paradigm. Besides providing support for these external applications, the PubSub infrastructure allows additional event handling functionality not available in the commercial EMR. The open, standards-based nature of the design should allow other organizations to implement the system in their information systems environment.

During the last decade, software supporting healthcare delivery has proliferated. This software can be divided into electronic medical record (EHR) systems and applications that treat EHRs as platforms. These collect, manage, and interpret medical data, thereby adding value to associated EHRs. To reduce the burden of developing for multiple EHR platforms, a group of standards has evolved that allow software written for one vendor's EHR to be introduced into settings supported by other vendors. The Health Services Platform Consortium (HSPC) is a collaborative effort to advocate for standards that will make healthcare applications truly interoperable. In this document, we discuss the approach adopted by the consortium and the standards central to this approach. We discriminate between interoperability standards that support the plug-and-play transfer of applications from one vendor's EHR to another and knowledge portability standards that allow knowledge artifacts used in one software environment to be introduced effectively in others.

Outbreaks of infectious diseases such as influenza are a significant threat to human health. Because there are different strains of influenza which can cause independent outbreaks, and influenza can affect demographic groups at different rates and times, there is a need to recognize and characterize multiple outbreaks of influenza. This paper describes a Bayesian system that uses data from emergency department patient care reports to create epidemiological models of overlapping outbreaks of influenza. Clinical findings are extracted from patient care reports using natural language processing. These findings are analyzed by a case detection system to create disease likelihoods that are passed to a multiple outbreak detection system. We evaluated the system using real and simulated outbreaks. The results show that this approach can recognize and characterize overlapping outbreaks of influenza. We describe several extensions that appear promising., (Copyright © 2017 Elsevier Inc. All rights reserved.)

Background: To improve health outcomes and cut health care costs, we often need to conduct prediction/classification using large clinical datasets (aka, clinical big data), for example, to identify high-risk patients for preventive interventions. Machine learning has been proposed as a key technology for doing this. Machine learning has won most data science competitions and could support many clinical activities, yet only 15% of hospitals use it for even limited purposes. Despite familiarity with data, health care researchers often lack machine learning expertise to directly use clinical big data, creating a hurdle in realizing value from their data. Health care researchers can work with data scientists with deep machine learning knowledge, but it takes time and effort for both parties to communicate effectively. Facing a shortage in the United States of data scientists and hiring competition from companies with deep pockets, health care systems have difficulty recruiting data scientists. Building and generalizing a machine learning model often requires hundreds to thousands of manual iterations by data scientists to select the following: (1) hyper-parameter values and complex algorithms that greatly affect model accuracy and (2) operators and periods for temporally aggregating clinical attributes (eg, whether a patient's weight kept rising in the past year). This process becomes infeasible with limited budgets., Objective: This study's goal is to enable health care researchers to directly use clinical big data, make machine learning feasible with limited budgets and data scientist resources, and realize value from data., Methods: This study will allow us to achieve the following: (1) finish developing the new software, Automated Machine Learning (Auto-ML), to automate model selection for machine learning with clinical big data and validate Auto-ML on seven benchmark modeling problems of clinical importance; (2) apply Auto-ML and novel methodology to two new modeling problems crucial for care management allocation and pilot one model with care managers; and (3) perform simulations to estimate the impact of adopting Auto-ML on US patient outcomes., Results: We are currently writing Auto-ML's design document. We intend to finish our study by around the year 2022., Conclusions: Auto-ML will generalize to various clinical prediction/classification problems. With minimal help from data scientists, health care researchers can use Auto-ML to quickly build high-quality models. This will boost wider use of machine learning in health care and improve patient outcomes., (©Gang Luo, Bryan L Stone, Michael D Johnson, Peter Tarczy-Hornoch, Adam B Wilcox, Sean D Mooney, Xiaoming Sheng, Peter J Haug, Flory L Nkoy. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 29.08.2017.)

Objectives: This study evaluates the accuracy and portability of a natural language processing (NLP) tool for extracting clinical findings of influenza from clinical notes across two large healthcare systems. Effectiveness is evaluated on how well NLP supports downstream influenza case-detection for disease surveillance., Methods: We independently developed two NLP parsers, one at Intermountain Healthcare (IH) in Utah and the other at University of Pittsburgh Medical Center (UPMC) using local clinical notes from emergency department (ED) encounters of influenza. We measured NLP parser performance for the presence and absence of 70 clinical findings indicative of influenza. We then developed Bayesian network models from NLP processed reports and tested their ability to discriminate among cases of (1) influenza, (2) non-influenza influenza-like illness (NI-ILI), and (3) 'other' diagnosis., Results: On Intermountain Healthcare reports, recall and precision of the IH NLP parser were 0.71 and 0.75, respectively, and UPMC NLP parser, 0.67 and 0.79. On University of Pittsburgh Medical Center reports, recall and precision of the UPMC NLP parser were 0.73 and 0.80, respectively, and IH NLP parser, 0.53 and 0.80. Bayesian case-detection performance measured by AUROC for influenza versus non-influenza on Intermountain Healthcare cases was 0.93 (using IH NLP parser) and 0.93 (using UPMC NLP parser). Case-detection on University of Pittsburgh Medical Center cases was 0.95 (using UPMC NLP parser) and 0.83 (using IH NLP parser). For influenza versus NI-ILI on Intermountain Healthcare cases performance was 0.70 (using IH NLP parser) and 0.76 (using UPMC NLP parser). On University of Pisstburgh Medical Center cases, 0.76 (using UPMC NLP parser) and 0.65 (using IH NLP parser)., Conclusion: In all but one instance (influenza versus NI-ILI using IH cases), local parsers were more effective at supporting case-detection although performances of non-local parsers were reasonable.

Objective: Mining disease-specific associations from existing knowledge resources can be useful for building disease-specific ontologies and supporting knowledge-based applications. Many association mining techniques have been exploited. However, the challenge remains when those extracted associations contained much noise. It is unreliable to determine the relevance of the association by simply setting up arbitrary cut-off points on multiple scores of relevance; and it would be expensive to ask human experts to manually review a large number of associations. We propose that machine-learning-based classification can be used to separate the signal from the noise, and to provide a feasible approach to create and maintain disease-specific vocabularies., Method: We initially focused on disease-medication associations for the purpose of simplicity. For a disease of interest, we extracted potentially treatment-related drug concepts from biomedical literature citations and from a local clinical data repository. Each concept was associated with multiple measures of relevance (i.e., features) such as frequency of occurrence. For the machine purpose of learning, we formed nine datasets for three diseases with each disease having two single-source datasets and one from the combination of previous two datasets. All the datasets were labeled using existing reference standards. Thereafter, we conducted two experiments: (1) to test if adding features from the clinical data repository would improve the performance of classification achieved using features from the biomedical literature only, and (2) to determine if classifier(s) trained with known medication-disease data sets would be generalizable to new disease(s)., Results: Simple logistic regression and LogitBoost were two classifiers identified as the preferred models separately for the biomedical-literature datasets and combined datasets. The performance of the classification using combined features provided significant improvement beyond that using biomedical-literature features alone (p-value<0.001). The performance of the classifier built from known diseases to predict associated concepts for new diseases showed no significant difference from the performance of the classifier built and tested using the new disease's dataset., Conclusion: It is feasible to use classification approaches to automatically predict the relevance of a concept to a disease of interest. It is useful to combine features from disparate sources for the task of classification. Classifiers built from known diseases were generalizable to new diseases., (Copyright © 2017 Elsevier Inc. All rights reserved.)

Objectives: This study evaluates the accuracy and transferability of Bayesian case detection systems (BCD) that use clinical notes from emergency department (ED) to detect influenza cases., Methods: A BCD uses natural language processing (NLP) to infer the presence or absence of clinical findings from ED notes, which are fed into a Bayesain network classifier (BN) to infer patients' diagnoses. We developed BCDs at the University of Pittsburgh Medical Center (BCDUPMC) and Intermountain Healthcare in Utah (BCDIH). At each site, we manually built a rule-based NLP and trained a Bayesain network classifier from over 40,000 ED encounters between Jan. 2008 and May. 2010 using feature selection, machine learning, and expert debiasing approach. Transferability of a BCD in this study may be impacted by seven factors: development (source) institution, development parser, application (target) institution, application parser, NLP transfer, BN transfer, and classification task. We employed an ANOVA analysis to study their impacts on BCD performance., Results: Both BCDs discriminated well between influenza and non-influenza on local test cases (AUCs > 0.92). When tested for transferability using the other institution's cases, BCDUPMC discriminations declined minimally (AUC decreased from 0.95 to 0.94, p<0.01), and BCDIH discriminations declined more (from 0.93 to 0.87, p<0.0001). We attributed the BCDIH decline to the lower recall of the IH parser on UPMC notes. The ANOVA analysis showed five significant factors: development parser, application institution, application parser, BN transfer, and classification task., Conclusion: We demonstrated high influenza case detection performance in two large healthcare systems in two geographically separated regions, providing evidentiary support for the use of automated case detection from routinely collected electronic clinical notes in national influenza surveillance. The transferability could be improved by training Bayesian network classifier locally and increasing the accuracy of the NLP parser.

Objective: Healthcare communities have identified a significant need for disease-specific information. Disease-specific ontologies are useful in assisting the retrieval of disease-relevant information from various sources. However, building these ontologies is labor intensive. Our goal is to develop a system for an automated generation of disease-pertinent concepts from a popular knowledge resource for the building of disease-specific ontologies., Methods: A pipeline system was developed with an initial focus of generating disease-specific treatment vocabularies. It was comprised of the components of disease-specific citation retrieval, predication extraction, treatment predication extraction, treatment concept extraction, and relevance ranking. A semantic schema was developed to support the extraction of treatment predications and concepts. Four ranking approaches (i.e., occurrence, interest, degree centrality, and weighted degree centrality) were proposed to measure the relevance of treatment concepts to the disease of interest. We measured the performance of four ranks in terms of the mean precision at the top 100 concepts with five diseases, as well as the precision-recall curves against two reference vocabularies. The performance of the system was also compared to two baseline approaches., Results: The pipeline system achieved a mean precision of 0.80 for the top 100 concepts with the ranking by interest. There were no significant different among the four ranks (p=0.53). However, the pipeline-based system had significantly better performance than the two baselines., Conclusions: The pipeline system can be useful for an automated generation of disease-relevant treatment concepts from the biomedical literature., (Copyright © 2016 Elsevier Inc. All rights reserved.)

Objective: Disease-specific vocabularies are fundamental to many knowledge-based intelligent systems and applications like text annotation, cohort selection, disease diagnostic modeling, and therapy recommendation. Reference standards are critical in the development and validation of automated methods for disease-specific vocabularies. The goal of the present study is to design and test a generalizable method for the development of vocabulary reference standards from expert-curated, disease-specific biomedical literature resources., Methods: We formed disease-specific corpora from literature resources like textbooks, evidence-based synthesized online sources, clinical practice guidelines, and journal articles. Medical experts annotated and adjudicated disease-specific terms in four classes (i.e., causes or risk factors, signs or symptoms, diagnostic tests or results, and treatment). Annotations were mapped to UMLS concepts. We assessed source variation, the contribution of each source to build disease-specific vocabularies, the saturation of the vocabularies with respect to the number of used sources, and the generalizability of the method with different diseases., Results: The study resulted in 2588 string-unique annotations for heart failure in four classes, and 193 and 425 respectively for pulmonary embolism and rheumatoid arthritis in treatment class. Approximately 80% of the annotations were mapped to UMLS concepts. The agreement among heart failure sources ranged between 0.28 and 0.46. The contribution of these sources to the final vocabulary ranged between 18% and 49%. With the sources explored, the heart failure vocabulary reached near saturation in all four classes with the inclusion of minimal six sources (or between four to seven sources if only counting terms occurred in two or more sources). It took fewer sources to reach near saturation for the other two diseases in terms of the treatment class., Conclusions: We developed a method for the development of disease-specific reference vocabularies. Expert-curated biomedical literature resources are substantial for acquiring disease-specific medical knowledge. It is feasible to reach near saturation in a disease-specific vocabulary using a relatively small number of literature sources., (Published by Elsevier B.V.)

Objective: Develop an efficient non-clinical method for identifying promising computer-based protocols for clinical study. An in silico comparison can provide information that informs the decision to proceed to a clinical trial. The authors compared two existing computer-based insulin infusion protocols: eProtocol-insulin from Utah, USA, and Glucosafe from Denmark., Materials and Methods: The authors used eProtocol-insulin to manage intensive care unit (ICU) hyperglycemia with intravenous (IV) insulin from 2004 to 2010. Recommendations accepted by the bedside clinicians directly link the subsequent blood glucose values to eProtocol-insulin recommendations and provide a unique clinical database. The authors retrospectively compared in silico 18,984 eProtocol-insulin continuous IV insulin infusion rate recommendations from 408 ICU patients with those of Glucosafe, the candidate computer-based protocol. The subsequent blood glucose measurement value (low, on target, high) was used to identify if the insulin recommendation was too high, on target, or too low., Results: Glucosafe consistently provided more favorable continuous IV insulin infusion rate recommendations than eProtocol-insulin for on target (64% of comparisons), low (80% of comparisons), or high (70% of comparisons) blood glucose. Aggregated eProtocol-insulin and Glucosafe continuous IV insulin infusion rates were clinically similar though statistically significantly different (Wilcoxon signed rank test P = .01). In contrast, when stratified by low, on target, or high subsequent blood glucose measurement, insulin infusion rates from eProtocol-insulin and Glucosafe were statistically significantly different (Wilcoxon signed rank test, P < .001), and clinically different., Discussion: This in silico comparison appears to be an efficient nonclinical method for identifying promising computer-based protocols., Conclusion: Preclinical in silico comparison analytical framework allows rapid and inexpensive identification of computer-based protocol care strategies that justify expensive and burdensome clinical trials., (© The Author 2015. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For Permissions, please email: journals.permissions@oup.com.)

Natural language processing (NLP) technologies provide an opportunity to extract key patient data from free text documents within the electronic health record (EHR). We are developing a series of components from which to construct NLP pipelines. These pipelines typically begin with a component whose goal is to label sections within medical documents with codes indicating the anticipated semantics of their content. This Clinical Section Labeler prepares the document for further, focused information extraction. Below we describe the evaluation of six algorithms designed for use in a Clinical Section Labeler. These algorithms are trained with N-gram-based feature sets extracted from document sections and the document types. In the evaluation, 6 different Bayesian models were trained and used to assign one of 27 different topics to each section. A tree-augmented Bayesian network using the document type and N-grams derived from section headers proved most accurate in assigning individual sections appropriate section topics.

Research Objective: To develop scalable informatics infrastructure for normalization of both structured and unstructured electronic health record (EHR) data into a unified, concept-based model for high-throughput phenotype extraction., Materials and Methods: Software tools and applications were developed to extract information from EHRs. Representative and convenience samples of both structured and unstructured data from two EHR systems-Mayo Clinic and Intermountain Healthcare-were used for development and validation. Extracted information was standardized and normalized to meaningful use (MU) conformant terminology and value set standards using Clinical Element Models (CEMs). These resources were used to demonstrate semi-automatic execution of MU clinical-quality measures modeled using the Quality Data Model (QDM) and an open-source rules engine., Results: Using CEMs and open-source natural language processing and terminology services engines-namely, Apache clinical Text Analysis and Knowledge Extraction System (cTAKES) and Common Terminology Services (CTS2)-we developed a data-normalization platform that ensures data security, end-to-end connectivity, and reliable data flow within and across institutions. We demonstrated the applicability of this platform by executing a QDM-based MU quality measure that determines the percentage of patients between 18 and 75 years with diabetes whose most recent low-density lipoprotein cholesterol test result during the measurement year was <100 mg/dL on a randomly selected cohort of 273 Mayo Clinic patients. The platform identified 21 and 18 patients for the denominator and numerator of the quality measure, respectively. Validation results indicate that all identified patients meet the QDM-based criteria., Conclusions: End-to-end automated systems for extracting clinical information from diverse EHR systems require extensive use of standardized vocabularies and terminologies, as well as robust information models for storing, discovering, and processing that information. This study demonstrates the application of modular and open-source resources for enabling secondary use of EHR data through normalization into standards-based, comparable, and consistent format for high-throughput phenotyping to identify patient cohorts.

Objective: Natural language processing (NLP) tasks are commonly decomposed into subtasks, chained together to form processing pipelines. The residual error produced in these subtasks propagates, adversely affecting the end objectives. Limited availability of annotated clinical data remains a barrier to reaching state-of-the-art operating characteristics using statistically based NLP tools in the clinical domain. Here we explore the unique linguistic constructions of clinical texts and demonstrate the loss in operating characteristics when out-of-the-box part-of-speech (POS) tagging tools are applied to the clinical domain. We test a domain adaptation approach integrating a novel lexical-generation probability rule used in a transformation-based learner to boost POS performance on clinical narratives., Methods: Two target corpora from independent healthcare institutions were constructed from high frequency clinical narratives. Four leading POS taggers with their out-of-the-box models trained from general English and biomedical abstracts were evaluated against these clinical corpora. A high performing domain adaptation method, Easy Adapt, was compared to our newly proposed method ClinAdapt., Results: The evaluated POS taggers drop in accuracy by 8.5-15% when tested on clinical narratives. The highest performing tagger reports an accuracy of 88.6%. Domain adaptation with Easy Adapt reports accuracies of 88.3-91.0% on clinical texts. ClinAdapt reports 93.2-93.9%., Conclusions: ClinAdapt successfully boosts POS tagging performance through domain adaptation requiring a modest amount of annotated clinical data. Improving the performance of critical NLP subtasks is expected to reduce pipeline error propagation leading to better overall results on complex processing tasks.

Objectives: To present a system that uses knowledge stored in a medical ontology to automate the development of diagnostic decision support systems. To illustrate its function through an example focused on the development of a tool for diagnosing pneumonia., Materials and Methods: We developed a system that automates the creation of diagnostic decision-support applications. It relies on a medical ontology to direct the acquisition of clinic data from a clinical data warehouse and uses an automated analytic system to apply a sequence of machine learning algorithms that create applications for diagnostic screening. We refer to this system as the ontology-driven diagnostic modeling system (ODMS). We tested this system using samples of patient data collected in Salt Lake City emergency rooms and stored in Intermountain Healthcare's enterprise data warehouse., Results: The system was used in the preliminary development steps of a tool to identify patients with pneumonia in the emergency department. This tool was compared with a manually created diagnostic tool derived from a curated dataset. The manually created tool is currently in clinical use. The automatically created tool had an area under the receiver operating characteristic curve of 0.920 (95% CI 0.916 to 0.924), compared with 0.944 (95% CI 0.942 to 0.947) for the manually created tool., Discussion: Initial testing of the ODMS demonstrates promising accuracy for the highly automated results and illustrates the route to model improvement., Conclusions: The use of medical knowledge, embedded in ontologies, to direct the initial development of diagnostic computing systems appears feasible.

The relationship between patient disease status and the presence or absence of body mass index (BMI) data in the electronic health record (EHR) has not been characterized. We conducted a descriptive study of the completeness of BMI data for three patient cohorts. Cross-sectional descriptions of BMI presence rates per patient were compared between a cohort having at least one ICD-9-CM code for diabetes mellitus (DM) versus a cohort with no diagnosis constraints. Conversely, frequencies of encounter diagnoses were compared among subgroups having BMI recorded or not in both cohorts described and a third cohort having DM codes from a separate organization's EHR. The data demonstrate a correlation with presence of BMI and higher disease status. This effect may bias the cohort average BMIs, which appear higher than expected. When EHR BMI data are repurposed for research, biases in the selective recording of BMI may affect the results.

The Strategic Health IT Advanced Research Projects (SHARP) Program, established by the Office of the National Coordinator for Health Information Technology in 2010 supports research findings that remove barriers for increased adoption of health IT. The improvements envisioned by the SHARP Area 4 Consortium (SHARPn) will enable the use of the electronic health record (EHR) for secondary purposes, such as care process and outcomes improvement, biomedical research and epidemiologic monitoring of the nation's health. One of the primary informatics problem areas in this endeavor is the standardization of disparate health data from the nation's many health care organizations and providers. The SHARPn team is developing open source services and components to support the ubiquitous exchange, sharing and reuse or 'liquidity' of operational clinical data stored in electronic health records. One year into the design and development of the SHARPn framework, we demonstrated end to end data flow and a prototype SHARPn platform, using thousands of patient electronic records sourced from two large healthcare organizations: Mayo Clinic and Intermountain Healthcare. The platform was deployed to (1) receive source EHR data in several formats, (2) generate structured data from EHR narrative text, and (3) normalize the EHR data using common detailed clinical models and Consolidated Health Informatics standard terminologies, which were (4) accessed by a phenotyping service using normalized data specifications. The architecture of this prototype SHARPn platform is presented. The EHR data throughput demonstration showed success in normalizing native EHR data, both structured and narrative, from two independent organizations and EHR systems. Based on the demonstration, observed challenges for standardization of EHR data for interoperable secondary use are discussed., (Copyright © 2012 Elsevier Inc. All rights reserved.)

The Arden Syntax is an HL7 standard language for representing medical knowledge as logic statements. Despite nearly 2 decades of availability, Arden Syntax has not been widely used. This has been attributed to the lack of a generally available compiler to implement the logic, to Arden's complex syntax, to the challenges of mapping local data to data references in the Medical Logic Modules (MLMs), or, more globally, to the general absence of decision support in healthcare computing. An XML representation (ArdenML) may partially address the technical challenges. MLMs created in ArdenML can be converted into executable files using standard transforms written in the Extensible Stylesheet Language Transformation (XSLT) language. As an example, we have demonstrated an approach to executing MLMs written in ArdenML using the Drools business rule management system. Extensions to ArdenML make it possible to generate a user interface through which an MLM developer can test for logical errors.

A semantic lexicon which associates words and phrases in text to concepts is critical for extracting and encoding clinical information in free text and therefore achieving semantic interoperability between structured and unstructured data in Electronic Health Records (EHRs). Directly using existing standard terminologies may have limited coverage with respect to concepts and their corresponding mentions in text. In this paper, we analyze how tokens and phrases in a large corpus distribute and how well the UMLS captures the semantics. A corpus-driven semantic lexicon, MedLex, has been constructed where the semantics is based on the UMLS assisted with variants mined and usage information gathered from clinical text. The detailed corpus analysis of tokens, chunks, and concept mentions shows the UMLS is an invaluable source for natural language processing. Increasing the semantic coverage of tokens provides a good foundation in capturing clinical information comprehensively. The study also yields some insights in developing practical NLP systems.

Sepsis is a systemic inflammatory state due to an infection, and is associated with very high mortality and morbidity. Early diagnosis and prompt antibiotic and supportive therapy is associated with improved outcomes. Our objective was to detect the presence of sepsis soon after the patient visits the emergency department. We used Dynamic Bayesian Networks, a temporal probabilistic technique to model a system whose state changes over time. We built, trained and tested the model using data of 3,100 patients admitted to the emergency department, and measured the accuracy of detecting sepsis using data collected within the first 3 hours, 6 hours, 12 hours and 24 hours after admission. The area under the curve was 0.911, 0.915, 0.937 and 0.944 respectively. We describe the data, data preparation techniques, model, results, various statistical measures and the limitations of our experiments. We also briefly discuss techniques to improve accuracy, and the generalizability of our methods to other diseases.

A standards-based, service-oriented architecture for clinical decision support (CDS) has the potential to significantly enhance CDS scalability and robustness. To enable such a CDS architecture, the Health Level 7 CDS Work Group reviewed the literature, hosted multi-stakeholder discussions, and consulted domain experts to identify and prioritize the services and capabilities required from clinical information systems (CISs) to enable service-oriented CDS. In addition, relevant available standards were identified. Through this process, ten CIS services and eight CIS capabilities were identified as being important for enabling scalable, service-oriented CDS. In particular, through a survey of 46 domain experts, five services and capabilities were identified as being especially critical: 1) the use of standard information models and terminologies; 2) the ability to leverage a Decision Support Service (DSS); 3) support for a clinical data query service; 4) support for an event subscription and notification service; and 5) support for a user communication service.

Purpose: To compare colonoscopy screening/surveillance rates by level of risk for colorectal cancer based on age, personal history of adenomatous polyps or colorectal cancer, or family history of colorectal cancer., Methods: Participants were aged 30-90 years, were seen within 5 years at Intermountain Healthcare, and had family history in the Utah Population Database. Colonoscopy rates were measured for those with/without risk factors., Results: Among those aged 60-69 years, 48.4% had colonoscopy in the last 10 years, with rates declining after age 70 years. Percentages of those having had a colonoscopy in the last 10 years generally increased by risk level from 38.5% in those with a familial relative risk <1.0 to 47.6% in those with a familial relative risk >3.0. Compared with those with no family history, the odds ratio for being screened according to guidelines was higher for those with one first-degree relative diagnosed with colorectal cancer ≥ 60 years or two affected second-degree relatives (1.54, 95% confidence interval: 1.46-1.61) than those with one affected first-degree relative diagnosed <60 years or ≥2 affected first-degree relatives (1.25, 95% confidence interval: 1.14-1.37)., Conclusions: Compliance with colonoscopy guidelines was higher for those with familial risk but did not correspond with the degree of risk.

Objective: To understand how the source of information affects different adverse event (AE) surveillance methods., Design: Retrospective analysis of inpatient adverse drug events (ADEs) and hospital-associated infections (HAIs) detected by either a computerized surveillance system (CSS) or manual chart review (MCR)., Measurement: Descriptive analysis of events detected using the two methods by type of AE, type of information about the AE, and sources of the information., Results: CSS detected more HAIs than MCR (92% vs 34%); however, a similar number of ADEs was detected by both systems (52% vs 51%). The agreement between systems was greater for HAIs than ADEs (26% vs 3%). The CSS missed events that did not have information in coded format or that were described only in physician narratives. The MCR detected events missed by CSS using information in physician narratives. Discharge summaries were more likely to contain information about AEs than any other type of physician narrative, followed by emergency department reports for HAIs and general consult notes for ADEs. Some ADEs found by MCR were detected by CSS but not verified by a clinician., Limitations: Inability to distinguish between CSS false positives and suspected AEs for cases in which the clinician did not document their assessment in the CSS., Conclusion: The effect that information source has on different surveillance methods depends on the type of AE. Integrating information from physician narratives with CSS using natural language processing would improve the detection of ADEs more than HAIs.

Purpose: Using a large, retrospective cohort from the Utah Population Database, we assess how well family history predicts who will acquire colorectal cancer during a 20-year period., Methods: Individuals were selected between ages 35 and 80 with no prior record of colorectal cancer diagnosis, as of the year 1985. Numbers of colorectal cancer-affected relatives and diagnosis ages were collected. Familial relative risk and absolute risk estimates were calculated. Colorectal cancer diagnoses in the cohort were counted between years 1986 and 2005. Cox regression and Harrell's C were used to measure the discriminatory power of resulting models., Results: A total of 431,153 individuals were included with 5,334 colorectal cancer diagnoses. Familial relative risk ranged from 0.83 to 12.39 and 20-year absolute risk from 0.002 to 0.21. With familial relative risk as the only predictor, Harrell's C = 0.53 and with age only, Harrell's C = 0.66. Familial relative risk combined with age produced a Harrell's C = 0.67., Conclusion: Family history by itself is not a strong predictor of exactly who will acquire colorectal cancer within 20 years. However, stratification of risk using absolute risk probabilities may be more helpful in focusing screening on individuals who are more likely to develop the disease.

Personalized medicine will require detailed clinical patient profiles, and a particular focus on capturing data that is useful in forecasting risk. A detailed family health history is considered a critical component of these profiles, insomuch that it has been coined as 'the best genetic test available'. Despite this, tools aimed at capturing this information for use in electronic health records have been characterized as inadequate. In this manuscript we detail the creation of a patient-facing family health history tool known as OurFamilyHealth, whose long-term emphasis is to facilitate risk assessment and clinical decision support. We present the rationale for such a tool, describe its development and release as a component of Intermountain Healthcare's patient portal, and detail early usage statistics surrounding the application. Data derived from the tool since its release are also compared against family history charting patterns in Intermountain's electronic health records, revealing differences in data availability.

Adequate control of serum glucose in critically ill patients is a complex problem requiring continuous monitoring and intervention, which have a direct effect on clinical outcomes. Understanding temporal relationships can help to improve our knowledge of complex disease processes and their response to treatment. We discuss a Dynamic Bayesian Network (DBN) model that we created using the open-source Projeny toolkit to represent various clinical variables and the temporal and atemporal relationships underlying insulin and glucose homeostasis. We evaluated this model by comparing the DBN model's insulin dose predictions against those of a rule-based protocol (eProtocol-insulin) currently used in the ICU. The results suggest that the DBN model's predictions are as effective as or better than those of the rule-based protocol. The limitations of our methods are discussed, with a brief note on their generalizability.

Infobuttons have been proven as an effective means for providing quick, context-specific links to pertinent information resources at the point of care. Current infobutton manager implementations, however, lack the ability to exchange metadata, are limited to a relatively small set of information providers, and are targeted primarily for a clinician audience. As part of a local effort to implement infobuttons for patient use via a tethered personal health record, we present a series of metadata extraction routines. These routines were constructed to extract key pieces of information from health information providers on the Internet, including content coverage, language availability, and readability scores. The extraction routines were tested using thirty different disease conditions against eight different providers. The routines yielded 183 potential infobutton targets and associated metadata for each. The capabilities of the extraction routines will be expanded to cover new types of metadata in the future.

The family health history has long been recognized as an effective way of understanding individuals' susceptibility to familial disease; yet electronic tools to support the capture and use of these data have been characterized as inadequate. As part of an ongoing effort to build patient-facing tools for entering detailed family health histories, we have compiled a set of concepts specific to familial disease using multi-source sampling. These concepts were abstracted by analyzing family health history data patterns in our enterprise data warehouse, collection patterns of consumer personal health records, analyses from the local state health department, a healthcare data dictionary, and concepts derived from genetic-oriented consumer education materials. Collectively, these sources yielded a set of more than 500 unique disease concepts, represented by more than 2500 synonyms for supporting patients in entering coded family health histories. We expect that these concepts will be useful in providing meaningful data and education resources for patients and providers alike.

Background & Aims: Colorectal cancer (CRC) risk estimates based on family history typically include only close relatives. We report familial relative risk (FRR) in probands with various combinations, or constellations, of affected relatives, extending to third-degree., Methods: A population-based resource that includes a computerized genealogy linked to statewide cancer records was used to identify genetic relationships among CRC cases and their first-, second-, and third-degree relatives (FDRs, SDRs, and TDRs). FRRs were estimated by comparing the observed number of affected persons with a particular family history constellation to the expected number, based on cohort-specific CRC rates., Results: A total of 2,327,327 persons included in > or =3 generation family histories were analyzed; 10,556 had a diagnosis of CRC. The FRR for CRC in persons with > or =1 affected FDR = 2.05 (95% CI, 1.96-2.14), consistent with published estimates. In the absence of a positive first-degree family history, considering both affected SDRs and TDRs, only 1 constellation had an FRR estimate that was significantly >1.0 (0 affected FDRs, 1 affected SDR, 2 affected TDRs; FRR = 1.33; 95% CI, 1.13-1.55). The FRR for persons with 1 affected FDR, 1 affected SDR, and 0 affected TDRs was 1.88 (95% CI, 1.59-2.20), increasing to FRR = 3.28 (95% CI, 2.44-4.31) for probands with 1 affected FDR, 1 affected SDR, and > or =3 affected TDRs., Conclusions: Increased numbers of affected FDRs influences risk much more than affected SDRs or TDRs. However, when combined with a positive first-degree family history, a positive second- and third-degree family history can significantly increase risk., (Copyright 2010 AGA Institute. Published by Elsevier Inc. All rights reserved.)

Natural language processing applications that extract information from text rely on semantic representations. The objective of this paper is to describe a methodology for creating a semantic representation for information that will be automatically extracted from textual clinical records. We illustrate two of the four steps of the methodology in this paper using the case study of encoding information from dictated dental exams: (1) develop an initial representation from a set of training documents and (2) iteratively evaluate and evolve the representation while developing annotation guidelines. Our approach for developing and evaluating a semantic representation is based on standard principles and approaches that are not dependent on any particular domain or type of semantic representation.

Study Objective: The goals of this investigation were to study the temporal relationships between the demands for key resources in the emergency department (ED) and the inpatient hospital, and to develop multivariate forecasting models., Methods: Hourly data were collected from three diverse hospitals for the year 2006. Descriptive analysis and model fitting were carried out using graphical and multivariate time series methods. Multivariate models were compared to a univariate benchmark model in terms of their ability to provide out-of-sample forecasts of ED census and the demands for diagnostic resources., Results: Descriptive analyses revealed little temporal interaction between the demand for inpatient resources and the demand for ED resources at the facilities considered. Multivariate models provided more accurate forecasts of ED census and of the demands for diagnostic resources., Conclusion: Our results suggest that multivariate time series models can be used to reliably forecast ED patient census; however, forecasts of the demands for diagnostic resources were not sufficiently reliable to be useful in the clinical setting.

Family history (FH) is an important risk factor for many diseases and its assessment can be a powerful tool for identifying and stratifying patients at risk. As part of an initiative to improve FH collection and decision support at Intermountain Healthcare we conducted a literature review on FH data collection and reviewed a number of current Web-based tools. An ideal list of features was produced to help implement goals of the initiative.

In critically ill patients control of blood sugar levels with IV insulin has been shown to improve clinical outcomes in the intensive care units. We have developed an analytical framework with which to evaluate and compare IV insulin-treatment models and protocols. Performance of the analytical framework is demonstrated using protocols published by others and new protocols under development by our group.

Current trends have brought resurgent interest in developing consumer-oriented tools that gather patient-entered clinical data. Family health history data has long been recognized as valuable for risk assessment in primary care, but has gained renewed attention recently as part of IT-oriented efforts in personalized medicine. In order to better understand the breadth of data collected in consumer-oriented web applications, we evaluated their collection patterns using the recommendations issued by the American Health Information Community (AHIC).

Constantly changing diabetes care standards makes it challenging to deliver care adapted to the unique condition of the individual patient. The availability of large amounts of data from patient's electronic medical records makes it possible to individualize diabetes management. Initial design of a "patient-specific" hybrid system (physiological-causal probabilistic) of adaptive diabetes models and insulin treatment algorithms will be presented. The system is uniquely derived and tested using a diabetes data mart of about 33,000 patients.

Objective: Infobuttons are decision support tools that provide links within electronic medical record systems to relevant content in online information resources. The aim of infobuttons is to help clinicians promptly meet their information needs. The objective of this study was to determine whether infobutton links that direct to specific content topics ("topic links") are more effective than links that point to general overview content ("nonspecific links")., Design: Randomized controlled trial with a control and an intervention group. Clinicians in the control group had access to nonspecific links, while those in the intervention group had access to topic links., Measurements: Infobutton session duration, number of infobutton sessions, session success rate, and the self-reported impact that the infobutton session produced on decision making., Results: The analysis was performed on 90 subjects and 3,729 infobutton sessions. Subjects in the intervention group spent 17.4% less time seeking for information (35.5 seconds vs. 43 seconds, p = 0.008) than those in the control group. Subjects in the intervention group used infobuttons 20.5% (22 sessions vs. 17.5 sessions, p = 0.21) more often than in the control group, but the difference was not significant. The information seeking success rate was equally high in both groups (89.4% control vs. 87.2% intervention, p = 0.99). Subjects reported a high positive clinical impact (i.e., decision enhancement or knowledge update) in 62% of the sessions. Limitations The exclusion of users with a low frequency of infobutton use and the focus on medication-related information needs may limit the generalization of the results. The session outcomes measurement was based on clinicians' self-assessment and therefore prone to bias., Conclusion: The results support the hypothesis that topic links are more efficient than nonspecific links regarding the time seeking for information. It is unclear whether the statistical difference demonstrated will result in a clinically significant impact. However, the overall results confirm previous evidence that infobuttons are effective at helping clinicians to answer questions at the point of care and demonstrate a modest incremental change in the efficiency of information delivery for routine users of this tool.

Purpose: A new model expressing Arden Syntax with the eXtensible Markup Language (XML) was developed to increase its portability., Methods: Every example was manually parsed and reviewed until the schema and the style sheet were considered to be optimized. When the first schema was finished, several MLMs in Arden Syntax Markup Language (ArdenML) were validated against the schema. They were then transformed to HTML formats with the style sheet, during which they were compared to the original text version of their own MLM. When faults were found in the transformed MLM, the schema and/or style sheet was fixed. This cycle continued until all the examples were encoded into XML documents. The original MLMs were encoded in XML according to the proposed XML schema and reverse-parsed MLMs in ArdenML were checked using a public domain Arden Syntax checker., Results: Two hundred seventy seven examples of MLMs were successfully transformed into XML documents using the model, and the reverse-parse yielded the original text version of MLMs. Two hundred sixty five of the 277 MLMs showed the same error patterns before and after transformation, and all 11 errors related to statement structure were resolved in XML version. The model uses two syntax checking mechanisms, first an XML validation process, and second, a syntax check using an XSL style sheet., Conclusion: Now that we have a schema for ArdenML, we can also begin the development of style sheets for transformation ArdenML into other languages.

Purpose: To improve the completeness and timeliness of an electronic problem list, we have developed a system using Natural Language Processing (NLP) to automatically extract potential medical problems from clinical, free-text documents; these problems are then proposed for inclusion in an electronic problem list management application., Methods: A prospective randomized controlled evaluation of the Automatic Problem List (APL) system in an intensive care unit and in a cardiovascular surgery unit is reported here. A total of 247 patients were enrolled: 76 in an initial control phase and 171 in the randomized controlled trial that followed. During this latter phase, patients were randomly assigned to a control or an intervention group. All patients had their documents analyzed by the system, but the medical problems discovered were only proposed in the problem list for intervention patients. We measured the sensitivity, specificity, positive and negative predictive values, likelihood ratios and the timeliness of the problem lists., Results: Our system significantly increased the sensitivity of the problem lists in the intensive care unit, from about 9% to 41%, and even 77% if problems automatically proposed but not acknowledged by users were also considered. Timeliness of addition of problems to the list was greatly improved, with a time between a problem's first mention in a clinical document and its addition to the problem list reduced from about 6 days to less than 2 days. No significant effect was observed in the cardiovascular surgery unit.

Objective: Infobuttons are decision support tools that offer links to information resources based on the context of the interaction between a clinician and an electronic medical record (EMR) system. The objective of this study was to explore machine learning and web usage mining methods to produce classification models for the prediction of information resources that might be relevant in a particular infobutton context., Design: Classification models were developed and evaluated with an infobutton usage dataset. The performance of the models was measured and compared with a reference implementation in a series of experiments., Measurements: Level of agreement (kappa) between the models and the resources that clinicians actually used in each infobutton session., Results: The classification models performed significantly better than the reference implementation (p<.0001). The performance of these models tended to decrease over time, probably due to a phenomenon known as concept drift. However, the performance of the models remained stable when concept drift handling techniques were used., Conclusions: The results suggest that classification models are a promising method for the prediction of information resources that a clinician would use to answer patient care questions.