Search

Objective: To develop and validate an automated method for determining the set of patients for whom a given primary care physician holds overall clinical responsibility.Design: The study included all adult patients (16,185) seen at least once in an ambulatory setting during a three-year period by 18 primary care physicians in ten practices. The physicians indicated whether they considered themselves to be the physician primarily responsible for the overall clinical care of each visiting patient. Statistical models were constructed to predict the physicians' designations using predictor variables derived from electronically available appointment schedules and demographic information.Measurements: Predictive accuracy was assessed primarily using the area under the receiver-operating characteristic curve (AUC), and secondarily using positive predictive value (PPV) and sensitivity.Results: A minimal set of six variables was identified as predictive of the physicians' designations. The constructed model had a median AUC for individual physicians of 0.92 (interquartile interval: 0.90-0.96), a PPV of 0.94 (interquartile interval: 0.87-0.95), and a sensitivity of 0.95 (interquartile interval: 0.87-0.97).Conclusion: A statistical model using a minimal set of commonly available electronic data can accurately predict the set of patients for whom a physician holds primary clinical responsibility. Further research examining the generalization of the model to other settings would be valuable. [ABSTRACT FROM AUTHOR]

Objectives: Electronic health records (EHR) can allow for the generation of large cohorts of individuals with given diseases for clinical and genomic research. A rate-limiting step is the development of electronic phenotype selection algorithms to find such cohorts. This study evaluated the portability of a published phenotype algorithm to identify rheumatoid arthritis (RA) patients from EHR records at three institutions with different EHR systems.Materials and Methods: Physicians reviewed charts from three institutions to identify patients with RA. Each institution compiled attributes from various sources in the EHR, including codified data and clinical narratives, which were searched using one of two natural language processing (NLP) systems. The performance of the published model was compared with locally retrained models.Results: Applying the previously published model from Partners Healthcare to datasets from Northwestern and Vanderbilt Universities, the area under the receiver operating characteristic curve was found to be 92% for Northwestern and 95% for Vanderbilt, compared with 97% at Partners. Retraining the model improved the average sensitivity at a specificity of 97% to 72% from the original 65%. Both the original logistic regression models and locally retrained models were superior to simple billing code count thresholds.Discussion: These results show that a previously published algorithm for RA is portable to two external hospitals using different EHR systems, different NLP systems, and different target NLP vocabularies. Retraining the algorithm primarily increased the sensitivity at each site.Conclusion: Electronic phenotype algorithms allow rapid identification of case populations in multiple sites with little retraining. [ABSTRACT FROM AUTHOR]

Objective: To develop and evaluate a data-driven process to generate suggestions for improving alert criteria using explainable artificial intelligence (XAI) approaches., Methods: We extracted data on alerts generated from January 1, 2019 to December 31, 2020, at Vanderbilt University Medical Center. We developed machine learning models to predict user responses to alerts. We applied XAI techniques to generate global explanations and local explanations. We evaluated the generated suggestions by comparing with alert's historical change logs and stakeholder interviews. Suggestions that either matched (or partially matched) changes already made to the alert or were considered clinically correct were classified as helpful., Results: The final dataset included 2 991 823 firings with 2689 features. Among the 5 machine learning models, the LightGBM model achieved the highest Area under the ROC Curve: 0.919 [0.918, 0.920]. We identified 96 helpful suggestions. A total of 278 807 firings (9.3%) could have been eliminated. Some of the suggestions also revealed workflow and education issues., Conclusion: We developed a data-driven process to generate suggestions for improving alert criteria using XAI techniques. Our approach could identify improvements regarding clinical decision support (CDS) that might be overlooked or delayed in manual reviews. It also unveils a secondary purpose for the XAI: to improve quality by discovering scenarios where CDS alerts are not accepted due to workflow, education, or staffing issues., (© The Author(s) 2024. Published by Oxford University Press on behalf of the American Medical Informatics Association.)

Background: Large community cohorts are useful for lung cancer research, allowing for the analysis of risk factors and development of predictive models., Objective: A robust methodology for (1) identifying lung cancer and pulmonary nodules diagnoses as well as (2) associating multimodal longitudinal data with these events from electronic health record (EHRs) is needed to optimally curate cohorts at scale., Methods: In this study, we leveraged (1) SNOMED concepts to develop ICD-based decision rules for building a cohort that captured lung cancer and pulmonary nodules and (2) clinical knowledge to define time windows for collecting longitudinal imaging and clinical concepts. We curated three cohorts with clinical data and repeated imaging for subjects with pulmonary nodules from our Vanderbilt University Medical Center., Results: Our approach achieved an estimated sensitivity 0.930 (95% CI: [0.879, 0.969]), specificity of 0.996 (95% CI: [0.989, 1.00]), positive predictive value of 0.979 (95% CI: [0.959, 1.000]), and negative predictive value of 0.987 (95% CI: [0.976, 0.994]) for distinguishing lung cancer from subjects with SPNs., Conclusion: This work represents a general strategy for high-throughput curation of multi-modal longitudinal cohorts at risk for lung cancer from routinely collected EHRs.

The rising popularity of artificial intelligence in healthcare is highlighting the problem that a computational model achieving super-human clinical performance at its training sites may perform substantially worse at new sites. In this perspective, we argue that we should typically expect this failure to transport, and we present common sources for it, divided into those under the control of the experimenter and those inherent to the clinical data-generating process. Of the inherent sources we look a little deeper into site-specific clinical practices that can affect the data distribution, and propose a potential solution intended to isolate the imprint of those practices on the data from the patterns of disease cause and effect that are the usual target of probabilistic clinical models., (© 2024. The Author(s).)

Treatments ideally mitigate pathogenesis, or the detrimental effects of the root causes of disease. However, existing definitions of treatment effect fail to account for pathogenic mechanism. We therefore introduce the Treated Root causal Effects (TRE) metric which measures the ability of a treatment to modify root causal effects. We leverage TREs to automatically identify treatment targets and cluster patients who respond similarly to treatment. The proposed algorithm learns a partially linear causal model to extract the root causal effects of each variable and then estimates TREs for target discovery and downstream subtyping. We maintain interpretability even without assuming an invertible structural equation model. Experiments across a range of datasets corroborate the generality of the proposed approach., Competing Interests: Declaration of competing interest None Declared., (Copyright © 2024 Elsevier Ltd. All rights reserved.)

Transformer-based models, capable of learning better global dependencies, have recently demonstrated exceptional representation learning capabilities in computer vision and medical image analysis. Transformer reformats the image into separate patches and realizes global communication via the self-attention mechanism. However, positional information between patches is hard to preserve in such 1D sequences, and loss of it can lead to sub-optimal performance when dealing with large amounts of heterogeneous tissues of various sizes in 3D medical image segmentation. Additionally, current methods are not robust and efficient for heavy-duty medical segmentation tasks such as predicting a large number of tissue classes or modeling globally inter-connected tissue structures. To address such challenges and inspired by the nested hierarchical structures in vision transformer, we proposed a novel 3D medical image segmentation method (UNesT), employing a simplified and faster-converging transformer encoder design that achieves local communication among spatially adjacent patch sequences by aggregating them hierarchically. We extensively validate our method on multiple challenging datasets, consisting of multiple modalities, anatomies, and a wide range of tissue classes, including 133 structures in the brain, 14 organs in the abdomen, 4 hierarchical components in the kidneys, inter-connected kidney tumors and brain tumors. We show that UNesT consistently achieves state-of-the-art performance and evaluate its generalizability and data efficiency. Particularly, the model achieves whole brain segmentation task complete ROI with 133 tissue classes in a single network, outperforming prior state-of-the-art method SLANT27 ensembled with 27 networks. Our model performance increases the mean DSC score of the publicly available Colin and CANDI dataset from 0.7264 to 0.7444 and from 0.6968 to 0.7025, respectively. Code, pre-trained models, and use case pipeline are available at: https://github.com/MASILab/UNesT., Competing Interests: Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Yucheng Tang reports financial support was provided by Vanderbilt University. Yucheng Tang reports a relationship with Vanderbilt University that includes: employment., (Copyright © 2023 Elsevier B.V. All rights reserved.)

The accuracy of predictive models for solitary pulmonary nodule (SPN) diagnosis can be greatly increased by incorporating repeat imaging and medical context, such as electronic health records (EHRs). However, clinically routine modalities such as imaging and diagnostic codes can be asynchronous and irregularly sampled over different time scales which are obstacles to longitudinal multimodal learning. In this work, we propose a transformer-based multimodal strategy to integrate repeat imaging with longitudinal clinical signatures from routinely collected EHRs for SPN classification. We perform unsupervised disentanglement of latent clinical signatures and leverage time-distance scaled self-attention to jointly learn from clinical signatures expressions and chest computed tomography (CT) scans. Our classifier is pretrained on 2,668 scans from a public dataset and 1,149 subjects with longitudinal chest CTs, billing codes, medications, and laboratory tests from EHRs of our home institution. Evaluation on 227 subjects with challenging SPNs revealed a significant AUC improvement over a longitudinal multimodal baseline (0.824 vs 0.752 AUC), as well as improvements over a single cross-section multimodal scenario (0.809 AUC) and a longitudinal imaging-only scenario (0.741 AUC). This work demonstrates significant advantages with a novel approach for co-learning longitudinal imaging and non-imaging phenotypes with transformers. Code available at https://github.com/MASILab/lmsignatures.

Background An artificial intelligence (AI) algorithm has been developed for fully automated body composition assessment of lung cancer screening noncontrast low-dose CT of the chest (LDCT) scans, but the utility of these measurements in disease risk prediction models has not been assessed. Purpose To evaluate the added value of CT-based AI-derived body composition measurements in risk prediction of lung cancer incidence, lung cancer death, cardiovascular disease (CVD) death, and all-cause mortality in the National Lung Screening Trial (NLST). Materials and Methods In this secondary analysis of the NLST, body composition measurements, including area and attenuation attributes of skeletal muscle and subcutaneous adipose tissue, were derived from baseline LDCT examinations by using a previously developed AI algorithm. The added value of these measurements was assessed with sex- and cause-specific Cox proportional hazards models with and without the AI-derived body composition measurements for predicting lung cancer incidence, lung cancer death, CVD death, and all-cause mortality. Models were adjusted for confounding variables including age; body mass index; quantitative emphysema; coronary artery calcification; history of diabetes, heart disease, hypertension, and stroke; and other PLCO M2012 lung cancer risk factors. Goodness-of-fit improvements were assessed with the likelihood ratio test. Results Among 20 768 included participants (median age, 61 years [IQR, 57-65 years]; 12 317 men), 865 were diagnosed with lung cancer and 4180 died during follow-up. Including the AI-derived body composition measurements improved risk prediction for lung cancer death (male participants: χ 2 = 23.09, P < .001; female participants: χ 2 = 15.04, P = .002), CVD death (males: χ 2 = 69.94, P < .001; females: χ 2 = 16.60, P < .001), and all-cause mortality (males: χ 2 = 248.13, P < .001; females: χ 2 = 94.54, P < .001), but not for lung cancer incidence (male participants: χ 2 = 2.53, P = .11; female participants: χ 2 = 1.73, P = .19). Conclusion The body composition measurements automatically derived from baseline low-dose CT examinations added predictive value for lung cancer death, CVD death, and all-cause death, but not for lung cancer incidence in the NLST. Clinical trial registration no. NCT00047385 © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Fintelmann in this issue.

Features learned from single radiologic images are unable to provide information about whether and how much a lesion may be changing over time. Time-dependent features computed from repeated images can capture those changes and help identify malignant lesions by their temporal behavior. However, longitudinal medical imaging presents the unique challenge of sparse, irregular time intervals in data acquisition. While self-attention has been shown to be a versatile and efficient learning mechanism for time series and natural images, its potential for interpreting temporal distance between sparse, irregularly sampled spatial features has not been explored. In this work, we propose two interpretations of a time-distance vision transformer (ViT) by using (1) vector embeddings of continuous time and (2) a temporal emphasis model to scale self-attention weights. The two algorithms are evaluated based on benign versus malignant lung cancer discrimination of synthetic pulmonary nodules and lung screening computed tomography studies from the National Lung Screening Trial (NLST). Experiments evaluating the time-distance ViTs on synthetic nodules show a fundamental improvement in classifying irregularly sampled longitudinal images when compared to standard ViTs. In cross-validation on screening chest CTs from the NLST, our methods (0.785 and 0.786 AUC respectively) significantly outperform a cross-sectional approach (0.734 AUC) and match the discriminative performance of the leading longitudinal medical imaging algorithm (0.779 AUC) on benign versus malignant classification. This work represents the first self-attention-based framework for classifying longitudinal medical images. Our code is available at https://github.com/tom1193/time-distance-transformer.

Objective: Patients with indeterminate pulmonary nodules (IPN) with an intermediate to a high probability of lung cancer generally undergo invasive diagnostic procedures. Chest computed tomography image and clinical data have been in estimating the pretest probability of lung cancer. In this study, we apply a deep learning network to integrate multi-modal data from CT images and clinical data (including blood-based biomarkers) to improve lung cancer diagnosis. Our goal is to reduce uncertainty and to avoid morbidity, mortality, over- and undertreatment of patients with IPNs., Method: We use a retrospective study design with cross-validation and external-validation from four different sites. We introduce a deep learning framework with a two-path structure to learn from CT images and clinical data. The proposed model can learn and predict with single modality if the multi-modal data is not complete. We use 1284 patients in the learning cohort for model development. Three external sites (with 155, 136 and 96 patients, respectively) provided patient data for external validation. We compare our model to widely applied clinical prediction models (Mayo and Brock models) and image-only methods (e.g., Liao et al. model)., Results: Our co-learning model improves upon the performance of clinical-factor-only (Mayo and Brock models) and image-only (Liao et al.) models in both cross-validation of learning cohort (e.g., , Auc: 0.787 (ours) vs. 0.707-0.719 (baselines), results reported in validation fold and external-validation using three datasets from University of Pittsburgh Medical Center (e.g., 0.918 (ours) vs. 0.828-0.886 (baselines)), Detection of Early Cancer Among Military Personnel (e.g., 0.712 (ours) vs. 0.576-0.709 (baselines)), and University of Colorado Denver (e.g., 0.847 (ours) vs. 0.679-0.746 (baselines)). In addition, our model achieves better re-classification performance (cNRI 0.04 to 0.20) in all cross- and external-validation sets compared to the Mayo model., Conclusions: Lung cancer risk estimation in patients with IPNs can benefit from the co-learning of CT image and clinical data. Learning from more subjects, even though those only have a single modality, can improve the prediction accuracy. An integrated deep learning model can achieve reasonable discrimination and re-classification performance., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2022 Elsevier Ltd. All rights reserved.)

Heart failure (HF) has no cure and, for HF with preserved ejection fraction (HFpEF), no life-extending treatments. Defining the clinical epidemiology of HF could facilitate earlier identification of high-risk individuals. We define the clinical epidemiology of HF subtypes (HFpEF and HF with reduced ejection fraction [HFrEF]), identified among 2.7 million individuals receiving routine clinical care. Differences in patterns and rates of accumulation of comorbidities, frequency of hospitalization, use of specialty care, were defined for each HF subtype. Among 28,156 HF cases, 8322 (30%) were HFpEF and 11,677 (42%) were HFrEF. HFpEF was the more prevalent subtype among older women. 177 Phenotypes differentially associated with HFpEF versus HFrEF. HFrEF was more frequently associated with diagnoses related to ischemic cardiac injury while HFpEF was associated more with non-cardiac comorbidities and HF symptoms. These comorbidity patterns were frequently present 3 years prior to a HFpEF diagnosis. HF subtypes demonstrated distinct patterns of clinical co-morbidities and disease progression. For HFpEF, these comorbidities were often non-cardiac and manifested prior to the onset of a HF diagnosis. Recognizing these comorbidity patterns, along the care continuum, may present a window of opportunity to identify individuals at risk for developing incident HFpEF., (© 2021. The Author(s).)

Objective: Simulating electronic health record data offers an opportunity to resolve the tension between data sharing and patient privacy. Recent techniques based on generative adversarial networks have shown promise but neglect the temporal aspect of healthcare. We introduce a generative framework for simulating the trajectory of patients' diagnoses and measures to evaluate utility and privacy., Materials and Methods: The framework simulates date-stamped diagnosis sequences based on a 2-stage process that 1) sequentially extracts temporal patterns from clinical visits and 2) generates synthetic data conditioned on the learned patterns. We designed 3 utility measures to characterize the extent to which the framework maintains feature correlations and temporal patterns in clinical events. We evaluated the framework with billing codes, represented as phenome-wide association study codes (phecodes), from over 500 000 Vanderbilt University Medical Center electronic health records. We further assessed the privacy risks based on membership inference and attribute disclosure attacks., Results: The simulated temporal sequences exhibited similar characteristics to real sequences on the utility measures. Notably, diagnosis prediction models based on real versus synthetic temporal data exhibited an average relative difference in area under the ROC curve of 1.6% with standard deviation of 3.8% for 1276 phecodes. Additionally, the relative difference in the mean occurrence age and time between visits were 4.9% and 4.2%, respectively. The privacy risks in synthetic data, with respect to the membership and attribute inference were negligible., Conclusion: This investigation indicates that temporal diagnosis code sequences can be simulated in a manner that provides utility and respects privacy., (© The Author(s) 2020. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com.)

Study Objective: A challenge in reducing unwanted care variation is effectively managing the wide variety of performed surgical procedures. While an organization may perform thousands of types of cases, privacy and logistical constraints prevent review of previous cases to learn about prior practices. To bridge this gap, we developed a system for extracting key data from anesthesia records. Our objective was to determine whether usage of the system would improve case planning performance for anesthesia residents., Design: Randomized, cross-over trial., Setting: Vanderbilt University Medical Center., Measurements: We developed a web-based, data visualization tool for reviewing de-identified anesthesia records. First year anesthesia residents were recruited and performed simulated case planning tasks (e.g., selecting an anesthetic type) across six case scenarios using a randomized, cross-over design after a baseline assessment. An algorithm scored case planning performance based on care components selected by residents occurring frequently among prior anesthetics, which was scored on a 0-4 point scale. Linear mixed effects regression quantified the tool effect on the average performance score, adjusting for potential confounders., Main Results: We analyzed 516 survey questionnaires from 19 residents. The mean performance score was 2.55 ± SD 0.32. Utilization of the tool was associated with an average score improvement of 0.120 points (95% CI 0.060 to 0.179; p < 0.001). Additionally, a 0.055 point improvement due to the "learning effect" was observed from each assessment to the next (95% CI 0.034 to 0.077; p < 0.001). Assessment score was also significantly associated with specific case scenarios (p < 0.001)., Conclusions: This study demonstrated the feasibility of developing of a clinical data visualization system that aggregated key anesthetic information and found that the usage of tools modestly improved residents' performance in simulated case planning., (Copyright © 2020 Elsevier Inc. All rights reserved.)

Background: The data visualization literature asserts that the details of the optimal data display must be tailored to the specific task, the background of the user, and the characteristics of the data. The general organizing principle of a concept-oriented display is known to be useful for many tasks and data types., Objectives: In this project, we used general principles of data visualization and a co-design process to produce a clinical display tailored to a specific cognitive task, chosen from the anesthesia domain, but with clear generalizability to other clinical tasks. To support the work of the anesthesia-in-charge (AIC) our task was, for a given day, to depict the acuity level and complexity of each patient in the collection of those that will be operated on the following day. The AIC uses this information to optimally allocate anesthesia staff and providers across operating rooms., Methods: We used a co-design process to collaborate with participants who work in the AIC role. We conducted two in-depth interviews with AICs and engaged them in subsequent input on iterative design solutions., Results: Through a co-design process, we found (1) the need to carefully match the level of detail in the display to the level required by the clinical task, (2) the impedance caused by irrelevant information on the screen such as icons relevant only to other tasks, and (3) the desire for a specific but optional trajectory of increasingly detailed textual summaries., Conclusion: This study reports a real-world clinical informatics development project that engaged users as co-designers. Our process led to the user-preferred design of a single binary flag to identify the subset of patients needing further investigation, and then a trajectory of increasingly detailed, text-based abstractions for each patient that can be displayed when more information is needed., Competing Interests: None declared., (Thieme. All rights reserved.)

Model calibration, critical to the success and safety of clinical prediction models, deteriorates over time in response to the dynamic nature of clinical environments. To support informed, data-driven model updating strategies, we present and evaluate a calibration drift detection system. Methods are developed for maintaining dynamic calibration curves with optimized online stochastic gradient descent and for detecting increasing miscalibration with adaptive sliding windows. These methods are generalizable to support diverse prediction models developed using a variety of learning algorithms and customizable to address the unique needs of clinical use cases. In both simulation and case studies, our system accurately detected calibration drift. When drift is detected, our system further provides actionable alerts by including information on a window of recent data that may be appropriate for model updating. Simulations showed these windows were primarily composed of data accruing after drift onset, supporting the potential utility of the windows for model updating. By promoting model updating as calibration deteriorates rather than on pre-determined schedules, implementations of our drift detection system may minimize interim periods of insufficient model accuracy and focus analytic resources on those models most in need of attention., (Copyright © 2020 Elsevier Inc. All rights reserved.)

Background: Suboptimal information display in electronic health records (EHRs) is a notorious pain point for users. Designing an effective display is difficult, due in part to the complex and varied nature of clinical practice., Objective: This article aims to understand the goals, constraints, frustrations, and mental models of inpatient medical providers when accessing EHR data, to better inform the display of clinical information., Methods: A multidisciplinary ethnographic study of inpatient medical providers., Results: Our participants' primary goal was usually to assemble a clinical picture around a given question, under the constraints of time pressure and incomplete information. To do so, they tend to use a mental model of multiple layers of abstraction when thinking of patients and disease; they prefer immediate pattern recognition strategies for answering clinical questions, with breadth-first or depth-first search strategies used subsequently if needed; and they are sensitive to data relevance, completeness, and reliability when reading a record., Conclusion: These results conflict with the ubiquitous display design practice of separating data by type (test results, medications, notes, etc.), a mismatch that is known to encumber efficient mental processing by increasing both navigation burden and memory demands on users. A popular and obvious solution is to select or filter the data to display exactly what is presumed to be relevant to the clinical question, but this solution is both brittle and mistrusted by users. A less brittle approach that is more aligned with our users' mental model could use abstraction to summarize details instead of filtering to hide data. An abstraction-based approach could allow clinicians to more easily assemble a clinical picture, to use immediate pattern recognition strategies, and to adjust the level of displayed detail to their particular needs. It could also help the user notice unanticipated patterns and to fluidly shift attention as understanding evolves., Competing Interests: None declared., (Georg Thieme Verlag KG Stuttgart · New York.)

Purpose: To provide pharmacists and other clinicians with a basic understanding of the underlying principles and practical applications of artificial intelligence (AI) in the medication-use process., Summary: "Artificial intelligence" is a general term used to describe the theory and development of computer systems to perform tasks that normally would require human cognition, such as perception, language understanding, reasoning, learning, planning, and problem solving. Following the fundamental theorem of informatics, a better term for AI would be "augmented intelligence," or leveraging the strengths of computers and the strengths of clinicians together to obtain improved outcomes for patients. Understanding the vocabulary of and methods used in AI will help clinicians productively communicate with data scientists to collaborate on developing models that augment patient care. This primer includes discussion of approaches to identifying problems in practice that could benefit from application of AI and those that would not, as well as methods of training, validating, implementing, evaluating, and maintaining AI models. Some key limitations of AI related to the medication-use process are also discussed., Conclusion: As medication-use domain experts, pharmacists play a key role in developing and evaluating AI in healthcare. An understanding of the core concepts of AI is necessary to engage in collaboration with data scientists and critically evaluating its place in patient care, especially as clinical practice continues to evolve and develop., (© American Society of Health-System Pharmacists 2020. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.)

In evolving clinical environments, the accuracy of prediction models deteriorates over time. Guidance on the design of model updating policies is limited, and there is limited exploration of the impact of different policies on future model performance and across different model types. We implemented a new data-driven updating strategy based on a nonparametric testing procedure and compared this strategy to two baseline approaches in which models are never updated or fully refit annually. The test-based strategy generally recommended intermittent recalibration and delivered more highly calibrated predictions than either of the baseline strategies. The test-based strategy highlighted differences in the updating requirements between logistic regression, L1-regularized logistic regression, random forest, and neural network models, both in terms of the extent and timing of updates. These findings underscore the potential improvements in using a data-driven maintenance approach over "one-size fits all" to sustain more stable and accurate model performance over time., (©2019 AMIA - All rights reserved.)

Semi-supervised methods have an increasing impact on computer vision tasks to make use of scarce labels on large datasets, yet these approaches have not been well translated to medical imaging. Of particular interest, the MixMatch method achieves significant performance improvement over popular semi-supervised learning methods with scarce labels in the CIFAR-10 dataset. In a complementary approach, Nullspace Tuning on equivalence classes offers the potential to leverage multiple subject scans when the ground truth for the subject is unknown. This work is the first to (1) explore MixMatch with Nullspace Tuning in the context of medical imaging and (2) characterize the impacts of the methods with diminishing labels. We consider two distinct medical imaging domains: skin lesion diagnosis and lung cancer prediction. In both cases we evaluate models trained with diminishing labeled data using supervised, MixMatch, and Nullspace Tuning methods as well as MixMatch with Nullspace Tuning together. MixMatch with Nullspace Tuning together is able to achieve an AUC of 0.755 in lung cancer diagnosis with only 200 labeled subjects on the National Lung Screening Trial and a balanced multi-class accuracy of 77% with only 779 labeled examples on HAM10000. This performance is similar to that of the fully supervised methods when all labels are available. In advancing data driven methods in medical imaging, it is important to consider the use of current state-of-the-art semi-supervised learning methods from the greater machine learning community and their impact on the limitations of data acquisition and annotation.

Objective: Clinical prediction models require updating as performance deteriorates over time. We developed a testing procedure to select updating methods that minimizes overfitting, incorporates uncertainty associated with updating sample sizes, and is applicable to both parametric and nonparametric models., Materials and Methods: We describe a procedure to select an updating method for dichotomous outcome models by balancing simplicity against accuracy. We illustrate the test's properties on simulated scenarios of population shift and 2 models based on Department of Veterans Affairs inpatient admissions., Results: In simulations, the test generally recommended no update under no population shift, no update or modest recalibration under case mix shifts, intercept correction under changing outcome rates, and refitting under shifted predictor-outcome associations. The recommended updates provided superior or similar calibration to that achieved with more complex updating. In the case study, however, small update sets lead the test to recommend simpler updates than may have been ideal based on subsequent performance., Discussion: Our test's recommendations highlighted the benefits of simple updating as opposed to systematic refitting in response to performance drift. The complexity of recommended updating methods reflected sample size and magnitude of performance drift, as anticipated. The case study highlights the conservative nature of our test., Conclusions: This new test supports data-driven updating of models developed with both biostatistical and machine learning approaches, promoting the transportability and maintenance of a wide array of clinical prediction models and, in turn, a variety of applications relying on modern prediction tools., (© The Author(s) 2019. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com.)

Objective: Active Learning (AL) attempts to reduce annotation cost (ie, time) by selecting the most informative examples for annotation. Most approaches tacitly (and unrealistically) assume that the cost for annotating each sample is identical. This study introduces a cost-aware AL method, which simultaneously models both the annotation cost and the informativeness of the samples and evaluates both via simulation and user studies., Materials and Methods: We designed a novel, cost-aware AL algorithm (Cost-CAUSE) for annotating clinical named entities; we first utilized lexical and syntactic features to estimate annotation cost, then we incorporated this cost measure into an existing AL algorithm. Using the 2010 i2b2/VA data set, we then conducted a simulation study comparing Cost-CAUSE with noncost-aware AL methods, and a user study comparing Cost-CAUSE with passive learning., Results: Our cost model fit empirical annotation data well, and Cost-CAUSE increased the simulation area under the learning curve (ALC) scores by up to 5.6% and 4.9%, compared with random sampling and alternate AL methods. Moreover, in a user annotation task, Cost-CAUSE outperformed passive learning on the ALC score and reduced annotation time by 20.5%-30.2%., Discussion: Although AL has proven effective in simulations, our user study shows that a real-world environment is far more complex. Other factors have a noticeable effect on the AL method, such as the annotation accuracy of users, the tiredness of users, and even the physical and mental condition of users., Conclusion: Cost-CAUSE saves significant annotation cost compared to random sampling., (© The Author(s) 2019. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com.)

Composite models that combine medical imaging with electronic medical records (EMR) improve predictive power when compared to traditional models that use imaging alone. The digitization of EMR provides potential access to a wealth of medical information, but presents new challenges in algorithm design and inference. Previous studies, such as Phenome Wide Association Study (PheWAS), have shown that EMR data can be used to investigate the relationship between genotypes and clinical conditions. Here, we introduce Phenome-Disease Association Study to extend the statistical capabilities of the PheWAS software through a custom Python package, which creates diagnostic EMR signatures to capture system-wide co-morbidities for a disease population within a given time interval. We investigate the effect of integrating these EMR signatures with radiological data to improve diagnostic classification in disease domains known to have confounding factors because of variable and complex clinical presentation. Specifically, we focus on two studies: First, a study of four major optic nerve related conditions; and second, a study of diabetes. Addition of EMR signature vectors to radiologically derived structural metrics improves the area under the curve (AUC) for diagnostic classification using elastic net regression, for diseases of the optic nerve. For glaucoma, the AUC improves from 0.71 to 0.83, for intrinsic optic nerve disease it increases from 0.72 to 0.91, for optic nerve edema it increases from 0.95 to 0.96, and for thyroid eye disease from 0.79 to 0.89. The EMR signatures recapitulate known comorbidities with diabetes, such as abnormal glucose, but do not significantly modulate image-derived features. In summary, EMR signatures present a scalable and readily applicable.

We discuss a decision-theoretic approach to building a panel-based, preemptive genotyping program. The method is based on findings that a large percentage of patients are prescribed medications that are known to have pharmacogenetic associations, and over time, a substantial proportion are prescribed additional such medication. Preemptive genotyping facilitates genotype-guided therapy at the time medications are prescribed; panel-based testing allows providers to reuse previously collected genetic data when a new indication arises. Because it is cost-prohibitive to conduct panel-based genotyping on all patients, we describe a three-step approach to identify patients with the highest anticipated benefit. First, we construct prediction models to estimate the risk of being prescribed one of the target medications using readily available clinical data. Second, we use literature-based estimates of adverse event rates, variant allele frequencies, secular death rates, and costs to construct a discrete event simulation that estimates the expected benefit of having an individual's genetic data in the electronic health record after an indication has occurred. Finally, we combine medication prescription risk with expected benefit of genotyping once a medication is indicated to calculate the expected benefit of preemptive genotyping. For each patient-clinic visit, we calculate this expected benefit across a range of medications and select patients with the highest expected benefit overall. We build a proof of concept implementation using a cohort of patients from a single academic medical center observed from July 2010 through December 2012. We then apply the results of our modeling strategy to show the extent to which we can improve clinical and economic outcomes in a cohort observed from January 2013 through December 2015.

Purpose: Manually tracing regions of interest (ROIs) within the liver is the de facto standard method for measuring liver attenuation on computed tomography (CT) in diagnosing nonalcoholic fatty liver disease (NAFLD). However, manual tracing is resource intensive. To address these limitations and to expand the availability of a quantitative CT measure of hepatic steatosis, we propose the automatic liver attenuation ROI-based measurement (ALARM) method for automated liver attenuation estimation., Methods: The ALARM method consists of two major stages: (a) deep convolutional neural network (DCNN)-based liver segmentation and (b) automated ROI extraction. First, liver segmentation was achieved using our previously developed SS-Net. Then, a single central ROI (center-ROI) and three circles ROI (periphery-ROI) were computed based on liver segmentation and morphological operations. The ALARM method is available as an open source Docker container (https://github.com/MASILab/ALARM)., Results: Two hundred and forty-six subjects with 738 abdomen CT scans from the African American-Diabetes Heart Study (AA-DHS) were used for external validation (testing), independent from the training and validation cohort (100 clinically acquired CT abdominal scans). From the correlation analyses, the proposed ALARM method achieved Pearson correlations = 0.94 with manual estimation on liver attenuation estimations. When evaluating the ALARM method for detection of nonalcoholic fatty liver disease (NAFLD) using the traditional cut point of < 40 HU, the center-ROI achieved substantial agreements (Kappa = 0.79) with manual estimation, while the periphery-ROI method achieved "excellent" agreement (Kappa = 0.88) with manual estimation. The automated ALARM method had reduced variability compared to manual measurements as indicated by a smaller standard deviation., Conclusions: We propose a fully automated liver attenuation estimation method termed ALARM by combining DCNN and morphological operations, which achieved "excellent" agreement with manual estimation for fatty liver detection. The entire pipeline is implemented as a Docker container which enables users to achieve liver attenuation estimation in five minutes per CT exam., (© 2019 American Association of Physicists in Medicine.)

Objective: Standards such as the Logical Observation Identifiers Names and Codes (LOINC®) are critical for interoperability and integrating data into common data models, but are inconsistently used. Without consistent mapping to standards, clinical data cannot be harmonized, shared, or interpreted in a meaningful context. We sought to develop an automated machine learning pipeline that leverages noisy labels to map laboratory data to LOINC codes., Materials and Methods: Across 130 sites in the Department of Veterans Affairs Corporate Data Warehouse, we selected the 150 most commonly used laboratory tests with numeric results per site from 2000 through 2016. Using source data text and numeric fields, we developed a machine learning model and manually validated random samples from both labeled and unlabeled datasets., Results: The raw laboratory data consisted of >6.5 billion test results, with 2215 distinct LOINC codes. The model predicted the correct LOINC code in 85% of the unlabeled data and 96% of the labeled data by test frequency. In the subset of labeled data where the original and model-predicted LOINC codes disagreed, the model-predicted LOINC code was correct in 83% of the data by test frequency., Conclusion: Using a completely automated process, we are able to assign LOINC codes to unlabeled data with high accuracy. When the model-predicted LOINC code differed from the original LOINC code, the model prediction was correct in the vast majority of cases. This scalable, automated algorithm may improve data quality and interoperability, while substantially reducing the manual effort currently needed to accurately map laboratory data.

The widely used American Society of Anesthesiologists Physical Status (ASA PS) classification is subjective, requires manual clinician review to score, and has limited granularity. Our objective was to develop a system that automatically generates an ASA PS with finer granularity by creating a continuous ASA PS score. Supervised machine learning methods were used to create a model that predicts a patient's ASA PS on a continuous scale using the patient's home medications and comorbidities. Three different types of predictive models were trained: regression models, ordinal models, and classification models. The performance and agreement of each model to anesthesiologists were compared by calculating the mean squared error (MSE), rounded MSE and Cohen's Kappa on a holdout set. To assess model performance on continuous ASA PS, model rankings were compared to two anesthesiologists on a subset of ASA PS 3 case pairs. The random forest regression model achieved the best MSE and rounded MSE. A model consisting of three random forest classifiers (split model) achieved the best Cohen's Kappa. The model's agreement with our anesthesiologists on the ASA PS 3 case pairs yielded fair to moderate Kappa values. The results suggest that the random forest split classification model can predict ASA PS with agreement similar to that of anesthesiologists reported in literature and produce a continuous score in which agreement in accurately judging granularity is fair to moderate.

Advanced regression and machine learning models can provide personalized risk predictions to support clinical decision-making. We aimed to understand whether modeling methods impact the tendency of calibration to deteriorate as patient populations shift over time, with the goal of informing model updating practices. We developed models for 30-day hospital mortality using seven common regression and machine learning methods. Models were developed on 2006 admissions to Department of Veterans Affairs hospitals and validated on admissions in 2007-2013. All models maintained discrimination. Calibration was stable for the neural network model and declined for all other models. The L-2 penalized logistic regression and random forest models experienced smaller magnitudes of calibration drift than the other regression models. Calibration drift was linked with a changing case mix rather than shifts in predictoroutcome associations or outcome rate. Model updating protocols will need to be tailored to variations in calibration drift across methods.

Objective: Predictive analytics create opportunities to incorporate personalized risk estimates into clinical decision support. Models must be well calibrated to support decision-making, yet calibration deteriorates over time. This study explored the influence of modeling methods on performance drift and connected observed drift with data shifts in the patient population., Materials and Methods: Using 2003 admissions to Department of Veterans Affairs hospitals nationwide, we developed 7 parallel models for hospital-acquired acute kidney injury using common regression and machine learning methods, validating each over 9 subsequent years., Results: Discrimination was maintained for all models. Calibration declined as all models increasingly overpredicted risk. However, the random forest and neural network models maintained calibration across ranges of probability, capturing more admissions than did the regression models. The magnitude of overprediction increased over time for the regression models while remaining stable and small for the machine learning models. Changes in the rate of acute kidney injury were strongly linked to increasing overprediction, while changes in predictor-outcome associations corresponded with diverging patterns of calibration drift across methods., Conclusions: Efficient and effective updating protocols will be essential for maintaining accuracy of, user confidence in, and safety of personalized risk predictions to support decision-making. Model updating protocols should be tailored to account for variations in calibration drift across methods and respond to periods of rapid performance drift rather than be limited to regularly scheduled annual or biannual intervals., (© The Author 2017. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For Permissions, please email: journals.permissions@oup.com)

Objective: Phenotyping algorithms applied to electronic health record (EHR) data enable investigators to identify large cohorts for clinical and genomic research. Algorithm development is often iterative, depends on fallible investigator intuition, and is time- and labor-intensive. We developed and evaluated 4 types of phenotyping algorithms and categories of EHR information to identify hypertensive individuals and controls and provide a portable module for implementation at other sites., Materials and Methods: We reviewed the EHRs of 631 individuals followed at Vanderbilt for hypertension status. We developed features and phenotyping algorithms of increasing complexity. Input categories included International Classification of Diseases, Ninth Revision (ICD9) codes, medications, vital signs, narrative-text search results, and Unified Medical Language System (UMLS) concepts extracted using natural language processing (NLP). We developed a module and tested portability by replicating 10 of the best-performing algorithms at the Marshfield Clinic., Results: Random forests using billing codes, medications, vitals, and concepts had the best performance with a median area under the receiver operator characteristic curve (AUC) of 0.976. Normalized sums of all 4 categories also performed well (0.959 AUC). The best non-NLP algorithm combined normalized ICD9 codes, medications, and blood pressure readings with a median AUC of 0.948. Blood pressure cutoffs or ICD9 code counts alone had AUCs of 0.854 and 0.908, respectively. Marshfield Clinic results were similar., Conclusion: This work shows that billing codes or blood pressure readings alone yield good hypertension classification performance. However, even simple combinations of input categories improve performance. The most complex algorithms classified hypertension with excellent recall and precision., (© The Author 2016. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For Permissions, please email: journals.permissions@oup.com.)

Objectives: Named entity recognition (NER), a sequential labeling task, is one of the fundamental tasks for building clinical natural language processing (NLP) systems. Machine learning (ML) based approaches can achieve good performance, but they often require large amounts of annotated samples, which are expensive to build due to the requirement of domain experts in annotation. Active learning (AL), a sample selection approach integrated with supervised ML, aims to minimize the annotation cost while maximizing the performance of ML-based models. In this study, our goal was to develop and evaluate both existing and new AL methods for a clinical NER task to identify concepts of medical problems, treatments, and lab tests from the clinical notes., Methods: Using the annotated NER corpus from the 2010 i2b2/VA NLP challenge that contained 349 clinical documents with 20,423 unique sentences, we simulated AL experiments using a number of existing and novel algorithms in three different categories including uncertainty-based, diversity-based, and baseline sampling strategies. They were compared with the passive learning that uses random sampling. Learning curves that plot performance of the NER model against the estimated annotation cost (based on number of sentences or words in the training set) were generated to evaluate different active learning and the passive learning methods and the area under the learning curve (ALC) score was computed., Results: Based on the learning curves of F-measure vs. number of sentences, uncertainty sampling algorithms outperformed all other methods in ALC. Most diversity-based methods also performed better than random sampling in ALC. To achieve an F-measure of 0.80, the best method based on uncertainty sampling could save 66% annotations in sentences, as compared to random sampling. For the learning curves of F-measure vs. number of words, uncertainty sampling methods again outperformed all other methods in ALC. To achieve 0.80 in F-measure, in comparison to random sampling, the best uncertainty based method saved 42% annotations in words. But the best diversity based method reduced only 7% annotation effort., Conclusion: In the simulated setting, AL methods, particularly uncertainty-sampling based approaches, seemed to significantly save annotation cost for the clinical NER task. The actual benefit of active learning in clinical NER should be further evaluated in a real-time setting., (Copyright © 2015 Elsevier Inc. All rights reserved.)

Sampling repeated clinical laboratory tests with appropriate timing is challenging because the latent physiologic function being sampled is in general nonstationary. When ordering repeated tests, clinicians adopt various simple strategies that may or may not be well suited to the behavior of the function. Previous research on this topic has been primarily focused on cost-driven assessments of oversampling. But for monitoring physiologic state or for retrospective analysis, undersampling can be much more problematic than oversampling. In this paper we analyze hundreds of observation sequences of four different clinical laboratory tests to provide principled, data-driven estimates of undersampling and oversampling, and to assess whether the sampling adapts to changing volatility of the latent function. To do this, we developed a new method for fitting a Gaussian process to samples of a nonstationary latent function. Our method includes an explicit estimate of the latent function's volatility over time, which is deterministically related to its nonstationarity. We find on average that the degree of undersampling is up to an order of magnitude greater than oversampling, and that only a small minority are sampled with an adaptive strategy.

Acute coronary syndrome (ACS) accounts for 1.36 million hospitalizations and billions of dollars in costs in the United States alone. A major challenge to diagnosing and treating patients with suspected ACS is the significant symptom overlap between patients with and without ACS. There is a high cost to over- and under-treatment. Guidelines recommend early risk stratification of patients, but many tools lack sufficient accuracy for use in clinical practice. Prognostic indices often misrepresent clinical populations and rely on curated data. We used random forest and elastic net on 20,078 deidentified records with significant missing and noisy values to develop models that outperform existing ACS risk prediction tools. We found that the random forest (AUC = 0.848) significantly outperformed elastic net (AUC=0.818), ridge regression (AUC = 0.810), and the TIMI (AUC = 0.745) and GRACE (AUC = 0.623) scores. Our findings show that random forest applied to noisy and sparse data can perform on par with previously developed scoring metrics.

The episodic, irregular and asynchronous nature of medical data render them difficult substrates for standard machine learning algorithms. We would like to abstract away this difficulty for the class of time-stamped categorical variables (or events ) by modeling them as a renewal process and inferring a probability density over non-parametric longitudinal intensity functions that modulate the process. Several methods exist for inferring such a density over intensity functions, but either their constraints prevent their use with our potentially bursty event streams, or their time complexity renders their use intractable on our long-duration observations of high-resolution events, or both. In this paper we present a new efficient and flexible inference method that uses direct numeric integration and smooth interpolation over Gaussian processes. We demonstrate that our direct method is up to twice as accurate and two orders of magnitude more efficient than the best existing method (thinning). Importantly, our direct method can infer intensity functions over the full range of bursty to memoryless to regular events, which thinning and many other methods cannot do. Finally, we apply the method to clinical event data and demonstrate a simple example application facilitated by the abstraction.

Objective: To create a computable MEDication Indication resource (MEDI) to support primary and secondary use of electronic medical records (EMRs)., Materials and Methods: We processed four public medication resources, RxNorm, Side Effect Resource (SIDER) 2, MedlinePlus, and Wikipedia, to create MEDI. We applied natural language processing and ontology relationships to extract indications for prescribable, single-ingredient medication concepts and all ingredient concepts as defined by RxNorm. Indications were coded as Unified Medical Language System (UMLS) concepts and International Classification of Diseases, 9th edition (ICD9) codes. A total of 689 extracted indications were randomly selected for manual review for accuracy using dual-physician review. We identified a subset of medication-indication pairs that optimizes recall while maintaining high precision., Results: MEDI contains 3112 medications and 63 343 medication-indication pairs. Wikipedia was the largest resource, with 2608 medications and 34 911 pairs. For each resource, estimated precision and recall, respectively, were 94% and 20% for RxNorm, 75% and 33% for MedlinePlus, 67% and 31% for SIDER 2, and 56% and 51% for Wikipedia. The MEDI high-precision subset (MEDI-HPS) includes indications found within either RxNorm or at least two of the three other resources. MEDI-HPS contains 13 304 unique indication pairs regarding 2136 medications. The mean±SD number of indications for each medication in MEDI-HPS is 6.22 ± 6.09. The estimated precision of MEDI-HPS is 92%., Conclusions: MEDI is a publicly available, computable resource that links medications with their indications as represented by concepts and billing codes. MEDI may benefit clinical EMR applications and reuse of EMR data for research.

Inferring precise phenotypic patterns from population-scale clinical data is a core computational task in the development of precision, personalized medicine. The traditional approach uses supervised learning, in which an expert designates which patterns to look for (by specifying the learning task and the class labels), and where to look for them (by specifying the input variables). While appropriate for individual tasks, this approach scales poorly and misses the patterns that we don't think to look for. Unsupervised feature learning overcomes these limitations by identifying patterns (or features) that collectively form a compact and expressive representation of the source data, with no need for expert input or labeled examples. Its rising popularity is driven by new deep learning methods, which have produced high-profile successes on difficult standardized problems of object recognition in images. Here we introduce its use for phenotype discovery in clinical data. This use is challenging because the largest source of clinical data - Electronic Medical Records - typically contains noisy, sparse, and irregularly timed observations, rendering them poor substrates for deep learning methods. Our approach couples dirty clinical data to deep learning architecture via longitudinal probability densities inferred using Gaussian process regression. From episodic, longitudinal sequences of serum uric acid measurements in 4368 individuals we produced continuous phenotypic features that suggest multiple population subtypes, and that accurately distinguished (0.97 AUC) the uric-acid signatures of gout vs. acute leukemia despite not being optimized for the task. The unsupervised features were as accurate as gold-standard features engineered by an expert with complete knowledge of the domain, the classification task, and the class labels. Our findings demonstrate the potential for achieving computational phenotype discovery at population scale. We expect such data-driven phenotypes to expose unknown disease variants and subtypes and to provide rich targets for genetic association studies.

Understanding of medications-disease relationships is critical to distinguish indications from adverse effects, and medication exposures serve as important markers of disease and severity in electronic medical records (EMR). We created a computable medication-indication (MEDI) resource by applying natural language processing and ontology relationships to four public medication resources. Physicians evaluated accuracy of medication-indication relationships. MEDI contained 3,112 medications and 63,343 medication-indication pairs derived from the four resources, whose precisions varied from 56-94%. The MEDI high precision subset (MEDI-HPS) includes indications found within either RxNorm or ≥2 resources and had an estimated precision of 92%. MEDI-HPS contains 13,304 unique indication pairs for 2,136 medications. MEDI is a free, computable resource that links medications with their indications as represented by formal concepts and may assist clinical and research uses of EMR data.

Inferring precise phenotypic patterns from population-scale clinical data is a critical computational task of personalized medicine. The dominant approach uses supervised learning, in which a human expert specifies which patterns to look for (by designating a learning task and class labels) and where to look for them (by constructing input features). This scales poorly and misses the unexpected patterns, which are the most informative. Unsupervised feature learning overcomes these limitations by identifying patterns (or features) that form a compact, expressive representation of their source data without depending on class labels or human input. Unsupervised features have become the state of the art for image and speech recognition. In this work we introduce the idea of using them for the problem of phenotype recognition. We present an unsupervised approach for inferring broadly useful, data-driven micro-phenotypes in the form of continuous, longitudinal features learned from noisy, sparse, and irregular EMR data. In our demonstration project, unsupervised features learned from longitudinal laboratory values were as accurate (0.96 AUC) in a classification task unknown to the learning algorithm as gold-standard features engineered by an expert with complete knowledge of the domain, the data, and the class labels.

The goal of data anonymization is to allow the release of scientifically useful data in a form that protects the privacy of its subjects. This requires more than simply removing personal identifiers from the data, because an attacker can still use auxiliary information to infer sensitive individual information. Additional perturbation is necessary to prevent these inferences, and the challenge is to perturb the data in a way that preserves its analytic utility.No existing anonymization algorithm provides both perfect privacy protection and perfect analytic utility. We make the new observation that anonymization algorithms are not required to operate in the original vector-space basis of the data, and many algorithms can be improved by operating in a judiciously chosen alternate basis. A spectral basis derived from the data's eigenvectors is one that can provide substantial improvement. We introduce the term spectral anonymization to refer to an algorithm that uses a spectral basis for anonymization, and we give two illustrative examples.We also propose new measures of privacy protection that are more general and more informative than existing measures, and a principled reference standard with which to define adequate privacy protection.

Background: Evaluating the quality of care provided by individual primary care physicians (PCPs) may be limited by failing to know which patients the PCP feels personally responsible for., Objective: To develop and validate a model for linking patients to specific PCPs., Design: Retrospective convenience sample., Participants: Eighteen PCPs from 10 practice sites within an academic adult primary care network., Measurements: Each PCP reviewed the records for all outpatients seen over the preceding 3 years (16,435 patients reviewed) and designated each patient as "My Patient" or "Not My Patient." Using this reference standard, we developed an algorithm with logistic regression modeling to predict "My Patient" using development and validation subsets drawn from the same patient set. Quality of care was then assessed by "My Patient" or "Not My Patient" designation by analyzing cancer screening test rates., Results: Overall, PCPs designated 11,226 patients (68.3%, range per provider 15% to 93%) to be "My Patient." The model accurately categorized patients in development and validation subsets (combined sensitivity 80.4%, specificity 93.7%, and positive predictive value 96.5%). To achieve positive predictive values of > 90% for individual PCPs, the model excluded 19.6% of PCP "My Patients" (range 5.5% to 75.3%). Cancer screening rates were higher among model-predicted "My Patients.", Conclusions: Nearly one-third of patients seen were considered "Not My Patient" by the PCP, although this proportion varied widely. We developed and validated a simple model to link specific patients and PCPs. Such efforts may help effectively target interventions to improve primary care quality.

Receiver operating characteristic (ROC) curves are frequently used in biomedical informatics research to evaluate classification and prediction models for decision support, diagnosis, and prognosis. ROC analysis investigates the accuracy of a model's ability to separate positive from negative cases (such as predicting the presence or absence of disease), and the results are independent of the prevalence of positive cases in the study population. It is especially useful in evaluating predictive models or other tests that produce output values over a continuous range, since it captures the trade-off between sensitivity and specificity over that range. There are many ways to conduct an ROC analysis. The best approach depends on the experiment; an inappropriate approach can easily lead to incorrect conclusions. In this article, we review the basic concepts of ROC analysis, illustrate their use with sample calculations, make recommendations drawn from the literature, and list readily available software.

As a guide to optimizing the geometry of bright light treatment, 12 healthy subjects were studied three times in the laboratory from 11 p.m. to 2 a.m. On three evenings, in counterbalanced orders, subjects received 500 lux in the upper visual field, 500 lux in the lower visual field, or 5 lux while watching television. In the upper visual field, 500 lux significantly suppressed melatonin, as compared to 500 lux in the lower visual field or to 5 lux. In the lower visual field, 500 lux produced intermediate suppression of borderline significance. The results suggest that bright light treatment of depression or circadian phase disorders might be most effective when applied in the upper visual field.