Search

Objectives: It is widely acknowledged that the performance of diagnostic and prognostic prediction models should be assessed in external validation studies with independent data from "different but related" samples as compared with that of the development sample. We developed a framework of methodological steps and statistical methods for analyzing and enhancing the interpretation of results from external validation studies of prediction models. Study Design and Setting: We propose to quantify the degree of relatedness between development and validation samples on a scale ranging from reproducibility to transportability by evaluating their corresponding case-mix differences. We subsequently assess the models' performance in the validation sample and interpret the performance in view of the case-mix differences. Finally, we may adjust the model to the validation setting. Results: We illustrate this three-step framework with a prediction model for diagnosing deep venous thrombosis using three validation samples with varying case mix. While one external validation sample merely assessed the model's reproducibility, two other samples rather assessed model transportability. The performance in all validation samples was adequate, and the model did not require extensive updating to correct for miscalibration or poor fit to the validation settings. Conclusion: The proposed framework enhances the interpretation of findings at external validation of prediction models. (C) 2015 The Authors. Published by Elsevier Inc.

Item does not contain fulltext, AIMS/HYPOTHESIS: The aim of this work was to investigate whether measurement of the mean common carotid intima-media thickness (CIMT) improves cardiovascular risk prediction in individuals with diabetes. METHODS: We performed a subanalysis among 4,220 individuals with diabetes in a large ongoing individual participant data meta-analysis involving 56,194 subjects from 17 population-based cohorts worldwide. We first refitted the risk factors of the Framingham heart risk score on the individuals without previous cardiovascular disease (baseline model) and then expanded this model with the mean common CIMT (CIMT model). The absolute 10 year risk for developing a myocardial infarction or stroke was estimated from both models. In individuals with diabetes we compared discrimination and calibration of the two models. Reclassification of individuals with diabetes was based on allocation to another cardiovascular risk category when mean common CIMT was added. RESULTS: During a median follow-up of 8.7 years, 684 first-time cardiovascular events occurred among the population with diabetes. The C statistic was 0.67 for the Framingham model and 0.68 for the CIMT model. The absolute 10 year risk for developing a myocardial infarction or stroke was 16% in both models. There was no net reclassification improvement with the addition of mean common CIMT (1.7%; 95% CI -1.8, 3.8). There were no differences in the results between men and women. CONCLUSIONS/INTERPRETATION: There is no improvement in risk prediction in individuals with diabetes when measurement of the mean common CIMT is added to the Framingham risk score. Therefore, this measurement is not recommended for improving individual cardiovascular risk stratification in individuals with diabetes.

Item does not contain fulltext, CONTEXT: The evidence that measurement of the common carotid intima-media thickness (CIMT) improves the risk scores in prediction of the absolute risk of cardiovascular events is inconsistent. OBJECTIVE: To determine whether common CIMT has added value in 10-year risk prediction of first-time myocardial infarctions or strokes, above that of the Framingham Risk Score. DATA SOURCES: Relevant studies were identified through literature searches of databases (PubMed from 1950 to June 2012 and EMBASE from 1980 to June 2012) and expert opinion. STUDY SELECTION: Studies were included if participants were drawn from the general population, common CIMT was measured at baseline, and individuals were followed up for first-time myocardial infarction or stroke. DATA EXTRACTION: Individual data were combined into 1 data set and an individual participant data meta-analysis was performed on individuals without existing cardiovascular disease. RESULTS: We included 14 population-based cohorts contributing data for 45,828 individuals. During a median follow-up of 11 years, 4007 first-time myocardial infarctions or strokes occurred. We first refitted the risk factors of the Framingham Risk Score and then extended the model with common CIMT measurements to estimate the absolute 10-year risks to develop a first-time myocardial infarction or stroke in both models. The C statistic of both models was similar (0.757; 95% CI, 0.749-0.764; and 0.759; 95% CI, 0.752-0.766). The net reclassification improvement with the addition of common CIMT was small (0.8%; 95% CI, 0.1%-1.6%). In those at intermediate risk, the net reclassification improvement was 3.6% in all individuals (95% CI, 2.7%-4.6%) and no differences between men and women. CONCLUSION: The addition of common CIMT measurements to the Framingham Risk Score was associated with small improvement in 10-year risk prediction of first-time myocardial infarction or stroke, but this improvement is unlikely to be of clinical importance.

Background: Diagnostic and prognostic literature is overwhelmed with studies reporting univariable predictor-outcome associations. Currently, methods to incorporate such information in the construction of a prediction model are underdeveloped and unfamiliar to many researchers. Methods. This article aims to improve upon an adaptation method originally proposed by Greenland (1987) and Steyerberg (2000) to incorporate previously published univariable associations in the construction of a novel prediction model. The proposed method improves upon the variance estimation component by reconfiguring the adaptation process in established theory and making it more robust. Different variants of the proposed method were tested in a simulation study, where performance was measured by comparing estimated associations with their predefined values according to the Mean Squared Error and coverage of the 90% confidence intervals. Results: Results demonstrate that performance of estimated multivariable associations considerably improves for small datasets where external evidence is included. Although the error of estimated associations decreases with increasing amount of individual participant data, it does not disappear completely, even in very large datasets. Conclusions: The proposed method to aggregate previously published univariable associations with individual participant data in the construction of a novel prediction models outperforms established approaches and is especially worthwhile when relatively limited individual participant data are available.