251. Evaluation of predictive model performance of an existing model in the presence of missing data
- Author
-
Matthew J. Schipper, Pin Li, Jeremy M. G. Taylor, Daniel E. Spratt, and R. Jeffery Karnes
- Subjects
Male ,Statistics and Probability ,Receiver operating characteristic ,Epidemiology ,Computer science ,Inverse probability weighting ,Missing data ,01 natural sciences ,Article ,010104 statistics & probability ,03 medical and health sciences ,0302 clinical medicine ,Inverse probability ,ROC Curve ,Brier score ,Data Interpretation, Statistical ,Covariate ,Statistics ,Humans ,Computer Simulation ,030212 general & internal medicine ,Imputation (statistics) ,0101 mathematics ,Predictive modelling ,Probability - Abstract
In medical research, the Brier score (BS) and the area under the receiver operating characteristic (ROC) curves (AUC) are two common metrics used to evaluate prediction models of a binary outcome, such as using biomarkers to predict the risk of developing a disease in the future. The assessment of an existing prediction models using data with missing covariate values is challenging. In this article, we propose inverse probability weighted (IPW) and augmented inverse probability weighted (AIPW) estimates of AUC and BS to handle the missing data. An alternative approach uses multiple imputation (MI), which requires a model for the distribution of the missing variable. We evaluated the performance of IPW and AIPW in comparison with MI in simulation studies under missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR) scenarios. When there are missing observations in the data, MI and IPW can be used to obtain unbiased estimates of BS and AUC if the imputation model for the missing variable or the model for the missingness is correctly specified. MI is more efficient than IPW. Our simulation results suggest that AIPW can be more efficient than IPW, and also achieves double robustness from miss-specification of either the missingness model or the imputation model. The outcome variable should be included in the model for the missing variable under all scenarios, while it only needs to be included in missingness model if the missingness depends on the outcome. We illustrate these methods using an example from prostate cancer.
- Published
- 2021
- Full Text
- View/download PDF