Pairing Machine Learning and Clinical Psychology: How You Evaluate Predictive Performance Matters

Machine learning is being utilized at an increasing rate in clinical psychology. Applying machine learning comes with a number of challenges, both in deciding which algorithms to test, and how to evaluate the predictive performance. We focus on this last component, demonstrating across both a simulation and empirical example that the method researchers choose to evaluate prediction with machine learning can have large consequences for the substantive conclusions. More specifically, we demonstrate that one method of evaluation that has been used repeatedly in clinical research, the optimism corrected bootstrap, can result in extremely biased results when paired with specific machine learning algorithms. We conclude with providing recommendations for researchers and a discussion of additional best practices.