Search

There is significant interest in using neuroimaging data to predict behavior. The predictive models are often interpreted by the computation of feature importance, which quantifies the predictive relevance of an imaging feature. Tian and Zalesky (2021) suggest that feature importance estimates exhibit low split-half reliability, as well as a trade-off between prediction accuracy and feature importance reliability across parcellation resolutions. However, it is unclear whether the trade-off between prediction accuracy and feature importance reliability is universal. Here, we demonstrate that, with a sufficient sample size, feature importance (operationalized as Haufe-transformed weights) can achieve fair to excellent split-half reliability. With a sample size of 2600 participants, Haufe-transformed weights achieve average intra-class correlation coefficients of 0.75, 0.57 and 0.53 for cognitive, personality and mental health measures respectively. Haufe-transformed weights are much more reliable than original regression weights and univariate FC-behavior correlations. Original regression weights are not reliable even with 2600 participants. Intriguingly, feature importance reliability is strongly positively correlated with prediction accuracy across phenotypes. Within a particular behavioral domain, there is no clear relationship between prediction performance and feature importance reliability across regression models. Furthermore, we show mathematically that feature importance reliability is necessary, but not sufficient, for low feature importance error. In the case of linear models, lower feature importance error is mathematically related to lower prediction error. Therefore, higher feature importance reliability might yield lower feature importance error and higher prediction accuracy. Finally, we discuss how our theoretical results relate with the reliability of imaging features and behavioral measures. Overall, the current study provides empirical and theoretical insights into the relationship between prediction accuracy and feature importance reliability.

A fundamental goal across the neurosciences is the characterization of relationships linking brain anatomy, functioning, and behavior. Although various MRI modalities have been developed to probe these relationships, direct comparisons of their ability to predict behavior have been lacking. Here, we compared the ability of anatomical T1, diffusion and functional MRI (fMRI) to predict behavior at an individual level. Cortical thickness, area and volume were extracted from anatomical T1 images. Diffusion Tensor Imaging (DTI) and approximate Neurite Orientation Dispersion and Density Imaging (NODDI) models were fitted to the diffusion images. The resulting metrics were projected to the Tract-Based Spatial Statistics (TBSS) skeleton. We also ran probabilistic tractography for the diffusion images, from which we extracted the stream count, average stream length, and the average of each DTI and NODDI metric across tracts connecting each pair of brain regions. Functional connectivity (FC) was extracted from both task and resting-state fMRI. Individualized prediction of a wide range of behavioral measures were performed using kernel ridge regression, linear ridge regression and elastic net regression. Consistency of the results were investigated with the Human Connectome Project (HCP) and Adolescent Brain Cognitive Development (ABCD) datasets. In both datasets, FC-based models gave the best prediction performance, regardless of regression model or behavioral measure. This was especially true for the cognitive component. Furthermore, all modalities were able to predict cognition better than other behavioral components. Combining all modalities improved prediction of cognition, but not other behavioral components. Finally, across all behaviors, combining resting and task FC yielded prediction performance similar to combining all modalities. Overall, our study suggests that in the case of healthy children and young adults, behaviorally-relevant information in T1 and diffusion features might reflect a subset of the variance captured by FC.