Search

The estimated accuracy of a classifier is a random quantity with variability. A common practice in supervised machine learning, is thus to test if the estimated accuracy is significantly better than chance level. This method of signal detection is particularly popular in neuroimaging and genetics. We provide evidence that using a classifier's accuracy as a test statistic can be an underpowered strategy for finding differences between populations, compared to a bona fide statistical test. It is also computationally more demanding than a statistical test. Via simulation, we compare test statistics that are based on classification accuracy, to others based on multivariate test statistics. We find that the probability of detecting differences between two distributions is lower for accuracy-based statistics. We examine several candidate causes for the low power of accuracy-tests. These causes include: the discrete nature of the accuracy-test statistic, the type of signal accuracy-tests are designed to detect, their inefficient use of the data, and their suboptimal regularization. When the purpose of the analysis is the evaluation of a particular classifier, not signal detection, we suggest several improvements to increase power. In particular, to replace V-fold cross-validation with the Leave-One-Out Bootstrap., (© The Author 2019. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.)

Trauma-related psychopathology has been associated with an intense emotional reaction to stressful event. Emotional responses have evolved to signal the presence of risks to be avoided or of rewards to be approached in the environment. Thus, individuals' sensitivity to signals of risk and reward may affect the level of stress vulnerability. Stress, however, can modify these sensitivities as well. In the current functional magnetic resonance imaging (fMRI) study, we prospectively probed the neural correlates of such sensitivities in 24 healthy soldiers by using an interactive game that encompasses risky and rewarding intervals both pre-exposure and post-exposure to stressful military service. As expected, risky and rewarding intervals elicited selective responses in the amygdala and nucleus accumbens (Nacc), respectively. Furthermore, increased post-traumatic stress disorder symptoms post-exposure (i.e., stress vulnerability) corresponded to greater amygdala's response to risk both pre-exposure and post-exposure and to decreased NAcc response to reward only post-exposure. By combining these regional responsivities post-exposure, we accurately identified all the most vulnerable soldiers. Imbalanced neural responsivity to risk and reward following exposure to stress may therefore constitute a marker for stress vulnerability. Such identification of vulnerability biomarkers can aid future diagnostic and therapeutic efforts by allowing early detection of vulnerability as well as follow up on patient's treatment progression.