Search

IntroductionStatistical effect sizes are systematically overestimated in small samples, leading to poor generalizability and replicability of findings in all areas of research. Due to the large number of variables, this is particularly problematic in neuroimaging research. While cross-validation is frequently used in multivariate machine learning approaches to assess model generalizability and replicability, the benefits for mass-univariate brain analysis are yet unclear. We investigated the impact of cross-validation on effect size estimation in univariate voxel-based brain-wide associations, using body mass index (BMI) as an exemplary predictor.MethodsA total of n=3401 adults were pooled from three independent cohorts. Brain-wide associations between BMI and gray matter structure were tested using a standard linear mass-univariate voxel-based approach. First, a traditional non-cross-validated analysis was conducted to identify brain-wide effect sizes in the total sample (as an estimate of a realistic reference effect size). The impact of sample size (bootstrapped samples ranging from n=25 to n=3401) and cross-validation on effect size estimates was investigated across selected voxels with differing underlying effect sizes (including the brain-wide lowest effect size). Linear effects were estimated within training sets and then applied to unseen test set data, using 5-fold cross-validation. Resulting effect sizes (explained variance) were investigated.ResultsAnalysis in the total sample (n=3401) without cross-validation yielded mainly negative correlations between BMI and gray matter density with a maximum effect size ofR2p=.036 (peak voxel in the cerebellum). Effects were overestimated exponentially with decreasing sample size, with effect sizes up toR2p=.535 in samples of n=25 for the voxel with the brain-wide largest effect and up toR2p=.429 for the voxel with the brain-wide smallest effect. When applying cross-validation, linear effects estimated in small samples did not generalize to an independent test set. For the largest brain-wide effect a minimum sample size of n=100 was required to start generalizing (explained variance >0 in unseen data), while n=400 were needed for smaller effects ofR2p=.005 to generalize. For a voxel with an underlying null effect, linear effects found in non-cross-validated samples did not generalize to test sets even with the maximum sample size of n=3401. Effect size estimates obtained with and without cross-validation approached convergence in large samples.DiscussionCross-validation is a useful method to counteract the overestimation of effect size particularly in small samples and to assess the generalizability of effects. Train and test set effect sizes converge in large samples which likely reflects a good generalizability for models in such samples. While linear effects start generalizing to unseen data in samples of n>100 for large effect sizes, the generalization of smaller effects requires larger samples (n>400). Cross-validation should be applied in voxel-based mass-univariate analysis to foster accurate effect size estimation and improve replicability of neuroimaging findings. We provide open-source python code for this purpose (https://osf.io/cy7fp/?view_only=a10fd0ee7b914f50820b5265f65f0cdb).

BACKGROUND: Altered brain structural connectivity has been implicated in the pathophysiology of psychiatric disorders including schizophrenia (SZ), bipolar disorder (BD), and major depressive disorder (MDD). However, it is unknown which part of these connectivity abnormalities are disorder specific and which are shared across the spectrum of psychotic and affective disorders. We investigated common and distinct brain connectivity alterations in a large sample (N = 1743) of patients with SZ, BD, or MDD and healthy control (HC) subjects.METHODS: This study examined diffusion-weighted imaging-based structural connectome topology in 720 patients with MDD, 112 patients with BD, 69 patients with SZ, and 842 HC subjects (mean age of all subjects: 35.7 years). Graph theory-based network analysis was used to investigate connectome organization. Machine learning algorithms were trained to classify groups based on their structural connectivity matrices.RESULTS: Groups differed significantly in the network metrics global efficiency, clustering, present edges, and global connectivity strength with a converging pattern of alterations between diagnoses (e.g., efficiency: HC > MDD > BD > SZ, false discovery rate-corrected p = .028). Subnetwork analysis revealed a common core of edges that were affected across all 3 disorders, but also revealed differences between disorders. Machine learning algorithms could not discriminate between disorders but could discriminate each diagnosis from HC. Furthermore, dysconnectivity patterns were found most pronounced in patients with an early disease onset irrespective of diagnosis.CONCLUSIONS: We found shared and specific signatures of structural white matter dysconnectivity in SZ, BD, and MDD, leading to commonly reduced network efficiency. These results showed a compromised brain communication across a spectrum of major psychiatric disorders.

Psychiatric disorders show heterogeneous clinical manifestations and disease trajectories, with current classification systems not accurately reflecting their molecular etiology. This heterogeneity impedes timely and targeted treatment. Our study aimed to identify diagnostically mixed psychiatric patient clusters that share clinical and genetic features and may profit from similar therapeutic interventions. We used unsupervised high-dimensional data clustering on deep clinical data to identify transdiagnostic groups in a discovery sample (N=1250) of healthy controls and patients diagnosed with depression, bipolar disorder, schizophrenia, schizoaffective disorder, and other psychiatric disorders. We observed five diagnostically mixed clusters and ordered them based on severity. The least impaired cluster 0, containing most healthy controls, was characterized by general well-being. Clusters 1-3 differed predominantly regarding levels of maltreatment, depression, daily functioning, and parental bonding. Cluster 4 contained most patients diagnosed with psychotic disorders and exhibited the highest severity in many dimensions, including medication load. MDD patients were present in all clusters, indicating that we captured different disease stages or subtypes. We replicated all but the smallest cluster 1 in an independent sample (N=622). Next, we analyzed genetic differences between clusters using polygenic scores (PGS) and the psychiatric family history. These genetic variables differed mainly between clusters 0 and 4 (prediction AUC=81%; significant PGS: cross-disorder psychiatric risk, schizophrenia, and educational attainment). Our results confirm that psychiatric disorders consist of heterogeneous subtypes sharing molecular factors and symptoms. The identification of transdiagnostic clusters advances our understanding of the heterogeneity of psychiatric disorders and may support the development of personalized treatment regimes.

Psychiatric disorders show heterogeneous symptoms and trajectories, with current nosology not accurately reflecting their molecular etiology and the variability and symptomatic overlap within and between diagnostic classes. This heterogeneity impedes timely and targeted treatment. Our study aimed to identify psychiatric patient clusters that share clinical and genetic features and may profit from similar therapies. We used high-dimensional data clustering on deep clinical data to identify transdiagnostic groups in a discovery sample (N = 1250) of healthy controls and patients diagnosed with depression, bipolar disorder, schizophrenia, schizoaffective disorder, and other psychiatric disorders. We observed five diagnostically mixed clusters and ordered them based on severity. The least impaired cluster 0, containing most healthy controls, showed general well-being. Clusters 1–3 differed predominantly regarding levels of maltreatment, depression, daily functioning, and parental bonding. Cluster 4 contained most patients diagnosed with psychotic disorders and exhibited the highest severity in many dimensions, including medication load. Depressed patients were present in all clusters, indicating that we captured different disease stages or subtypes. We replicated all but the smallest cluster 1 in an independent sample (N = 622). Next, we analyzed genetic differences between clusters using polygenic scores (PGS) and the psychiatric family history. These genetic variables differed mainly between clusters 0 and 4 (prediction area under the receiver operating characteristic curve (AUC) = 81%; significant PGS: cross-disorder psychiatric risk, schizophrenia, and educational attainment). Our results confirm that psychiatric disorders consist of heterogeneous subtypes sharing molecular factors and symptoms. The identification of transdiagnostic clusters advances our understanding of the heterogeneity of psychiatric disorders and may support the development of personalized treatments.