Search

Replicating results (i.e. obtaining consistent results using a new independent dataset) is an essential part of good science. As replicability has consequences for theories derived from empirical studies, it is of utmost importance to better understand the underlying mechanisms influencing it. A popular tool for non-invasive neuroimaging studies is functional magnetic resonance imaging (fMRI). While the effect of underpowered studies is well documented, the empirical assessment of the interplay between sample size and replicability of results for task-based fMRI studies remains limited. In this work, we extend existing work on this assessment in two ways. Firstly, we use a large database of 1400 subjects performing four types of tasks from the IMAGEN project to subsample a series of independent samples of increasing size. Secondly, replicability is evaluated using a multi-dimensional framework consisting of 3 different measures: (un)conditional test-retest reliability, coherence and stability. We demonstrate not only a positive effect of sample size, but also a trade-off between spatial resolution and replicability. When replicability is assessed voxelwise or when observing small areas of activation, a larger sample size than typically used in fMRI is required to replicate results. On the other hand, when focussing on clusters of voxels, we observe a higher replicability. In addition, we observe variability in the size of clusters of activation between experimental paradigms or contrasts of parameter estimates within these.

Given the increasing amount of neuroimaging studies, there is a growing need to summarize published results. Coordinate-based meta-analyses use the locations of statistically significant local maxima with possibly the associated effect sizes to aggregate studies. In this paper, we investigate the influence of key characteristics of a coordinate-based meta-analysis on (1) the balance between false and true positives and (2) the activation reliability of the outcome from a coordinate-based meta-analysis. More particularly, we consider the influence of the chosen group level model at the study level [fixed effects, ordinary least squares (OLS), or mixed effects models], the type of coordinate-based meta-analysis [Activation Likelihood Estimation (ALE) that only uses peak locations, fixed effects, and random effects meta-analysis that take into account both peak location and height] and the amount of studies included in the analysis (from 10 to 35). To do this, we apply a resampling scheme on a large dataset (N = 1,400) to create a test condition and compare this with an independent evaluation condition. The test condition corresponds to subsampling participants into studies and combine these using meta-analyses. The evaluation condition corresponds to a high-powered group analysis. We observe the best performance when using mixed effects models in individual studies combined with a random effects meta-analysis. Moreover the performance increases with the number of studies included in the meta-analysis. When peak height is not taken into consideration, we show that the popular ALE procedure is a good alternative in terms of the balance between type I and II errors. However, it requires more studies compared to other procedures in terms of activation reliability. Finally, we discuss the differences, interpretations, and limitations of our results.

1 Introduction The reproducibility of fMRI studies suffers from small sample sizes, an overly stringent focus on minimizing type I errors (at the cost of power) and topological instability (Durnez et al., 2014; Roels et al., 2015). It is therefore increasingly recognized that further progress in understanding brain function will require integration of data across studies using meta-analyses (Wager et al., 2007). Methods for coordinate-based meta-analyses are designed to determine locations of activity based on studies that only report peak voxels whose evidence survive a selected threshold. In this contribution, we study the impact of analysis choices made at the individual study level on the coordinate-based meta-analysis methods. More specifically, we focus on pooling of subjects in which we consider (1) fixed effects pooling (no between-subjects variability), (2) ordinary least squares or OLS (homogeneous within-subjects variances) and (3) mixed effects pooling (estimating both). We use a large database to set up a simulation study and implement 3 coordinate- based meta-analysis techniques: activation likelihood estimation (ALE, Turkeltaub et al., 2002) and two weighted average meta-analyses. ALE calculates the convergence of activation at a specific voxel. The weighted average meta-analysis is either implemented as a fixed effects meta-analysis (assumes no between study variability) or a random effects meta-analysis (seed based d-mapping, SBdM, Radua et al. (2012)). 2 Method The design of the study is depicted in figure 1. We use data of 1400 subjects from the processed math > language contrast of the IMAGEN project (Schumann et al., 2010). In one iteration, 200 subjects are sampled that go into a ‘ground truth’ for which activation is assessed through mixed effects pooling with the False Discovery Rate (FDR) controlled at level 0.001. The other 200 subjects go into a test condition and are then sampled as 10 separate studies. Within each study, we apply the three different pooling methods and we obtain peak locations (FDR control at level 0.05). These study results are then combined in meta-analyses. For ALE, images are either uncorrected for multiple testing or corrected through FDR control (‘pID’ assumes independent or positive de- pendent test statistics among voxels, ‘pID’ makes no assumption on the joint distribution of test statistics). All meta-analyses are thresholded to control the type I error rate or FDR at level 0.05. The meta-analytic results are compared with the ground truth (200 different subjects) and a benchmark (group analysis as in ground truth but using the same 200 subjects as in the meta-analyses). We calculate the overlap-of-activation, power and false positive rate, FPR (Maitra, 2010). In total, 30 iterations are performed. 3 Results Results are given in figure 2. At study level, the OLS pooling method shows consistently lower levels of overlap, power and FPR while the mixed effects pooling method has the highest levels of overlap, power and FPR. For the meta-analysis methods, the fixed and random effects analyses show higher levels of overlap and power compared to ALE though with a slight increase in FPR. 4 Conclusions With this study, we show how analytical choices at the individual study level impact results on the meta-analytical level. More specifically, OLS leads to more conservative results while fixed/mixed effects pooling is more liberal. The different performance of the meta-analysis methods requires further research. One possible explanation is the implemented kernel sizes to model local maxima that differ between the methods.

Neuroscientists are increasingly aware of the limited reproducibility of functional Magnetic Resonance Imaging (fMRI) studies due to an inherent low signal to noise ratio, small sample sizes, questionable research practices and a strong focus on minimizing type I errors (Durnez et al., 2014). To overcome this, researchers now recognize the value of open science initiatives which leads to an increased amount of open access data and a higher inclusion of null results (Pernet and Poline, 2016). This in turn enables the integration of data across studies to increase the understanding of human brain function. In this contribution, we investigate two approaches of pooling data across studies. Traditionally, researchers can rely on methods for meta-analysis to aggregate effect sizes while accounting for within and between study variability. A second approach is to use a three-level linear model with an additional study level, an extension of the two-stage mixed effects model commonly used in fMRI to aggregate trials and subjects within individual studies. We compare both procedures in terms of the average standardized bias, length and coverage of confidence intervals using both simulations and resting state fMRI (under the null hypothesis). Results indicate that the three-level model does not outperform meta-analysis techniques.

Background Carp (2012) demonstrated the large variability that is present in the method sections of fMRI studies. This methodological variability between studies limits reproducible research. New method Evaluation protocols for methods used in fMRI should include data-analytical stability measures quantifying the variability in results following choices in the methods. Data-analytical stability can be seen as a proxy for reproducibility. To illustrate how one can perform such evaluations, we study two competing approaches for topological feature based inference (random field theory and permutation based testing) and two competing methods for smoothing (Gaussian smoothing and adaptive smoothing). We compare these approaches from the perspective of data-analytical stability in real data, and additionally consider validity and reliability in simulations. Results There is clear evidence that choices in the methods impact the validity, reliability and stability of the results. For the particular comparison studied, we find that permutation based methods render the most valid results. For stability and reliability, the performance of different smoothing and inference types depends on the setting. However, while being more reliable, adaptive smoothing can evoke less stable results when using larger kernel width, especially with cluster size based permutation inference. Comparison with existing methods While existing evaluation methods focus on validity and reliability, we show that data-analytical stability enables to further distinguish between performance of different methods. Conclusion Data-analytical stability is an important additional criterion that can easily be incorporated in evaluation protocols.