Heterogeneity in ecological and evolutionary meta-analyses: Its magnitude and implications

Meta-analysis is the gold standard for synthesis in ecology and evolution. Together with estimating overall effect magnitudes, meta-analyses estimate differences between effect sizes via heterogeneity statistics. It is widely hypothesized that heterogeneity will be present in ecological/evolutionary meta-analyses due to the system-specific nature of biological phenomena. Despite driving recommended best practices, the generality of heterogeneity in ecological data has never been systematically reviewed. We reviewed 700 studies, finding 325 that used formal meta-analysis, of which total heterogeneity was reported in fewer than 40%. We used second-order meta-analysis to collate heterogeneity statistics from 86 studies. Our analysis revealed that the median and mean heterogeneity, expressed as I2, are 84.67% and 91.69%, respectively. These estimates are well above "high" heterogeneity (i.e., 75%), based on widely adopted benchmarks. We encourage reporting heterogeneity in the forms of I2 and the estimated variance components (e.g., τ2) as standard practice. These statistics provide vital insights in to the degree to which effect sizes vary, and provide the statistical support for the exploration of predictors of effect-size magnitude. Along with standard meta-regression techniques that fit moderator variables, multi-level models now allow partitioning of heterogeneity among correlated (e.g., phylogenetic) structures that exist within data.