Generalization of the QST framework in hierarchically structured populations: Impacts of inbreeding and dominance

Q(ST) is a differentiation parameter based on the decomposition of the genetic variance of a trait. In the case of additive inheritance and absence of selection, it is analogous to the genic differentiation measured on individual loci, F(ST). Thus, Q(ST)−F(ST) comparison is used to infer selection: selective divergence when Q(ST) > F(ST), or convergence when Q(ST) < F(ST). The definition of Q-statistics was extended to two-level hierarchical population structures with Hardy–Weinberg equilibrium. Here, we generalize the Q-statistics framework to any hierarchical population structure. First, we developed the analytical definition of hierarchical Q-statistics for populations not at Hardy–Weinberg equilibrium. We show that the Q-statistics values obtained with the Hardy–Weinberg definition are lower than their corresponding F-statistics when F(IS) > 0 (higher when FIS < 0). Then, we used an island model simulation approach to investigate the impact of inbreeding and dominance on the Q(ST)−F(ST) framework in a hierarchical population structure. We show that, while differentiation at the lower hierarchical level (Q(SR)) is a monotonic function of migration, differentiation at the upper level (Q(RT)) is not. In the case of additive inheritance, we show that inbreeding inflates the variance of Q(RT), which can increase the frequency of Q(RT) > F(RT) cases. We also show that dominance drastically reduces Q-statistics below F-statistics for any level of the hierarchy. Therefore, high values of Q-statistics are good indicators of selection, but low values are not in the case of dominance.