Impact of statistical significance and sample size on conclusions in sports science research - an analysis on the example of the relative age effect.

Purpose : The null hypothesis significance test (NHST) is a commonly applied statistical method for detecting effects in science, despite it being repeatedly criticized. Detractors argue that by focusing exclusively on NHST results, scientists fail to consider descriptive results, potentially leading to misinformed policy makers. They also point out that the influence of sample size on statistical power is often overlooked. This paper investigates whether this critique holds true in sport science research by analyzing the conclusions in publications about the relative age effect (RAE) – an effect manifested in biased birth date patterns. Method : In an extensive content analysis, 7,247 samples listed in 647 sources were recorded and analyzed using binary logistic regression. Results : Findings show discrepancies between NHST results and birth patterns. Authors in RAE research rely more heavily on NHST results than on birth patterns to draw their conclusions regarding the presence of a RAE. In addition, findings indicate that NHST results are influenced by sample size, birth pattern, and the interaction of both. This interaction leads to a RAE more often being suspected in large samples than small samples, even though birth patterns are more evenly distributed in large samples. Conclusion : As large samples are more likely to represent recreational sport and small samples are more likely to represent elite sport, the strong orientation towards NHST results for conclusions can lead to misinformation about the location of substantial RAEs. Applications in Sport : Similar reliance on NHST results and potential misinformation are also to be expected in other topics in sport research, where characteristics like elite status tend to accumulate in certain sample sizes. Decision-makers in sport should contextualize research findings. Researchers should use NHST appropriately and carefully and combine it with other statistical measures.Key Words: null hypothesis significance testing, birthdate effect, metascience [ABSTRACT FROM AUTHOR]