Search

Researchers commonly make dichotomous claims based on continuous test statistics. Many have branded the practice as a misuse of statistics and criticize scientists for the widespread application of hypothesis tests to tentatively reject a hypothesis (or not) depending on whether a p-value is below or above an alpha level. Although dichotomous claims are rarely explicitly defended, we argue they play an important epistemological and pragmatic role in science. The epistemological function of dichotomous claims consists in transforming data into quasibasic statements, which are tentatively accepted singular facts that can corroborate or falsify theoretical claims. This transformation requires a prespecified methodological decision procedure such as Neyman-Pearson hypothesis tests. From the perspective of methodological falsificationism these decision procedures are necessary, as probabilistic statements (e.g., continuous test statistics) cannot function as falsifiers of substantive hypotheses. The pragmatic function of dichotomous claims is to facilitate scrutiny and criticism among peers by generating contestable claims, a process referred to by Popper as “conjectures and refutations.” We speculate about how the surprisingly widespread use of a 5% alpha level might have facilitated this pragmatic function. Abandoning dichotomous claims, for example because researchers commonly misuse p-values, would sacrifice their crucial epistemic and pragmatic functions.

Robust scientific knowledge is contingent upon replication of original findings. However, replicating researchers are constrained by resources, and will almost always have to choose one replication effort to focus on from a set of potential candidates. To select a candidate efficiently in these cases, we need methods for deciding which out of all candidates considered would be the most useful to replicate, given some overall goal researchers wish to achieve. In this article we assume that the overall goal researchers wish to achieve is to maximize the utility gained by conducting the replication study. We then propose a general rule for study selection in replication research based on the replication value of the set of claims considered for replication. The replication value of a claim is defined as the maximum expected utility we could gain by conducting a replication of the claim, and is a function of (a) the value of being certain about the claim, and (b) uncertainty about the claim based on current evidence. We formalize this definition in terms of a causal decision model, utilizing concepts from decision theory and causal graph modeling. We discuss the validity of using replication value as a measure of expected utility gain, and we suggest approaches for deriving quantitative estimates of replication value. Our goal in this article is not to define concrete guidelines for study selection, but to provide the necessary theoretical foundations on which such concrete guidelines could be built.Translational Abstract Replication-redoing a study using the same procedures-is an important part of checking the robustness of claims in the psychological literature. The practice of replicating original studies has been woefully devalued for many years, but this is now changing. Recent calls for improving the quality of research in psychology has generated a surge of interest in funding, conducting, and publishing replication studies. Because many studies have never been replicated, and researchers have limited time and money to perform replication studies, researchers must decide which studies are the most important to replicate. This way scientists learn the most, given limited resources. In this article, we lay out what it means to think about what is the most important thing to replicate, and we propose a general decision rule for picking a study to replicate. That rule depends on a concept we call replication value. Replication value is a function of the importance of the study, and how uncertain we are about the findings. In this article we explain how researchers can think precisely about the value of replication studies. We then discuss when and how it makes sense to use replication value as a measure of how valuable a replication study would be, and we discuss factors that funders, journals, or scientists could consider when determining how valuable a replication study is.

Exercise physiology and sport science have traditionally made use of the null hypothesis of no difference to make decisions about experimental interventions. In this article, we aim to review current statistical approaches typically used by exercise physiologists and sport scientists for the design and analysis of experimental interventions and to highlight the importance of including equivalence and non-inferiority studies, which address different research questions from deciding whether an effect is present. Initially, we briefly describe the most common approaches, along with their rationale, to investigate the effects of different interventions. We then discuss the main steps involved in the design and analysis of equivalence and non-inferiority studies, commonly performed in other research fields, with worked examples from exercise physiology and sport science scenarios. Finally, we provide recommendations to exercise physiologists and sport scientists who would like to apply the different approaches in future research. We hope this work will promote the correct use of equivalence and non-inferiority designs in exercise physiology and sport science whenever the research context, conditions, applications, researchers’ interests or reasonable beliefs justify these approaches.

Meta-analysis is one of the most useful and powerful research approaches, however, this relevance relies on its credibility. In recent years different concerns about the credibility of psychological research have emerged. Analytical reproducibility of scientific results could be considered as the minimal threshold of it. In this study, our purpose was to assess the analytical reproducibility and data reusability of a set of published meta-analyses. From a random sample of 100 papers containing at least one meta-analysis on the effectiveness of interventions in psychology used in a previous study, 217 meta-analyses were selected. We first tried to retrieve the original data by recovering a data file, recoding the data from document files (pdf, doc), or on request. Second, through a multi-stage workflow, we tried to reproduce the main results of each meta-analysis using these data. The original data were retrieved for 146 meta-analyses from different sources. Of these, in the first stage 52 showed a discrepancy larger than 5% in the main results, in 25 of them, this discrepancy was solved with minor adjustments, or correction of coding errors. In the remaining 27, different issues were identified in an in-depth review of the papers, such as reporting inconsistencies, lack of some data, or transcription errors. Current practices of data sharing in meta-analyses hamper the reusability of meta-analytic data. On the other hand, the implementation of new tools would help to avoid certain errors in the meta-analysis reporting process.

Habits often conflict with goal-directed behaviors and this phenomenon has been convincingly demonstrated in instrumental learning experiments. Recent computational models explain habit-goal conflicts as the competitions between two learning systems, arbitrated by a central unit. In this paper, we show that habit-goal conflicts can be more parsimoniously explained by a dynamic integration of habit and goal values in a sequential sampling model, without any arbitration. A computational model was developed by extending the multialternative decision field theory with the assumptions that habits bias starting positions of preference accumulation, and that goal importance and goal relevance determine sampling probabilities of goal-related attributes. Simulation studies demonstrated our approach’s ability to qualitatively reproduce important empirical findings from three paradigms – classic devaluation, devaluation with a concurrent schedule, and reversal learning, and to predict gradual changes in decision times. Implications of our work for a unified theory of instrumental learning and value-based decision-making are discussed.

Psychologists must be able to test both for the presence of an effect and for the absence of an effect. In addition to testing against zero, researchers can use the two one-sided tests (TOST) procedure to test for equivalence and reject the presence of a smallest effect size of interest (SESOI). The TOST procedure can be used to determine if an observed effect is surprisingly small, given that a true effect at least as extreme as the SESOI exists. We explain a range of approaches to determine the SESOI in psychological science and provide detailed examples of how equivalence tests should be performed and reported. Equivalence tests are an important extension of the statistical tools psychologists currently use and enable researchers to falsify predictions about the presence, and declare the absence, of meaningful effects.

Researchers often rely on analysis of variance (ANOVA) when they report results of experiments. To ensure that a study is adequately powered to yield informative results with an ANOVA, researchers can perform an a priori power analysis. However, power analysis for factorial ANOVA designs is often a challenge. Current software solutions do not allow power analyses for complex designs with several within-participants factors. Moreover, power analyses often need [Formula: see text] or Cohen’s f as input, but these effect sizes are not intuitive and do not generalize to different experimental designs. We have created the R package Superpower and online Shiny apps to enable researchers without extensive programming experience to perform simulation-based power analysis for ANOVA designs of up to three within- or between-participants factors. Predicted effects are entered by specifying means, standard deviations, and, for within-participants factors, the correlations. The simulation provides the statistical power for all ANOVA main effects, interactions, and individual comparisons. The software can plot power across a range of sample sizes, can control for multiple comparisons, and can compute power when the homogeneity or sphericity assumption is violated. This Tutorial demonstrates how to perform a priori power analysis to design informative studies for main effects, interactions, and individual comparisons and highlights important factors that determine the statistical power for factorial ANOVA designs.

Determining a lack of association between an outcome variable and a number of different explanatory variables is frequently necessary in order to disregard a proposed model (i.e., to confirm the lack of a meaningful association between an outcome and predictors). Despite this, the literature rarely offers information about, or technical recommendations concerning, the appropriate statistical methodology to be used to accomplish this task. This paper introduces non-inferiority tests for ANOVA and linear regression analyses, which correspond to the standard widely used F test for η ^ 2 and R2 , respectively. A simulation study is conducted to examine the Type I error rates and statistical power of the tests, and a comparison is made with an alternative Bayesian testing approach. The results indicate that the proposed non-inferiority test is a potentially useful tool for 'testing the null'.

An important step when designing an empirical study is to justify the sample size that will be collected. The key aim of a sample size justification for such studies is to explain how the collected data is expected to provide valuable information given the inferential goals of the researcher. In this overview article six approaches are discussed to justify the sample size in a quantitative empirical study: 1) collecting data from (almost) the entire population, 2) choosing a sample size based on resource constraints, 3) performing an a-priori power analysis, 4) planning for a desired accuracy, 5) using heuristics, or 6) explicitly acknowledging the absence of a justification. An important question to consider when justifying sample sizes is which effect sizes are deemed interesting, and the extent to which the data that is collected informs inferences about these effect sizes. Depending on the sample size justification chosen, researchers could consider 1) what the smallest effect size of interest is, 2) which minimal effect size will be statistically significant, 3) which effect sizes they expect (and what they base these expectations on), 4) which effect sizes would be rejected based on a confidence interval around the effect size, 5) which ranges of effects a study has sufficient power to detect based on a sensitivity power analysis, and 6) which effect sizes are expected in a specific research area. Researchers can use the guidelines presented in this article, for example by using the interactive form in the accompanying online Shiny app, to improve their sample size justification, and hopefully, align the informational value of a study with their inferential goals.

This study investigates PhD candidates’ (N = 391) perceptions about their research environment at a Dutch university in terms of the research climate, (un)ethical supervisory practices, and questionable research practices. We assessed whether their perceptions are related to career considerations. We gathered quantitative self-report estimations of the perceptions of PhD candidates using an online survey tool and then conducted descriptive and within-subject correlation analysis of the results. While most PhD candidates experience fair evaluation processes, openness, integrity, trust, and freedom in their research climate, many report lack of time and support, insufficient supervision, and witness questionable research practices. Results based on Spearman correlations indicate that those who experience a less healthy research environment (including experiences with unethical supervision, questionable practices, and barriers to responsible research), more often consider leaving academia and their current PhD position.

Falsificationist and confirmationist approaches provide two well-established ways of evaluating generalizability. Yarkoni rejects both and invents a third approach we call neo-operationalism. His proposal cannot work for the hypothetical concepts psychologists use, because the universe of operationalizations is impossible to define, and hypothetical concepts cannot be reduced to their operationalizations. We conclude that he is wrong in his generalizability-crisis diagnosis.

Psychological science would become more efficient if researchers implemented sequential designs where feasible. Miller and Ulrich (2020) propose an independent segments procedure where data can be analyzed at a prespecified number of equally spaced looks while controlling the Type I error rate. Such procedures already exist in the sequential analysis literature, and in this commentary, I reflect on whether psychologists should choose to adopt these existing procedures instead. I believe limitations in the independent segments procedure make it relatively unattractive. Being forced to stop for futility based on a bound not chosen to control Type II errors, or reject a smallest effect size of interest in an equivalence test, limits the inferences one can make. Having to use a prespecified number of equally spaced looks is logistically inconvenient. And not having the flexibility to choose α and β spending functions limits the possibility to design efficient studies based on the goal and limitations of the researcher. Recent software packages such as rpact (Wassmer & Pahlke, 2019) make sequential designs equally easy to perform as the independent segments procedure. While learning new statistical methods always takes time, I believe psychological scientists should start on a path that will not limit them in the flexibility and inferences their statistical procedure provides. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Criteria for recognizing and rewarding scientists typically focus on individual contributions. This creates a conflict between what is best for scientists’ careers and what is best for science. In this paper, we show how principles from the theory of multilevel selection provide a toolkit for modifying incentives to better align individual and collective interests. A core principle is the need to shift the level at which selection operates, from individuals to the groups in which individuals are embedded. This principle is used in several fields to improve collective outcomes, including animal husbandry, professional sports, and professional organizations. Shifting the level of selection has the potential to ameliorate several problems in contemporary science, including accounting for scientists’ indirect contributions, reducing individual-level competition, and promoting specialization. We discuss the difficulties associated with changing the level of selection and outline directions for future development in this domain.

The recommendations by Muff and colleagues are an incoherent approach to statistical inferences, and should only be used if one wants to signal a misunderstanding of p-values. Coherent alternatives to quantify evidence exist, such as likelihoods and Bayes factors. Therefore, researchers should not follow the recommendation by Muff and colleagues to report p = 0.08 as ‘weak evidence’, p = 0.03 as ‘moderate evidence’, and p = 0.168 as ‘no evidence’.

The default use of an alpha level of 0.05 is suboptimal for two reasons. First, decisions based on data can be made more efficiently by choosing an alpha level that minimizes the combined Type 1 and Type 2 error rate. Second, it is possible that in studies with very high statistical power p-values lower than the alpha level can be more likely when the null hypothesis is true, than when the alternative hypothesis is true (i.e., Lindley's paradox). This manuscript explains two approaches that can be used to justify a better choice of an alpha level than relying on the default threshold of 0.05. The first approach is based on the idea to either minimize or balance Type 1 and Type 2 error rates. The second approach lowers the alpha level as a function of the sample size to prevent Lindley's paradox. An R package and Shiny app are provided to perform the required calculations. Both approaches have their limitations (e.g., the challenge of specifying relative costs and priors), but can offer an improvement to current practices, especially when sample sizes are large. The use of alpha levels that have a better justification should improve statistical inferences and can increase the efficiency and informativeness of scientific research.

Researchers are generally required to report and interpret effect sizes and associated confidence intervals. When comparing two independent groups, the most commonly used estimator of effect size is Cohen’s ds where sample mean difference is divided by the pooled standard deviation. However, computing the pooled error term is not valid when both groups do not share common population variances. Furthermore, the assumption of equal population variances is unlikely in many psychological fields. Consequently, researchers shift to the use of Welch’s t-test over Student’s t-test in the context of hypothesis testing. Meanwhile, the question which effect size to report when equal variances are not assumed remains open. Based on Monte Carlo simulations, we compare Hedges’ gs (i.e. Cohen’s ds with correction for bias) to Glass’s gs, Shieh’s gs and Hedges’ g_s^*. Comparisons are made under normality as well as under realistic deviations from the assumptions of normality and equal variances. Although it is not directly related with Welch’s t-test (unlike Shieh’s gs), we recommend the use of Hedges’ g_s^* because it shows better properties than all other estimators. Practical recommendations, R package and Shiny App in order to compute effect size estimators and confidence intervals are provided.

Despite the growing popularity of digital health interventions, limitations of traditional behavior change theories and a lack of theory integration hinder theory-driven behavior change applications. In this paper, we aim to review theories relevant to lifestyle behavior change from the broader psychology literature and then integrate these theories into a new theoretical framework called adaptive decision-making to address two specific problems. First, our framework represents lifestyle behaviors at two levels—one of individual daily decisions (action level) and one of larger behavioral episodes (reflection level)—to more closely match the temporal characteristics of lifestyle behaviors and their associated digital data. Second, the framework connects decision-making theories and learning theories to explain how behaviors and cognitive constructs dynamically influence each other, making it a suitable scaffold for building computational models. We map common digital intervention techniques onto the behavioral and cognitive processes in the framework and discuss possible contributions of the framework to both theory development and digital intervention design.

The Action-sentence Compatibility Effect (ACE) is a well-known demonstration of the role of motor activity in the comprehension of language. Participants are asked to make sensibility judgments on sentences by producing movements toward the body or away from the body. The ACE is the finding that movements are faster when the direction of the movement (e.g., toward) matches the direction of the action in the to-be-judged sentence (e.g., Art gave you the pen describes action toward you). We report on a pre-registered, multi-lab replication of one version of the ACE. The results show that none of the 18 labs involved in the study observed a reliable ACE, and that the meta-analytic estimate of the size of the ACE was essentially zero. ispartof: Psychonomic Bulletin & Review vol:29 issue:2 pages:613-626 ispartof: location:United States status: published

Despite the growing popularity of digital health interventions, limitations of traditional behavior change theories and a lack of theory integration hinder theory-driven behavior change applications. In this paper, we aim to review theories relevant to lifestyle behavior change from the broader psychology literature and then integrate these theories into a new theoretical framework called adaptive decision-making to address two specific problems. First, our framework represents lifestyle behaviors at two levels-one of individual daily decisions (action level) and one of larger behavioral episodes (reflection level)-to more closely match the temporal characteristics of lifestyle behaviors and their associated digital data. Second, the framework connects decision-making theories and learning theories to explain how behaviors and cognitive constructs dynamically influence each other, making it a suitable scaffold for building computational models. We map common digital intervention techniques onto the behavioral and cognitive processes in the framework and discuss possible contributions of the framework to both theory development and digital intervention design.

When studies with positive results that support the tested hypotheses have a higher probability of being published than studies with negative results, the literature will give a distorted view of the evidence for scientific claims. Psychological scientists have been concerned about the degree of distortion in their literature due to publication bias and inflated Type-1 error rates. Registered Reports were developed with the goal to minimise such biases: In this new publication format, peer review and the decision to publish take place before the study results are known. We compared the results in the full population of published Registered Reports in Psychology (N = 71 as of November 2018) with a random sample of hypothesis-testing studies from the standard literature (N = 152) by searching 633 journals for the phrase ‘test* the hypothes*’ (replicating a method by Fanelli, 2010). Analysing the first hypothesis reported in each paper, we found 96% positive results in standard reports, but only 44% positive results in Registered Reports. The difference remained nearly as large when direct replications were excluded from the analysis (96% vs 50% positive results). This large gap suggests that psychologists underreport negative results to an extent that threatens cumulative science. Although our study did not directly test the effectiveness of Registered Reports at reducing bias, these results show that the introduction of Registered Reports has led to a much larger proportion of negative results appearing in the published literature compared to standard reports.

Academic journals provide a key quality-control mechanism in science. Yet, information asymmetries and conflicts of interests incentivize scientists to deceive journals about the quality of their research. How can honesty be ensured, despite incentives for deception? Here, we address this question by applying the theory of honest signaling to the publication process. Our models demonstrate that several mechanisms can ensure honest journal submission, including differential benefits, differential costs, and costs to resubmitting rejected papers. Without submission costs, scientists benefit from submitting all papers to high-ranking journals, unless papers can only be submitted a limited number of times. Counterintuitively, our analysis implies that inefficiencies in academic publishing (e.g., arbitrary formatting requirements, long review times) can serve a function by disincentivizing scientists from submitting low-quality work to high-ranking journals. Our models provide simple, powerful tools for understanding how to promote honest paper submission in academic publishing.

Surveys indicate that researchers generally have a positive attitude towards open peer review when this consists of making reviews available alongside published articles. Researchers are more negative about revealing the identity of reviewers. They worry reviewers will be less likely to express criticism if their identity is known to authors. Experiments suggest that reviewers are somewhat less likely to recommend rejection when they are told their identity will be communicated to authors, than when they will remain anonymous. One recent study revealed reviewers in five journals who voluntarily signed their reviews gave more positive recommendations than those who did not sign their reviews. We replicate and extend this finding by analyzing 12010 open reviews in PeerJ and 4188 reviews in the Royal Society Open Science where authors can voluntarily sign their reviews. These results based on behavioral data from real peer reviews across a wide range of scientific disciplines demonstrate convincingly that reviewers’ decision to sign is related to their recommendation. The proportion of signed reviews was higher for more positive recommendations, than for more negative recommendations. We also share all 23649 text-mined reviews as raw data underlying our results that can be re-used by researchers interested in peer review.

This tutorial illustrates how to design, analyze, and report group sequential designs. In these designs, groups of observations are collected and repeatedly analyzed, while controlling error rates. Compared to a fixed sample size design, where data is analyzed only once, group sequential designs offer the possibility to stop the study at interim looks at the data either for efficacy or futility. Hence, they provide greater flexibility and are more efficient in the sense that due to early stopping the expected sample size is smaller as compared to the sample size in the design with no interim look. In this tutorial we illustrate how to use the R package 'rpact' and the associated Shiny app to design studies that control the Type I error rate when repeatedly analyzing data, even when neither the number of looks at the data, nor the exact timing of looks at the data, is specified. Specifically for *t*-tests, we illustrate how to perform an a-priori power analysis for group sequential designs, and explain how to stop the data collection for futility by rejecting the presence of an effect of interest based on a beta-spending function. Finally, we discuss how to report adjusted effect size estimates and confidence intervals. The recent availability of accessible software such as 'rpact' makes it possible for psychologists to benefit from the efficiency gains provided by group sequential designs.

For almost half a century, Paul Meehl educated psychologists about how the mindless use of null-hypothesis significance tests made research on theories in the social sciences basically uninterpretable (Meehl, 1990). In response to the replication crisis, reforms in psychology have focused on formalising procedures for testing hypotheses. These reforms were necessary and impactful. However, as an unexpected consequence, psychologists have begun to realise that they may not be ready to test hypotheses. Forcing researchers to prematurely test hypotheses before they have established a sound ‘derivation chain’ between test and theory is counterproductive. Instead, various non-confirmatory research activities should be used to obtain the inputs necessary to make hypothesis tests informative. Before testing hypotheses, researchers should spend more time forming concepts, developing valid measures, establishing the causal relationships between concepts and their functional form, and identifying boundary conditions and auxiliary assumptions. Providing these inputs should be recognised and incentivised as a crucial goal in and of itself. In this article, we discuss how shifting the focus to non-confirmatory research can tie together many loose ends of psychology’s reform movement and help us lay the foundation to develop strong, testable theories, as Paul Meehl urged us to.

To move beyond the limitations of null-hypothesis tests, statistical approaches have been developed where the observed data are compared against a range of values that are equivalent to the absence of a meaningful effect. Specifying a range of values around zero allows researchers to statistically reject the presence of effects large enough to matter, and prevents practically insignificant effects from being interpreted as a statistically significant difference. We compare the behavior of the recently proposed second generation p-value (Blume, D’Agostino McGowan, Dupont, & Greevy, 2018) with the more established Two One-Sided Tests (TOST) equivalence testing procedure (Schuirmann, 1987). We show that the two approaches yield almost identical results under optimal conditions. Under suboptimal conditions (e.g., when the confidence interval is wider than the equivalence range, or when confidence intervals are asymmetric) the second generation p-value becomes difficult to interpret. The second generation p-value is interpretable in a dichotomous manner (i.e., when the SGPV equals 0 or 1 because the confidence intervals lies completely within or outside of the equivalence range), but this dichotomous interpretation does not require calculations. We conclude that equivalence tests yield more consistent p-values, distinguish between datasets that yield the same second generation p-value, and allow for easier control of Type I and Type II error rates.

To guard against rushed and sloppy science, build pressure testing into your research. To guard against rushed and sloppy science, build pressure testing into your research.

Magnitude based inference (MBI) was widely adopted by sport science researchers as an alternative to null hypothesis significance tests. It has been criticized for lacking a theoretical framework, mixing Bayesian and frequentist thinking, and encouraging researchers to run small studies with high Type 1 error rates. MBI terminology describes the position of confidence intervals in relation to smallest meaningful effect sizes. We show these positions correspond to combinations of one-sided tests of hypotheses about the presence or absence of meaningful effects, and formally describe MBI as a multiple decision procedure. MBI terminology operates as if tests are conducted at multiple alpha levels. We illustrate how error rates can be controlled by limiting each one-sided hypothesis test to a single alpha level. To provide transparent error control in a Neyman-Pearson framework and encourage the use of standard statistical software, we recommend replacing MBI with one-sided tests against smallest meaningful effects, or pairs of such tests as in equivalence testing. Researchers should pre-specify their hypotheses and alpha levels, perform a priori sample size calculations, and justify all assumptions. Our recommendations show researchers what tests to use and how to design and report their statistical analyses to accord with standard frequentist practice.

Statistical reforms in psychology and surrounding domains have been largely motivated by doubts surrounding the methodological and statistical rigor of researchers. In light of high rates of p-value misunderstandings, against the backdrop systematically raised criticisms of NHST, some authors have deemed misconceptions about p-value “impervious to correction” (Haller & Krauss, 2002, p. 1), while others have advocated for the entire abandonment of statistical significance in place of alternative methods of inference (e.g., Amrhein, Greenland, & McShane, 2019). Surprisingly little work, however, has empirically investigated the extent to which statistical misconceptions can be improved reliably within individuals, nor whether these improvements are transient or maintained. The current study (N = 2,320) evaluated baseline misconception rates of p-value, confidence interval, and Bayes factor interpretations among online learners, as well as rates of improvement in accuracy across an 8-week massive open online course (MOOC). Results demonstrated statistically significant improvements in accuracy rates, across all three concepts, for immediate learning (first post-test), as well as support for retained learning until week 8 (second post-test). The current work challenges the idea that statistical misconceptions are impervious to correction, and highlights the importance of pinpointing those more problematic misunderstandings that may warrant more efforts to support improvement. Before we abandon one of the most widely used approaches to statistical inferences in place of an alternative, it is worth exploring whether it is first possible to improve existing misunderstandings amid current methods used.

Contains fulltext : 228242.pdf (Publisher’s version ) (Open Access) Drawing on the concept of a gale of creative destruction in a capitalistic economy, we argue that initiatives to assess the robustness of findings in the organizational literature should aim to simultaneously test competing ideas operating in the same theoretical space. In other words, replication efforts should seek not just to support or question the original findings, but also to replace them with revised, stronger theories with greater explanatory power. Achieving this will typically require adding new measures, conditions, and subject populations to research designs, in order to carry out conceptual tests of multiple theories in addition to directly replicating the original findings. To illustrate the value of the creative destruction approach for theory pruning in organizational scholarship, we describe recent replication initiatives re-examining culture and work morality, working parents’ reasoning about day care options, and gender discrimination in hiring decisions. Significance statement It is becoming increasingly clear that many, if not most, published research findings across scientific fields are not readily replicable when the same method is repeated. Although extremely valuable, failed replications risk leaving a theoretical void - reducing confidence the original theoretical prediction is true, but not replacing it with positive evidence in favor of an alternative theory. We introduce the creative destruction approach to replication, which combines theory pruning methods from the field of management with emerging best practices from the open science movement, with the aim of making replications as generative as possible. In effect, we advocate for a Replication 2.0 movement in which the goal shifts from checking on the reliability of past findings to actively engaging in competitive theory testing and theory building. Scientific transparency statement The materials, code, and data for this article are posted publicly on the Open Science Framework, with links provided in the article. 19 p.

[No abstract available] Funding Agencies|Flinders University; RU, Radboud Universiteit; NIU, Northern Illinois University; University of Nottingham; Universität zu Köln; Charité – Universitätsmedizin Berlin; SLU, Saint Louis University; TU/e, Technische Universiteit Eindhoven; University of Leeds; IUPUI, Indiana University-Purdue University Indianapolis; UiT, Universitetet i Tromsø; EUR, Erasmus Universiteit Rotterdam; UvT, Universiteit van Tilburg; University of Oxford; UNIL, Université de Lausanne; SEBM, Society for Experimental Biology and Medicine; NYU, New York University; UiT, Universitetet i Tromsø; RWTH Aachen University; KU Leuven; Universität Bielefeld; University of Louisiana at Lafayette; UA, University of Arkansas; Genentech Foundation for Biomedical Sciences; University of Bristol; UV, University of Virginia; City of Mobile; Université de Fribourg; Universiteit Gent; Yale University; VT, Virginia Polytechnic Institute and State University; DIBS, Duke Institute for Brain Sciences, Duke University; UCB UK; University of Leeds; BIBS, Brown Institute for Brain Science; UU, Universiteit Utrecht; Keele University; HRD, Division of Human Resource Development; WVU, West Virginia University; King’s College London; NTU, Nottingham Trent University; Rijksuniversiteit Groningen; KACST, King Abdulaziz City for Science and Technology; University of Edinburgh; UCONN, University of Connecticut; University of Kent; University of Cambridge; UNLV, University of Nevada, Las Vegas; Universiteit Leiden; UBC, University of British Columbia; HGS, Human Genome Sciences; AV CR, Akademie Ved Ceské Republiky; IUPUI, Indiana University-Purdue University Indianapolis; UK, University of Kentucky; University of Texas at Austin; DRS, Dahlem Research School, Freie Universität Berlin; Bangor University; RAND Corporation; University of Glasgow; University of Chester; CONICET, Consejo Nacional de Investigaciones Científicas y Técnicas; Massey University; UC, University of California; NIH, National Institutes of Health; UFSC, Universidade Federal de Santa Catarina; OU, Open University; Electroencephalography and Clinical Neuroscience Society; UNAB, Universidad Autónoma de Bucaramanga; KACST, King Abdulaziz City for Science and Technology; SU, Stanford University; CSIR, Council for Scientific and Industrial Research; FSU, Florida State University; UW, University of Wisconsin-Madison; LUMC, Leids Universitair Medisch Centrum; CERSI, Center of Excellence in Regulatory Science and Innovation; DePaul University; Baylor University; SU, Sahlgrenska Universitetssjukhuset; NINDS, National Institute of Neurological Disorders and Stroke

Scientists should be able to provide support for the absence of a meaningful effect. Currently researchers often incorrectly conclude an effect is absent based a non-significant result. A widely recommended approach within a Frequentist framework is to test for equivalence. In equivalence tests, such as the Two One-Sided Tests (TOST) procedure discussed in this article, an upper and lower equivalence bound is specified based on the smallest effect size of interest. The TOST procedure can be used to statistically reject the presence of effects large enough to be considered worthwhile. This practical primer with accompanying spreadsheet and R package enables psychologists to easily perform equivalence tests (and power analyses) by setting equivalence bounds based on standardized effect sizes, and provides recommendations to pre-specify equivalence bounds. Extending your statistical toolkit with equivalence tests might very well be the easiest way for psychologists to improve their statistical and theoretical inferences.

When comparing two independent groups, psychology researchers commonly use Student's t-Tests. Assumptions of normality and homogeneity of variance underlie this test. More often than not, when these conditions are not met, Student's t-Test can be severely biased and lead to invalid statistical inferences. Moreover, we argue that the assumption of equal variances will seldom hold in psychological research, and choosing between Student's t-Test and Welch's t-Test based on the outcomes of a test of the equality of variances often fails to provide an appropriate answer. We show that the Welch's t-Test provides a better control of Type 1 error rates when the assumption of homogeneity of variance is not met, and it loses little robustness compared to Student's t-Test when the assumptions are met. We argue that Welch's t-Test should be used as a default strategy., SCOPUS: ar.j, info:eu-repo/semantics/published

While great interest in health effects of natural product (NP) including dietary supplements and foods persists, promising preclinical NP research is not consistently translating into actionable clinical trial (CT) outcomes. Generally considered the gold standard for assessing safety and efficacy, CTs, especially phase III CTs, are costly and require rigorous planning to optimize the value of the information obtained. More effective bridging from NP research to CT was the goal of a September,2018transdisciplinaryworkshop. Participants emphasized that replicability and likelihood of successful translation depend on rigor in experimental design, interpretation, and reporting across the continuum of NP research. Discussions spanned good practices for NP characterization and quality control; use and interpretation of models (computational through in vivo) with strong clinical predictive validity; controls for experimental artefacts, especially for in vitro interrogation of bioactivity and mechanisms of action; rigorous assessment and interpretation of prior research; transparency in all reporting; and prioritization of research questions. Natural product clinical trials prioritized based on rigorous, convergent supporting data and current public health needs are most likely to be informative and ultimately affect public health. Thoughtful, coordinated implementation of these practices should enhance the knowledge gained from future NP research.

For over two centuries researchers have been criticized for using research practices that makes it easier to present data in line with what they wish to be true. With the rise of the internet it has become easier to preregister the theoretical and empirical basis for predictions, the experimental design, the materials, and the analysis code. Whether the practice of preregistration is valuable depends on your philosophy of science. Here, I provide a conceptual analysis of the value of preregistration for psychological science from an error statistical philosophy (Mayo, 2018). Preregistration has the goal to allow others to transparently evaluate the capacity of a test to falsify a prediction, or the severity of a test. Researchers who aim to test predictions with severity should find value in the practice of preregistration. I differentiate the goal of preregistration from positive externalities, discuss how preregistration itself does not make a study better or worse compared to a non-preregistered study, and highlight the importance of evaluating the usefulness of a tool such as preregistration based on an explicit consideration of your philosophy of science.

While great interest in health effects of natural product (NP) foods and dietary supplements persists, promising preclinical NP research is not consistently translating into actionable clinical trial (CT) outcomes. Generally considered the gold standard for assessing safety and efficacy, CTs,especially Phase III CTs, are costly and require rigorous planning to optimize the value of the information obtained. More effective bridging from NP research to CT was the goal of a September, 2018 transdisciplinary workshop. Participants emphasized that replicability and likelihood of successful translation depend on rigor in experimental design, interpretation, and reporting across the continuum of NP research. Discussions spanned good practices for: NPcharacterization and quality control; use and interpretation of models (computational through in vivo) with strong clinical predictive validity; controls for experimental artefacts, especially for in vitro interrogation of bioactivity and mechanisms of action; rigorous assessment and interpretation of prior research; transparency in all reporting; and prioritization of research questions. NPCTsprioritized based on rigorous, convergent supporting data and current public health needs are most likely to be informative and ultimately affect public health. Thoughtful, coordinated implementation of these practices should enhance the knowledge gained from future NP research.

OBJECTIVES: To discuss good practices and criteria for optimal design and interpretation of pre-clinical and clinical natural product (NP) research in order to increase benefit from our investment in NP clinical trials (CT). Background: Large, randomized, controlled CT often fail to reject the null hypothesis or show rigorous evidence of benefit. This includes recent large, NIH-supported CT of nutrients such as vitamin D and selenium, and of botanical dietary supplements. Negative and positive outcomes may be equally important for public health, but because large CT cost at least $20 M each, plus opportunity costs, it is important that CT designs maximize the yield of actionable information regardless of outcome. METHODS: Experts and stakeholders from academia, government and the private sector collaboratively developed a broadly attended workshop in which good practices to enhance rigor, reproducibility and translational relevance were discussed. RESULTS: N/A. CONCLUSIONS: Critical issues in CT design include product identity, reproducibility and pharmacology (where feasible), power to test a primary outcome significant to consumers, and placebo controls. When basing a CT on traditional use or prior in vitro or in vivo studies, similarity of product (e.g., source identity, methods of preparation, form and intake), health outcome and population (e.g., age, sex, genetics, diet and environment), require careful consideration. Appropriate controls for known types of in vitro assay interference (e.g., aggregation, membrane disruption, protein denaturation) are imperative. Compounds with limited bioavailability, or activity only at concentrations above those achievable by ingestion, are likely poor candidates for dietary CT. Translational validity of model systems should be carefully assessed. Appropriate analyses (e.g., p-curve and meta-regression methods) should be used to obtain bias-corrected effect size estimates, and to identify research areas where the evidence base may be weaker than published findings suggest. Finally, CT prioritization should consider expected impact on public health, and whether known NP causal mechanisms of action are such that useful information, e.g., on product bioavailability or biological activity, are generated even if the completed CT fails to reject the null hypothesis. FUNDING SOURCES: NIH, FDA, USDA.

NOW PUBLISHED: https://doi.org/10.1177%2F2515245920917961. The idea that in behavioural research everything correlates with everything else was a niche area of the scientific literature for over half a century. With the increasing availability of large datasets in psychology, and the heightened interest in falsifiability and null results, the ‘crud’ factor has however become more relevant than ever before. It is often referenced by researchers to discount minute – but statistically significant – effects that are deemed too small to be considered meaningful. This review tracks the history of the crud factor and examines how its use in the scientific literature has developed to this day. It highlights a common and deep-seated lack of understanding about what the crud factor is, whether it can be proven to exist or estimated, and how it should be interpreted. This makes the crud factor a convenient tool for psychologists to disregard unwanted results, even though the presence of a crud factor should be a large inconvenience for the discipline. To inspire a concerted effort to take the crud factor more seriously, this review clarifies the definitions of important concepts, highlights current pitfalls and poses questions that need to be addressed to ultimately improve our understanding of the crud factor. Such work will be necessary to develop the crud factor into a useful concept encouraging improved psychological research and theory corroboration practices.

Ongoing technological developments have made it easier than ever before for scientists to share their data, materials, and analysis code. Sharing data and analysis code makes it easier for other researchers to reuse or check published research. However, these benefits will emerge only if researchers can reproduce the analyses reported in published articles and if data are annotated well enough so that it is clear what all variable and value labels mean. Because most researchers are not trained in computational reproducibility, it is important to evaluate current practices to identify those that can be improved. We examined data and code sharing for Registered Reports published in the psychological literature from 2014 to 2018 and attempted to independently computationally reproduce the main results in each article. Of the 62 articles that met our inclusion criteria, 41 had data available, and 37 had analysis scripts available. Both data and code for 36 of the articles were shared. We could run the scripts for 31 analyses, and we reproduced the main results for 21 articles. Although the percentage of articles for which both data and code were shared (36 out of 62, or 58%) and the percentage of articles for which main results could be computationally reproduced (21 out of 36, or 58%) were relatively high compared with the percentages found in other studies, there is clear room for improvement. We provide practical recommendations based on our observations and cite examples of good research practices in the studies whose main results we reproduced.

Researchers often lack knowledge about how to deal with outliers when analyzing their data. Even more frequently, researchers do not pre-specify how they plan to manage outliers. In this paper we aim to improve research practices by outlining what you need to know about outliers. We start by providing a functional definition of outliers. We then lay down an appropriate nomenclature/classification of outliers. This nomenclature is used to understand what kinds of outliers can be encountered and serves as a guideline to make appropriate decisions regarding the conservation, deletion, or recoding of outliers. These decisions might impact the validity of statistical inferences as well as the reproducibility of our experiments. To be able to make informed decisions about outliers you first need proper detection tools. We remind readers why the most common outlier detection methods are problematic and recommend the use of the median absolute deviation to detect univariate outliers, and of the Mahalanobis-MCD distance to detect multivariate outliers. An R package was created that can be used to easily perform these detection tests. Finally, we promote the use of pre-registration to avoid flexibility in data analysis when handling outliers. 'Publishers note: due to a typesetting error, this paper was originally published with incorrect table numbering, where tables 2, 3, and 4 were incorrectly labelled. This was corrected soon after publication.'

Due to the strong overreliance on p-values in the scientific literature some researchers have argued that p-values should be abandoned or banned, and that we need to move beyond p-values and embrace practical alternatives. When proposing alternatives to p-values statisticians often commit the ‘Statistician’s Fallacy’, where they declare which statistic researchers really ‘want to know’. Instead of telling researchers what they want to know, statisticians should teach researchers which questions they can ask. In some situations, the answer to the question they are most interested in will be the p-value. As long as null-hypothesis tests have been criticized, researchers have suggested to include minimum-effect tests and equivalence tests in our statistical toolbox, and these tests (even though they return p-values) have the potential to greatly improve the questions researchers ask. It is clear there is room for improvement in how we teach p-values. If anyone really believes p-values are an important cause of problems in science, preventing the misinterpretation of p-values by developing better evidence-based education and user-centered statistical software should be a top priority. Telling researchers which statistic they should use has distracted us from examining more important questions, such as asking researchers what they want to know when they do scientific research. Before we can improve our statistical inferences, we need to improve our statistical questions.

Effect sizes are an important outcome of quantitative research, but few guidelines exist that explain how researchers can determine which effect sizes are meaningful. Psychologists often want to study effects that are large enough to make a difference to people's subjective experience. Thus, subjective experience is one way to gauge the meaningfulness of an effect. We propose and illustrate one method for how to quantify the smallest subjectively experienced difference—the smallest change in an outcome measure that individuals consider to be meaningful enough in their subjective experience such that they are willing to rate themselves as feeling different—using an anchor-based method with a global rating of change question applied to the positive and negative affect scale. We provide a step-by-step guide for the questions that researchers need to consider in deciding whether and how to use the anchor-based method, and we make explicit the assumptions of the method that future research can examine. For researchers interested in people's subjective experiences, this anchor-based method provides one way to specify a smallest effect size of interest, which allows researchers to interpret observed results in terms of their theoretical and practical significance.

When data analysts operate within different statistical frameworks (e.g., frequentist versus Bayesian, emphasis on estimation versus emphasis on testing), how does this impact the qualitative conclusions that are drawn for real data? To study this question empirically we selected from the literature two simple scenarios --involving a comparison of two proportions and a Pearson correlation-- and asked four teams of statisticians to provide a concise analysis and a qualitative interpretation of the outcome. The results showed considerable overall agreement; nevertheless, this agreement did not appear to diminish the intensity of the subsequent debate over which statistical framework is more appropriate to address the questions at hand.

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely highpowered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied. UCR::Vicerrectoría de Investigación::Unidades de Investigación::Ciencias Sociales::Instituto de Investigaciones Psicológicas (IIP)

In this commentary, we re-examine the use of a mouse-tracking method for revealing attribute processing speed difference in dietary self-control (Sullivan et al. 2015; Lim et al., 2018). Through re-analyses of Sullivan et al. (2015)’s data and a simulation study, it can be shown that the attribute-angle correlations in the empirical data, which were used to estimate processing speeds, are attributed primarily to their common correlations with choice. The simulation study further suggests that when we account for the choice-mediated attribute-angle correlations, the data patterns used for supporting the original hypothesis can be produced by implementing a plausible alternative mechanism unrelated to processing speeds. The mouse-tracking method therefore fails to provide clear evidence for processing speed difference as a cognitive mechanism of self-control. Researchers should be cautious when using the mouse-tracking method to estimate attribute processing speeds.

Although context effects have repeatedly been demonstrated, it remains difficult to predict how context features influence the associative meaning of concepts. In a recent series of Experiments (see Part 1, Schietecat, Lakens, IJsselsteijn, & de Kort, 2018), we proposed and tested the dimension-specificity hypothesis for understanding and predicting context-dependent cross-modal associations between saturation, brightness, and aggression. In the current manuscript, Part 2, we aim to further test the dimension-specificity hypothesis by predicting the context-dependency of the meaning of the color red observed in the literature. The results of a series of five experiments revealed that the associations between red and valence could be predicted based on the activated dimensions of meaning (i.e., the evaluation or activity dimension) through the opposing concepts in the task. In the context of green, red was associated with negativity, whereas in context of blue, red was mostly associated with activity. Therefore, red was related to both aggression (a negative concept) and enthusiasm (a positive concept), depending on the context.

Although researchers have repeatedly shown that the meaning of the same concept can vary across different contexts, it has proven difficult to predict when people will assign which meaning to a concept, and which associations will be activated by a concept. Building on the affective theory of meaning (Osgood, Suci, & Tannenbaum, 1957) and the polarity correspondence principle (Proctor & Cho, 2006), we propose the dimension-specificity hypothesis with the aim to understand and predict the context-dependency of cross-modal associations. We present three sets of experiments in which we use the dimension-specificity hypothesis to predict the cross-modal associations that should emerge between aggression-related concepts and saturation and brightness. The dimension-specificity hypothesis predicts that cross-modal associations emerge depending upon which affective dimension of meaning (i.e., the evaluation, activity, or potency dimension) is most salient in a specific context. The salience of dimensions of meaning is assumed to depend upon the relative conceptual distances between bipolar opposed concept pairs (e.g., good vs. bad). The dimension-specificity hypothesis proposes that plus and minus polarities will be attributed to the bipolar concepts, and associations between concrete and affective abstract concepts that share plus or minus polarities will become activated. Our data support the emergence of dimension-specific polarity attributions. We discuss the potential of dimension-specific polarity attributions to understand and predict how the context influences the emergence of cross-modal associations.

OBJECTIVES: To study the statistical power of randomized clinical trials and examine developments over time.STUDY DESIGN AND SETTING: We analyzed the statistical power in 136,212 clinical trials between 1975 and 2014 extracted from meta-analyses from the Cochrane database of systematic reviews. We determined study power to detect standardized effect sizes, where power was based on the meta-analyzed effect size. Average power, effect size, and temporal patterns were examined for all meta-analyses and a subset of significant meta-analyses.RESULTS: The number of trials with power ≥80% was low (7%) but increased over time: from 5% in 1975-1979 to 9% in 2010-2014. In significant meta-analyses, the proportion of trials with sufficient power increased from 9% to 15% in these years (median power increased from 16% to 23%). This increase was mainly due to increasing sample sizes, while effect sizes remained stable with a median Cohen's h of 0.09 (interquartile range 0.04-0.22) and a median Cohen's d of 0.20 (0.11-0.40).CONCLUSION: This study demonstrates that sufficient power in clinical trials is still problematic, although the situation is slowly improving. Our data encourage further efforts to increase statistical power in clinical trials to guarantee rigorous and reproducible evidence-based medicine.