Search

Study preregistration has become increasingly popular in psychology, but its effectiveness in restricting potentially biasing researcher degrees of freedom remains unclear. We used an extensive protocol to assess the strictness of preregistrations and the consistency between preregistration and publications of 300 preregistered psychology studies. We found that preregistrations often lack methodological details and that undisclosed deviations from preregistered plans are frequent. Combining the strictness and consistency results highlights that biases due to researcher degrees of freedom are prevalent and likely in many preregistered studies. More comprehensive registration templates typically yielded stricter and hence better preregistrations. We did not find that effectiveness of preregistrations differed over time or between original and replication studies. Furthermore, we found that operationalizations of variables were generally more effectively preregistered than other study parts. Inconsistencies between preregistrations and published studies were mainly encountered for data collection procedures, statistical models, and exclusion criteria. Our results indicate that, to unlock the full potential of preregistration, researchers in psychology should aim to write stricter preregistrations, adhere to these preregistrations more faithfully, and more transparently report any deviations from the preregistrations. This could be facilitated by training and education to improve preregistration skills, as well as the development of more comprehensive templates.

Supplementary file containing materials and additional sensitivity analysis

We propose to develop an automated journal transparency tool that users could employ to obtain information about a given scholarly journal’s operations and transparency policies. As we envision it, users (e.g., researchers and clinicians) could then use this data to make a more informed decision about whether or not they want to engage with the journal (e.g., read it, submit manuscripts to it, cite work published there). Here we seek to obtain user preferences for the proposed journal transparency tool from the clinician and researcher community. This tool is part of a wider initiative whereby we are using a user-centered design strategy, to obtain stakeholder preference for patients and publishers (e.g., publishers, journals, indexing services, marketing, and communication).

This is a retrospective observational study in which we compare the prevalence of statistical reporting inconsistencies over time in two journals that implemented statcheck in their peer review process (Psychological Science and the Journal of Experimental Social Psychology) and two matched control journals (the Journal of Experimental Psychology: General and the Journal of Personality and Social Psychology).

This study assesses the extent of selective hypothesis reporting in psychological research by comparing the hypotheses found in a set of 459 preregistrations to the hypotheses found in the corresponding papers. We found that more than half of the preregistered studies we assessed contain omitted hypotheses (N = 224; 52.2%) or added hypotheses (N = 227; 56.8%), and about one-fifth of studies contain changed hypotheses (N = 82; 19%). We found only a small number of studies with demoted hypotheses (N = 2; 1%) and no studies with promoted hypotheses. In all, 59% of studies include at least one hypothesis in one or more of these categories, indicating a substantial bias in presenting and selecting hypotheses by researchers and/or reviewers/editors. Contrary to our expectations, we found that added hypotheses and changed hypotheses were not more likely to be statistically significant than non-selectively reported hypotheses. For the other types of selective hypothesis reporting, no powerful test of the relationship with statistical significance could be carried out. Finally, we found that replication studies were less likely to include selectively reported hypotheses than original studies. Thus, selective hypothesis reporting is problematically common in psychological research and may partly be explained by the fact that authors do not specifically and clearly enough formulate hypotheses in preregistrations and papers. We urge researchers, reviewers, and editors to ensure that hypotheses outlined in preregistrations are clearly formulated and accurately presented in the corresponding papers.

This project will perform multiverse analysis on ~67 results in psychology to document variability in effect size estimates based on researcher degrees of freedom and to test whether reported outcomes are larger than outcomes averaged over multiverses within the studies.

In this component you can find all files that are relevant for the preregistration of this project.

This project contains materials for a planned stereotype threat study in 2016 conducted in Dutch high schools.

Stage 1 IPA at Royal Society Open Science

This is the preregistration for the main study of the "Measurement Invariance Testing in Psychology" project. The preregistration for the pilot study can be found here: https://osf.io/6rfy5.

A comparison of the quality and strength of evidence of studies that removed outliers and those who did not remove outliers

This registration is a frozen version of the project done for improved long-term stability upon publication.

Money priming theory proposes that exposure to money-related stimuli can affect people’s thoughts, feelings, motivations and behaviors (for a review, see Vohs, 2015). Our study answers the call for a comprehensive meta-analysis examining the available evidence on money priming (Vadillo, Hardwicke and Shanks, 2016).

The aim of this study is to test the fairness of a Dutch achievement test with respect to gender, to investigate if gender differences in test scores are due to differences in actual latent ability or to irrelevant group characteristics. Tests can be unfair and have measurement bias for a number of reasons, but in the present study we focus on the potential influence of motivational factors in test unfairness. An appropriate method to detect measurement bias is to test for measurement invariance with Multiple Group Confirmatory Factor Analyses (MGCFAs). MGCFAs are performed to test three models for measurement invariance: a two-factor model of cognitive abilities, a one-factor model of motivation and a three-factor model of cognitive abilities and motivation. Test results from a standardization sample of 6195 Dutch students on the Drempeltest and Leermotivatietest are used to answer the research questions.

A meta-analytic project on Cyberball studies of ostracism.

See Document

This preregistration contains details on the data and models that we employ to investigate whether 1) the relationship between missingness propensity and mathematics ability; 2) whether women perform worse than men do under time limits; and 3) whether women are at a double disadvantage on speeded tests under stereotype threat conditions.