Search

Author guidelines for journals could help to promote transparency, openness, and reproducibility

Social scientists should adopt higher transparency standards to improve the quality and credibility of research.

Open data collected from research participants creates a tension between scholarly values of transparency and sharing, on the one hand, and privacy and security, on the other hand. A common solution is to make data sets anonymous by removing personally identifying information (e.g., names or worker IDs) before sharing. However, ostensibly anonymized data sets may be at risk of re-identification if they include demographic information. In the present article, we provide researchers with broadly applicable guidance and tangible tools so that they can engage in open science practices without jeopardizing participants' privacy. Specifically, we (a) review current privacy standards, (b) describe computer science data protection frameworks and their adaptability to the social sciences, (c) provide practical guidance for assessing and addressing re-identification risk, (d) introduce two open-source algorithms developed for psychological scientists-MinBlur and MinBlurLite-to increase privacy while maintaining the integrity of open data, and (e) highlight aspects of ethical data sharing that require further attention. Ultimately, the risk of re-identification should not dissuade engagement with open science practices. Instead, technical innovations should be developed and harnessed so that science can be as open as possible to promote transparency and sharing and as closed as necessary to maintain privacy and security. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental findings using rigour-enhancing practices: confirmatory tests, large sample sizes, preregistration and methodological transparency. In contrast to past systematic replication efforts that reported replication rates averaging 50%, replication attempts here produced the expected effects with significance testing (P < 0.05) in 86% of attempts, slightly exceeding the maximum expected replicability based on observed effect sizes and sample sizes. When one lab attempted to replicate an effect discovered by another lab, the effect size in the replications was 97% that in the original study. This high replication rate justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries., (© 2023. The Author(s).)

Replication-an important, uncommon, and misunderstood practice-is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress.

We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication - it is hard to assess whether reported findings are credible., Competing Interests: TE, AD, BN Employed by the nonprofit Center for Open Science that has a mission to increase openness, integrity, and reproducibility of research, NP, EI Employed by and hold shares in Science Exchange Inc, (© 2021, Errington et al.)

Replicability is an important feature of scientific research, but aspects of contemporary research culture, such as an emphasis on novelty, can make replicability seem less important than it should be. The Reproducibility Project: Cancer Biology was set up to provide evidence about the replicability of preclinical research in cancer biology by repeating selected experiments from high-impact papers. A total of 50 experiments from 23 papers were repeated, generating data about the replicability of a total of 158 effects. Most of the original effects were positive effects (136), with the rest being null effects (22). A majority of the original effect sizes were reported as numerical values (117), with the rest being reported as representative images (41). We employed seven methods to assess replicability, and some of these methods were not suitable for all the effects in our sample. One method compared effect sizes: for positive effects, the median effect size in the replications was 85% smaller than the median effect size in the original experiments, and 92% of replication effect sizes were smaller than the original. The other methods were binary - the replication was either a success or a failure - and five of these methods could be used to assess both positive and null effects when effect sizes were reported as numerical values. For positive effects, 40% of replications (39/97) succeeded according to three or more of these five methods, and for null effects 80% of replications (12/15) were successful on this basis; combining positive and null effects, the success rate was 46% (51/112). A successful replication does not definitively confirm an original finding or its theoretical interpretation. Equally, a failure to replicate does not disconfirm a finding, but it does suggest that additional investigation is needed to establish its reliability., Competing Interests: TE, AD Employed by the Center for Open Science, a non-profit organization that has a mission to increase openness, integrity, and reproducibility of research, MM No competing interests declared, CS Was employed by the Center for Open Science, a non-profit organization that has a mission to increase openness, integrity, and reproducibility of research, NP Was employed by and holds shares in Science Exchange Inc, EI Employed by and holds shares in Science Exchange Inc, BN Employed by the nonprofit Center for Open Science that has a mission to increase openness, integrity, and reproducibility of research, (© 2021, Errington et al.)

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research., Competing Interests: BA, BS, GN, Ov, CA, Mv, JB, DB, UB, RB, LB, NB, EC, AC, NC, AD, Nv, CD, Jv, AD, GD, GE, MG, RH, SH, FH, JH, MJ, KJ, AK, MK, YK, DL, JM, DM, MM, BN, RP, Dv, JR, MS, AS, TS, MS, DS, RS, DS, BS, SS, JS, EU, JW, EW No competing interests declared, BN Executive Director of the Center for Open Science, a non-profit technology and culture change organization with a mission to increase openness, integrity, and reproducibility of research, (© 2021, Aczel et al.)

In registered reports (RRs), initial peer review and in-principle acceptance occur before knowing the research outcomes. This combats publication bias and distinguishes planned from unplanned research. How RRs could improve the credibility of research findings is straightforward, but there is little empirical evidence. Also, there could be unintended costs such as reducing novelty. Here, 353 researchers peer reviewed a pair of papers from 29 published RRs from psychology and neuroscience and 57 non-RR comparison papers. RRs numerically outperformed comparison papers on all 19 criteria (mean difference 0.46, scale range -4 to +4) with effects ranging from RRs being statistically indistinguishable from comparison papers in novelty (0.13, 95% credible interval [-0.24, 0.49]) and creativity (0.22, [-0.14, 0.58]) to sizeable improvements in rigour of methodology (0.99, [0.62, 1.35]) and analysis (0.97, [0.60, 1.34]) and overall paper quality (0.66, [0.30, 1.02]). RRs could improve research quality while reducing publication bias and ultimately improve the credibility of the published literature., (© 2021. The Author(s), under exclusive licence to Springer Nature Limited.)

There is evidence that prediction markets are useful tools to aggregate information on researchers' beliefs about scientific results including the outcome of replications. In this study, we use prediction markets to forecast the results of novel experimental designs that test established theories. We set up prediction markets for hypotheses tested in the Defense Advanced Research Projects Agency's (DARPA) Next Generation Social Science (NGS2) programme. Researchers were invited to bet on whether 22 hypotheses would be supported or not. We define support as a test result in the same direction as hypothesized, with a Bayes factor of at least 10 (i.e. a likelihood of the observed data being consistent with the tested hypothesis that is at least 10 times greater compared with the null hypothesis). In addition to betting on this binary outcome, we asked participants to bet on the expected effect size (in Cohen's d ) for each hypothesis. Our goal was to recruit at least 50 participants that signed up to participate in these markets. While this was the case, only 39 participants ended up actually trading. Participants also completed a survey on both the binary result and the effect size. We find that neither prediction markets nor surveys performed well in predicting outcomes for NGS2., (© 2021 The Authors.)

Sharing clinical trial data can provide value to research participants and communities by accelerating the development of new knowledge and therapies as investigators merge data sets to conduct new analyses, reproduce published findings to raise standards for original research, and learn from the work of others to generate new research questions. Nonprofit funders, including disease advocacy and patient-focused organizations, play a pivotal role in the promotion and implementation of data sharing policies. Funders are uniquely positioned to promote and support a culture of data sharing by serving as trusted liaisons between potential research participants and investigators who wish to access these participants' networks for clinical trial recruitment. In short, nonprofit funders can drive policies and influence research culture. The purpose of this paper is to detail a set of aspirational goals and forward thinking, collaborative data sharing solutions for nonprofit funders to fold into existing funding policies. The goals of this paper convey the complexity of the opportunities and challenges facing nonprofit funders and the appropriate prioritization of data sharing within their organizations and may serve as a starting point for a data sharing toolkit for nonprofit funders of clinical trials to provide the clarity of mission and mechanisms to enforce the data sharing practices their communities already expect are happening., (©Timothy Coetzee, Mad Price Ball, Marc Boutin, Abby Bronson, David T Dexter, Rebecca A English, Patricia Furlong, Andrew D Goodman, Cynthia Grossman, Adrian F Hernandez, Jennifer E Hinners, Lynn Hudson, Annie Kennedy, Mary Jane Marchisotto, Elizabeth Myers, W Benjamin Nowell, Brian A Nosek, Todd Sherer, Carolyn Shore, Ida Sim, Luba Smolensky, Christopher Williams, Julie Wood, Sharon F Terry, Lynn Matrisian. Originally published in Journal of Participatory Medicine (http://jopm.jmir.org), 29.03.2021.)

This meta-analysis evaluated theoretical predictions from balanced identity theory (BIT) and evaluated the validity of zero points of Implicit Association Test (IAT) and self-report measures used to test these predictions. Twenty-one researchers contributed individual subject data from 36 experiments (total N = 12,773) that used both explicit and implicit measures of the social-cognitive constructs. The meta-analysis confirmed predictions of BIT's balance-congruity principle and simultaneously validated interpretation of the IAT's zero point as indicating absence of preference between two attitude objects. Statistical power afforded by the sample size enabled the first confirmations of balance-congruity predictions with self-report measures. Beyond these empirical results, the meta-analysis introduced a within-study statistical test of the balance-congruity principle, finding that it had greater efficiency than the previous best method. The meta-analysis's full data set has been publicly archived to enable further studies of interrelations among attitudes, stereotypes, and identities.

Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of "researcher degrees of freedom" aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimated effect sizes. Preregistration is a mechanism for reducing such degrees of freedom by specifying designs and analysis plans before observing the research outcomes. The effectiveness of preregistration may depend, in part, on whether the process facilitates sufficiently specific articulation of such plans. In this preregistered study, we compared 2 formats of preregistration available on the OSF: Standard Pre-Data Collection Registration and Prereg Challenge Registration (now called "OSF Preregistration," http://osf.io/prereg/). The Prereg Challenge format was a "structured" workflow with detailed instructions and an independent review to confirm completeness; the "Standard" format was "unstructured" with minimal direct guidance to give researchers flexibility for what to prespecify. Results of comparing random samples of 53 preregistrations from each format indicate that the "structured" format restricted the opportunistic use of researcher degrees of freedom better (Cliff's Delta = 0.49) than the "unstructured" format, but neither eliminated all researcher degrees of freedom. We also observed very low concordance among coders about the number of hypotheses (14%), indicating that they are often not clearly stated. We conclude that effective preregistration is challenging, and registration formats that provide effective guidance may improve the quality of research., Competing Interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: DM, CS, and BN are employees of the Center for Open Science (COS), a non-profit technology and culture change company with a mission to increase openness, integrity, and reproducibility of research. COS operates the OSF and the Prereg Challenge.

Preprints increase accessibility and can speed scholarly communication if researchers view them as credible enough to read and use. Preprint services do not provide the heuristic cues of a journal's reputation, selection, and peer-review processes that, regardless of their flaws, are often used as a guide for deciding what to read. We conducted a survey of 3759 researchers across a wide range of disciplines to determine the importance of different cues for assessing the credibility of individual preprints and preprint services. We found that cues related to information about open science content and independent verification of author claims were rated as highly important for judging preprint credibility, and peer views and author information were rated as less important. As of early 2020, very few preprint services display any of the most important cues. By adding such cues, services may be able to help researchers better assess the credibility of preprints, enabling scholars to more confidently use preprints, thereby accelerating scientific communication and discovery., Competing Interests: C.S., T.E. and B.N. are paid employees of the non-profit Center for Open Science that has a mission to increase openness, integrity and reproducibility of research including offering services supporting preprints., (© 2020 The Authors.)

Credibility of scientific claims is established with evidence for their replicability using new data. According to common understanding, replication is repeating a study's procedure and observing whether the prior finding recurs. This definition is intuitive, easy to apply, and incorrect. We propose that replication is a study for which any outcome would be considered diagnostic evidence about a claim from prior research. This definition reduces emphasis on operational characteristics of the study and increases emphasis on the interpretation of possible outcomes. The purpose of replication is to advance theory by confronting existing understanding with new evidence. Ironically, the value of replication may be strongest when existing understanding is weakest. Successful replication provides evidence of generalizability across the conditions that inevitably differ from the original study; Unsuccessful replication indicates that the reliability of the finding may be more constrained than recognized previously. Defining replication as a confrontation of current theoretical expectations clarifies its important, exciting, and generative role in scientific progress., Competing Interests: We have read the journal’s policy and the authors of this manuscript have the following competing interests: BAN and TME are employees of the Center for Open Science, a nonprofit technology and culture change organization with a mission to increase openness, integrity, and reproducibility of research.

We report a randomized trial of a research ethics training intervention designed to enhance ethics communication in university science and engineering laboratories, focusing specifically on authorship and data management. The intervention is a project-based research ethics curriculum that was designed to enhance the ability of science and engineering research laboratory members to engage in reason giving and interpersonal communication necessary for ethical practice. The randomized trial was fielded in active faculty-led laboratories at two US research-intensive institutions. Here, we show that laboratory members perceived improvements in the quality of discourse on research ethics within their laboratories and enhanced awareness of the relevance and reasons for that discourse for their work as measured by a survey administered over 4 mo after the intervention. This training represents a paradigm shift compared with more typical module-based or classroom ethics instruction that is divorced from the everyday workflow and practices within laboratories and is designed to cultivate a campus culture of ethical science and engineering research in the very work settings where laboratory members interact., Competing Interests: Competing interest statement: C.K.S., T.M.E., and B.A.N. are employed by the nonprofit Center for Open Science that has a mission to increase openness, integrity, and reproducibility of research and which maintains the open-source Open Science Framework (OSF) used in the intervention., (Copyright © 2020 the Author(s). Published by PNAS.)

Preregistration clarifies the distinction between planned and unplanned research by reducing unnoticed flexibility. This improves credibility of findings and calibration of uncertainty. However, making decisions before conducting analyses requires practice. During report writing, respecting both what was planned and what actually happened requires good judgment and humility in making claims., (Copyright © 2019 Elsevier Ltd. All rights reserved.)

Most scientific research is conducted by small teams of investigators who together formulate hypotheses, collect data, conduct analyses, and report novel findings. These teams operate independently as vertically integrated silos. Here we argue that scientific research that is horizontally distributed can provide substantial complementary value, aiming to maximize available resources, promote inclusiveness and transparency, and increase rigor and reliability. This alternative approach enables researchers to tackle ambitious projects that would not be possible under the standard model. Crowdsourced scientific initiatives vary in the degree of communication between project members from largely independent work curated by a coordination team to crowd collaboration on shared activities. The potential benefits and challenges of large-scale collaboration span the entire research process: ideation, study design, data collection, data analysis, reporting, and peer review. Complementing traditional small science with crowdsourced approaches can accelerate the progress of science and improve the quality of scientific research.

Using a novel technique known as network meta-analysis, we synthesized evidence from 492 studies (87,418 participants) to investigate the effectiveness of procedures in changing implicit measures, which we define as response biases on implicit tasks. We also evaluated these procedures' effects on explicit and behavioral measures. We found that implicit measures can be changed, but effects are often relatively weak (| d s| < .30). Most studies focused on producing short-term changes with brief, single-session manipulations. Procedures that associate sets of concepts, invoke goals or motivations, or tax mental resources changed implicit measures the most, whereas procedures that induced threat, affirmation, or specific moods/emotions changed implicit measures the least. Bias tests suggested that implicit effects could be inflated relative to their true population values. Procedures changed explicit measures less consistently and to a smaller degree than implicit measures and generally produced trivial changes in behavior. Finally, changes in implicit measures did not mediate changes in explicit measures or behavior. Our findings suggest that changes in implicit measures are possible, but those changes do not necessarily translate into changes in explicit measures or behavior. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Social judgment is shaped by multiple biases operating simultaneously, but most bias-reduction interventions target only a single social category. In seven preregistered studies (total N > 7,000), we investigated whether asking participants to avoid one social bias affected that and other social biases. Participants selected honor society applicants based on academic credentials. Applicants also differed on social categories irrelevant for selection: attractiveness and ingroup status. Participants asked to avoid potential bias in one social category showed small but reliable reductions in bias for that category ( r = .095), but showed near-zero bias reduction on the unmentioned social category ( r = .006). Asking participants to avoid many possible social biases or alerting them to bias without specifically identifying a category did not consistently reduce bias. The effectiveness of interventions for reducing social biases may be highly specific, perhaps even contingent on explicitly and narrowly identifying the potential source of bias.

Serious concerns about the way research is organized collectively are increasingly being raised. They include the escalating costs of research and lower research productivity, low public trust in researchers to report the truth, lack of diversity, poor community engagement, ethical concerns over research practices, and irreproducibility. Open science (OS) collaborations comprise of a set of practices including open access publication, open data sharing and the absence of restrictive intellectual property rights with which institutions, firms, governments and communities are experimenting in order to overcome these concerns. We gathered two groups of international representatives from a large variety of stakeholders to construct a toolkit to guide and facilitate data collection about OS and non-OS collaborations. Ultimately, the toolkit will be used to assess and study the impact of OS collaborations on research and innovation. The toolkit contains the following four elements: 1) an annual report form of quantitative data to be completed by OS partnership administrators; 2) a series of semi-structured interview guides of stakeholders; 3) a survey form of participants in OS collaborations; and 4) a set of other quantitative measures best collected by other organizations, such as research foundations and governmental or intergovernmental agencies. We opened our toolkit to community comment and input. We present the resulting toolkit for use by government and philanthropic grantors, institutions, researchers and community organizations with the aim of measuring the implementation and impact of OS partnership across these organizations. We invite these and other stakeholders to not only measure, but to share the resulting data so that social scientists and policy makers can analyse the data across projects., Competing Interests: No competing interests were disclosed., (Copyright: © 2019 Gold ER et al.)

Being able to replicate scientific findings is crucial for scientific progress 1-15 . We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015 16-36 . The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.

Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes-a process called preregistration. Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting. Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings., Competing Interests: Conflict of interest statement: B.A.N., A.C.D., and D.T.M. are employed by the nonprofit Center for Open Science that has as its mission to increase openness, integrity, and reproducibility of research.

The first results from the Reproducibility Project: Cancer Biology suggest that there is scope for improving reproducibility in pre-clinical cancer research., Competing Interests: The authors declare that no competing interests exist.

Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research., Competing Interests: Competing interests M.R.M, together with C.D.C and D.V.M.B., has received funding from the BBSRC (grant number BB/N019660/1) to convene a workshop on advanced methods for reproducible science, and is chair of the CHDI Foundation Independent Statistical Standing Committee. B.A.N. is executive director of the non-profit Center for Open Science with a mission to increase openness, integrity and reproducibility of research. N.P.d.S. leads the NC3Rs programme of work on experimental design, which developed the ARRIVE guidelines and Experimental Design Assistant. J.J.W. is director, experimental design, at CHDI Management/CHDI Foundation, a non-profit biomedical research organization exclusively dedicated to developing therapeutics for Huntington’s disease. The other authors declare no competing interests.

Implicit preferences are malleable, but does that change last? We tested 9 interventions (8 real and 1 sham) to reduce implicit racial preferences over time. In 2 studies with a total of 6,321 participants, all 9 interventions immediately reduced implicit preferences. However, none were effective after a delay of several hours to several days. We also found that these interventions did not change explicit racial preferences and were not reliably moderated by motivations to respond without prejudice. Short-term malleability in implicit preferences does not necessarily lead to long-term change, raising new questions about the flexibility and stability of implicit preferences. (PsycINFO Database Record, ((c) 2016 APA, all rights reserved).)

Purpose: One challenge academic health centers face is to advance female faculty to leadership positions and retain them there in numbers equal to men, especially given the equal representation of women and men among graduates of medicine and biological sciences over the last 10 years. The purpose of this study is to investigate the explicit and implicit biases favoring men as leaders, among both men and women faculty, and to assess whether these attitudes change following an educational intervention., Method: The authors used a standardized, 20-minute educational intervention to educate faculty about implicit biases and strategies for overcoming them. Next, they assessed the effect of this intervention. From March 2012 through April 2013, 281 faculty members participated in the intervention across 13 of 18 clinical departments., Results: The study assessed faculty members' perceptions of bias as well as their explicit and implicit attitudes toward gender and leadership. Results indicated that the intervention significantly changed all faculty members' perceptions of bias (P < .05 across all eight measures). Although, as expected, explicit biases did not change following the intervention, the intervention did have a small but significant positive effect on the implicit biases surrounding women and leadership of all participants regardless of age or gender (P = .008)., Conclusions: These results suggest that providing education on bias and strategies for reducing it can serve as an important step toward reducing gender bias in academic medicine and, ultimately, promoting institutional change, specifically the promoting of women to higher ranks.

Open access, open data, open source and other open scholarship practices are growing in popularity and necessity. However, widespread adoption of these practices has not yet been achieved. One reason is that researchers are uncertain about how sharing their work will affect their careers. We review literature demonstrating that open research is associated with increases in citations, media attention, potential collaborators, job opportunities and funding opportunities. These findings are evidence that open research practices bring significant benefits to researchers relative to more traditional closed practices.

Beginning January 2014, Psychological Science gave authors the opportunity to signal open data and materials if they qualified for badges that accompanied published articles. Before badges, less than 3% of Psychological Science articles reported open data. After badges, 23% reported open data, with an accelerating trend; 39% reported open data in the first half of 2015, an increase of more than an order of magnitude from baseline. There was no change over time in the low rates of data sharing among comparison journals. Moreover, reporting openness does not guarantee openness. When badges were earned, reportedly available data were more likely to be actually available, correct, usable, and complete than when badges were not earned. Open materials also increased to a weaker degree, and there was more variability among comparison journals. Badges are simple, effective signals to promote open practices and improve preservation of data and materials by using independent repositories.

Replication is vital for increasing precision and accuracy of scientific claims. However, when replications "succeed" or "fail," they could have reputational consequences for the claim's originators. Surveys of United States adults (N = 4,786), undergraduates (N = 428), and researchers (N = 313) showed that reputational assessments of scientists were based more on how they pursue knowledge and respond to replication evidence, not whether the initial results were true. When comparing one scientist that produced boring but certain results with another that produced exciting but uncertain results, opinion favored the former despite researchers' belief in more rewards for the latter. Considering idealized views of scientific practices offers an opportunity to address incentives to reward both innovation and verification.

Gilbert et al. conclude that evidence from the Open Science Collaboration's Reproducibility Project: Psychology indicates high reproducibility, given the study methodology. Their very optimistic assessment is limited by statistical misconceptions and by causal inferences from selectively interpreted, correlational data. Using the Reproducibility Project: Psychology data, both optimistic and pessimistic conclusions about reproducibility are possible, and neither are yet warranted., (Copyright © 2016, American Association for the Advancement of Science.)

Societal perceptions may factor into the high rates of nontreatment in patients with lung cancer. To determine whether bias exists toward lung cancer, a study using the Implicit Association Test method of inferring subconscious attitudes and stereotypes from participant reaction times to visual cues was initiated. Participants were primarily recruited from an online survey panel based on US census data. Explicit attitudes regarding lung and breast cancer were derived from participants' ratings (n = 1778) regarding what they thought patients experienced in terms of guilt, shame, and hope (descriptive statements) and from participants' opinions regarding whether patients ought to experience such feelings (normative statements). Participants' responses to descriptive and normative statements about lung cancer were compared with responses to statements about breast cancer. Analyses of responses revealed that the participants were more likely to agree with negative descriptive and normative statements about lung cancer than breast cancer (P<0.001). Furthermore, participants had significantly stronger implicit negative associations with lung cancer compared with breast cancer; mean response times in the lung cancer/negative conditions were significantly shorter than in the lung cancer/positive conditions (P<0.001). Patients, caregivers, healthcare providers, and members of the general public had comparable levels of negative implicit attitudes toward lung cancer. These results show that lung cancer was stigmatized by patients, caregivers, healthcare professionals, and the general public. Further research is needed to investigate whether implicit and explicit attitudes and stereotypes affect patient care.

Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.

Objectives: We examined providers' implicit and explicit attitudes toward lesbian and gay people by provider gender, sexual identity, and race/ethnicity., Methods: We examined attitudes toward heterosexual people versus lesbian and gay people in Implicit Association Test takers: 2338 medical doctors, 5379 nurses, 8531 mental health providers, 2735 other treatment providers, and 214,110 nonproviders in the United States and internationally between May 2006 and December 2012. We characterized the sample with descriptive statistics and calculated Cohen d, a standardized effect size measure, with 95% confidence intervals., Results: Among heterosexual providers, implicit preferences always favored heterosexual people over lesbian and gay people. Implicit preferences for heterosexual women were weaker than implicit preferences for heterosexual men. Heterosexual nurses held the strongest implicit preference for heterosexual men over gay men (Cohen d = 1.30; 95% confidence interval = 1.28, 1.32 among female nurses; Cohen d = 1.38; 95% confidence interval = 1.32, 1.44 among male nurses). Among all groups, explicit preferences for heterosexual versus lesbian and gay people were weaker than implicit preferences., Conclusions: Implicit preferences for heterosexual people versus lesbian and gay people are pervasive among heterosexual health care providers. Future research should investigate how implicit sexual prejudice affects care.

Women's representation in science has changed substantially, but unevenly, over the past 40 years. In health and biological sciences, for example, women's representation among U.S. scientists is now on par with or greater than men's, while in physical sciences and engineering they remain a clear minority. We investigated whether variation in proportions of women in scientific disciplines is related to differing levels of male-favoring explicit or implicit stereotypes held by students and scientists in each discipline. We hypothesized that science-is-male stereotypes would be weaker in disciplines where women are better represented. This prediction was tested with a sample of 176,935 college-educated participants (70% female), including thousands of engineers, physicians, and scientists. The prediction was supported for the explicit stereotype, but not for the implicit stereotype. Implicit stereotype strength did not correspond with disciplines' gender ratios, but, rather, correlated with two indicators of disciplines' scientific intensity, positively for men and negatively for women. From age 18 on, women who majored or worked in disciplines perceived as more scientific had substantially weaker science-is-male stereotypes than did men in the same disciplines, with gender differences larger than 0.8 standard deviations in the most scientifically-perceived disciplines. Further, particularly for women, differences in the strength of implicit stereotypes across scientific disciplines corresponded with the strength of scientific values held by women in the disciplines. These results are discussed in the context of dual process theory of mental operation and balanced identity theory. The findings point to the need for longitudinal study of the factors' affecting development of adults' and, especially, children's implicit gender stereotypes and scientific identity.