Search

The replication crisis and subsequent credibility revolution in psychology have highlighted many suboptimal research practices such as p‐hacking, overgeneralizing, and a lack of transparency. These practices may have been employed reflexively but upon reflection, they are hard to defend. We suggest that current practices for reporting and discussing study limitations are another example of an area where there is much room for improvement. In this article, we call for more rigorous reporting of study limitations in social and personality psychology articles, and we offer advice for how to do this. We recommend that authors consider what the best argument is against their conclusions (which we call the “steel‐person principle”). We consider limitations as threats to construct, internal, external, and statistical conclusion validity (Shadish et al., 2002), and offer some examples for better practice reporting of common study limitations. Our advice has its own limitations — both our representation of current practices and our recommendations are largely based on our own metaresearch and opinions. Nevertheless, we hope that we can prompt researchers to write more deeply and clearly about the limitations of their research, and to hold each other to higher standards when reviewing each other's work.

Every research project has limitations. The limitations that authors acknowledge in their articles offer a glimpse into some of the concerns that occupy a field's attention. We examine the types of limitations authors discuss in their published articles by categorizing them according to the four validities framework and investigate whether the field's attention to each of the four validities has shifted from 2010 to 2020. We selected one journal in social and personality psychology (Social Psychological and Personality Science; SPPS), the subfield most in the crosshairs of psychology's replication crisis. We sampled 440 articles (with half of those articles containing a subsection explicitly addressing limitations), and we identified and categorized 831 limitations across the 440 articles. Articles with limitations sections reported more limitations than those without (avg. 2.6 vs. 1.2 limitations per article). Threats to external validity were the most common type of reported limitation (est. 52% of articles), and threats to statistical conclusion validity were the least common (est. 17% of articles). Authors reported slightly more limitations over time. Despite the extensive attention paid to statistical conclusion validity in the scientific discourse throughout psychology's credibility revolution, our results suggest that concerns about statistics-related issues were not reflected in social and personality psychologists' reported limitations. The high prevalence of limitations concerning external validity might suggest it is time that we improve our practices in this area, rather than apologizing for these limitations after the fact. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

BACKGROUND: Scientists are increasingly concerned with making their work easy to verify and build upon. Associated practices include sharing data, materials, and analytic scripts, and preregistering protocols. This shift towards increased transparency and rigor has been referred to as a "credibility revolution." The credibility of empirical legal research has been questioned in the past due to its distinctive peer review system and because the legal background of its researchers means that many often are not trained in study design or statistics. Still, there has been no systematic study of transparency and credibility-related characteristics of published empirical legal research. METHODS: To fill this gap and provide an estimate of current practices that can be tracked as the field evolves, we assessed 300 empirical articles from highly ranked law journals including both faculty-edited journals and student-edited journals. RESULTS: We found high levels of article accessibility (86%, 95% CI = [82%, 90%]), especially among student-edited journals (100%). Few articles stated that a study's data are available (19%, 95% CI = [15%, 23%]). Statements of preregistration (3%, 95% CI = [1%, 5%]) and availability of analytic scripts (6%, 95% CI = [4%, 9%]) were very uncommon. (i.e., they collected new data using the study's reported methods, but found results inconsistent or not as strong as the original). CONCLUSION: We suggest that empirical legal researchers and the journals that publish their work cultivate norms and practices to encourage research credibility. Our estimates may be revisited to track the field's progress in the coming years.

What research practices should be considered acceptable? Historically, scientists have set the standards for what constitutes acceptable research practices. However, there is value in considering non-scientists' perspectives, including research participants'. 1873 participants from MTurk and university subject pools were surveyed after their participation in one of eight minimal-risk studies. We asked participants how they would feel if (mostly) common research practices were applied to their data: p-hacking/cherry-picking results, selective reporting of studies, Hypothesizing After Results are Known (HARKing), committing fraud, conducting direct replications, sharing data, sharing methods, and open access publishing. An overwhelming majority of psychology research participants think questionable research practices (e.g. p-hacking, HARKing) are unacceptable (68.3-81.3%), and were supportive of practices to increase transparency and replicability (71.4-80.1%). A surprising number of participants expressed positive or neutral views toward scientific fraud (18.7%), raising concerns about data quality. We grapple with this concern and interpret our results in light of the limitations of our study. Despite the ambiguity in our results, we argue that there is evidence (from our study and others') that researchers may be violating participants' expectations and should be transparent with participants about how their data will be used.

Replication-an important, uncommon, and misunderstood practice-is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress.

For several decades, leading behavioral scientists have offered strong criticisms of the common practice of null hypothesis significance testing as producing spurious findings without strong theoretical or empirical support. But only in the past decade has this manifested as a full-scale replication crisis. We consider some possible reasons why, on or about December 2010, the behavioral sciences changed.

While the social sciences have made impressive progress in adopting transparent research practices that facilitate verification, replication, and reuse of materials, the problem of publication bias persists. Bias on the part of peer reviewers and journal editors, as well as the use of outdated research practices by authors, continues to skew literature toward statistically significant effects, many of which may be false positives. To mitigate this bias, we propose a framework to enable authors to report all results efficiently (RARE), with an initial focus on experimental and other prospective empirical social science research that utilizes public study registries. This framework depicts an integrated system that leverages the capacities of existing infrastructure in the form of public registries, institutional review boards, journals, and granting agencies, as well as investigators themselves, to efficiently incentivize full reporting and thereby, improve confidence in social science findings. In addition to increasing access to the results of scientific endeavors, a well-coordinated research ecosystem can prevent scholars from wasting time investigating the same questions in ways that have not worked in the past and reduce wasted funds on the part of granting agencies.

Personality is not the most popular subfield of psychology. But, in one way or another, personality psychologists have played an outsized role in the ongoing "credibility revolution" in psychology. Not only have individual personality psychologists taken on visible roles in the movement, but our field's practices and norms have now become models for other fields to emulate (or, for those who share Baumeister's (2016, https://doi.org/10.1016/j.jesp.2016.02.003) skeptical view of the consequences of increasing rigor, a model for what to avoid). In this article we discuss some unique features of our field that may have placed us in an ideal position to be leaders in this movement. We do so from a subjective perspective, describing our impressions and opinions about possible explanations for personality psychology's disproportionate role in the credibility revolution. We also discuss some ways in which personality psychology remains less-than-optimal, and how we can address these flaws.

Academic journals provide a key quality-control mechanism in science. Yet, information asymmetries and conflicts of interests incentivize scientists to deceive journals about the quality of their research. How can honesty be ensured, despite incentives for deception? Here, we address this question by applying the theory of honest signaling to the publication process. Our models demonstrate that several mechanisms can ensure honest journal submission, including differential benefits, differential costs, and costs to resubmitting rejected papers. Without submission costs, scientists benefit from submitting all papers to high-ranking journals, unless papers can only be submitted a limited number of times. Counterintuitively, our analysis implies that inefficiencies in academic publishing (e.g., arbitrary formatting requirements, long review times) can serve a function by disincentivizing scientists from submitting low-quality work to high-ranking journals. Our models provide simple, powerful tools for understanding how to promote honest paper submission in academic publishing.

Fields closely related to empirical legal research (ELR) are enhancing their methods to improve the credibility of their findings. This includes making data, analysis codes and other materials openly available on digital repositories and preregistering studies. There are numerous benefits to these practices, such as research being easier to find and access through digital research methods. However, ELR appears to be lagging cognate fields. This may be partly due to a lack of field-specific meta-research and guidance. We sought to fill that gap by first evaluating credibility indicators in ELR, including a review of guidelines for legal journals. This review finds considerable room for improvement in how law journals regulate ELR. The remainder of the article provides practical guidance for the field. We start with general recommendations for empirical legal researchers and then turn to recommendations aimed at three commonly used empirical legal methods: content analyses of judicial decisions, surveys and qualitative studies. We end with suggestions for journals and law schools.

Science is often perceived to be a self-correcting enterprise. In principle, the assessment of scientific claims is supposed to proceed in a cumulative fashion, with the reigning theories of the day progressively approximating truth more accurately over time. In practice, however, cumulative self-correction tends to proceed less efficiently than one might naively suppose. Far from evaluating new evidence dispassionately and infallibly, individual scientists often cling stubbornly to prior findings. Here we explore the dynamics of scientific self-correction at an individual rather than collective level. In 13 written statements, researchers from diverse branches of psychology share why and how they have lost confidence in one of their own published findings. We qualitatively characterize these disclosures and explore their implications. A cross-disciplinary survey suggests that such loss-of-confidence sentiments are surprisingly common among members of the broader scientific population yet rarely become part of the public record. We argue that removing barriers to self-correction at the individual level is imperative if the scientific community as a whole is to achieve the ideal of efficient self-correction.

Contains fulltext : 214974.pdf (publisher's version ) (Open Access), In the version of this article initially published, the surname of author Marcus Munafò was misspelled. The error has been corrected in the HTML and PDF versions of the article.

Contains fulltext : 214941.pdf (publisher's version ) (Open Access), We present a consensus-based checklist to improve and document the transparency of research reports in social and behavioural research. An accompanying online application allows users to complete the form and generate a report that they can submit with their manuscript or post to a public repository.

We explore the development of habitual responding within the colour-word contingency learning paradigm, in which participants respond to the colour of neutral words. Each word is most often presented in one colour. Learning is indicated by faster responses to the colour when the word is presented in the expected rather than in the unexpected colour. In Experiment 1, participants took part in two sessions, separated by one day. Critically, one set of words was trained across both days, and other new sets of words were introduced at various time points. Overall performance was faster on trials with overtrained words. Additionally, contingency effects were larger for overtrained words than for words introduced on Day 2. Removing the contingency had a similar impact on the learning effect for overtrained and new words. However, during a counterconditioning phase, where the words were made predictive of new colours, the previous contingency continued to influence performance for overtrained words but not for more recently introduced words. Relatedly, the new contingency was not acquired for the overtrained words. The reverse pattern was observed for recently-introduced words, with the newly-introduced contingency rapidly acquired and the influence of the old contingency quickly extinguished. In Experiments 2 and 3, however, both new and old learning effects were observed for both overtrained and recently-acquired contingencies. The net results suggest that while contingency learning effects are highly pliable during initial and subsequent learning, early-acquired contingency knowledge is maintained after removal of the contingency. Implications for models of learning are discussed.

There is ample evidence that watching sports induces strong emotions that translate into manifold consequential behaviours. However, it is rather ill-understood how exactly spectators’ emotions unfold during soccer matches and what determines their intensity. To address these questions, we used the 2018 FIFA World Cup as a natural quasi-experiment to conduct a pre-registered study on spectators’ emotional experiences. Employing an app-based experience-sampling design, we tracked 251 German spectators during the tournament and assessed high-resolution changes in core affect (valence, activation) throughout soccer matches. Across the three German matches, multi-level models revealed that all spectators exhibited strong changes on both affective dimensions in response to Germany’s performance. Although fans experienced slightly more intense affect than non-fans, particularly during losses, this moderating effect was very small in comparison to the magnitude of the affective fluctuations that occurred independent of fan identity. Taken together, the findings suggest group emotions (collectively felt emotion irrespective of individual affiliation) rather than group-affiliation based emotions (individually felt emotion because of an affiliated group), as the dominant process underlying spectator affect during the 2018 FIFA World Cup.

Under what conditions does prosocial spending promote happiness? In a series of appropriately powered and pre-registered experiments, the present research revisited the role of impact, social connection, and perceived choice in maximizing the emotional benefits of spending money on others. In two exploratory studies, we found that happy (vs. less happy) prosocial spending experiences were marked by higher levels of impact, social connection and perceived choice (Study 1a and 1b). Consistent with these initial findings, three pre-registered studies confirmed that spending money on others was particularly rewarding when people were able to see the difference their generosity made (Study 2); when they felt a sense of social connection to the person or cause they were helping (Study 3); and when they felt that the decision to help was freely chosen (Study 4). Together, our findings corroborate previous research on impact, social connection and perceived choice, and highlight the importance of considering these key variables when evaluating old and new evidence on the emotional benefits of prosocial spending. In addition, our findings suggest that charitable organizations and policymakers should review their current solicitation strategies and pay more attention to people’s sense of impact, connection and choice when seeking charitable donations.

The words that people use have been found to reflect stable psychological traits, but less is known about the extent to which everyday fluctuations in spoken language reflect transient psychological states. We explored within-person associations between spoken words and self-reported state emotion among 185 participants who wore the Electronically Activated Recorder (EAR; an unobtrusive audio recording device) and completed experience sampling reports of their positive and negative emotions 4 times per day for 7 days (1,579 observations). We examined language using the Linguistic Inquiry and Word Count program (LIWC; theoretically created dictionaries) and open-vocabulary themes (clusters of data-driven semantically-related words). Although some studies give the impression that LIWC's positive and negative emotion dictionaries can be used as indicators of emotion experience, we found that when computed on spoken language, LIWC emotion scores were not significantly associated with self-reports of state emotion experience. Exploration of other categories of language variables suggests a number of hypotheses about substantive everyday correlates of momentary positive and negative emotion that can be tested in future studies. These findings (a) suggest that LIWC positive and negative emotion dictionaries may not capture self-reported subjective emotion experience when applied to everyday speech, (b) emphasize the importance of establishing the validity of language-based measures within one's target domain, (c) demonstrate the potential for developing new hypotheses about personality processes from the open-ended words that are used in everyday speech, and (d) extend perspectives on intraindividual variability to the domain of spoken language. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Relationship satisfaction can be assessed in retrospection, as a global evaluation, or as a momentary state. In two experience sampling studies (N = 130, N = 510) the specificities of these assessment modalities are examined. We show that 1) compared to other summary statistics like the median, the mean of relationship satisfaction states describes retrospective and global evaluations best (but the difference to some other summary statistics was negligible); 2) retrospection introduces an overestimation of the average annoyance in the relationship reported on a momentary basis, which results in an overall negative mean-level bias for retrospective relationship satisfaction; 3) this bias is most strongly moderated by global relationship satisfaction at the time of retrospection; 4) snapshots of momentary relationship satisfaction get representative of global evaluations after approximately two weeks of sampling. The findings extend the recall bias reported in the literature for retrospection of negative affect to the domain of relationship evaluations and assist researchers in designing efficient experience sampling studies.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

We present a consensus-based checklist to improve and document the transparency of research reports in social and behavioural research. An accompanying online application allows users to complete the form and generate a report that they can submit with their manuscript or post to a public repository.

The authors evaluate the quality of research reported in major journals in social-personality psychology by ranking those journals with respect to their N-pact Factors (NF)-the statistical power of the empirical studies they publish to detect typical effect sizes. Power is a particularly important attribute for evaluating research quality because, relative to studies that have low power, studies that have high power are more likely to (a) to provide accurate estimates of effects, (b) to produce literatures with low false positive rates, and (c) to lead to replicable findings. The authors show that the average sample size in social-personality research is 104 and that the power to detect the typical effect size in the field is approximately 50%. Moreover, they show that there is considerable variation among journals in sample sizes and power of the studies they publish, with some journals consistently publishing higher power studies than others. The authors hope that these rankings will be of use to authors who are choosing where to submit their best work, provide hiring and promotion committees with a superior way of quantifying journal quality, and encourage competition among journals to improve their NF rankings.

Although self-knowledge is an unquestioned good in many philosophical traditions, testing this assumption scientifically has posed a challenge because of the difficulty of measuring individual differences in self-knowledge. In this study, we used a novel, naturalistic, and objective criterion to determine individuals' degree of self-knowledge. Specifically, self-knowledge was measured as the congruence between people's beliefs about how they typically behave and their actual behavior as measured with unobtrusive audio recordings from daily life. We found that this measure of self-knowledge was positively correlated with informants' perceptions of relationship quality. These results suggest that self-knowledge is interpersonally advantageous. Given the importance of relationships for our social species, self-knowledge could have great social value that has heretofore been overlooked.

How can self-knowledge of personality be improved? What path is the most fruitful source for learning about our true selves? Previous research has noted two main avenues for learning about the self: looking inward (e.g., introspection) and looking outward (e.g., feedback). Although most of the literature on these topics does not directly measure the accuracy of self-perceptions (i.e., self-knowledge), we review these paths and their potential for improving self-knowledge. We come to the conclusion that explicit feedback, a largely unexamined path, is likely a fruitful avenue for learning about one's own personality. Specifically, we suggest that self-knowledge might be fully realized through the use of explicit feedback from close, knowledgeable others. As such, we conclude that the road to self-knowledge likely cannot be traveled alone but must be traveled with close others who can help shed light on our blind spots.