Search

Model comparison in Bayesian mixed models is becoming popular in psychological science. Here we develop a set of nested models that account for order restrictions across individuals in psychological tasks. An order-restricted model addresses the question 'Does Everybody', as in, 'Does everybody show the usual Stroop effect', or ‘Does everybody respond more quickly to intense noises than subtle ones.’ The crux of the modeling is the instantiation of 10s or 100s of order restrictions simultaneously, one for each participant. To our knowledge, the problem is intractable in frequentist contexts but relatively straightforward in Bayesian ones. We develop a Bayes factor model-comparison strategy using Zellner and colleagues’ default g-priors appropriate for assessing whether effects obey equality and order restrictions. We apply the methodology to seven data sets from Stroop, Simon, and Eriksen interference tasks. Not too surprisingly, we find that everybody Stroops—that is, for all people congruent colors are truly named more quickly than incongruent ones. But, perhaps surprisingly, we find these order constraints are violated for some people in the Simon task, that is, for these people spatially incongruent responses occur truly more quickly than congruent ones! Implications of the modeling and conjectures about the task-related differences are discussed.This paper was written in R-Markdown with code for data analysis integrated into the text. The Markdown script isopen and freely available at https://github.com/PerceptionAndCognitionLab/ctx-indiff. The data are also open and freely available at https://github.com/PerceptionCognitionLab/data0/tree/master/contexteffects.

Separating the signal in a test from the irrelevant noise is a challenge for all measurement. Low test reliability limits test validity, attenuates important relationships, and can lead to regression artifacts. Multiple approaches to the assessment and improvement of reliability are discussed. The advantages and disadvantages of several different approaches to reliability are considered. Practical advice on how to assess reliability using open source software is provided.

Wagenmakers et al. addressed the illogic use of p-values in 'Psychological Science under Scrutiny'. While historical criticisms mostly deal with the illogical nature of null hypothesis significance testing (NHST), Wagenmakers et al. generalize such argumentation to the p-value itself. Unfortunately, Wagenmakers et al. misinterpret the formal logic basis of tests of significance (and, by extension, of tests of acceptance). This article highlights three instances where such logical interpretation fails and provides plausible corrections and further clarification.

Longitudinal panel data obtained from multiple individuals measured at multiple time points are crucial for psychological research. To analyze such data, a variety of modeling approaches such as hierarchical linear modeling or linear structural equation modeling are available. Such traditional parametric approaches are based on a relatively strong set of assumptions, which are often not met in practice. We present a flexible modeling approach for longitudinal data that is based on the Bayesian statistical learning method Gaussian Process Regression. We term this novel approach Gaussian Process Panel Modeling (GPPM). We show that GPPM subsumes most common modeling approaches for longitudinal data such as linear structural equation models and state-space models as special cases but also extends the space of expressible models beyond them. GPPM offers great flexibility in model specification, facilitates both parametric and nonparametric modeling in a single framework, enables continuous-time modeling as well as person-specific predictions, and offers a modular system that allows the user to piece together hypotheses about change by selecting from and combining predefined types of trajectories or dynamics. We demonstrate the utility of GPPM based on a selection of models and data sets.

Currently, two main approaches exist to distinguish differential susceptibility from diathesis-stress and vantage sensitivity in genotype × environment interaction (G×E) research: Regions of significance (RoS) and competitive-confirmatory approaches. Each is limited by their single-gene/single-environment foci given that most phenotypes are the product of multiple interacting genetic and environmental factors. We thus addressed these two concerns in a recently developed R package (LEGIT) for constructing G×E interaction models with latent genetic and environmental scores using alternating optimization. Herein we test, by means of computer simulation, diverse G×E models in the context of both single and multiple genes and environments. Results indicate that the RoS and competitive-confirmatory approaches were highly accurate when the sample size was large, whereas the latter performed better in small samples and for small effect sizes. The confirmatory approach generally had good accuracy (a) when effect size was moderate and N ≥ 500 and (b) when effect size was large and N ≥ 250, whereas RoS performed poorly. Computational tools to determine the type of G×E of multiple genes and environments are provided as extensions in our LEGIT R package.

Microaggressions–subtle slights that communicate implicit bias–have become a widespread concern in recent years.However, the empirical credibility of microaggression theory has been questioned due to a lack of conceptual clarity and the prevalence of methodological biases within microaggression research. Challenging the empirical validity of the traditional demographics-based microaggression concept, this study examined the potential for cross-spectrum microaggression perception being a method for derogating opposition viewpoints, consistent with existing work on ideological prejudice. Using an experimental online survey (N= 404), there was a significant association between participants’ ideological orientation and their judgements of the ‘perpetrators’and ‘victims’of microaggressions when ‘victims’were associated with leftist causes, but not right-wing causes. These associations were not moderated by emotional investment in ideological orientation (i.e., collective narcissism). Implications for microaggression theory, diversity training programs, and the study of politically-salient individual differences are subsequently discussed.

Seeking to address the lack of research reproducibility, Benjamin et al. propose a pragmatic solution: use a stricter 0.005 standard for statistical significance when claiming evidence of new discoveries. Notwithstanding its potential impact, the proposal has motivated a large mass of authors to dispute it. In reality, a simpler and better suited alternative proposal exists, perhaps the only one the authors could be entitled to make from their Jeffresian perspective: Use JASP, the stand-alone, free-to-download, R-based statistical software with user-friendly GUI for learning more about your error statistics and your beliefs.

Forbes, Wright, Markon, and Krueger (2017) make a compelling case for proceeding cautiously with respect to the overinterpretation and dissemination of results using the increasingly popular approach of creating “networks” from co-occurrences of psychopathology symptoms. We commend the authors on their initial investigation and their utilization of cross-validation techniques in an effort to capture the stability of a variety of network estimation methods. Such techniques get at the heart of establishing “reproducibility”, an increasing focus of concern in both psychology (e.g., Pashler & Wegenmakers, 2012) and science more generally (e.g., Baker, 2016). However, as we will show, the problem is likely worse (or at least more complicated) than they initially indicated. Specifically, for multivariate binary data, the marginal distributions enforce a large degree of structure on the data. We show that some expected measurements – such as commonly used centrality statistics – can have substantially higher values than what would usually be expected. As such, we propose a nonparametric approach to generate confidence intervals through Monte Carlo simulation. We apply the proposed methodology to the National Comorbidity Survey - Replication, provided by Forbes et al, finding that the many of the results are indistinguishable from what would be expected by chance. Further, we discuss the problem of multiple testing and potential issues of applying methods developed for one-mode networks (e.g., ties within a single set of observations) to two-mode networks (e.g., ties between two distinct sets of entities). When taken together, these issues indicate that the psychometric network models should be employed with extreme caution and interpreted guardedly.

Moral psychology has relied nearly exclusively on text stimuli in the development and testing of theories. However, text stimuli lack the rich variety of morally-relevant social and contextual cues available in everyday interactions. A consequence of this pervasive ecological invalidity may be that moral psychological theories are mischaracterized by an overreliance on cue-impoverished moral stimuli. We address this limitation by developing a cue-rich Moral and Affective Film Set (MAAFS). We crowd-sourced videos of moral behaviours, using previously validated text stimuli and definitions of moral foundations as a guide for content. Crowd-sourced clips were rated by 322 American and 253 Australian participants on a range of moral and affective dimensions, including wrongness, moral foundation relevance, punishment, arousal, discrete emotion-relevance, clarity, previous exposure, and how weird/uncommon the moral acts were. Sixty nine moral videos formed the final stimulus set. Ratings confirmed that the videos are reliably rated as morally wrong, feature a variety of moral concerns and that the moral content is clear. The validation process revealed features that make the MAAFS useful for future research: (1) the MAAFS includes a range of videos that depict everyday transgressions, (2) certain videos evoke negative emotions at an intensity comparable to mood induction films, (3) the videos are largely novel: participants had never seen more than 90% of the videos. We anticipate the MAAFS will be a particularly valuable tool for researchers in moral psychology who seek to explore the structure of morality in scenarios that approximate real-life. However, the MAAFS may be valuable for other fields of psychology, for example, affective scientists may use these videos as a mood induction procedure. The complete stimulus set, links to videos, and normative statistics can be accessed at osf.io/8w3en.

The use of quantitative, qualitative, and mixed methods approaches has been foundational to research on emerging adulthood, yet there remain many unresolved methodological issues pertaining to how to handle qualitative data. The purpose of this article is to review best practices for coding and establishing reliability when working with narrative data. In doing so, we highlight how establishing reliability must be seen as an evolving process, rather than simply a focus on the end product. The review is divided into three broad sections. In the first section, we discuss relatively more quantitatively focused methods of coding and establishing reliability, whereas in the second section we discuss relatively more qualitatively focused methods. In the final section, we provide recommendations for researchers interested in coding narrative and other types of open-ended data. This article is intended to serve as an essential resource for researchers working on a variety of topics related to emerging adulthood and beyond.

Psychology is a complicated science. It has no general axioms or mathematical proofs, is rarely directly observable, and has the privilege of being the only discipline in which the content under investigation (i.e. human psychological phenomena) are the very tools utilized to conduct this investigation. For these reasons, it is easy to be seduced by the idea that our psychological theories, limited by our cognitive capacities, accurately reflect a far more complex landscape. Like the Flatlanders in Edwin Abbot’s famous short story (1884), we may be led to believe that the parsimony offered by our low-dimensional theories reflects the reality of a much higher-dimensional problem. Here we contest that this “Flatland fallacy” leads us to seek out simplified explanations of complex phenomena, limiting our capacity as scientists to build and communicate useful models of human psychology. We suggest that this fallacy can be overcome through (1) the use of quantitative models which force researchers to formalize their theories to overcome this fallacy and (2) improved quantitative training which can build new norms for conducting psychological research.

Humans have variable preferences over the uncertainty of potential outcomes during decision-making. To date, most uncertainty research has focused on how individuals form subjective values of risky and ambiguous choices where uncertainty results from known or unknown outcome variability, rather than exploring situations where outcome values are uncertain. Here we examine ‘vagueness,’ defined as a specific form of uncertainty in which a player is informed only that the outcome of a decision lies somewhere within a known range of values. We use a model-based approach and an experimental gambling task to examine whether individuals evaluate uncertainty in outcome values differently from uncertainty in the probability of the outcomes. Our results show that subjective valuation of vagueness, risk, and ambiguity depend upon individuals’ distinct preferences for the first and second moments of gamble outcomes, indicating that individuals have preferences for uncertainty in mean outcome values and outcome variability, respectively, that guide decision-making across types of uncertain situations.

The recent field-wide emphasis on power has brought the number of participants used in experiments into focus. Cognitive psychologists follow a design tradition where few participants perform many trials each. We ask if one wishes to increase power, is it better to add trials or to add participants. The answer is straightforward---greatest power is achieved by using more people, and the gain from adding people is greater than the gain from adding trials. In light of these results, the cognitive design tradition seems less than ideal. Yet, there are conditions where one may trade people for trials with only a minor decrement in power. Under these conditions, the limiting factor is the trial variability rather than variability across people in the population. These conditions are highly plausible, and we present a stochastic-dominance theoretical argument as to why. We think dominance holds in most cognitive effects, for example, in the Stroop effect. Dominance here is the statement that all people truly identify congruent colors faster than incongruent ones. Under this dominance assumption where everyone's true effect is in the same direction, small mean effects imply a small degree of variability across the population. It is this degree of homogeneity, the consequence of dominance, that licenses the cognitive and psychophysical design traditions.

Developing robust statistical methods is an important goal for psychological science. Whereas classical methods (i.e., sampling distributions, p-values, etc.) have been thoroughly characterized, Bayesian robust methods remain relatively uncommon in practice and methodological literatures. Here we propose a robust Bayesian model (BHS t ) that accommodates heterogeneous (H) variances by predicting the scale parameter on the log scale and tail-heaviness with a Student-t likelihood (S t). Through simulations with normative and contaminated (i.e., heavy-tailed) data, we demonstrate that BHS t has consistent frequentist properties in terms of type I error, power, and mean squared error compared to three classical robust methods. With a motivating example, we illustrate Bayesian inferential methods such as approximate leave-one-out cross-validation and posterior predictive checks. We end by suggesting areas of improvement for BHS t and discussing Bayesian robust methods in practice.

Self-reported discriminatory experiences differ slightly by US social racial groups, but why? Using the public-use Add Health dataset, the matter was investigated. It was found that when accounting for cognitive ability (IQ) that differs by social race group, there no longer seemed to be any detectable differences in self-reported discrimination, nor any differences related to skin tone. This seemed to be due to a slight correlation between self-reported discrimination and cognitive ability (r = -.14).Furthermore, the validity of social racial groups were examined for the prediction of self-reported income. It was found that other-perceived social race and skin tone did not predict income when IQ was included in the model, but self-perceived social race did. Results were overall congruent with meritocracy and incongruent with racial discrimination models.

Simple heuristics are often regarded as tractable decision strategies because they ignore a great deal of information in the input data. One puzzle is why heuristics can outperform full-information models, such as linear regression, which make full use of the available information. These "less-is-more'' effects, in which a relatively simpler model outperforms a more complex model, are prevalent throughout cognitive science, and are frequently argued to demonstrate an inherent advantage of simplifying computation or ignoring information. In contrast, we show at the computational level (where algorithmic restrictions are set aside) that it is never optimal to discard information. Through a formal Bayesian analysis, we prove that popular heuristics, such as tallying and Take-the Best, are formally equivalent to Bayesian inference under the limit of infinitely strong priors. Varying the strength of the prior yields a continuum of Bayesian models with the heuristics at one end and ordinary regression at the other. Critically, intermediate models perform better across all our simulations, suggesting that down-weighting information with the appropriate prior is preferable to entirely ignoring it. Rather than because of their simplicity, our analyses suggest heuristics perform well because they implement strong priors that approximate the actual structure of the environment. We end by considering how new heuristics could be derived by infinitely strengthening the priors of other Bayesian models. These formal results have implications for work in psychology, machine learning and economics.

Forbes, Wright, Markon, and Krueger (2017) state that “psychopathology networks have limited replicability” and that “popular network analysis methods produce unreliable results”. These conclusions are based on an assessment of the replicability of four different network models for symptoms of major depression and generalized anxiety across two samples; in addition, Forbes et al. (2017) analyze the stability of the network models within the samples using split-halves. Our re-analysis of the same data with the same methods led to results directly opposed to those of Forbes et al. (2017): All network models replicate very well across the two datasets and across the split-halves. We trace the differences between Forbes et al.’s (2017) results and our own to the fact that they did not appear to accurately implement all network models, and used debatable metrics to assess replicability. In particular, Forbes et al. (2017) deviate from existing estimation routines for relative importance networks, do not acknowledge the fact that the skip-structure used in the interviews strongly distorted correlations between symptoms, and incorrectly assume that network structures and metrics should not only be expected to be the same across the different samples, but also across the different network models used. In addition to a comprehensive re-analysis of the data, we end with a discussion of best practices concerning future research into the replicability of psychometric networks.

In France, little data are available on mental health and well-being in academia, and nothing has been published about PhD students. From studies abroad, we know that doing a PhD is a difficult experience resulting in high attrition rates with significant financial and human costs. Here we focused on PhD students in biology at university Lyon 1. A first study aimed at measuring the mental health and well-being of PhD students using several generalist and PhD-specific tools. Our results on 136 participants showed that a large fraction of the PhD students experience abnormal levels of stress, depression and anxiety, and their mean well-being score is significantly lower than that of a British reference sample. French PhD student well-being is specifically affected by career uncertainty, perceived lack of progress in the PhD and perceived lack of competence, which points towards possible cultural differences of experiencing a PhD in France and the UK. In a second study, we carried out a positive psychology intervention. Comparing the scores of the test and control groups showed a clear effect of the intervention on reducing anxiety. We discuss our results and the possible future steps to improve French PhD students’ well-being.

Objective: Experiencing depression symptoms, even at mild to moderate levels, is associated with maladaptive outcomes for adolescents. We used network analysis to evaluate which symptoms (and associations between symptoms) are most central to adolescent depression.Method: Participants were part of a large, diverse community sample (N = 1,409) of adolescents between the ages of 13-19 years old. Network analysis was used to identify the most central symptoms (nodes) and associations between symptoms (edges) assessed by the Children’s Depression Inventory (CDI). We also evaluated these centrality indicators for network robustness using stability and accuracy tests, associated symptom centrality with mean levels of symptoms, and examined potential differences between the structure and connectivity of depression networks in boys and girls. Results: The most central symptoms in the network were self-hatred, loneliness, sadness, and pessimism. The strongest associations between symptoms were sadness-crying, anhedonia-school dislike, sadness-loneliness, school work difficulty-school performance decrement, self-hatred-negative body image, sleep disturbance-fatigue, and self-deprecation-self-blame. The network was robust to stability and accuracy tests. Notably, symptom centrality and mean levels of symptoms were not associated. Boys and girls’ networks did not differ in levels of connectivity, though the link between body image and self-hatred was stronger in girls than boys. Conclusions: Self-hatred, loneliness, sadness, and pessimism were the most central symptoms in adolescent depression networks, suggesting these symptoms (and associations between symptoms) should be prioritized in theoretical models of adolescent depression and could also serve as important treatment targets for adolescent depression interventions.

The inclusion of partial regression coefficients in the Ferguson (2015a) meta-analysis published in Perspectives on Psychological Science caught our eye. While some anecdotal support was given in this paper to justify this practice, no empirical rationale was offered. This is odd, because a great deal of methodological literature cautions against this practice. To better address this concern, we present evidence through both computational examples and Monte Carlo simulations to suggest that the inclusion of partial relationships in meta-analytic models represents a statistical misspecification that obfuscates the ability to estimate the true population effect size. Additionally, we discuss three means of remediating this issue that specifically address the possibility of statistical control in meta-analysis.

Current theories of mathematical cognition offer competing accounts of the interplay between encoding and calculation in mental arithmetic. Additive models propose that manipulations of problem format do not interact with the cognitive processes used in calculation. Alternatively, interactive models suppose that format manipulations have a direct effect on calculation processes. In the present study, we tested these competing models by fitting participants’ RT distributions in an arithmetic verification task with a single- boundary accumulator model (the shifted Wald distribution). We found that in addition to providing a more complete description of RT distributions, the accumulator model afforded a potentially more sensitive test of format effects. Specifically, we found that format affected drift rate, which implies that problem format has a direct impact on calculation processes. These data give further support for an interactive model of mental arithmetic.

We propose analytic review as a solution to the problem of misreporting statistical results in psychological science. Analytic review requires authors submitting manuscripts for publication to also submit the data file and syntax used during analyses. Regular reviewers or statistical experts then review reported analyses, in order to verify that the analyses reported were actually conducted, and that the statistical values are accurately reported. We begin by describing the problem of misreporting in psychology, and then introduce the basic analytic review process. We then highlight both primary and secondary benefits of adopting analytic review, and describe different permutations of the analytic review system, each with its own strengths and limitations. We conclude by attempting to dispel three anticipated concerns about analytic review, namely: analytic review will increase the workload placed on scholars, analytic review will infringe on the traditional peer-review process, and analytic review will hurt the image of the discipline of psychology. Although implementing analytic review will add one more step to the bureaucratic publication process, we believe it can be implemented in an efficient manner that would greatly assist in decreasing the frequency and impact of misreporting, while also providing secondary benefits in other domains of scientific integrity.

Sexuality researchers frequently use exploratory factor analysis (EFA) to illuminate the distinguishable theoretical constructs assessed by a set of variables. EFA entails a substantive number of analytic decisions to be made with respect to sample size determination, and how factors are extracted, rotated, and retained. The available analytic options, however, are not all equally empirically rigorous. We discuss the commonly available options for conducting EFA and which options constitute best practices for EFA. We also present the results of a methodological review of the analytic options for EFA used by sexuality researchers in more than 200 EFAs, published in more than 160 articles and chapters from 1974 to 2014, in a sample of sexuality research journals. Our review reveals that best practices for EFA are actually those least frequently used by sexuality researchers. We introduce freely available analytic resources to help make it easier for sexuality researchers to adhere to best practices when conducting EFAs in their own research.

An interesting feature of circumplex models of psychological constructs is that they may yield a sinusoidal (i.e., sine wave) pattern of relations between the elements in the model and variables that are external to the model. Thus far, however, the fit to a sinusoidal pattern has been evaluated with visual inspection or arbitrary means of parsing patterns of association. In this research, we developed and validated a sinusoidal fit index (SFI). We then applied the SFI to Schwartz’s original (1992) and revised (Schwartz et al., 2012) models of human values and the circumplex model of interpersonal problems (Alden et al., 1990). Examination of data from the European Social Survey revealed adequate to very good sinusoidal fit of Schwartz’s model to two-thirds of the variables in the survey. Data from published papers revealed very good fit to all variables in the model of interpersonal problems. The results also suggested potential adjustments to the scoring of human values and to the ordering of the values. Discussion focuses on implications for research on values, applications to other circumplex models of individual difference, and broad implications for detecting sinusoidal patterns in data.

In 2000 Nicholas J. Mackintosh (2000) published an article in "Nature" referring to the concept of general intelligence ("g") claiming that there is clear empirical evidence for the existence of the g factor and psychologists are "united in their support of g". Surprisingly, his view remained yet unchallenged although this issue is by no means as clear-cut as Mackintosh argues. Let us therefore attempt to clarify some common but unfortunately major misconceptions about g, which Mackintosh, following Jensen's (1998) precedent, recounted in his "Nature" article. The bottom line is that Spearman's g does not exist, that this has been known and acknowledged by leading scholars (Guttman, 1992; Thurstone, 1947) of factor analysis for decades so that the task of objectively defining human intelligence remains unfinished., Typos corrected

Developmental studies of hormones and behavior often include littermates—rodent siblings that share early-life experiences and genes. Due to between-litter variation (i.e., litter effects), the statistical assumption of independent observations is untenable. In two literatures—natural variation in maternal care and prenatal stress—entire litters are categorized based on maternal behavior or experimental condition. Here, we (1) review both literatures; (2) simulate false positive rates for commonly used statistical methods in each literature; and (3) characterize small sample performance of multilevel models (MLM) and generalized estimating equations (GEE). We found that the assumption of independence was routinely violated (>85%), false positives (α = 0.05) exceeded nominal levels (up to 0.70), and power (1−β) rarely surpassed 0.80 (even for optimistic sample and effect sizes). Additionally, we show that MLMs and GEEs have adequate performance for common research designs. We discuss implications for the extant literature, the field of behavioral neuroendocrinology, and provide recommendations.

This tutorial on Bayesian inference targets psychological researchers who are trained in the null hypothesis testing approach and use of SPSS software. There a number ofexcellent quality tutorials on Bayesian inference, but their problem is that, they assume mathematical knowledge that most psychological researchers do not possess. Thistutorial starts from the idea that Bayesian inference is not more difficult than the traditional approach, but before being introduced to probability theory notation is necessary for the newcomer to understand simple probability principles, which could be explained without mathematical formulas or probability notation. For this purpose in this tutorial I use a simple tool-the parameter-data table-to explain how probability theory can easily be used to make inferences in research. Then I compare the Bayesian and the null hypothesis testing approach using the same tool. Only after having introduced these principles I show the formulas and notations and explain how they relate to the parameter-data table. It is to be expected that this tutorial will increase the use of Bayesian inference by psychological researchers. Moreover, Bayesian researchers may use this tutorial to teach Bayesian inference to undergraduate or postgraduate students.

© 2014 American Psychological Association. Psychological distance and abstraction both represent key variables of considerable interest to researchers across cognitive, social, and developmental psychology. Moreover, largely inspired by construal level theory, numerous experiments across multiple fields have now connected these 2 constructs, examining how psychological distance affects the level of abstraction at which people mentally represent the world around them. The time is clearly ripe for a quantitative synthesis to shed light on the relation between these constructs and investigate potential moderators. To this end, we conducted 2 meta-analyses of research examining the effects of psychological distance on abstraction and its downstream consequences. Across 106 papers containing a total of 267 experiments, our results showed a reliable and medium-sized effect of psychological distance on both level of abstraction in mental representation and the downstream consequences of abstraction. Importantly, these effects replicate across time, researchers, and settings. Our analyses also identified several key moderators, including the size of the difference in distance between 2 levels of a temporal distance manipulation and the dependent variable's capacity to tap processing of both abstract and concrete features (rather than only one or the other). We discuss theoretical and methodological implications, and highlight promising avenues for future research.

Due to converging evidence in animals and healthy human populations, oxytocin has been identified as potentially having therapeutic properties. As such, numerous randomized controlled trials have investigated the efficacy of intranasal oxytocin (IN-OT) on reducing psychiatric symptoms in clinical populations. As results have been mixed, meta-analytic reviews seeking to synthesize the extant literature have been published.One such review was published in Psychiatry Research (Hofmann et al., 2015). The authors concluded that IN-OT significantly improved psychiatric symptoms and found significant effects on depression, anxiety, psychotic symptoms, and general psychopathology. We found several errors in this paper and, when corrected, resulted in all null results (no effect of IN-OT)which suggests that the conclusions of Hofmann et al.(2015) are incorrect. The current letter therefore has three aims: (1) we will outline several errors and raise questions regarding their analysis; and (2) we will perform a meta-analysis using the same primary studies and similar methods; and (3) we will conclude by stating the importance of this letter

In recent years, the network approach to psychopathology has been advanced as an alternative way of conceptualizing mental disorders. In this approach, mental disorders arise from direct interactions between symptoms. Although the network approach has led to many novel methodologies and substantive applications, it has not yet been fully articulated as a scientific theory of mental disorders. The present paper aims to develop such a theory, by postulating a limited set of theoretical principles regarding the structure and dynamics of symptom networks. At the heart of the theory lies the notion that symptoms of psychopathology are causally connected through myriads of biological, psychological and societal mechanisms. If these causal relations are sufficiently strong, symptoms can generate a level of feedback that renders them self-sustaining. In this case, the network can get stuck in a disorder state. The network theory holds that this is a general feature of mental disorders, which can therefore be understood as alternative stable states of strongly connected symptom networks. This idea naturally leads to a comprehensive model of psychopathology, encompassing a common explanatory model for mental disorders, as well as novel definitions of associated concepts such as mental health, resilience, vulnerability and liability. In addition, the network theory has direct implications for how to understand diagnosis and treatment, and suggests a clear agenda for future research in psychiatry and associated disciplines.

Effect size indices are valuable to research in applied psychology, but traditional measures (e.g. Cohen’s d or Pearson’s r) are limited by their ability to convey practical information about intervention effectiveness. Researchers rely on concepts such as ‘standardized mean difference’ or ‘explained variance’ to express effectiveness. Practitioners, policymakers, and lay-people prefer concepts such as frequencies. Partial solutions have been offered by rules-of-thumb (e.g. Cohen’s categories of ‘small’, ‘moderate’ and ‘large’), but such pigeon-holing is somewhat arbitrary and of little nuance. We review, and provide a tutorial on, the Numbers Needed for Treat (NNT) statistic—an effect size index that originated in the medical literature. NNT fills the communicative gap between research and practice, and is particularly suited to gauge the impact of a behavior change intervention on a population level. NNT is defined as the number of people who need to be exposed to an intervention to achieve the desired change in one more individual, relative to a control condition. NNT has informational advantages: 1) it communicates effect magnitude in a frequency-format (number of people) making the impact of an intervention on a population transparent, and 2) it considers the population behavior base-rate to estimate this number. We adapt and extend the NNT index to suit applied psychology endeavors, and argue that the measure can strengthen the translation of intervention research to practice. The statistical procedure to estimate the NNT is explained, illustrated with concrete examples, and supplemented by script and functions to calculate the index in the R - environment.

We present a data set containing 705 between-study heterogeneity estimates τ2 as reported in 61 articles published in 'Psychological Bulletin' from 1990–2013. The data set also includes information about the number and type of effect sizes, the 'Q'- and 'I'2-statistics, and publication bias. The data set is stored in the Open Science Framework repository (https://osf.io/wyhve/) and can be used for several purposes: (1) to compare a specific heterogeneity estimate to the distribution of between-study heterogeneity estimates in psychology; (2) to construct an informed prior distribution for the between-study heterogeneity in psychology; (3) to obtain realistic population values for Monte Carlo simulations investigating the performance of meta-analytic methods. Funding statement: This research was supported by the ERC project “Bayes or Bust”.

Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 ( Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.

Systems factorial technology (Townsend:Nozawa,1995) is a leading methodology for assessing the processing of multiple-feature items. By using certain experimental designs and analyses, researchers can assess wether features are processed in serial, in parallel, or coactively. Current practice is to categorize each individual as displaying one of these three architectures. We argue this approach implicitly assumes heterogeneity of processing strategies across participants. A more scientifically meaningful approach may be to first ask whether all people are serial or parallel or coactive before assuming heterogeneity. We develop a series of Bayesian hierarchical models that captures both situations where everyone follows a common architecture and, alternatively, where there is heterogeneity in architecture. These models use g-prior structures that make computation of Bayes factors convenient. We report an application to investigate Miller's (1956) notion of chunking. We asked participants to compare object that are composed of separable features simultaneously, a perception task, and sequentially, a memory task. We assessed whether processing changed across the perception and memory tasks with the notion that participants might have to chunk features to store them, and that this chunking might make processing more efficient. The answer is ``no." We find a serial architecture for processing for highly separable features (size of circle and the orientation of its diameter) in both the perception and memory tasks. We also find parallel processing for less separable features (first and second digit in a two-digit number) in both perception and memory tasks. Taken together, while processing may depend on the separability of features, it does not vary across perception and memory. As importantly, we find that all people had the same processing strategy; that is models that stated no heterogeneity outperformed those with heterogeneity. This result indicates that architecture may be universal in this setting and not under strategic control.

This paper will discuss the concept of Bayes factors as inferential tools that can directly replace NHST in the day-to-day work of developmental researchers. A Bayes factor indicates the degree which data observed should increase (or decrease) our support for one hypothesis in comparison to another. This framework allows researchers to not just reject but also produce evidence in favor of null hypotheses. Bayes factor alternatives to common tests used by developmental psychologists are available in easy-to-use software. However, we note that Bayesian estimation (rather than Bayes factors) may be a more appealing and general framework when a point null hypothesis is a priori implausible.

Bayesian structural equation modeling (BSEM) has recently gained popularity because it enables researchers to fit complex models and solve some of the issues often encountered in classical maximum likelihood estimation, such as nonconvergence and inadmissible solutions. An important component of any Bayesian analysis is the prior distribution of the unknown model parameters. Often, researchers rely on default priors, which are constructed in an automatic fashion without requiring substantive prior information. However, the prior can have a serious influence on the estimation of the model parameters, which affects the mean squared error, bias, coverage rates, and quantiles of the estimates. In this article, we investigate the performance of three different default priors: noninformative improper priors, vague proper priors, and empirical Bayes priors-with the latter being novel in the BSEM literature. Based on a simulation study, we find that these three default BSEM methods may perform very differently, especially with small samples. A careful prior sensitivity analysis is therefore needed when performing a default BSEM analysis. For this purpose, we provide a practical step-by-step guide for practitioners to conducting a prior sensitivity analysis in default BSEM. Our recommendations are illustrated using a well-known case study from the structural equation modeling literature, and all code for conducting the prior sensitivity analysis is available in the online supplemental materials. (PsycINFO Database Record

We introduce the fundamental tenets of Bayesian inference, which derive from two basic laws of probability theory. We cover the interpretation of probabilities, discrete and continuous versions of Bayes' rule, parameter estimation, and model comparison. Using seven worked examples, we illustrate these principles and set up some of the technical background for the rest of this special issue of Psychonomic Bulletin & Review. Supplemental material is available via https://osf.io/wskex/.

© 2017, Psychonomic Society, Inc. Cognitive biases, such as the anchoring bias, pose a serious challenge to rational accounts of human cognition. We investigate whether rational theories can meet this challenge by taking into account the mind’s bounded cognitive resources. We asked what reasoning under uncertainty would look like if people made rational use of their finite time and limited cognitive resources. To answer this question, we applied a mathematical theory of bounded rationality to the problem of numerical estimation. Our analysis led to a rational process model that can be interpreted in terms of anchoring-and-adjustment. This model provided a unifying explanation for ten anchoring phenomena including the differential effect of accuracy motivation on the bias towards provided versus self-generated anchors. Our results illustrate the potential of resource-rational analysis to provide formal theories that can unify a wide range of empirical results and reconcile the impressive capacities of the human mind with its apparently irrational cognitive biases.

This technical report compares alternative computational models of numerical estimation using Bayesian model selection. We find that people's estimates are best explained by a resource-rational model of anchoring and adjustment according to which the number of adjustments increases with error cost but decreases with time cost so as to achieve an optimal speed-accuracy tradeoff.

The unrealistic high rate of positive results within psychology increased the attention for replication research. Researchers who conduct a replication and want to statistically combine the results of their replication with a statistically significant original study encounter problems when using traditional meta-analysis techniques. The original study’s effect size is most probably overestimated because of it being statistically significant and this bias is not taken into consideration in traditional meta-analysis. We developed a hybrid method that does take statistical significance of the original study into account and enables (a) accurate effect size estimation, (b) estimation of a confidence interval, and (c) testing of the null hypothesis of no effect. We analytically approximate the performance of the hybrid method and describe its good statistical properties. Applying the hybrid method to the data of the Reproducibility Project Psychology (Open Science Collaboration, 2015) demonstrated that the conclusions based on the hybrid method are often in line with those of the replication, suggesting that many published psychological studies have smaller effect sizes than reported in the original study and that some effects may be even absent. We offer hands-on guidelines for how to statistically combine an original study and replication, and developed a web-based application (https://rvanaert.shinyapps.io/hybrid) for applying the hybrid method.

The vast majority of published results in the literature is statistically significant, which raises concerns about their reliability. The Reproducibility Project Psychology (RPP) and Experimental Economics Replication Project (EE-RP) both replicated a large number of published studies in psychology and economics. The original study and replication were statistically significant in 36.1% in RPP and 68.8% in EE-RP suggesting many null effects among the replicated studies. However, evidence in favor of the null hypothesis cannot be examined with null hypothesis significance testing. We developed a Bayesian meta-analysis method called snapshot hybrid that is easy to use and understand and quantifies the amount of evidence in favor of a zero, small, medium and large effect. The method computes posterior model probabilities for a zero, small, medium, and large effect and adjusts for publication bias by taking into account that the original study is statistically significant. We first analytically approximate the methods performance, and demonstrate the necessity to control for the original study’s significance to enable the accumulation of evidence for a true zero effect. Then we applied the method to the data of RPP and EE-RP, showing that the underlying effect sizes of the included studies in EE-RP are generally larger than in RPP, but that the sample sizes of especially the included studies in RPP are often too small to draw definite conclusions about the true effect size. We also illustrate how snapshot hybrid can be used to determine the required sample size of the replication akin to power analysis in null hypothesis significance testing and present an easy to use web application (https://rvanaert.shinyapps.io/snapshot/) and R code for applying the method.

This paper is in press at Journal of Applied Psychology.Abstract:Meta-analytic structural equation modeling (MASEM) is becoming increasingly popular for testing theoretical models from a pool of correlation matrices in management and organizational studies. One limitation of the conventional MASEM approaches is that the proposed structural equation models are only tested on the average correlation matrix. It remains unclear how far the proposed models can be generalized to other populations when the correlation matrices are heterogeneous. Recently, Yu, Downes, Carter, and O’Boyle (2016) proposed a full information MASEM approach to address this limitation by fitting structural equation models from the correlation matrices generated from a parametric bootstrap. However, their approach suffers from several conceptual issues and technical errors. In this study, we reran some of the simulations in Yu et al. by correcting all of the errors in their original studies. The findings showed that bootstrap credible intervals (CVs) work reasonably well, while test statistics and goodness-of-fit indices do not. We advise researchers on what they can and cannot achieve by applying the full information MASEM approach. We recommend fitting MASEM with the TSSEM approach, which works well for the simulation studies. If researchers want to inspect the heterogeneity of the parameters, they may use the bootstrap CVs from the full information MASEM approach. All of these analyses were implemented in the open-source R statistical platform; researchers can easily apply and verify the findings. This paper concludes with several future directions to address the issue of heterogeneity in MASEM.

A key goal in research is to use data to assess competing hypotheses or theories. An alternative to the conventional significance testing is Bayesian model comparison. The main idea is that competing theories are represented by statistical models. In the Bayesian framework, these models then yield predictions about data even before the data are seen. How well the data match the predictions under competing models may be calculated, and the ratio of these matches—the Bayes factor—is used to assess the evidence for one model compared to another. We illustrate the process of going from theories to models and to predictions in the context of two hypothetical examples about how exposure to media affects attitudes toward refugees.

Recent controversies in psychology have highlighted disagreements about how science works and how science proceeds. Despite the existence of tools and data that can help to address some of these disagreements, psychologists have been slow to provide empirical evidence that speaks to these issues. Indeed, there is essentially no work on how scientists communicate their findings. In this paper, I use language from scientific literature to answer questions about scientific behavior. Specifically, I find that features of data analysis practices to be associated with two narrative features of these manuscripts -- hedging (diminishing claims) and boosting (amplifying claims). The use of hedging and boosting were found to be associated with statistical errors. Hedging and boosting were also found to be associated with and statistical overfitting, though this relationship is more uncertain. Additionally male first-authors are less likely to hedge. No other characteristics of the author were found to be associated with these linguistics features. Though the work is exploratory, the results are largely consistent with a linguistic overfitting hypothesis, in which the authors of a paper use language to amplify and diminish findings in a paper in order to create a more compelling narrative around theory. The flexibility of language is a powerful tool used to communicate research, but there is a fine line between effective interpretation of data and convenient overinterpretation, and it is worthwhile to be cognizant of when that line may be crossed.

Studies of highly and profoundly gifted children typically involve small sample sizes, as the population is relatively rare, and many statistical methods cannot handle these small sample sizes well. However, topological data analysis (TDA) tools are robust, even with very small samples, and can provide useful information as well as robust statistical tests.This study demonstrates these capabilities on data simulated from previous talent search results (small and large samples), as well as a subset of data from Ruf’s cohort of gifted children. TDA methods show strong, robust performance and uncover insight into sample characteristics and subgroups, including the appearance of similar subgroups across assessment populations.

Human behavioral data often shows patterns of sudden change overtime. Sometimes the causes of these step changes are internal, such as learning curves that change abruptly when a learner implements a new rule. Sometimes the cause is external, such as when people's opinions about a topic change in response to a new relevant event. Detecting change points in sequences of binary data is basic statistical problem, with many existing solutions, but they seem rarely to be used in psychological modeling. We develop a simple and flexible Bayesian approach to modeling step changes in cognition, implemented as a graphical model in JAGS. The model is able to infer how many change points are justified by the data, as well as the location of the change points. The basic model is also easily extended to include latent-mixture and hierarchical structures, allowing it to be tailored to specific cognitive modeling problems. We demonstrate the adequacy of the basic model by applying it to the classic Lindisfarne Scribes problem, and the flexibility of the modeling approach through two new applications. The first involves a latent-mixture model to determine if individuals learn categories incrementally or in discrete stages.The second involves a hierarchical model of crowd-sourced predictions about the winner of the U.S. National Football League's Most Valuable Player for the 2016{2017 season.

The belief-bias effect is one of the most-studied biases in reasoning. A recent study of the phenomenon using the signal detection theory (SDT) model called into question all theoretical accounts of belief-bias by demonstrating that belief-based differences in the ability to discriminate between valid and invalid syllogisms may be an artifact stemming from the use of inappropriate linear measurement models such as analysis of variance (Dube, Rotello, & Heit, 2010). The discrepancy between Dube et al.'s (2010) results and the previous three decades of work, together with former's methodological criticisms suggests the need to revisit earlier results, this time collecting confidence-rating responses. Using a hierarchical-Bayesian meta-analysis, we reanalyzed a corpus of twenty-two confidence-rating studies (N = 993). The results indicated that extensive replications using confidence-rating data are unnecessary as the observed Receiver Operating Characteristic functions are not systematically asymmetric. These results were subsequently corroborated with a novel experimental design based on SDT's generalized area theorem. Although the meta-analysis confirms that believability does not influence discriminability unconditionally, it also confirmed previous results that factors such as individual differences mediate the effect. The main point is that data from previous and future studies can be safely analyzed using appropriate hierarchical methods which do not require confidence-ratings. More generally, our results set a new standard for analyzing data and evaluating theories in reasoning. Important methodological and theoretical considerations for future work on belief bias and related domains are discussed.

The growing literature conceptualizing mental disorders like Posttraumatic Stress Disorder (PTSD) as networks of interacting symptoms faces three key challenges. Prior studies predominantly used (a) small samples with low power for precise estimation, (b) non-clinical samples, and (c) single samples. This renders network structures in clinical data, and the extent to which networks replicate across datasets, unknown. To overcome these limitations, the present cross-cultural multisite study estimated regularized partial correlation networks of 16 PTSD symptoms across four datasets of traumatized patients receiving treatment for PTSD (total N=2,782). Despite differences in culture, trauma-type and severity of the samples, considerable similarities emerged, with moderate to high correlations between symptom profiles (0.43-0.82), network structures (0.62-0.74), and centrality estimates (0.63-0.75). We discuss the importance of future replicability efforts to improve clinical psychological science, and provide code, model output, and correlation matrices to make the results of this paper fully reproducible.

‘The fallacy of placing confidence in confidence intervals’ (Morey et al., 2016, Psychonomic Bulletin & Review, doi: 10.3758/s13423-015-0947-8) delved into a much needed technical and philosophical dissertation regarding the differences between typical (mis)interpretations of frequentist confidence intervals and the typical correct interpretation of Bayesian credible intervals. My contribution here partly strengthens the authors’ argument, partly closes some gaps they left open, and concludes with a note of attention to the possibility that there may be distinctions without real practical differences in the ultimate use of estimation by intervals, namely when assuming a common ground of uninformative priors and intervals as ranges of values instead of as posterior distributions per se.

The R package “statcheck” (Epskamp & Nuijten, 2016) is a tool to extract statistical results from articles and check whether the reported p-value matches the accompanying test statistic and degrees of freedom. A previous study showed high interrater reliabilities (between .76 and .89) between statcheck and manual coding of inconsistencies (.76 - .89; Nuijten, Hartgerink, Van Assen, Epskamp, & Wicherts, 2016). Here we present an additional, detailed study of the validity of statcheck. In Study 1, we calculated its sensitivity and specificity. We found that statcheck’s sensitivity (true positive rate) and specificity (true negative rate) were high: between 85.3% and 100%, and between 96.0% and 100%, respectively, depending on the assumptions and settings. The overall accuracy of statcheck ranged from 96.2% to 99.9%. In Study 2, we investigated statcheck’s ability to deal with statistical corrections for multiple testing or violations of assumptions in articles. We found that the prevalence of corrections for multiple testing or violations of assumptions in psychology was higher than we initially estimated in Nuijten et al. (2016). Although we found numerous reporting inconsistencies in results corrected for violations of the sphericity assumption, we demonstrate that inconsistencies associated with statistical corrections are not what is causing the high estimates of the prevalence of statistical reporting inconsistencies in psychology.