Search

Missing data are ubiquitous in psychological research. They may come about as an unwanted result of coding or computer error, participants' non-response or absence, or missing values may be intentional, as in planned missing designs. We discuss the effects of missing data on χ²-based goodness-of-fit indices in Structural Equation Modeling (SEM), specifically on the Root Mean Squared Error of Approximation (RMSEA). We use simulations to show that naive implementations of the RMSEA have a downward bias in the presence of missing data and, thus, overestimate model goodness-of-fit. Unfortunately, many state-of-the-art software packages report the biased form of RMSEA. As a consequence, the scientific community may have been accepting a much larger fraction of models with non-acceptable model fit. We propose a bias-correction for the RMSEA based on information-theoretic considerations that take into account the expected misfit of a person with fully observed data. The corrected RMSEA is asymptotically independent of the proportion of missing data for misspecified models. Importantly, results of the corrected RMSEA computation are identical to naive RMSEA if there are no missing data.

Ordinal variables, while extremely common in Psychology, are almost exclusively analysed with statistical models that falsely assume them to be metric. This practice can lead to distorted effect size estimates, inflated error rates, and other problems. We argue for the application of ordinal models that make appropriate assumptions about the variables under study. In this tutorial article, we first explain the three major ordinal model classes; the cumulative, sequential and adjacent category models. We then show how to fit ordinal models in a fully Bayesian framework with the R package brms, using data sets on stem cell opinions and marriage time courses. Two appendices provide detailed mathematical derivations of the models, and a third practical example that connects ordinal models to Signal Detection Theory with confidence rating data. Ordinal models provide better theoretical interpretation and numerical inference from ordinal data, and we recommend their widespread adoption in Psychology.

[Correction Notice: An Erratum for this article was reported in Vol 115(6) of Journal of Personality and Social Psychology (see record 2018-63189-002). In the article, the original supplemental material has been revised to include a clarifying note to the Tests of model fit over larger sample (Samples C-G) section and post-peer review analyses added to the Post-peer review Factor Analytic Tests section. All versions of this article have been corrected.] Philosophers and behavioral scientists refer to wisdom as unbiased reasoning that guides one toward a balance of interests and promotes a good life. However, major instruments developed to test wisdom appear biased, and it is unclear whether they capture balance-related tendencies. We examined whether shifting from global, de-contextualized reports to state-level reports about concrete situations provides a less biased method to assess wise reasoning (e.g., intellectual humility, recognition of uncertainty and change, consideration of the broader context at hand and perspectives of others, integration of these perspectives or compromise), which may be aligned with the notion of balancing interests. Results of a large-scale psychometric investigation (N = 4,463) revealed that the novel Situated WIse Reasoning Scale (SWIS) is reliable and appears independent of psychological biases (attribution bias, bias blind spot, self-deception, and impression management), whereas global wisdom reports are subject to such biases. Moreover, SWIS scores were positively related to indices of living well (e.g., adaptive emotion regulation, mindfulness), and balancing of cooperative and self-protective interests, goals (influence-vs.-adjustment), and causal inferences about conflict (attribution to the self-vs.-other party). In contrast, global wisdom reports were unrelated or negatively related to balance-related measures. Notably, people showed modest within-person consistency in wise reasoning across situations or over time, suggesting that a single-shot measurement may be insufficient for whole understanding of trait-level wisdom. We discuss theoretical and practical implications for research on wisdom, judgment and decision making, well-being, and prosociality. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

This article introduces the NetworkToolbox package for R. Network analysis offers an intuitive perspective on complex phenomena via models depicted by nodes (variables) and edges (correlations). The ability of networks to model complexity has made them the standard approach for modeling the intricate interactions in the brain. Similarly, networks have become an increasingly attractive model for studying the complexity of psychological and psychopathological phenomena. NetworkToolbox aims to provide psychological researchers with state-of-the-art methods and measures for estimating and analyzing brain, cognitive, and psychometric data. In this article, I introduce NetworkToolbox and provide a manual for applying the package’s functions to psychometric (personality) data.

Psychological data often violate the normality assumptions made by commonly used statistical methods. These violations are addressed in a variety of ways such as transformations or assuming the employed method is robust to violations. Here we argue that data transformations are unnecessary at best and severely misleading at worst. An alternative approach is to use a Bayesian model that permits skewness and other perturbations to classical assumptions (e.g., heteroskedasticity). Through simulation, we demonstrate that a Bayesian skew-normal model has optimal frequentist properties (i.e., "type 1" error, "power", unbiasedness) compared to normal-assumptive models with or without transformation. Furthermore, the Bayesian skew-normal model has greater predictive utility, as indicated by posterior predictive checking and approximate leave-one-out cross-validation. After an applied example, we discuss practical implications of our findings for psychological science in general, and specifically how Bayesian modeling can improve reproducibility in psychology.

A wide variety of effect size indices have been proposed for quantifying the magnitude of treatment effects in single-case designs. Commonly used measures include parametric indices such as the standardized mean difference as well as nonoverlap measures such as the percentage of nonoverlapping data, improvement rate difference, and nonoverlap of all pairs. Currently, little is known about the properties of these indices when applied to behavioral data collected by systematic direct observation, even though systematic direct observation is the most common method for outcome measurement in single-case research. This study uses Monte Carlo simulation to investigate the properties of several widely used single-case effect size measures when applied to systematic direct observation data. Results indicate that the magnitude of the nonoverlap measures and of the standardized mean difference can be strongly influenced by procedural details of the study's design, which is a significant limitation to using these indices as effect sizes for meta-analysis of single-case designs. A less widely used parametric index, the log response ratio, has the advantage of being insensitive to sample size and observation session length, although its magnitude is influenced by the use of partial interval recording. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

State trace analyses assess the latent dimensionality of a cognitive process by asking whether the means of two dependent variables conform to a monotonic function across a set of conditions. Using an assumption of independence between the measures, recently proposed statistical tests address bivariate measurement error, allowing both frequentist and Bayesian analyses of monotonicity (e.g., Davis-Stober, Morey, Gretton, & Heathcote, 2016; Kalish, Dunn, Burdakov, & Sysoev, 2016). However, statistical inference can be biased by unacknowledged dependencies between measures, particularly when the data are insufficient to overwhelm an incorrect prior assumption of independence. To address this limitation, we developed a hierarchical Bayesian model that explicitly models the separate roles of subject, item, and trial-level dependencies between two measures. Assessment of monotonicity is then performed by fitting separate models that do or do not allow a non-monotonic relation between the condition effects (i.e., same versus different rank orders). The Widely Applicable Information Criterion (WAIC) and Pseudo Bayesian Model Averaging – both cross validation measures of model fit – are used for model comparison, providing an inferential conclusion regarding the dimensionality of the latent psychological space. We validated this new state trace analysis technique using model recovery simulation studies, which assumed different ground truths regarding monotonicity and the direction/magnitude of the subject- and trial-level dependence. We also provide an example application of this new technique to a visual object learning study that compared performance on a visual retrieval task (forced choice part recognition) versus a verbal retrieval task (cued recall).

Steinley, Hoffman, Brusco, and Sher (2017) proposed a new method for evaluating the performance of psychological network models: fixed-margin sampling. The authors investigated LASSO regularized Ising models (eLasso) by generating random datasets with the same margins as the original binary dataset, and concluded that many estimated eLasso parameters are not distinguishable from those that would be expected if the data were generated by chance. We argue that fixed-margin sampling cannot be used for this purpose, as it generates data under a particular null-hypothesis: a unidimensional factor model with interchangeable indicators (i.e., the Rasch model). We show this by discussing relevant psychometric literature and by performing simulation studies. Results indicate that while eLasso correctly estimated network models and estimated almost no edges due to chance, fixed-margin sampling performed poorly in classifying true effects as “interesting” (Steinley et al. 2017, p. 1004). Further simulation studies indicate that fixed-margin sampling offers a powerful method for highlighting local misfit from the Rasch model, but performs only moderately in identifying global departures from the Rasch model. We conclude that fixed-margin sampling is not up to the task of assessing if results from estimated Ising models or other multivariate psychometric models are due to chance.

The analysis of longitudinal neuroimaging data within the massively univariate framework provides the opportunity to study empirical questions about neurodevelopment. Missing outcome data are an all-to-common feature of any longitudinal study, a feature that, if handled improperly, can reduce statistical power and lead to biased parameter estimates. The goal of this paper is to provide conceptual clarity of the issues and non-issues that arise from analyzing incomplete data in longitudinal studies with particular focus on neuroimaging data. This paper begins with a review of the hierarchy of missing data mechanisms and their relationship to likelihood-based methods, a review that is necessary not just for likelihood-based methods, but also for multiple-imputation methods. Next, the paper provides a series of simulation studies with designs common in longitudinal neuroimaging studies to help illustrate missing data concepts regardless of interpretation. Finally, two applied examples are used to demonstrate the sensitivity of inferences under different missing data assumptions and how this may change the substantive interpretation. The paper concludes with a set of guidelines for analyzing incomplete longitudinal data that can improve the validity of research findings in developmental neuroimaging research. Keywords: Neuroimaging, Missing data, Likelihood, Longitudinal data

An important tool in the advancement of cognitive science are quantitative models that represent different cognitive variables in terms of model parameters. To evaluate such models, their parameters are typically tested for relationships with behavioral and physiological variables that are thought to reflect specific cognitive processes. However, many models do not come equipped with the statistical framework needed to relate model parameters to covariates. Instead, researchers often revert to classifying participants into groups depending on their values on the covariates, and subsequently comparing the estimated model parameters between these groups. Here we develop a comprehensive solution to the covariate problem in the form of a Bayesian regression framework. Our framework can be easily added to existing cognitive models and allows researchers to quantify the evidential support for relationships between covariates and model parameters using Bayes factors. Moreover, we present a simulation study that demonstrates the superiority of the Bayesian regression framework to the conventional classification-based approach.

The Cognitive Reflection Test, measuring intuition inhibition and cognitive reflection, has become extremely popular because it reliably predicts reasoning performance, decision-making, and beliefs. Across studies, the response format of CRT items sometimes differs, based on the assumed construct equivalence of tests with open-ended versus multiple-choice items (the equivalence hypothesis). Evidence and theoretical reasons, however, suggest that the cognitive processes measured by these response formats and their associated performances might differ (the nonequivalence hypothesis). We tested the two hypotheses experimentally by assessing the performance in tests with different response formats and by comparing their predictive and construct validity. In a between-subjects experiment (n = 452), participants answered stem-equivalent CRT items in an open-ended, a two-option, or a four-option response format and then completed tasks on belief bias, denominator neglect, and paranormal beliefs (benchmark indicators of predictive validity), as well as on actively open-minded thinking and numeracy (benchmark indicators of construct validity). We found no significant differences between the three response formats in the numbers of correct responses, the numbers of intuitive responses (with the exception of the two-option version, which had a higher number than the other tests), and the correlational patterns of the indicators of predictive and construct validity. All three test versions were similarly reliable, but the multiple-choice formats were completed more quickly. We speculate that the specific nature of the CRT items helps build construct equivalence among the different response formats. We recommend using the validated multiple-choice version of the CRT presented here, particularly the four-option CRT, for practical and methodological reasons. Supplementary materials and data are available at https://osf.io/mzhyc/ .

Evaluation of model fit is critically important for every structural equation model (SEM), and sophisticated methods have been developed for this task. Among them are the χ² goodness-of-fit test, decomposition of the χ², derived measures like the popular root mean square error of approximation (RMSEA) or comparative fit index (CFI), or inspection of residuals or modification indices. Many of these methods provide a global approach to model fit evaluation: A single index is computed that quantifies the fit of the entire SEM to the data. In contrast, graphical criteria like d-separation or trek-separation allow derivation of implications that can be used for local fit evaluation, an approach that is hardly ever applied. We provide an overview of local fit evaluation from the viewpoint of SEM practitioners. In the presence of model misfit, local fit evaluation can potentially help in pinpointing where the problem with the model lies. For models that do fit the data, local tests can identify the parts of the model that are corroborated by the data. Local tests can also be conducted before a model is fitted at all, and they can be used even for models that are globally underidentified. We discuss appropriate statistical local tests, and provide applied examples. We also present novel software in R that automates this type of local fit evaluation. (PsycINFO Database Record

BayeFit is a module for Python that allows users to fit, plot, and extract parameter estimates from models of psychophysical data using Bayesian inference. The module makes extensive use of the module PyStan, which in turn makes extensive use of the package Stan, a powerful tool for Bayesian inference using a No-U-Turn sampler. Although PyStan provides an interface between Python and Stan, defining models for use with PyStan can be difficult for most users given the models are defined in C++ language. BayesFit removes this barrier to entry by providing researchers with functions that streamline the process of defining psychophysical models, obtaining fits, extracting outputs, and visualizing fitted models. The source code for BayesFit is available at https://github.com/slugocm/bayesfit. This module can be expanded and reused as newer versions of Python are developed to ensure researchers always have a tool available to ease the process of fitting models to psychophysical data.

With the development of increasingly automated vehicles (AVs) comes the increasingly difficult challenge of comprehensively validating these for acceptable, and ideally beneficial, impacts on the transport system. There is a growing consensus that virtual testing, where simulated AVs are deployed in simulated traffic, will be key for cost-effective testing and optimization. The least mature model components in such simulations are those generating the behavior of human agents in or around the AVs. In this paper, human models and virtual testing applications are presented for two example scenarios: (i) a human pedestrian deciding whether to cross a street in front of an approaching automated vehicle, with or without external human–machine interface elements, and (ii) an AV handing over control to a human driver in a critical rear-end situation. These scenarios have received much recent research attention, yet simulation-ready human behavior models are lacking. They are discussed here in the context of existing models of perceptual decision-making, situational awareness, and traffic interactions. It is argued that the human behavior in question might be usefully conceptualized as a number of interrelated decision processes, not all of which are necessarily directly associated with externally observable behavior. The results show that models based on this type of framework can reproduce qualitative patterns of behavior reported in the literature for the two addressed scenarios, and it is demonstrated how computer simulations based on the models, once these have been properly validated, could allow prediction and optimization of AV impacts on traffic flow and traffic safety.

Background Participant retention strategies that minimise attrition in longitudinal cohort studies have evolved considerably in recent years. This study aimed to assess, via systematic review and meta-analysis, the effectiveness of both traditional strategies and contemporary innovations for retention adopted by longitudinal cohort studies in the past decade. Methods Health research databases were searched for retention strategies used within longitudinal cohort studies published in the 10-years prior, with 143 eligible longitudinal cohort studies identified (141 articles; sample size range: 30 to 61,895). Details on retention strategies and rates, research designs, and participant demographics were extracted. Meta-analyses of retained proportions were performed to examine the association between cohort retention rate and individual and thematically grouped retention strategies. Results Results identified 95 retention strategies, broadly classed as either: barrier-reduction, community-building, follow-up/reminder, or tracing strategies. Forty-four of these strategies had not been identified in previous reviews. Meta-regressions indicated that studies using barrier-reduction strategies retained 10% more of their sample (95%CI [0.13 to 1.08]; p = .01); however, studies using follow-up/reminder strategies lost an additional 10% of their sample (95%CI [− 1.19 to − 0.21]; p = .02). The overall number of strategies employed was not associated with retention. Conclusions Employing a larger number of retention strategies may not be associated with improved retention in longitudinal cohort studies, contrary to earlier narrative reviews. Results suggest that strategies that aim to reduce participant burden (e.g., flexibility in data collection methods) might be most effective in maximising cohort retention. Electronic supplementary material The online version of this article (10.1186/s12874-018-0586-7) contains supplementary material, which is available to authorized users.

We present a mixed-effects location scale model (MELSM) for examining the daily dynamics of affect in dyads. The MELSM includes person and time-varying variables to predict the location, or individual means, and the scale, or within-person variances. It also incorporates a submodel to account for between-person variances. The dyadic specification can accommodate individual and partner effects in both the location and the scale components, and allows random effects for all location and scale parameters. All covariances among the random effects, within and across the location and the scale are also estimated. These covariances offer new insights into the interplay of individual mean structures, intra-individual variability, and the influence of partner effects on such factors. To illustrate the model, we use data from 274 couples who provided daily ratings on their positive and negative emotions toward their relationship - up to 90 consecutive days. The model is fit using Hamiltonian Monte Carlo methods, and includes subsets of predictors in order to demonstrate the flexibility of this approach. We conclude with a discussion on the usefulness and the limitations of the MELSM for dyadic research.

Reliability is a fundamental problem for measurement in all of science. Al- though defined in multiple ways, and estimated in even more ways, the basic concepts are straight forward and need to be understood by methodologists as well as practioners. Reliability theory is not just for the psychometrician estimating latent variables, it is for everyone who wants to make inferences from measures of individuals or of groups. Easy to use, open source software is applied to examples of real data, and comparisons are made between the many types of reliability

The present study examined the dimensional structure underlying the Multidimensional Schizotypy Scale (MSS) and its brief version (MSS-B). The MSS and MSS-B were developed to assess current multidimensional conceptualizations of schizotypy. We used Exploratory Graph Analysis (EGA) to evaluate the dimensional structure of the scales in two large, independent samples (n = 6,265 and n = 1,000). We then used Confirmatory Factor Analysis (CFA) to compare the fit of the theoretical dimensions with the EGA dimensions. For the MSS, EGA identified four dimensions: positive schizotypy, two dimensions of negative schizotypy (affective and social anhedonia), and disorganized schizotypy. For the MSS-B, EGA identified three dimensions, which corresponded to the theorized positive, negative, and disorganized dimensions. Based on the MSS’s EGA dimensions, we also estimated a four-factor model for the MSS-B. The CFA comparison found that the four-factor model fit significantly better than the theoretical three-factor model for both the MSS and MSS-B. In short, we propose that the four-factor model supports the theoretical model and offers a more nuanced interpretation of the negative schizotypy dimension. Our findings offer new implications for future research on the MSS and MSS-B dimensions that may provide differential associations with interview and questionnaire measures.

Individual differences in general speed lead to a positive correlation between the mean and standard deviation of mean latency. This “coarse” scaling effect causes the mean latency difference (MLD) to be spuriously correlated with general speed. Within individuals, the correlation between the mean and standard deviation of trial latencies leads contrasted distributions to increase their overlap as an MLD of fixed width is translated to the right. To address this “fine” scaling effect, contrasts based on within subject latency transformations including the logarithm, standardization, and ranking were evaluated and turned out to be distinctly superior to the MLD. Notably, the mean gaussian rank latency difference was internally consistent, eliminated fine scaling, meliorated coarse scaling, reduced correlations with general speed, increased statistical power to detect within subject and between group effects, and has the potential to increase the validity of inferences drawn from response latency data.

Cross-validation (CV) is increasingly popular as a generic method to adjudicate between mathematical models of cognition and behavior. In order to measure model generalizability, CV quantifies out-of-sample predictive performance, and the CV preference goes to the model that predicted the out-of-sample data best. The advantages of CV include theoretic simplicity and practical feasibility. Despite its prominence, however, the limitations of CV are often underappreciated. Here we demonstrate the limitations of a particular form of CV --Bayesian leave-one-out cross-validation or LOO-- with three concrete examples. In each example, a data set of infinite size is perfectly in line with the predictions of a simple model (i.e., a general law or invariance). Nevertheless, LOO shows bounded and relatively modest support for the simple model. We conclude that CV is not a panacea for model selection.

Creativity assessment with open-ended production tasks relies heavily on scoring the quality of a subject’s ideas. This creates a faceted measurement structure involving persons, tasks (and ideas within tasks), and raters. Most studies, however, do not model possible systematic differences among raters. The present study examines individual rater differences in the context of a planned-missing design and its association with reliability and validity of creativity assessments. It applies the many-facet Rasch model (MFRM) to model and correct for these differences. We reanalyzed data from 2 studies (Ns = 132 and 298) where subjects produced metaphors, alternate uses for common objects, and creative instances. Each idea was scored by several raters. We simulated several conditions of reduced load on raters where they scored subsets of responses. We then compared the reliability and validity of IRT estimated scores (original vs. IRT adjusted scores) on various conditions of missing data. Results show that (a) raters vary substantially on the lenient-severity dimension, so rater differences should be modeled; (b) when different combinations of raters assess different subsets of ideas, systematic rater differences confound subjects’ scores, increasing measurement error and lowering criterion validity with external variables; and (c) MFRM adjustments effectively correct for rater effects, thus increasing correlations of scores obtained from partial with scores obtained with full data. We conclude that MFRM is a powerful means to model rater differences and reduce rater load in creativity research.

Mediation analysis plays a central role in the investigation of causal mechanisms, where the total, direct and indirect influence of one variable on another is typically assessed using path-tracing rule}. Two recent developments promise to revolutionize mediation analysis: first, the interventionist causal inference framework has allowed researchers to define and investigate mediation in terms of interventions on variables rather than path-tracing rules; second, researchers have begun to use Continuous-Time (CT) models to investigate mediation in longitudinal data. This paper shows how CT mediation can be treated using an interventionist approach. The links between interventionist and path-tracing definitions in traditional longitudinal SEM models and in CT models are explored in detail. The treatment of mediation from diverse strands of literature is unified, by showing how total, direct and indirect effects in different frameworks can be defined as qualitatively different types of interventions on the values of variables.

Stress is considered to be among the leading causes of various psychopathological problems, whose outcomes are often difficult to foresee. In order to prevent consequent maladaptive behaviors, the present paper introduces a novel, three-phase architecture for analyzing the formation, handling and manifestation of stress in individuals. We specifically model the process of transforming stressful stimuli to cognitive schemas, and from these to subconscious defense mechanism manifestations. Such three-phase model has been implemented and validated via the PERSONA-80 test. Psychometric properties of PERSONA-80 have been evaluated on a sample of 83 individuals from general and clinical populations. The test showed good internal consistency and excellent predictive validity results, from which we derived the purified scale PERSONA-16. The potential of the developed instrument is the detection of the inability to produce mature defense mechanisms from an arbitrary state of mind, or the inability to preserve schemas when exposed to stressful conditions, useful in high-stress occupation screening and monitoring. To encourage widespread use and validation in different domains, we provide at the end of this text the complete PERSONA-16 instrument.

Psychological constructs, such as emotions, thoughts, and attitudes are often measured by asking individuals to reply to questions using closed-ended numerical rating scales. However, when asking people about their state of mind in a natural context ("How are you?"), we receive open-ended answers using words ("Fine and happy!") and not closed-ended answers using numbers ("7") or categories ("A lot"). Nevertheless, to date it has been difficult to objectively quantify responses to open-ended questions. We develop an approach using open-ended questions in which the responses are analyzed using natural language processing (Latent Semantic Analyses). This approach of using open-ended, semantic questions is compared with traditional rating scales in nine studies (N = 92-854), including two different study paradigms. The first paradigm requires participants to describe psychological aspects of external stimuli (facial expressions) and the second paradigm involves asking participants to report their subjective well-being and mental health problems. The results demonstrate that the approach using semantic questions yields good statistical properties with competitive, or higher, validity and reliability compared with corresponding numerical rating scales. As these semantic measures are based on natural language and measure, differentiate, and describe psychological constructs, they have the potential of complementing and extending traditional rating scales. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Paper details and compares 6 different modifications to the k-means algorithm on simulated data and applies disparate algorithms to a college risk behavior dataset to understand risk groups among college populations. Study was MS thesis at University of Miami, 2014.

We investigated the latent structure of narcissistic personality disorder by comparing dimensional, hybrid, and categorical latent variable models, using confirmatory factor analysis (CFA), nonparametric and semiparametric factor analysis, and latent class analysis, respectively. We first explored these models in a clinical sample and then preregistered replication analyses in 4 additional data sets (with national, undergraduate, community, and mixed community/clinical samples) to test whether the best-fitting model would generalize across different data sets with different sample compositions. A 1-factor CFA outperformed categorical models in fit and reliability, suggesting the criteria do not serve to distinguish a narcissist class or subtypes; rather, a narcissistic dimension underlies the narcissistic personality disorder construct. The CFA also outperformed hybrid models, indicating that people fall within the same continuous distribution, rather than composing homogenous groups of relative severity (nonparametric factor analysis) or pulling apart into mixtures of discrete distributions (semiparametric factor analysis) along that spectrum. (PsycINFO Database Record

Psychologists wrestle with how to best handle multiple comparisons, while maintaining a balance between false positives and false negatives. Undercorrection, such as ignoring the presence of multiple comparisons altogether, is known to yield an unacceptably high rate of false positives. Overcorrection, such as treating all tests as independent when they are not, results in overly conservative evaluations of statistical significance. This tutorial demonstrates $M_{eff}$ correction, a method for adjusting statistical significance thresholds for multiple comparisons, without the assumption of independence of tests. This method, in which the effective number of tests ($M_{eff}$) is estimated from the correlations among the variables being tested, was developed and validated in the field of genetics, but is based on statistical concepts (eigenvalues) that are very familiar to psychologists. $M_{eff}$ correction can be applied in psychological research to balance the necessity of correction for multiple comparisons with the concerns that arise from complex, correlated tests.

An important goal for psychological science is developing methods to characterize relationships between variables. The customary approach uses structural equation models to connect latent factors to a number of observed measurements. More recently, regularized partial correlation networks have been proposed as an alternative approach for characterizing relationships among variables through covariances in the precision matrix. While the graphical lasso (glasso) method has merged as the default network estimation method, it was optimized in fields outside of psychology with very different needs, such as high dimensional data where the number of variables (p) exceeds the number of observations (n). In this paper, we describe the glasso method in the context of the fields where it was developed, and then we demonstrate that the advantages of regularization diminish in settings where psychological networks are often fitted (p ≪ n). We first show that improved properties of the precision matrix, such as eigenvalue estimation, and predictive accuracy with cross-validation are not always appreciable. We then introduce non-regularized methods based on multiple regression, after which we characterize performance with extensive simulations. Our results demonstrate that the non-regularized methods consistently outperform glasso with respect to limiting false positives, and they provide more consistent performance across sparsity levels, sample composition (p=n), and partial correlation size. We end by reviewing recent findings in the statistics literature that suggest alternative methods often have superior than glasso, as well as suggesting areas for future research in psychology.

*THIS PAPER HAS NOT YET BEEN PEER REVIEWED* Listening to music is a strategy many people use to regulate their emotions, especially sadness. However, there is disagreement about whether listening to music is a healthy way to regulate emotions, with some research finding that sad music worsens a sad state, especially for people high in rumination. To further explore the immediate consequences of music listening when sad 128 young adults (41% male, aged 18 to 25 years) were induced into a sad emotional state prior to random assignment to listening of either self-selected music, experimenter-selected sad music, or no music. Results revealed that listening to either self-selectedor experimenter-selected music led to a decrease in sadness. No difference was found between groups at post-listening. However, participants who listened to self-selected music reported a return to baseline levels of sadness, while this did not occur for participants who listened to experimenter-selected or were in the no music control. Rumination was also measured but did not moderate the impact of music listening on sadness for either musiccondition. Furthermore, there was no impact of rumination on participants’ perceptions of sadness in music. These results support the notion that listening to sad music does not worsen a sad state—even for those high in rumination—although it does appear to slow the emotion regulation process in cases where sad music is not self-selected.

As the debate over best statistical practices continues in academic journals, conferences, and the blogosphere, working researchers (e.g., psychologists) need to figure out how much time and effort to invest in attending to experts’ arguments, how to design their next project, and how to craft a sustainable long-term strategy for data analysis and inference. The present special issue of the American Statistician promises help. In this article, we offer a modest proposal for a continued and informed use of the conventional p value without the pitfalls of statistical rituals. Other statistical indices should complement reporting, and extra-statistical (e.g., theoretical) judgments ought to be made with care and clarity.

Network theory, as a theoretical and methodological framework, is energizing many research fields, among which clinical psychology and psychiatry. Fundamental to the network theory of psychopathology is the role of specific symptoms and their interactions. Current statistical tools, however, fail to fully capture this constitutional property. We propose community detection tools as a means to evaluate the complex network structure of psychopathology, free from its original boundaries of distinct disorders. Unique to this approach is that symptoms can belong to multiple communities. Using a large community sample and spanning a broad range of symptoms (Symptom Checklist-90-Revised), we identified 18 communities of interconnected symptoms. The differential role of symptoms within and between communities offers a framework to study the clinical concepts of comorbidity, heterogeneity and hallmark symptoms. Symptoms with many and strong connections within a community, defined as stabilizing symptoms, could be thought of as the core of a community, whereas symptoms that belong to multiple communities, defined as communicating symptoms, facilitate the communication between problem areas. We propose that defining symptoms on their stabilizing and/or communicating role within and across communities accelerates our understanding of these clinical phenomena, central to research and treatment of psychopathology.

The internet-based assessment of response time (RT) and error rate (ERR) has recently become a well-validated alternative to traditional laboratory-based assessment, because methodological research has provided evidence for negligible setting- and setup-related differences in RT and ERR measures of central tendency. However, corresponding data on potential differences in the variability of such performance measures are still lacking, to date. Hence, the aim of this study was to conduct internet-based mental chronometry in both poorly standardized domestic and highly standardized laboratory environments and to compare the variabilities of the corresponding performance measures. Using the Millisecond Inquisit4Web software, 127 men and women completed three different RT-based cognitive paradigms (i.e., go/no-go, two-back, and number-letter). Each participant completed all paradigms in two environments (i.e., at home and in the laboratory), with a time lag of seven days and in a counterbalanced order. Mixed-effects modeling was employed to estimate the between-setting variability across a comprehensive set of performance measures, including conventional measures of central tendency (i.e., mean RT and ERR) and further measures characterizing the joint distribution of RT/ERR. The latter measures were estimated using the diffusion model. The results suggested negligible differences between the domestic and laboratory settings. Thus, this study provides novel evidence suggesting that the statistical power of internet-based mental chronometry is commonly not compromised by increased environmental variance. The within- and between-session reliabilities were in a satisfactory range-that is, comparable to performance measures collected offline in laboratory settings. In consequence, our results support the broad applicability, robustness, and cost efficiency of mental chronometry assessment using the internet.

Hypothesis testing is a special form of model selection. Once a pair of competing models is fully defined, their definition immediately leads to a measure of how strongly each model supports the data. The ratio of their support is often called the likelihood ratio or the Bayes factor. Critical in the model selection endeavor is the specification of the models. In the case of hypothesis testing, it is of the greatest importance that we specify exactly what is meant by a "null" hypothesis as well as the alternative to which it is contrasted, and that these are suitable instantiations of theoretical positions. Here, we provide an overview of different instantiations of null and alternative hypotheses that can be useful in practice, while the underlying method of likelihood comparison is universal and identical in all cases. An associated app can be found via https://osf.io/mvp53/.

Research at the individual level shows strong positive relationships between performance in video games and on intelligence tests. Together with evidence of above average IQs of players of traditional mental sports such as chess, this suggests that national IQs should be strongly related to national performance on mental sports. To investigate this, lists of top players for 12 different electronic sports (e-sports) and traditional mental sports were collected from a variety of sources (total n = 36k). Using a log count approach to control for population size, national cognitive ability/IQ was found to be a predictor (p

The words in children’s language learning environments are strongly predictive of cognitive development and school achievement. But how do we measure language environments and do so at the scale of the many words that children hear day-in and day-out? The quantity and quality of words in a child’s input is typically measured in terms of total amount of talk and the lexical diversity in that talk. There are disagreements in the literature whether amount or diversity is the more critical measure of the input. Here we analyze the properties of a large corpus (6.5 million words) of speech to children and simulate learning environments that differ in amount of talk per unit time, lexical diversity, and the contexts of talk. The central conclusion is that what researchers need to theoretically understand, measure, and change is not the total amount of words, or the diversity of words, but the function that relates total words to the diversity of words, and how that function changes across different contexts of talk.

This report describes two preliminary simulation studies that were included in earlierrevisions of Epskamp, Waldorp, Mõttus, and Borsboom (2017). These will now be includedin a future manuscript.

Cognitive measurement models decompose observed behavior into latent cognitive processes. For situations with more than one condition, such models allow to test hypotheses on the level of the latent processes. We propose a fully Bayesian ensemble model approach to test hypotheses on the level of the latent processes in situations in which multiple measurement models or model classes exist. In the first step, one needs to perform a Bayesian model selection step comparing the hypotheses within each model class. Aggregating the results of the first step yields ensemble posterior model probabilities. We provide an example for a working memory data set using an ensemble of a resource model and a slots model.

The debate between Bayesians and frequentist statisticians has been going on for decades. Whilst there are fundamental theoretical and philosophical differences between both schools of thought, we argue that in two most common situations the practical differences are negligible when off-the-shelf Bayesian analysis (i.e., using ‘objective’ priors) is used. We emphasize this reasoning by focusing on interval estimates: confidence intervals and credible intervals. We show that this is the case for the most common empirical situations in the social sciences, the estimation of a proportion of a binomial distribution and the estimation of the mean of a unimodal distribution. Numerical differences between both approaches are small, sometimes even smaller than those between two competing frequentist or two competing Bayesian approaches. We outline the ramifications of this for scientific practice.

Zwaan et al. and others discuss the importance of the inevitable differences between a replication experiment and the corresponding original experiment. But these discussions are not informed by a principled, quantitative framework for taking differences into account. Bayesian confirmation theory provides such a framework. It will not entirely solve the problem, but it will lead to new insights.

Borderline personality disorder (BPD) is characterized by impulsivity, emotion dysregulation, disturbed relationships, and identity disturbances. Despite the known variable co-occurrence of BPD symptoms, the possible causal relationships are not well understood. We addressed this by creating a hierarchical network model of BPD, which identifies the most likely acyclic causal pathways that are driving BPD development. Cross-sectional data was obtained from the Structured Clinical Interview-II (SCID-II), and possible causal relationships between symptoms were identified from conditional independence relations. The symptoms’ hierarchy values, assessing their role in causal pathways, was determined by a random walk-based algorithm. By analyzing the directed network of BPD symptoms, it was found that symptoms in initial stages of causal pathways were abandonment, physical fights, impulsivity, suicidal threats, identity disturbances, and affective instability. Based on the assessed role symptoms play in causal pathways of BPD development, specific symptoms can be targeted during early diagnosis and clinical assessment.

Investigating the bases of inter-individual differences in risk-taking is necessary to refine our cognitive and neural models of decision-making and to ultimately counter risky behaviours in real-life policy settings. However, recent evidence suggests that behavioural tasks fare poorly compared to standard questionnaires to measure individual differences in risk-taking. Crucially, the results were not improved using model-based measures of risk taking. Here we propose two possible - not mutually exclusive - explanations for these results. Finally, we suggest future avenues of research to improve the assessment of inter-individual differences in risk-taking that combine repeated online testing and the development of mechanistic computational models.

We have tested the feasibility of a method to prevent the occurrence of so-called Questionable Research Practices (QRP). A part from embedded pre-registration the major aspect of the system is real-time uploading of data on a secure server. We outline the method, discuss the drop-out treatment and compare it to the Born-open data method, and report on our preliminary experiences. We also discuss the extension of the data-integrity system from secure server to use of blockchain technology.

A recent preprint by Brown and Coyne titled, "No Evidence That Twitter Language Reliably Predicts Heart Disease: A Reanalysis of Eichstaedt et al." asserts to re-analyze our 2015 article published in Psychological Science, “Twitter Language Predicts Heart Disease Mortality”, disputing its primary findings. While we welcome scrutiny of the study, Brown and Coyne’s paper does not in fact report on a reanalysis, but rather presents a new analysis relating Twitter language to suicide instead of heart disease mortality. In our original article, we showed that Twitter language, fed into standard machine learning algorithms, was able to predict (i.e., estimate cross-sectionally) the out-of-sample heart disease rates of U.S. counties. Further, in a separate analysis, we found that the dictionaries and topics (i.e., sets of related words) which best predicted county atherosclerotic heart disease mortality rates included language related to education and income (e.g., “management,” “ideas,” “conference”) as well as negative social relationships (“hate”, “alone,” “jealous”), disengagement (“tired, “bored,” “sleepy”), negative emotions (“sorry,” “mad,” “sad”) as well as positive emotions (“great,” “happy,” “cool”) and psychological engagement (“learn,” “interesting,” “awake”). Beyond conducting a new analysis (correlating Twitter language with suicide rates), Brown and Coyne also detail a number of methodological limitations of group-level and social media-based studies. We discussed most of these limitations in our original article, but welcome this opportunity to emphasize some of the key aspects and qualifiers of our findings, considering each of their critiques and how they relate to our findings. Of particular note, even though we discuss our findings in the context of what is known about the etiology of heart disease at the individual level, we reiterate here a point made in our original paper: that individual-level causal inferences cannot be made from the cross-sectional and group-level analyses we presented. Our findings are intended to provide a new epidemiological tool to take advantage of large amounts of public data, and to complement, not replace, definitive health data collected through other means.We offer preliminary comments on the suicide language correlations: Previous studies have suggested that county-level suicides are relatively strongly associated with living in rural areas (Hirsch et al., 2006; Searles et al., 2014) and with county elevation (Kim et al., 2011; Brenner et al., 2011). When we control for these two confounds, we find the dictionary associations reported by Brown and Coyne are no longer significant. We conclude that their analysis is largely unrelated to our study and does not invalidate the findings of our original paper. In addition, we offer a replication of our original findings across more years, with a larger Twitter data set. We find that (a) Twitter language still predicts county atherosclerotic heart disease mortality with the same accuracy, and (b) the specific dictionary correlations we reported are largely unchanged on the new data set. To facilitate the reproduction by other researchers of our original work, we also re-release the data and code with which to reproduce our original findings, making it more user-friendly. We will do the same for this replication upon publication.

In the psychometric literature concerning spatial navigation, several questionnaires had been designed to assess the strategies that different individuals engage in when they encode and retrieve environmental knowledge from the first-person (egocentric) and third-person (allocentric) perspectives. Focusing on 12 survey-based strategy items of the Navigation Strategy Questionnaire (Zhong, 2013; Zhong & Kozhevnikov, 2016), this paper applies item response theory-based analysis to determine: (a) whether an item with a low rotated component loading of .27 (SURVEY12) can be considered for removal in future surveys (When I reconstruct my mental map, its environmental orientation is fixed and does not change with my imagined heading directions), and (b) whether a bifactor model can better describe the latent structure of the survey-based strategy. Results from the item level fit diagnostics, categorical response and item information curves showed that SURVEY12 can be indeed removed in future surveys, and that a bifactor model with three items assessing the selective preference for allocentric reference frame constituting a content factor offered a better model fit compared to a unidimensional model. Total score from these three items also correlated significantly with a pointing-to-landmarks task to the same relative magnitude as the total scores from all items, and all items excluding SURVEY12. These findings suggest that the three preference-related items can constitute a subscale for predicting spatial orientation performance in future studies. Moreoverr, it is also recommended that future studies employ spatial navigation tasks that engage survey-based strategy primarily, and design more items concerning the impact of environmental familiarity on survey-based strategy use.

"We propose to change the default P-value threshold forstatistical significance for claims of new discoveries from 0.05 to 0.005."

The shifted-Wald model is a popular analysis tool for one-choice reaction-time tasks. In its simplest version, the shifted-Wald model assumes a constant trial-independent drift rate parameter. However, the presence of endogenous processes—fluctuation in attention and motivation, fatigue and boredom—suggest that drift rate might vary across experimental trials. Here we show how across-trial variability in drift rate can be accounted for by assuming a trial-specific drift rate parameter that is governed by a positive-valued distribution. We consider two candidate distributions: the truncated normal distribution and the gamma distribution. For the resulting distributions of first-arrival times, we derive analytical and sampling-based solutions, and implement the models in a Bayesian framework. Recovery studies and an application to a data set comprised of 1469 participants suggest that (1) both mixture distributions yield similar results; (2) all model parameters can be recovered accurately except for the drift variance parameter; (3) despite poor recovery, the presence of the drift variance parameter facilitates accurate recovery of the remaining parameters; (4) shift, threshold, and drift mean parameters are correlated.

Definitions of the term “empathy” have long been discussed. Recent evidence suggests that “empathy” and “concern” (also known as “compassion”) are psychologically and physiologically distinct. These distinctions directly affect the psychometric measures used to assess empathy. A new measure called the Empathy Index (EI) has been proposed that assesses empathy strictly in terms of emotional contagion. However, this measure has not been validated using Brazilian Portuguese language. Our objective was therefore to translate, culturally adapt, and examine the psychometric properties of the EI in a Brazilian population. We also examined the EI’s relationship with the Interpersonal Reactivity Index (IRI). In Study 1 (N=483), we explored the relationship between EI and IRI through factor analysis. Empathy and concern loaded on different factors. The EI subscales showed adequate Cronbrach’s alphas and concern, but not empathy, predicted contributions during an economic game. In Study 2 (N=215), confirmatory factor analyses confirmed the factorial structure of EI and its relation with IRI. Therefore, we consider that the Brazilian version of the EI showed good evidence of both validity and reliability.

In this paper, we discuss the benefits of and how to utilize Bayesian statistics in studies of moral education. To demonstrate concrete examples of the applications of Bayesian statistics to studies of moral education, we reanalyzed two datasets previously collected: one small dataset collected from a moral educational intervention experiment, and one big dataset from a large-scale Defining Issues Test-2 survey. Results suggest that Bayesian analysis of datasets collected from moral educational studies can provide additional useful statistical information, particularly that associated with the strength of evidence supporting alternative hypotheses, which has not been provided by the classical frequentist approach focusing on P-values. Finally, we introduce several practical guidelines pertaining to how to utilize Bayesian statistics, including the utilization of newly developed free statistical software, JASP, and thresholding based on Bayes Factors, to scholars in the field of moral education.

The creation of abridged versions of research tools is a common, justifiable process, but unfortunately it is often carried out without due methodological care and regard for the consequences. Smith and collaborators (2000) have already written about the mistakes that can be made, but their article has not had much practical impact. There are two main mistakes commonly made by researchers: assuming the transferability of validity and reliability between the full and shortened versions and using less stringent criteria to assess the validity and reliability of short forms. These two problems manifest as nine sins committed during the construction of short forms. Here we present procedures designed to allow researchers to avoid these mistakes and create abridged versions of research tools that are as reliable as possible and to assess the costs of the various methods of abridging questionnaires. To this end we determine the expected length of the tool and weight the benefits of reduced questionnaire completion time against the loss of reliability. We also estimate the shared variance of the full and short versions and classification accuracy of the new, short version. We compared quality of short form obtained from three most common statistical techniques for abridging questionnaires. We analysed data from a sample of 519 persons; 309 (59.5%) completed the paper version of the Self-Narrative Inclination Questionnaire (IAN-R), and 210 (40.5%) participated in online tests. Abridgements based on factor loadings and Cronbach’s # were similarly effective and these two methods had a slight advantage over a method based on item response theory-based analyses of difficulty and discriminatory power.

In determining the need to directly replicate, it is crucial to first verify the original results through independent reanalysis of the data. Original results that appear erroneous and that cannot be reproduced by reanalysis offer little evidence to begin with, thereby diminishing the need to replicate. Sharing data and scripts is essential to ensure reproducibility.