Search

The current chapter briefly reviews the history of meta-analysis before turning to an assessment of its strengths and weaknesses, as well as considerations of alternatives to and modifications of traditional meta-analysis. It closes with recommendations to make meta-analyses more informative.

A new system for subjective rating of responses to divergent thinking tasks was tested using raters recruited from Amazon Mechanical Turk. The rationale for the study was to determine if such raters could provide reliable (aka generalizable) ratings from the perspective of generalizability theory. To promote reliability across the Alternative Uses and Consequence task prompts often used by researchers as measures of Divergent Thinking, two parallel scales were developed to facilitate feasibility and validity of ratings performed by laypeople. Generalizability and dependability studies were conducted separately for two scoring systems: the average-rating system and the snapshot system. Results showed that it is difficult to achieve adequate reliability using the snapshot system, while good reliability can be achieved on both task families using the average-rating system and a specific number of items and raters. Additionally, the construct validity of the average-rating system is generally good, with less validity for certain Consequences items. Recommendations for researchers wishing to adopt the new scales are discussed, along with broader issues of generalizability of subjective creativity ratings.

Using complete enumeration (e.g., generating all possible subsets of item combinations) to evaluate clustering problems has the benefit of locating globally optimal solutions automatically without the concern of sampling variability. The proposed method is meant to combine clustering variables in such a way as to create groups that are maximally different on a theoretically sound derivation variable(s). After the population of all unique sets is permuted, optimization on some predefined, user-specific function can occur. We apply this technique to optimizing the diagnosis of Alcohol Use Disorder. This is a unique application, from a clustering point of view, in that the decision rule for clustering observations into the diagnosis group relies on both the set of items being considered and a predefined threshold on the number of items required to be endorsed for the diagnosis to occur. In optimizing diagnostic rules, criteria set sizes can be reduced without a loss of significant information when compared to current and proposed, alternative, diagnostic schemes.

Reliability is a fundamental problem for measurement in all of science. Al- though defined in multiple ways, and estimated in even more ways, the basic concepts are straight forward and need to be understood by methodologists as well as practioners. Reliability theory is not just for the psychometrician estimating latent variables, it is for everyone who wants to make inferences from measures of individuals or of groups. Easy to use, open source software is applied to examples of real data, and comparisons are made between the many types of reliability

The present study examined the dimensional structure underlying the Multidimensional Schizotypy Scale (MSS) and its brief version (MSS-B). The MSS and MSS-B were developed to assess current multidimensional conceptualizations of schizotypy. We used Exploratory Graph Analysis (EGA) to evaluate the dimensional structure of the scales in two large, independent samples (n = 6,265 and n = 1,000). We then used Confirmatory Factor Analysis (CFA) to compare the fit of the theoretical dimensions with the EGA dimensions. For the MSS, EGA identified four dimensions: positive schizotypy, two dimensions of negative schizotypy (affective and social anhedonia), and disorganized schizotypy. For the MSS-B, EGA identified three dimensions, which corresponded to the theorized positive, negative, and disorganized dimensions. Based on the MSS’s EGA dimensions, we also estimated a four-factor model for the MSS-B. The CFA comparison found that the four-factor model fit significantly better than the theoretical three-factor model for both the MSS and MSS-B. In short, we propose that the four-factor model supports the theoretical model and offers a more nuanced interpretation of the negative schizotypy dimension. Our findings offer new implications for future research on the MSS and MSS-B dimensions that may provide differential associations with interview and questionnaire measures.

Individual differences in general speed lead to a positive correlation between the mean and standard deviation of mean latency. This “coarse” scaling effect causes the mean latency difference (MLD) to be spuriously correlated with general speed. Within individuals, the correlation between the mean and standard deviation of trial latencies leads contrasted distributions to increase their overlap as an MLD of fixed width is translated to the right. To address this “fine” scaling effect, contrasts based on within subject latency transformations including the logarithm, standardization, and ranking were evaluated and turned out to be distinctly superior to the MLD. Notably, the mean gaussian rank latency difference was internally consistent, eliminated fine scaling, meliorated coarse scaling, reduced correlations with general speed, increased statistical power to detect within subject and between group effects, and has the potential to increase the validity of inferences drawn from response latency data.

Cross-validation (CV) is increasingly popular as a generic method to adjudicate between mathematical models of cognition and behavior. In order to measure model generalizability, CV quantifies out-of-sample predictive performance, and the CV preference goes to the model that predicted the out-of-sample data best. The advantages of CV include theoretic simplicity and practical feasibility. Despite its prominence, however, the limitations of CV are often underappreciated. Here we demonstrate the limitations of a particular form of CV --Bayesian leave-one-out cross-validation or LOO-- with three concrete examples. In each example, a data set of infinite size is perfectly in line with the predictions of a simple model (i.e., a general law or invariance). Nevertheless, LOO shows bounded and relatively modest support for the simple model. We conclude that CV is not a panacea for model selection.

Creativity assessment with open-ended production tasks relies heavily on scoring the quality of a subject’s ideas. This creates a faceted measurement structure involving persons, tasks (and ideas within tasks), and raters. Most studies, however, do not model possible systematic differences among raters. The present study examines individual rater differences in the context of a planned-missing design and its association with reliability and validity of creativity assessments. It applies the many-facet Rasch model (MFRM) to model and correct for these differences. We reanalyzed data from 2 studies (Ns = 132 and 298) where subjects produced metaphors, alternate uses for common objects, and creative instances. Each idea was scored by several raters. We simulated several conditions of reduced load on raters where they scored subsets of responses. We then compared the reliability and validity of IRT estimated scores (original vs. IRT adjusted scores) on various conditions of missing data. Results show that (a) raters vary substantially on the lenient-severity dimension, so rater differences should be modeled; (b) when different combinations of raters assess different subsets of ideas, systematic rater differences confound subjects’ scores, increasing measurement error and lowering criterion validity with external variables; and (c) MFRM adjustments effectively correct for rater effects, thus increasing correlations of scores obtained from partial with scores obtained with full data. We conclude that MFRM is a powerful means to model rater differences and reduce rater load in creativity research.

Multiple-choice and continuous-response tasks pose a number of challenges for models of the decision process, from empirical challenges like context effects to computational demands imposed by choice sets with a large number of outcomes. This paper develops a general framework for constructing models of the cognitive processes underlying both inferential and preferential choice among any arbitrarily large number of alternatives. This geometric approach represents the alternatives in a choice set along with a decision maker's beliefs or preferences in a ``decision space,'' simultaneously capturing the support for different alternatives along with the similarity relations between the alternatives in the choice set. Support for the alternatives (represented as vectors) shifts over time according to the dynamics of the belief / preference state (represented as a point) until a stopping rule is met (state crosses a hyperplane) and the corresponding selection is made. I review stopping rules that guarantee optimality in multi-alternative inferential choice, minimizing response time for a desired level of accuracy, as well as methods for constructing the decision space, which can result in context effects when applied to preferential choice.

Mediation analysis plays a central role in the investigation of causal mechanisms, where the total, direct and indirect influence of one variable on another is typically assessed using path-tracing rule}. Two recent developments promise to revolutionize mediation analysis: first, the interventionist causal inference framework has allowed researchers to define and investigate mediation in terms of interventions on variables rather than path-tracing rules; second, researchers have begun to use Continuous-Time (CT) models to investigate mediation in longitudinal data. This paper shows how CT mediation can be treated using an interventionist approach. The links between interventionist and path-tracing definitions in traditional longitudinal SEM models and in CT models are explored in detail. The treatment of mediation from diverse strands of literature is unified, by showing how total, direct and indirect effects in different frameworks can be defined as qualitatively different types of interventions on the values of variables.

Stress is considered to be among the leading causes of various psychopathological problems, whose outcomes are often difficult to foresee. In order to prevent consequent maladaptive behaviors, the present paper introduces a novel, three-phase architecture for analyzing the formation, handling and manifestation of stress in individuals. We specifically model the process of transforming stressful stimuli to cognitive schemas, and from these to subconscious defense mechanism manifestations. Such three-phase model has been implemented and validated via the PERSONA-80 test. Psychometric properties of PERSONA-80 have been evaluated on a sample of 83 individuals from general and clinical populations. The test showed good internal consistency and excellent predictive validity results, from which we derived the purified scale PERSONA-16. The potential of the developed instrument is the detection of the inability to produce mature defense mechanisms from an arbitrary state of mind, or the inability to preserve schemas when exposed to stressful conditions, useful in high-stress occupation screening and monitoring. To encourage widespread use and validation in different domains, we provide at the end of this text the complete PERSONA-16 instrument.

Psychological constructs, such as emotions, thoughts, and attitudes are often measured by asking individuals to reply to questions using closed-ended numerical rating scales. However, when asking people about their state of mind in a natural context ("How are you?"), we receive open-ended answers using words ("Fine and happy!") and not closed-ended answers using numbers ("7") or categories ("A lot"). Nevertheless, to date it has been difficult to objectively quantify responses to open-ended questions. We develop an approach using open-ended questions in which the responses are analyzed using natural language processing (Latent Semantic Analyses). This approach of using open-ended, semantic questions is compared with traditional rating scales in nine studies (N = 92-854), including two different study paradigms. The first paradigm requires participants to describe psychological aspects of external stimuli (facial expressions) and the second paradigm involves asking participants to report their subjective well-being and mental health problems. The results demonstrate that the approach using semantic questions yields good statistical properties with competitive, or higher, validity and reliability compared with corresponding numerical rating scales. As these semantic measures are based on natural language and measure, differentiate, and describe psychological constructs, they have the potential of complementing and extending traditional rating scales. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Paper details and compares 6 different modifications to the k-means algorithm on simulated data and applies disparate algorithms to a college risk behavior dataset to understand risk groups among college populations. Study was MS thesis at University of Miami, 2014.

We investigated the latent structure of narcissistic personality disorder by comparing dimensional, hybrid, and categorical latent variable models, using confirmatory factor analysis (CFA), nonparametric and semiparametric factor analysis, and latent class analysis, respectively. We first explored these models in a clinical sample and then preregistered replication analyses in 4 additional data sets (with national, undergraduate, community, and mixed community/clinical samples) to test whether the best-fitting model would generalize across different data sets with different sample compositions. A 1-factor CFA outperformed categorical models in fit and reliability, suggesting the criteria do not serve to distinguish a narcissist class or subtypes; rather, a narcissistic dimension underlies the narcissistic personality disorder construct. The CFA also outperformed hybrid models, indicating that people fall within the same continuous distribution, rather than composing homogenous groups of relative severity (nonparametric factor analysis) or pulling apart into mixtures of discrete distributions (semiparametric factor analysis) along that spectrum. (PsycINFO Database Record

Psychologists wrestle with how to best handle multiple comparisons, while maintaining a balance between false positives and false negatives. Undercorrection, such as ignoring the presence of multiple comparisons altogether, is known to yield an unacceptably high rate of false positives. Overcorrection, such as treating all tests as independent when they are not, results in overly conservative evaluations of statistical significance. This tutorial demonstrates $M_{eff}$ correction, a method for adjusting statistical significance thresholds for multiple comparisons, without the assumption of independence of tests. This method, in which the effective number of tests ($M_{eff}$) is estimated from the correlations among the variables being tested, was developed and validated in the field of genetics, but is based on statistical concepts (eigenvalues) that are very familiar to psychologists. $M_{eff}$ correction can be applied in psychological research to balance the necessity of correction for multiple comparisons with the concerns that arise from complex, correlated tests.

An important goal for psychological science is developing methods to characterize relationships between variables. The customary approach uses structural equation models to connect latent factors to a number of observed measurements. More recently, regularized partial correlation networks have been proposed as an alternative approach for characterizing relationships among variables through covariances in the precision matrix. While the graphical lasso (glasso) method has merged as the default network estimation method, it was optimized in fields outside of psychology with very different needs, such as high dimensional data where the number of variables (p) exceeds the number of observations (n). In this paper, we describe the glasso method in the context of the fields where it was developed, and then we demonstrate that the advantages of regularization diminish in settings where psychological networks are often fitted (p ≪ n). We first show that improved properties of the precision matrix, such as eigenvalue estimation, and predictive accuracy with cross-validation are not always appreciable. We then introduce non-regularized methods based on multiple regression, after which we characterize performance with extensive simulations. Our results demonstrate that the non-regularized methods consistently outperform glasso with respect to limiting false positives, and they provide more consistent performance across sparsity levels, sample composition (p=n), and partial correlation size. We end by reviewing recent findings in the statistics literature that suggest alternative methods often have superior than glasso, as well as suggesting areas for future research in psychology.

*THIS PAPER HAS NOT YET BEEN PEER REVIEWED* Listening to music is a strategy many people use to regulate their emotions, especially sadness. However, there is disagreement about whether listening to music is a healthy way to regulate emotions, with some research finding that sad music worsens a sad state, especially for people high in rumination. To further explore the immediate consequences of music listening when sad 128 young adults (41% male, aged 18 to 25 years) were induced into a sad emotional state prior to random assignment to listening of either self-selected music, experimenter-selected sad music, or no music. Results revealed that listening to either self-selectedor experimenter-selected music led to a decrease in sadness. No difference was found between groups at post-listening. However, participants who listened to self-selected music reported a return to baseline levels of sadness, while this did not occur for participants who listened to experimenter-selected or were in the no music control. Rumination was also measured but did not moderate the impact of music listening on sadness for either musiccondition. Furthermore, there was no impact of rumination on participants’ perceptions of sadness in music. These results support the notion that listening to sad music does not worsen a sad state—even for those high in rumination—although it does appear to slow the emotion regulation process in cases where sad music is not self-selected.

Scholars have often characterized wisdom as a trait—a stable and invariable individual disposition. However, recent advances in behavioral sciences suggest that many features of wisdom can be viewed as situation-specific states—concrete instances of wise and not so wise behavior. We critically examine the evidence concerning the trait vs. state aspects of wisdom. First, we review the common individual qualities attributed to wisdom. The extent to which these qualities are seen as stable traits vs. variable states varies among both laypeople and scientists, and new evidence shows that pertinent folk theories about wisdom vary substantially across cultures as well. Next, we present empirical research on wisdom in everyday situations which demonstrates the systematic variability of wisdom-related characteristics as a function of situational demands and induced mindsets. To resolve the trait vs. state debate in wisdom, we argue for a whole trait theoretical framework, conceptualizing traits as a density distribution of states. Based on theoretical and empirical insights, we conclude by providing recommendations for best practices when measuring trait- and state-components of wisdom.

As the debate over best statistical practices continues in academic journals, conferences, and the blogosphere, working researchers (e.g., psychologists) need to figure out how much time and effort to invest in attending to experts’ arguments, how to design their next project, and how to craft a sustainable long-term strategy for data analysis and inference. The present special issue of the American Statistician promises help. In this article, we offer a modest proposal for a continued and informed use of the conventional p value without the pitfalls of statistical rituals. Other statistical indices should complement reporting, and extra-statistical (e.g., theoretical) judgments ought to be made with care and clarity.

Network theory, as a theoretical and methodological framework, is energizing many research fields, among which clinical psychology and psychiatry. Fundamental to the network theory of psychopathology is the role of specific symptoms and their interactions. Current statistical tools, however, fail to fully capture this constitutional property. We propose community detection tools as a means to evaluate the complex network structure of psychopathology, free from its original boundaries of distinct disorders. Unique to this approach is that symptoms can belong to multiple communities. Using a large community sample and spanning a broad range of symptoms (Symptom Checklist-90-Revised), we identified 18 communities of interconnected symptoms. The differential role of symptoms within and between communities offers a framework to study the clinical concepts of comorbidity, heterogeneity and hallmark symptoms. Symptoms with many and strong connections within a community, defined as stabilizing symptoms, could be thought of as the core of a community, whereas symptoms that belong to multiple communities, defined as communicating symptoms, facilitate the communication between problem areas. We propose that defining symptoms on their stabilizing and/or communicating role within and across communities accelerates our understanding of these clinical phenomena, central to research and treatment of psychopathology.

The internet-based assessment of response time (RT) and error rate (ERR) has recently become a well-validated alternative to traditional laboratory-based assessment, because methodological research has provided evidence for negligible setting- and setup-related differences in RT and ERR measures of central tendency. However, corresponding data on potential differences in the variability of such performance measures are still lacking, to date. Hence, the aim of this study was to conduct internet-based mental chronometry in both poorly standardized domestic and highly standardized laboratory environments and to compare the variabilities of the corresponding performance measures. Using the Millisecond Inquisit4Web software, 127 men and women completed three different RT-based cognitive paradigms (i.e., go/no-go, two-back, and number-letter). Each participant completed all paradigms in two environments (i.e., at home and in the laboratory), with a time lag of seven days and in a counterbalanced order. Mixed-effects modeling was employed to estimate the between-setting variability across a comprehensive set of performance measures, including conventional measures of central tendency (i.e., mean RT and ERR) and further measures characterizing the joint distribution of RT/ERR. The latter measures were estimated using the diffusion model. The results suggested negligible differences between the domestic and laboratory settings. Thus, this study provides novel evidence suggesting that the statistical power of internet-based mental chronometry is commonly not compromised by increased environmental variance. The within- and between-session reliabilities were in a satisfactory range-that is, comparable to performance measures collected offline in laboratory settings. In consequence, our results support the broad applicability, robustness, and cost efficiency of mental chronometry assessment using the internet.

© 2018 Elsevier Inc. For many years the Diffusion Decision Model (DDM) has successfully accounted for behavioral data from a wide range of domains. Important contributors to the DDM's success are the across-trial variability parameters, which allow the model to account for the various shapes of response time distributions encountered in practice. However, several researchers have pointed out that estimating the variability parameters can be a challenging task. Moreover, the numerous fitting methods for the DDM each come with their own associated problems and solutions. This often leaves users in a difficult position. In this collaborative project we invited researchers from the DDM community to apply their various fitting methods to simulated data and provide advice and expert guidance on estimating the DDM's across-trial variability parameters using these methods. Our study establishes a comprehensive reference resource and describes methods that can help to overcome the challenges associated with estimating the DDM's across-trial variability parameters. ispartof: JOURNAL OF MATHEMATICAL PSYCHOLOGY vol:87 pages:46-75 status: published

Hypothesis testing is a special form of model selection. Once a pair of competing models is fully defined, their definition immediately leads to a measure of how strongly each model supports the data. The ratio of their support is often called the likelihood ratio or the Bayes factor. Critical in the model selection endeavor is the specification of the models. In the case of hypothesis testing, it is of the greatest importance that we specify exactly what is meant by a "null" hypothesis as well as the alternative to which it is contrasted, and that these are suitable instantiations of theoretical positions. Here, we provide an overview of different instantiations of null and alternative hypotheses that can be useful in practice, while the underlying method of likelihood comparison is universal and identical in all cases. An associated app can be found via https://osf.io/mvp53/.

We estimated the predictive power of the dynamic items in the Finnish Risk and Needs Assessment Form (RITA), assessed by case-workers, for predicting recidivism. These 52 items were compared to static predictors including crime committed, prison history, and age. We used two machine learning methods (elastic net and random forest) for this purpose and compared them with logistic regression. Participants were 746 men that had, and 746 that had not, reoffended during matched follow-up periods from 0.5 to 5.8 years. Both RITA-items and static predictors predicted general and violent recidivism well (AUC = .73 – .79), but combining them increased discrimination only slightly (ΔAUC = 0.01 – 0.02) over static predictors alone. Calibration was good for all models. We argue that the results show strong potential for the RITA-items but that development is best focused on improving usability for identifying treatment targets and for updating risk assessments.

Research at the individual level shows strong positive relationships between performance in video games and on intelligence tests. Together with evidence of above average IQs of players of traditional mental sports such as chess, this suggests that national IQs should be strongly related to national performance on mental sports. To investigate this, lists of top players for 12 different electronic sports (e-sports) and traditional mental sports were collected from a variety of sources (total n = 36k). Using a log count approach to control for population size, national cognitive ability/IQ was found to be a predictor (p

The words in children’s language learning environments are strongly predictive of cognitive development and school achievement. But how do we measure language environments and do so at the scale of the many words that children hear day-in and day-out? The quantity and quality of words in a child’s input is typically measured in terms of total amount of talk and the lexical diversity in that talk. There are disagreements in the literature whether amount or diversity is the more critical measure of the input. Here we analyze the properties of a large corpus (6.5 million words) of speech to children and simulate learning environments that differ in amount of talk per unit time, lexical diversity, and the contexts of talk. The central conclusion is that what researchers need to theoretically understand, measure, and change is not the total amount of words, or the diversity of words, but the function that relates total words to the diversity of words, and how that function changes across different contexts of talk.

This report describes two preliminary simulation studies that were included in earlierrevisions of Epskamp, Waldorp, Mõttus, and Borsboom (2017). These will now be includedin a future manuscript.

Cognitive measurement models decompose observed behavior into latent cognitive processes. For situations with more than one condition, such models allow to test hypotheses on the level of the latent processes. We propose a fully Bayesian ensemble model approach to test hypotheses on the level of the latent processes in situations in which multiple measurement models or model classes exist. In the first step, one needs to perform a Bayesian model selection step comparing the hypotheses within each model class. Aggregating the results of the first step yields ensemble posterior model probabilities. We provide an example for a working memory data set using an ensemble of a resource model and a slots model.

The debate between Bayesians and frequentist statisticians has been going on for decades. Whilst there are fundamental theoretical and philosophical differences between both schools of thought, we argue that in two most common situations the practical differences are negligible when off-the-shelf Bayesian analysis (i.e., using ‘objective’ priors) is used. We emphasize this reasoning by focusing on interval estimates: confidence intervals and credible intervals. We show that this is the case for the most common empirical situations in the social sciences, the estimation of a proportion of a binomial distribution and the estimation of the mean of a unimodal distribution. Numerical differences between both approaches are small, sometimes even smaller than those between two competing frequentist or two competing Bayesian approaches. We outline the ramifications of this for scientific practice.

Decision Trees (DT), one method falling under the umbrella of Exploratory Data Mining (EDM; McArdle & Ritschard, 2013), has seen an increasing amount of use in psychological research. This can be attributed to its easily interpretable tree structure, along with its propensity to automatically capture interactions among predictors. However, the creation of tree structures comes at the expense of generalizability – namely that slight perturbations in the sample can result in dramatically different results. The purpose of this paper was to investigate this instability (reliability) and to provide metrics to quantify the propensity for the results to generalize. This was examined across multiple DT algorithms and datasets to determine whether there were differences in stability, and what factors accounted for these differences. There were large differences in both stability and predictive performance across algorithms, with some favoring simplistic, stable trees, and others large, unstable, but highly predictive models. Finally, this comparison was applied to three empirical examples – predicting dementia with various cognitive scales and demographic factors, using depression scales to predict suicidal ideation, and predicting science achievement in Kindergarten. In sum, we demonstrate that measuring the stability of DT algorithms has utility in comparing multiple algorithms and methods, providing additional information to researchers as to both the viability and usefulness of using DT algorithms in their research.

Zwaan et al. and others discuss the importance of the inevitable differences between a replication experiment and the corresponding original experiment. But these discussions are not informed by a principled, quantitative framework for taking differences into account. Bayesian confirmation theory provides such a framework. It will not entirely solve the problem, but it will lead to new insights.

Borderline personality disorder (BPD) is characterized by impulsivity, emotion dysregulation, disturbed relationships, and identity disturbances. Despite the known variable co-occurrence of BPD symptoms, the possible causal relationships are not well understood. We addressed this by creating a hierarchical network model of BPD, which identifies the most likely acyclic causal pathways that are driving BPD development. Cross-sectional data was obtained from the Structured Clinical Interview-II (SCID-II), and possible causal relationships between symptoms were identified from conditional independence relations. The symptoms’ hierarchy values, assessing their role in causal pathways, was determined by a random walk-based algorithm. By analyzing the directed network of BPD symptoms, it was found that symptoms in initial stages of causal pathways were abandonment, physical fights, impulsivity, suicidal threats, identity disturbances, and affective instability. Based on the assessed role symptoms play in causal pathways of BPD development, specific symptoms can be targeted during early diagnosis and clinical assessment.

Investigating the bases of inter-individual differences in risk-taking is necessary to refine our cognitive and neural models of decision-making and to ultimately counter risky behaviours in real-life policy settings. However, recent evidence suggests that behavioural tasks fare poorly compared to standard questionnaires to measure individual differences in risk-taking. Crucially, the results were not improved using model-based measures of risk taking. Here we propose two possible - not mutually exclusive - explanations for these results. Finally, we suggest future avenues of research to improve the assessment of inter-individual differences in risk-taking that combine repeated online testing and the development of mechanistic computational models.

We have tested the feasibility of a method to prevent the occurrence of so-called Questionable Research Practices (QRP). A part from embedded pre-registration the major aspect of the system is real-time uploading of data on a secure server. We outline the method, discuss the drop-out treatment and compare it to the Born-open data method, and report on our preliminary experiences. We also discuss the extension of the data-integrity system from secure server to use of blockchain technology.

A recent preprint by Brown and Coyne titled, "No Evidence That Twitter Language Reliably Predicts Heart Disease: A Reanalysis of Eichstaedt et al." asserts to re-analyze our 2015 article published in Psychological Science, “Twitter Language Predicts Heart Disease Mortality”, disputing its primary findings. While we welcome scrutiny of the study, Brown and Coyne’s paper does not in fact report on a reanalysis, but rather presents a new analysis relating Twitter language to suicide instead of heart disease mortality. In our original article, we showed that Twitter language, fed into standard machine learning algorithms, was able to predict (i.e., estimate cross-sectionally) the out-of-sample heart disease rates of U.S. counties. Further, in a separate analysis, we found that the dictionaries and topics (i.e., sets of related words) which best predicted county atherosclerotic heart disease mortality rates included language related to education and income (e.g., “management,” “ideas,” “conference”) as well as negative social relationships (“hate”, “alone,” “jealous”), disengagement (“tired, “bored,” “sleepy”), negative emotions (“sorry,” “mad,” “sad”) as well as positive emotions (“great,” “happy,” “cool”) and psychological engagement (“learn,” “interesting,” “awake”). Beyond conducting a new analysis (correlating Twitter language with suicide rates), Brown and Coyne also detail a number of methodological limitations of group-level and social media-based studies. We discussed most of these limitations in our original article, but welcome this opportunity to emphasize some of the key aspects and qualifiers of our findings, considering each of their critiques and how they relate to our findings. Of particular note, even though we discuss our findings in the context of what is known about the etiology of heart disease at the individual level, we reiterate here a point made in our original paper: that individual-level causal inferences cannot be made from the cross-sectional and group-level analyses we presented. Our findings are intended to provide a new epidemiological tool to take advantage of large amounts of public data, and to complement, not replace, definitive health data collected through other means.We offer preliminary comments on the suicide language correlations: Previous studies have suggested that county-level suicides are relatively strongly associated with living in rural areas (Hirsch et al., 2006; Searles et al., 2014) and with county elevation (Kim et al., 2011; Brenner et al., 2011). When we control for these two confounds, we find the dictionary associations reported by Brown and Coyne are no longer significant. We conclude that their analysis is largely unrelated to our study and does not invalidate the findings of our original paper. In addition, we offer a replication of our original findings across more years, with a larger Twitter data set. We find that (a) Twitter language still predicts county atherosclerotic heart disease mortality with the same accuracy, and (b) the specific dictionary correlations we reported are largely unchanged on the new data set. To facilitate the reproduction by other researchers of our original work, we also re-release the data and code with which to reproduce our original findings, making it more user-friendly. We will do the same for this replication upon publication.

In the psychometric literature concerning spatial navigation, several questionnaires had been designed to assess the strategies that different individuals engage in when they encode and retrieve environmental knowledge from the first-person (egocentric) and third-person (allocentric) perspectives. Focusing on 12 survey-based strategy items of the Navigation Strategy Questionnaire (Zhong, 2013; Zhong & Kozhevnikov, 2016), this paper applies item response theory-based analysis to determine: (a) whether an item with a low rotated component loading of .27 (SURVEY12) can be considered for removal in future surveys (When I reconstruct my mental map, its environmental orientation is fixed and does not change with my imagined heading directions), and (b) whether a bifactor model can better describe the latent structure of the survey-based strategy. Results from the item level fit diagnostics, categorical response and item information curves showed that SURVEY12 can be indeed removed in future surveys, and that a bifactor model with three items assessing the selective preference for allocentric reference frame constituting a content factor offered a better model fit compared to a unidimensional model. Total score from these three items also correlated significantly with a pointing-to-landmarks task to the same relative magnitude as the total scores from all items, and all items excluding SURVEY12. These findings suggest that the three preference-related items can constitute a subscale for predicting spatial orientation performance in future studies. Moreoverr, it is also recommended that future studies employ spatial navigation tasks that engage survey-based strategy primarily, and design more items concerning the impact of environmental familiarity on survey-based strategy use.

"We propose to change the default P-value threshold forstatistical significance for claims of new discoveries from 0.05 to 0.005."

The shifted-Wald model is a popular analysis tool for one-choice reaction-time tasks. In its simplest version, the shifted-Wald model assumes a constant trial-independent drift rate parameter. However, the presence of endogenous processes—fluctuation in attention and motivation, fatigue and boredom—suggest that drift rate might vary across experimental trials. Here we show how across-trial variability in drift rate can be accounted for by assuming a trial-specific drift rate parameter that is governed by a positive-valued distribution. We consider two candidate distributions: the truncated normal distribution and the gamma distribution. For the resulting distributions of first-arrival times, we derive analytical and sampling-based solutions, and implement the models in a Bayesian framework. Recovery studies and an application to a data set comprised of 1469 participants suggest that (1) both mixture distributions yield similar results; (2) all model parameters can be recovered accurately except for the drift variance parameter; (3) despite poor recovery, the presence of the drift variance parameter facilitates accurate recovery of the remaining parameters; (4) shift, threshold, and drift mean parameters are correlated.

Definitions of the term “empathy” have long been discussed. Recent evidence suggests that “empathy” and “concern” (also known as “compassion”) are psychologically and physiologically distinct. These distinctions directly affect the psychometric measures used to assess empathy. A new measure called the Empathy Index (EI) has been proposed that assesses empathy strictly in terms of emotional contagion. However, this measure has not been validated using Brazilian Portuguese language. Our objective was therefore to translate, culturally adapt, and examine the psychometric properties of the EI in a Brazilian population. We also examined the EI’s relationship with the Interpersonal Reactivity Index (IRI). In Study 1 (N=483), we explored the relationship between EI and IRI through factor analysis. Empathy and concern loaded on different factors. The EI subscales showed adequate Cronbrach’s alphas and concern, but not empathy, predicted contributions during an economic game. In Study 2 (N=215), confirmatory factor analyses confirmed the factorial structure of EI and its relation with IRI. Therefore, we consider that the Brazilian version of the EI showed good evidence of both validity and reliability.

In this paper, we discuss the benefits of and how to utilize Bayesian statistics in studies of moral education. To demonstrate concrete examples of the applications of Bayesian statistics to studies of moral education, we reanalyzed two datasets previously collected: one small dataset collected from a moral educational intervention experiment, and one big dataset from a large-scale Defining Issues Test-2 survey. Results suggest that Bayesian analysis of datasets collected from moral educational studies can provide additional useful statistical information, particularly that associated with the strength of evidence supporting alternative hypotheses, which has not been provided by the classical frequentist approach focusing on P-values. Finally, we introduce several practical guidelines pertaining to how to utilize Bayesian statistics, including the utilization of newly developed free statistical software, JASP, and thresholding based on Bayes Factors, to scholars in the field of moral education.

The creation of abridged versions of research tools is a common, justifiable process, but unfortunately it is often carried out without due methodological care and regard for the consequences. Smith and collaborators (2000) have already written about the mistakes that can be made, but their article has not had much practical impact. There are two main mistakes commonly made by researchers: assuming the transferability of validity and reliability between the full and shortened versions and using less stringent criteria to assess the validity and reliability of short forms. These two problems manifest as nine sins committed during the construction of short forms. Here we present procedures designed to allow researchers to avoid these mistakes and create abridged versions of research tools that are as reliable as possible and to assess the costs of the various methods of abridging questionnaires. To this end we determine the expected length of the tool and weight the benefits of reduced questionnaire completion time against the loss of reliability. We also estimate the shared variance of the full and short versions and classification accuracy of the new, short version. We compared quality of short form obtained from three most common statistical techniques for abridging questionnaires. We analysed data from a sample of 519 persons; 309 (59.5%) completed the paper version of the Self-Narrative Inclination Questionnaire (IAN-R), and 210 (40.5%) participated in online tests. Abridgements based on factor loadings and Cronbach’s # were similarly effective and these two methods had a slight advantage over a method based on item response theory-based analyses of difficulty and discriminatory power.

In determining the need to directly replicate, it is crucial to first verify the original results through independent reanalysis of the data. Original results that appear erroneous and that cannot be reproduced by reanalysis offer little evidence to begin with, thereby diminishing the need to replicate. Sharing data and scripts is essential to ensure reproducibility.

By systematically varying the number of subjects and the number of frames per subject, we explored the influence of training set size on appearance and shape-based approaches to facial action unit (AU) detection. Digital video and expert coding of spontaneous facial activity from 80 subjects (over 350,000 frames) were used to train and test support vector machine classifiers. Appearance features were shape-normalized SIFT descriptors and shape features were 66 facial landmarks. Ten-fold cross-validation was used in all evaluations. Number of subjects and number of frames per subject differentially affected appearance and shape-based classifiers. For appearance features, which are high-dimensional, increasing the number of training subjects from 8 to 64 incrementally improved performance, regardless of the number of frames taken from each subject (ranging from 450 through 3600). In contrast, for shape features, increases in the number of training subjects and frames were associated with mixed results. In summary, maximal performance was attained using appearance features from large numbers of subjects with as few as 450 frames per subject. These findings suggest that variation in the number of subjects rather than number of frames per subject yields most efficient performance.

Observational measurement plays an integral role in a variety of scientific endeavors within biology, psychology, sociology, education, medicine, and marketing. The current article provides an interdisciplinary primer on observational measurement; in particular, it highlights recent advances in observational methodology and the challenges that accompany such growth. First, we detail the various types of instrument that can be used to standardize measurements across observers. Second, we argue for the importance of validity in observational measurement and provide several approaches to validation based on contemporary validity theory. Third, we outline the challenges currently faced by observational researchers pertaining to measurement drift, observer reactivity, reliability analysis, and time/expense. Fourth, we describe recent advances in computer-assisted measurement, fully automated measurement, and statistical data analysis. Finally, we identify several key directions for future observational research to explore.

The editorial for a Special Issue of the journal Psychonomic Bulletin & Review.

With the advent of computers in education, and the ample availability of online learning and practice environments, enormous amounts of data on learning become available. The purpose of this paper is to present a decade of experience with analyzing and improving an online practice environment for math, which has thus far recorded over a billion responses. We present the methods we use to both steer and analyze this system in real-time, using scoring rules on accuracy and response times, a tailored rating system to provide both learners and items with current ability and difficulty ratings, and an adaptive engine that matches learners to items. Moreover, we explore the quality of fit by means of prediction accuracy and parallel item reliability. Limitations and pitfalls are discussed by diagnosing sources of misfit, like violations of unidimensionality and unforeseen dynamics. Finally, directions for development are discussed, including embedded learning analytics and a focus on online experimentation to evaluate both the system itself and the users’learning gains. Though many challenges remain open, we believe that large steps have been made in providing methods to efficiently manage and research educational big data from a massive online learning system.

NOTE: Accepted in principle at Meta-Psychology, submission number MP2018.892, link: https://osf.io/ps5ru/. Anyone can participate in peer review by sending the editor an email, or through discussion on social media. The preferred way of open commenting, however, is to use the hypothes.is integration at PsyArXiv and directly comment on this preprint. Editor: Rickard Carlsson, rickard.carlsson@lnu.seWebsite: https://open.lnu.se/index.php/metapsychology ABSTRACT: Scientific progress relies on the replication and reuse of research. However, despite an emerging culture of sharing code and data in psychology, the research practices needed to achieve computational reproducibility -- the quality of a research project entailing the provision of sufficient code, data and documentation to allow an independent researcher to re-obtain the project's results -- are not widely adopted. Historically, the ability to share and reuse computationally reproducible research was technically challenging and time-consuming. One welcome development on this front is the advent of containers, a technology intended to facilitate code sharing for software development. Containers, however, remain technically demanding and imperfectly suited for research applications. This editorial argues that the use of containers adapted for research can help foster a culture of reproducibiliy in psychology research. We will illustrate this by introducing Code Ocean, an online computational reproducibility platform. (Disclaimer: the authors work for Code Ocean.)

Developing meta-analytic methods is an important goal for psychological science. When there are few studies in particular, commonly used methods have several limitations, most notably of which is underestimating between-study variability. Although Bayesian methods are often recommended for small sample situations, their performance has not been thoroughly examined in the context of meta-analysis. Here, we characterize and apply weakly informativepriors for estimating meta-analytic models and demonstrate with extensive simulations that fully Bayesian methods overcome boundary estimates of exactly zero between-study variance, better maintain error rates, and have lower frequentist risk according toKullback-Leibler divergence. While our results show that combining evidence with few studiesis non-trivial, we argue that this is an important goal that deserves further considerationin psychology. Further, we suggest that frequentist properties can provide importantinformation for Bayesian modeling. We conclude with meta-analytic guidelines for appliedresearchers that can be implemented with the provided computer code.

Model comparison in Bayesian mixed models is becoming popular in psychological science. Here we develop a set of nested models that account for order restrictions across individuals in psychological tasks. An order-restricted model addresses the question 'Does Everybody', as in, 'Does everybody show the usual Stroop effect', or ‘Does everybody respond more quickly to intense noises than subtle ones.’ The crux of the modeling is the instantiation of 10s or 100s of order restrictions simultaneously, one for each participant. To our knowledge, the problem is intractable in frequentist contexts but relatively straightforward in Bayesian ones. We develop a Bayes factor model-comparison strategy using Zellner and colleagues’ default g-priors appropriate for assessing whether effects obey equality and order restrictions. We apply the methodology to seven data sets from Stroop, Simon, and Eriksen interference tasks. Not too surprisingly, we find that everybody Stroops—that is, for all people congruent colors are truly named more quickly than incongruent ones. But, perhaps surprisingly, we find these order constraints are violated for some people in the Simon task, that is, for these people spatially incongruent responses occur truly more quickly than congruent ones! Implications of the modeling and conjectures about the task-related differences are discussed.This paper was written in R-Markdown with code for data analysis integrated into the text. The Markdown script isopen and freely available at https://github.com/PerceptionAndCognitionLab/ctx-indiff. The data are also open and freely available at https://github.com/PerceptionCognitionLab/data0/tree/master/contexteffects.

Separating the signal in a test from the irrelevant noise is a challenge for all measurement. Low test reliability limits test validity, attenuates important relationships, and can lead to regression artifacts. Multiple approaches to the assessment and improvement of reliability are discussed. The advantages and disadvantages of several different approaches to reliability are considered. Practical advice on how to assess reliability using open source software is provided.