Search

Reliability is a crucial concept in psychometrics. Although it is typically estimated as a single fixed quantity, previous work suggests that reliability can vary across persons, groups, and covariates. We propose a novel method for estimating and modeling case-specific reliability without repeated measurements or parallel tests. The proposed method employs a “Reliability Factor” that models the error variance of each case across multiple indicators, thereby producing case-specific reliability estimates. Additionally, we use Gaussian process modeling to a estimate non-linear, non-monotonic function between the latent factor itself and the reliability of the measure, providing an analogue to test information functions in item response theory. The reliability factor model is a new tool for examining latent regions with poor conditional reliability, and correlates thereof, in a classical test theory framework.

A growing number of publications focus on estimating Gaussian graphical models (GGM, networks of partial correlation coefficients). At the same time, generalizibility and replicability of these highly parameterized models are debated, and sample sizes typically found in datasets may not be sufficient for estimating the underlying network structure. In addition, while recent work emerged that aims to compare networks based on different samples, these studies do not take potential cross-study heterogeneity into account. To this end, this paper introduces methods for estimating GGMs by aggregating over multiple datasets. We first introduce a general maximum likelihood estimation modeling framework in which all discussed models are embedded. This modeling framework is subsequently used to introduce meta-analytic Gaussian network aggregation (MAGNA). We discuss two variants: fixed-effects MAGNA, in which heterogeneity across studies is not taken into account, and random-effects MAGNA, which models sample correlations and takes heterogeneity into account. We assess the performance of MAGNA in large-scale simulation studies. Finally, we exemplify the method using four datasets of post-traumatic stress disorder (PTSD) symptoms, and summarize findings from a larger meta-analysis of PTSD symptom.

Measurement reliability is a fundamental concept in psychology. It is traditionally considered a stable property of a questionnaire, measurement device, or experimental task. Although intraclass correlation coefficients (ICC) are often used to assess reliability in repeated measure designs, their descriptive nature depends upon the assumption of a common within-person variance. This work focuses on the presumption that each individual is adequately described by the average within-person variance in hierarchical models. And thus whether reliability generalizes to the individual level, which leads directly into the notion of individually varying ICCs. In particular, we introduce a novel approach, using the Bayes factor, wherein a researcher can directly test for homogeneous within-person variance in hierarchical models. Additionally, we introduce a membership model that allows for classifying which (and how many) individuals belong to the common variance model. The utility of our methodology is demonstrated on cognitive inhibition tasks. We find that heterogeneous within-person variance is a defining feature of these tasks, and in one case, the ratio between the largest to smallest within-person variance exceeded 20. This translates into a tenfold difference in person-specific reliability! We also find that few individuals belong to the common variance model, and thus traditional reliability indices are potentially masking important individual variation. We discuss the implications of our findings and possible future directions. The methods are implemented in the R package vICC

Invariance of the measurement model (MM) between subjects and within subjects over time is a prerequisite for drawing valid inferences when studying dynamics of psychological factors in intensive longitudinal data. To conveniently evaluate this invariance, latent Markov factor analysis (LMFA) was proposed. LMFA combines a latent Markov model with mixture factor analysis: The Markov model captures changes in MMs over time by clustering subjects' observations into a few states and state-specific factor analyses reveal what the MMs look like. However, to estimate the model, Vogelsmeier, Vermunt, van Roekel, and De Roover (2019) introduced a one-step (full information maximum likelihood; FIML) approach that is counterintuitive for applied researchers and entails cumbersome model selection procedures in the presence of many covariates. In this paper, we simplify the complex LMFA estimation and facilitate the exploration of covariate effects on state memberships by splitting the estimation in three intuitive steps: (1) obtain states with mixture factor analysis while treating repeated measures as independent, (2) assign observations to the states, and (3) use these states in a discrete- or continuous-time latent Markov model taking into account classification errors. A real data example demonstrates the empirical value. ispartof: MULTIVARIATE BEHAVIORAL RESEARCH vol:58 issue:2 ispartof: location:United States status: published

Missing data are ubiquitous in psychological research. They may come about as an unwanted result of coding or computer error, participants' non-response or absence, or missing values may be intentional, as in planned missing designs. We discuss the effects of missing data on χ²-based goodness-of-fit indices in Structural Equation Modeling (SEM), specifically on the Root Mean Squared Error of Approximation (RMSEA). We use simulations to show that naive implementations of the RMSEA have a downward bias in the presence of missing data and, thus, overestimate model goodness-of-fit. Unfortunately, many state-of-the-art software packages report the biased form of RMSEA. As a consequence, the scientific community may have been accepting a much larger fraction of models with non-acceptable model fit. We propose a bias-correction for the RMSEA based on information-theoretic considerations that take into account the expected misfit of a person with fully observed data. The corrected RMSEA is asymptotically independent of the proportion of missing data for misspecified models. Importantly, results of the corrected RMSEA computation are identical to naive RMSEA if there are no missing data.

Ordinal variables, while extremely common in Psychology, are almost exclusively analysed with statistical models that falsely assume them to be metric. This practice can lead to distorted effect size estimates, inflated error rates, and other problems. We argue for the application of ordinal models that make appropriate assumptions about the variables under study. In this tutorial article, we first explain the three major ordinal model classes; the cumulative, sequential and adjacent category models. We then show how to fit ordinal models in a fully Bayesian framework with the R package brms, using data sets on stem cell opinions and marriage time courses. Two appendices provide detailed mathematical derivations of the models, and a third practical example that connects ordinal models to Signal Detection Theory with confidence rating data. Ordinal models provide better theoretical interpretation and numerical inference from ordinal data, and we recommend their widespread adoption in Psychology.

[Correction Notice: An Erratum for this article was reported in Vol 115(6) of Journal of Personality and Social Psychology (see record 2018-63189-002). In the article, the original supplemental material has been revised to include a clarifying note to the Tests of model fit over larger sample (Samples C-G) section and post-peer review analyses added to the Post-peer review Factor Analytic Tests section. All versions of this article have been corrected.] Philosophers and behavioral scientists refer to wisdom as unbiased reasoning that guides one toward a balance of interests and promotes a good life. However, major instruments developed to test wisdom appear biased, and it is unclear whether they capture balance-related tendencies. We examined whether shifting from global, de-contextualized reports to state-level reports about concrete situations provides a less biased method to assess wise reasoning (e.g., intellectual humility, recognition of uncertainty and change, consideration of the broader context at hand and perspectives of others, integration of these perspectives or compromise), which may be aligned with the notion of balancing interests. Results of a large-scale psychometric investigation (N = 4,463) revealed that the novel Situated WIse Reasoning Scale (SWIS) is reliable and appears independent of psychological biases (attribution bias, bias blind spot, self-deception, and impression management), whereas global wisdom reports are subject to such biases. Moreover, SWIS scores were positively related to indices of living well (e.g., adaptive emotion regulation, mindfulness), and balancing of cooperative and self-protective interests, goals (influence-vs.-adjustment), and causal inferences about conflict (attribution to the self-vs.-other party). In contrast, global wisdom reports were unrelated or negatively related to balance-related measures. Notably, people showed modest within-person consistency in wise reasoning across situations or over time, suggesting that a single-shot measurement may be insufficient for whole understanding of trait-level wisdom. We discuss theoretical and practical implications for research on wisdom, judgment and decision making, well-being, and prosociality. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Clinicians aim to discern causal relations among features of mental disorders. Yet observational data are insufficient to confirm these relations. In the present study, we describe a causal graph approach to illustrate how one can discern causal relations among features of obsessive-compulsive disorder (OCD) by applying the Invariant Causal Prediction (ICP) algorithm and the Hidden Invariant Causal Prediction (HICP) algorithm, supplemented by findings from a literature review. Applying these methods to symptom severity data from four time points (before, during [twice], and after behavioral treatment) we estimated a causal graph for OCD. The resulting causal graph reveals multiple cycles between aspects of OCD that may play a role in the maintenance of the disorder. We conclude that the proposed design has great potential and could benefit clinicians with some adjustments and extensions.

Background. Studies investigating the link between depressive symptoms and inflammation have yielded inconsistent results, which may be due to two factors. First, studies differed regarding the specific inflammatory markers studied and covariates accounted for. Second, specific depressive symptoms may be differentially related to inflammation. We address both challenges using network psychometrics.Methods. We estimated seven regularized mixed graphical models in the NESDA data (N=2321) to explore shared variances among 1) depression severity, modeled via depression sum-score, 9 DSM-5 symptoms, or 28 individual depressive symptoms; 2) inflammatory markers C-reactive protein (CRP), interleukin 6 (IL-6), and tumor necrosis factor α (TNF-α); 3) before and after adjusting for sex, age, body mass index (BMI), exercise, smoking, alcohol, and chronic diseases.Results. The depression sum-score was related to both IL-6 and CRP before, and only to IL-6 after covariate adjustment. When modeling the DSM-5 symptoms and CRP in a conceptual replication of Jokela et al. (2016), CRP was associated with ‘sleep problems’, ‘energy level’, and ‘weight/appetite changes’; only the first two links survived covariate adjustment. In a conservative model with all 38 variables, symptoms and markers were unrelated. Following recent psychometric work, we re-estimated the full model without regularization: the depressive symptoms ‘insomnia, ‘hypersomnia’, and ‘aches and pain’ showed unique positive relations to all inflammatory markers.Conclusions. We found evidence for differential relations between markers, depressive symptoms, and covariates. Associations between symptoms and markers were attenuated after covariate adjustment; BMI and sex consistently showed strong relations with inflammatory markers.

This article introduces the NetworkToolbox package for R. Network analysis offers an intuitive perspective on complex phenomena via models depicted by nodes (variables) and edges (correlations). The ability of networks to model complexity has made them the standard approach for modeling the intricate interactions in the brain. Similarly, networks have become an increasingly attractive model for studying the complexity of psychological and psychopathological phenomena. NetworkToolbox aims to provide psychological researchers with state-of-the-art methods and measures for estimating and analyzing brain, cognitive, and psychometric data. In this article, I introduce NetworkToolbox and provide a manual for applying the package’s functions to psychometric (personality) data.

Psychological data often violate the normality assumptions made by commonly used statistical methods. These violations are addressed in a variety of ways such as transformations or assuming the employed method is robust to violations. Here we argue that data transformations are unnecessary at best and severely misleading at worst. An alternative approach is to use a Bayesian model that permits skewness and other perturbations to classical assumptions (e.g., heteroskedasticity). Through simulation, we demonstrate that a Bayesian skew-normal model has optimal frequentist properties (i.e., "type 1" error, "power", unbiasedness) compared to normal-assumptive models with or without transformation. Furthermore, the Bayesian skew-normal model has greater predictive utility, as indicated by posterior predictive checking and approximate leave-one-out cross-validation. After an applied example, we discuss practical implications of our findings for psychological science in general, and specifically how Bayesian modeling can improve reproducibility in psychology.

Psychometric network analysis is an emerging tool to investigate the structure of psychological and psychopathological constructs. To date, most of the psychometric network literature has emphasized the measurement of constructs (e.g., dimensional structure); however, this represents only one aspect of psychometrics. In the present study, we explored whether network analysis could be used as a tool for scale development. To do so, we used a previously published dataset (N = 794) of four Openness to Experience inventories to clarify the facet structure of the construct and identify the conceptual coverage of each inventory. In short, 10 facets and 3 aspects (i.e., meso-facets) were identified but no single inventory adequately covered all facets or aspects. Therefore, we used network analysis, including two novel network measures (community closeness centrality and network coverage), to develop a short measure that comprehensively measured all facets and aspects of the construct. We then compared the network-derived short form to short forms that were developed using classical test theory (CTT) and item response theory (IRT). The network-derived short form demonstrated comparable reliability to the CTT- and IRT-based short forms but had better coverage of the conceptual space (defined by the four inventories). Finally, we validated the network-derived short form by comparing its correlations with outcome measures (personality and political conservatism) to that of the four-inventory item pool. We found that the network approach is a promising psychometric tool for scale development and we discuss its implications for future applications.

In continuous test norming, the test score distribution is estimated as a continuous function of predictor(s). A flexible approach for norm estimation is the use of generalized additive models for location, scale, and shape. It is unknown how sensitive their estimates are to model flexibility and sample size. Generally, a flexible model that fits at the population level has smaller bias than its restricted nonfitting version, yet it has larger sampling variability. We investigated how model flexibility relates to bias, variance, and total variability in estimates of normalized z scores under empirically relevant conditions, involving the skew Student t and normal distributions as population distributions. We considered both transversal and longitudinal assumption violations. We found that models with too strict distributional assumptions yield biased estimates, whereas too flexible models yield increased variance. The skew Student t distribution, unlike the Box–Cox Power Exponential distribution, appeared problematic to estimate for normally distributed data. Recommendations for empirical norming practice are provided.

Rasch analysis is a popular statistical tool for developing and validating instruments that aim to measure human performance, attitudes and perceptions. Despite the availability of various software packages, constructing a good instrument based on Rasch analysis is still considered to be a complex, labor-intensive task, requiring human expertise and bares the possibility results in different but equally suited instruments. In this paper, we propose a semi-automated method for Rasch analysis based on first principles that reduces the need for human input. To this end, we introduce a novel criterion, called in-plus-out-of-questionnaire log-likelihood (IPOQ-LL). On artificial datasets, we confirm that optimization of IPOQ-LL leads to the desired behavior in the case of multi-dimensional and inhomogeneous surveys. On three publicly available real-world datasets, our method leads to instruments that have, for all practical purposes, similar clinimetrical properties as those obtained by Rasch analysis experts through a manual procedure.

Mixed-effects models are becoming common in psychological science. Although they have many desirable features, there is still untapped potential. It is customary to view homogeneous variance as an assumption to satisfy. We argue to move beyond that perspective, and to view modeling within-person variance as an opportunity to gain a richer understanding of psychological processes. The technique to do so is based on the mixed-effects location scale model that can simultaneously estimate mixed-effects submodels to both the mean (location) and within-person variance (scale). We develop a framework that goes beyond assessing the submodels in isolation of one another and introduce a novel Bayesian hypothesis test for mean—variance correlations in the distribution of random effects. We first present a motivating example, which makes clear how the model can characterize mean—variance relations. We then apply the method to reaction times (RTs) gathered from 2 cognitive inhibition tasks. We find there are more individual differences in the within-person variance than the mean structure, as well as a complex web of structural mean—variance relations. This stands in contrast to the dominant view of within-person variance (i.e., “noise”). The results also point toward paradoxical within-person, as opposed to between-person, effects: several people had slower and less variable incongruent responses. This contradicts the typical pattern, wherein larger means tend to be associated with more variability. We conclude with future directions, spanning from methodological to theoretical inquires, that can be answered with the presented methodology.Impact StatementTranslational Abstract: Mixed-effects models are becoming common in psychological science. Although researchers typically focus on explaining the dependent variable, characterizing individual differences in within-person variance can provide valuable insights into behavioral consistency. The technique to do so is based on an extension to the mixed-effect model that can simultaneously predict both the mean and within-person variance. We introduce Bayesian methodology to test the covariance among individual difference parameters that capture mean and within-person variance relations. This is motivated by theoretical models that predict larger means will be associated with more variability. Our approach allows for confirmatory hypothesis testing of this proposed relation in the distribution of random effects. Hence, researchers can begin to test the interplay between an experimental effect on the mean and within-person variability in their repeated measurements. We apply our method to RTs gathered from two cognitive inhibition tasks. We find there are more individual differences in the within-person variance than the mean structure, as well as a complex web of structural mean-variance relations. We also uncovered strikingly large correlations between the experimental effect on the mean and within-person variability. Substantively this novel finding suggests that cognitive inhibition has important and highly interrelated signatures on both RT and RT consistency. The method is implemented in the user-friendly R package hypMuVar.

A test score on a psychological test is usually expressed as a normed score, representing its position relative to test scores in a reference population. These typically depend on predictor(s) such as age. The test score distribution conditional on predictors is estimated using regression, which may need large normative samples to estimate the relationships between the predictor(s) and the distribution characteristics properly. In this study, we examine to what extent this burden can be alleviated by using prior information in the estimation of new norms with Bayesian Gaussian distributional regression. In a simulation study, we investigate to what extent this norm estimation is more efficient and how robust it is to prior model deviations. We varied the prior type, prior misspecification and sample size. In our simulated conditions, using a fixed effects prior resulted in more efficient norm estimation than a weakly informative prior as long as the prior misspecification was not age dependent. With the proposed method and reasonable prior information, the same norm precision can be achieved with a smaller normative sample, at least in empirical problems similar to our simulated conditions. This may help test developers to achieve cost-efficient high-quality norms. The method is illustrated using empirical normative data from the IDS-2 intelligence test.

The current study examines the measurement properties and validity of a novel, abbreviated youth version of the UPPS-P Impulsive Behavior Scale that was developed to maintain measurement consistency with the existing adult short form. Specifically, we examined this scale's (a) factor structure; (b) measurement and structural invariance across four demographic characteristics: gender, ethnicity, household income, and parental education; and (c) correlates using a subset of 4,521 preadolescent (9- and 10-year old) children (53% male) from the baseline wave of the Adolescent Brain and Cognitive Development (ABCD) study, a large, community-based sample. Our findings supported a correlated 5-factor model, as well as a hierarchical model that recaptured the covariation among these 5 lower-order factors in three higher-order factors. Both of these models are consistent with the commonly observed structure of the UPPS-P among adults. We established measurement invariance across all demographic characteristics. Finally, our UPPS-P scales evidenced good convergent and discriminant validity with a broad swath of theoretically relevant external criteria, including self- and parent-reported personality and psychopathology, as well as lab-based neurocognitive tasks. Our findings indicate that we can assess multidimensional impulsivity in children reliably and validly by means of self-report, allowing assessment of this critical domain at early stages of development. We hope that this measure will facilitate the study of impulsivity in large-scale samples to begin to understand the evolution and long-term consequences of impulsivity. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Until now there has been no successful exploration of a priori unknown faceted structure by means of exploratory factor analysis (EFA) of the measured variables (items or tasks). For this reason, we investigate by means of a simulation study how well methods for factor rotation can identify a two-facet orthogonal simple structure. Samples were generated from orthogonal two-facet population factor models with 4 (2 factors per facet) to 12 factors (6 factors per facet) and submitted to factor analysis with subsequent Varimax, Equamax, Parsimax, Factor Parsimony, Tandem I, Tandem II, Infomax, and McCammon’s Minimum Entropy rotation. As a benchmark, orthogonal target rotation of the sample loadings towards the corresponding faceted population loadings was also investigated. The conditions were sample size (n = 400, 1,000), number of factors (q = 4-12), and main loading size (l = .40, .50, .60). Mean congruence coefficients of the sample loading matrices with the corresponding population loading matrices and the root mean squared error between sample loading matrices and corresponding population loading matrices were used as dependent measures. For less than six factors Infomax and McCammon’s Minimum Entropy rotation and for six and more factors Tandem II rotation yielded the highest similarity of sample loading matrices with faceted population loading matrices. Analysis of data of 393 participants that performed a test for the Berlin Model of Intelligence Structure revealed that the faceted structure of this model could be found by means of target rotation of task aggregates corresponding to the cross-products of the facets. Moreover, McCammon’s Minimum Entropy rotation resulted in a loading pattern corresponding to the model, although the factor for figural intelligence was only weakly represented. Implications for the identification of faceted models by means of factor rotation are discussed.

Network psychometrics is undergoing a time of methodological reflection. In part, this was spurred by the revelation that l1-regularization does not reduce spurious associations in partial correlation networks. In this work, we address another motivation for the widespread use of regularized estimation: the thought that it is needed to mitigate overfitting. We first clarify important aspects of overfitting and the bias-variance tradeoff that are especially relevant for the network literature, where the number of nodes or items in a psychometric scale are not largecompared to the number of observations (i.e., a low p/n ratio). This revealed that bias and especially variance are most problematic in p=n ratios rarely encountered. We then introduce a nonregularized method, based on classical hypothesis testing, that fulfills two desiderata: (1) reducing or controlling the false positives rate and (2) quelling concerns of overfitting by providing accurate predictions. These were the primary motivations for initially adopting the graphical lasso (glasso). In several simulation studies, our nonregularized method provided more than competitive predictive performance, and, in many cases, outperformed glasso. Itappears to be nonregularized, as opposed to regularized estimation, that best satisfies these desiderata. We then provide insights into using our methodology. Here we discuss the multiple comparisons problem in relation to prediction: stringent alpha levels, resulting in a sparse network, can deteriorate predictive accuracy. We end by emphasizing key advantages of our approach that make it ideal for both inference and prediction in network analysis.

La santé psychique des étudiants est un problème de santé publique qui semble sous-considéré, en partie en raison de l’hétérogénéité des résultats obtenus dans les différentes enquêtes. Cette hétérogénéité a différentes causes : imprécision dans les termes utilisés ; confusion dans les périodes de temps étudiées ; variabilité des questionnaires utilisés qui ne mesurent pas les mêmes symptômes et dont les seuils de détection varient selon les études ; modélisation d’analyse des données conduisant à des construits narratifs trop simplistes. Ces problématiques sont inhérentes tant au processus scientifique qu’à son utilisation dans le discours social., Students’ mental health is a public health problem that seems under-considered, partly because of the heterogeneity of the results obtained in the different surveys. This heterogeneity has different causes: inaccuracy in the terms used; confusion in the time periods studied; variability of questionnaires used that do not measure the same symptoms and whose detection thresholds vary by study; data analysis modeling leading to overly simplistic narrative constructs. This problematic is inherent in both the scientific process and its use in social discourse.

Relative to broad Big Five domains, personality facets provide incremental value in predicting life outcomes. Valid between-group comparisons of means and correlates of facet scores are contingent upon measurement invariance of personality measures. Research on culture and Big Five personality has been largely limited to cross-national comparisons of domains, without assessing measurement invariance across ethnoracial groups within the same country. Using the NEO Inventories, we tested facet-level measurement invariance between Euro ( N = 418, 63.2% women, Mage = 18.43) and Asian Americans ( N = 429, 56.6% women, Mage = 18.00). Multigroup exploratory factor analysis within a confirmatory factor analysis framework showed partial strong invariance. Assertiveness and activity did not load onto extraversion as strongly for Asian Americans. Self-consciousness showed a stronger cross-loading onto extraversion among Asian Americans than Euro Americans. Achievement striving, competence, warmth, tender-mindedness, and excitement seeking showed noninvariant intercepts across groups. Collectivistic values emphasizing interpersonal harmony and modesty should be considered when examining narrow and broad traits among Asian Americans.

Metamemory can be defined as the knowledge about one's memory capabilities and about strategies that can aid memory. In this paper, we describe the development and validation of the Eyewitness Metamemory Scale (EMS), tailored specifically for use in face memory and eyewitness identification settings. Participants (N = 800) completed the EMS and other measures on general metamemory. Results from exploratory and confirmatory factor analysis revealed good factorial validity, internal consistency, and content validity. The EMS items emerged into three distinct factors: memory contentment, memory discontentment, and memory strategies. The EMS is a brief and easily administrable questionnaire that might be used to assess self-ratings of face recognition capacity and use of strategies to encode faces.

Research synthesis is based on the assumption that when the same association between constructs is observed repeatedly in a field, the relationship is probably real, even if its exact magnitude can be debated. Yet this probability is not only a function of recurring results, but also of the quality and consistency in the empirical procedures that produced those results and that any meta-analysis necessarily inherits. Standardized protocols in data collection, analysis, and interpretation are important empirical properties, and a healthy sign of a discipline's maturity.This manuscript proposes that meta-analyses as typically applied in psychology benefit from complementing their aggregates of observed effect sizes by systematically examining the standardization of methodology that deterministically produced them. Potential units of analyses are described and two examples are offered to illustrate the benefits of such efforts. Ideally, this synergetic approach emphasizes the role of methods in advancing theory by improving the quality of meta-analytic inferences.

The validity of psychological measurement is a crucial auxiliary theory underlying many sexual science studies. Although many sexuality researchers are familiar with certain elements of psychological measurement, the field of psychological measurement is a developing and evolving literature, with concepts, applications, and techniques that do not always trickle down quickly into interdisciplinary fields like sexual science. The purpose of this Measurement Memo, therefore, is to connect sexual scientists to measurement-related issues, explanations, and resources that they may not otherwise encounter in their scholarly reading. Our review focuses on those carrying out psychological measurement using theories and methods of latent variable modeling, and we identify and summarize key ideas and references that serve as good launching points for sexual scientists to begin to improve their psychological measurement practices, for beginners and seasoned users alike.

In this article we describe the origins of the subtests that appear on the modern Stanford-Binet Intelligence Scales (SB5), Wechsler Preschool and Primary Scale of Intelligence (WPPSI-IV), Wechsler Intelligence Scale for Children (WISC-V), and Wechsler Adult Intelligence Scale (WAIS-IV). We found that the majority of these subtest formats were first created in 1908 or earlier and that only three have been created since 1980. We discuss the implications of these findings, which are that (1) many subtests have lengthy research histories that support their use in measuring intelligence; (2) many subtests have formats that predate modern theories of test creation, cognitive psychology, and intelligence; and (3) the history of many subtests is more complex than psychologists probably realize.

Introduction\ud Although risky drinking and alcohol dependence have been associated with spatial attentional biases, concerns have been raised about the reliability of the frequently-used dot-probe task. A form of anticipatory bias related to predictive cues has been found to be related to alcohol-related processes, and to have high reliability in the context of threat stimuli. It remains to be determined whether this anticipatory attentional bias also has good reliability for alcohol stimuli. Further, correlations with drinking-related individual differences need to be replicated.\ud \ud Material and Methods\ud 83 healthy adult participants were included, who completed the task and questionnaires on risky drinking (AUDIT-C), drinking motives (DMQ-R), reasons to abstain from drinking (RALD), and alcohol craving (ACQ). The task used a 400 ms Cue-Stimulus Interval, based on previous work. The Spearman-Brown split-half reliability of reaction time-based bias scores was calculated. The within-subject effect of probe location (predicted-alcohol versus predicted-non-alcohol) was tested using a paired-sample t-test. Correlations were calculated between bias scores and questionnaire scales; tests were one-sided for predicted effects and two-sided for exploratory effects.\ud \ud Results\ud The reliability was .81 (.74 after outlier removal). There was no overall bias. A predicted correlation between risky drinking and anticipatory bias towards alcohol was found, but no other predicted or exploratory effects.\ud \ud Discussion\ud The anticipatory attentional bias for alcohol is a reliably measurable individual difference, with some evidence that it is associated with risky drinking.\ud \ud Conclusions\ud Implicit behavioural measures of spatial attentional bias can in principle achieve high reliability. Further study of attentional biases using predictive cues would appear to be promising.

A wide variety of effect size indices have been proposed for quantifying the magnitude of treatment effects in single-case designs. Commonly used measures include parametric indices such as the standardized mean difference as well as nonoverlap measures such as the percentage of nonoverlapping data, improvement rate difference, and nonoverlap of all pairs. Currently, little is known about the properties of these indices when applied to behavioral data collected by systematic direct observation, even though systematic direct observation is the most common method for outcome measurement in single-case research. This study uses Monte Carlo simulation to investigate the properties of several widely used single-case effect size measures when applied to systematic direct observation data. Results indicate that the magnitude of the nonoverlap measures and of the standardized mean difference can be strongly influenced by procedural details of the study's design, which is a significant limitation to using these indices as effect sizes for meta-analysis of single-case designs. A less widely used parametric index, the log response ratio, has the advantage of being insensitive to sample size and observation session length, although its magnitude is influenced by the use of partial interval recording. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Decision-making competence (DMC) has potential utility in incrementally predicting decision-making behavior relative to general mental ability (g). However, prior research presents an incomplete depiction of the degree to which these two constructs overlap. This paper evaluated both theoretical and empirical discriminant validity evidence for DMC. Theoretically, DMC and g are distinct constructs though highly interconnected; both constructs exist within the intelligence nomological network. However, our empirical findings demonstrate scores on the Adult-Decision Making Competence assessment, are largely indistinguishable from g. Future research should treat DMC as an acquired skill as opposed to an underlying individual difference.

State trace analyses assess the latent dimensionality of a cognitive process by asking whether the means of two dependent variables conform to a monotonic function across a set of conditions. Using an assumption of independence between the measures, recently proposed statistical tests address bivariate measurement error, allowing both frequentist and Bayesian analyses of monotonicity (e.g., Davis-Stober, Morey, Gretton, & Heathcote, 2016; Kalish, Dunn, Burdakov, & Sysoev, 2016). However, statistical inference can be biased by unacknowledged dependencies between measures, particularly when the data are insufficient to overwhelm an incorrect prior assumption of independence. To address this limitation, we developed a hierarchical Bayesian model that explicitly models the separate roles of subject, item, and trial-level dependencies between two measures. Assessment of monotonicity is then performed by fitting separate models that do or do not allow a non-monotonic relation between the condition effects (i.e., same versus different rank orders). The Widely Applicable Information Criterion (WAIC) and Pseudo Bayesian Model Averaging – both cross validation measures of model fit – are used for model comparison, providing an inferential conclusion regarding the dimensionality of the latent psychological space. We validated this new state trace analysis technique using model recovery simulation studies, which assumed different ground truths regarding monotonicity and the direction/magnitude of the subject- and trial-level dependence. We also provide an example application of this new technique to a visual object learning study that compared performance on a visual retrieval task (forced choice part recognition) versus a verbal retrieval task (cued recall).

A majority of statistically educated scientists draw incorrect conclusions based on the most commonly used statistical technique: null hypothesis significance testing (NHST). Frequentist techniques are often claimed to be incorrectly interpreted as Bayesian outcomes, which suggests that a Bayesian framework may fit better to inferences researchers frequently want to make (Briggs, 2012). The current study set out to test this proposition. Firstly, we investigated whether there is a discrepancy between what researchers think they can conclude and what they want to be able to conclude from NHST. Secondly, we investigated to what extent researchers want to incorporate prior study results and their personal beliefs in their statistical inference. Results show the expected discrepancy between what researchers think they can conclude from NHST and what they want to be able to conclude. Furthermore, researchers were interested in incorporating prior study results, but not their personal beliefs, into their statistical inference.

Steinley, Hoffman, Brusco, and Sher (2017) proposed a new method for evaluating the performance of psychological network models: fixed-margin sampling. The authors investigated LASSO regularized Ising models (eLasso) by generating random datasets with the same margins as the original binary dataset, and concluded that many estimated eLasso parameters are not distinguishable from those that would be expected if the data were generated by chance. We argue that fixed-margin sampling cannot be used for this purpose, as it generates data under a particular null-hypothesis: a unidimensional factor model with interchangeable indicators (i.e., the Rasch model). We show this by discussing relevant psychometric literature and by performing simulation studies. Results indicate that while eLasso correctly estimated network models and estimated almost no edges due to chance, fixed-margin sampling performed poorly in classifying true effects as “interesting” (Steinley et al. 2017, p. 1004). Further simulation studies indicate that fixed-margin sampling offers a powerful method for highlighting local misfit from the Rasch model, but performs only moderately in identifying global departures from the Rasch model. We conclude that fixed-margin sampling is not up to the task of assessing if results from estimated Ising models or other multivariate psychometric models are due to chance.

Network theories have been put forward for psychopathology (in which mental disorders originate from causal relations between symptoms) and for personality (in which personality factors originate from coupled equilibria of cognitions, affect states, behaviours, and environments). Here, we connect these theoretical strands in an overarching personality–resilience–psychopathology model. In this model, factors in personality networks control the shape of the dynamical landscape in which symptom networks evolve; for example, the neuroticism item ‘I often feel blue’ measures a general tendency to experience negative affect, which is hypothesized to influence the threshold parameter of the symptom ‘depressed mood’ in the psychopathology network. Conversely, events at the level of the fast–evolving psychopathology network (e.g. a depressive episode) can influence the slow–evolving personality variables (e.g. by increasing feelings of worthlessness). We apply the theory to neuroticism and major depressive disorder. Through simulations, we show that the model can accommodate important phenomena, such as the strong relation between neuroticism and depression and individual differences in the change of neuroticism levels and development of depression over time. The results of the simulation are implemented in an online, interactive simulation tool. Implications for research into the relationship between personality and psychopathology are discussed. © 2020 The Authors. European Journal of Personality published by John Wiley & Sons Ltd on behalf of European Association of Personality Psychology

Invariant item ordering refers to the statement that if one item is harder than another for one person, then it is harder for all people. Whether item ordering holds is a psychological statement because it describes how people may qualitatively vary. Yet, modern item response theory (IRT) makes an a priori commitment to item ordering. The Rasch model, for example, posits that items must order. Conversely, the 2PL model posits that items never order. Needed is an IRT model where item ordering or its violation is a function of the data rather than an a priori commitment. We develop two-parameter shift-scale models for this purpose, and find that the two-parameter uniform offers many advantages. We show how item ordering may be assessed using Bayes factor model comparison, and discuss computational issues with shift-scale IRT models.

Risk taking is a complex heterogeneous construct that has proven difficult to assess, especially when using behavioral tasks. We present a new measure – the Assessment of Physical Risk Taking (APRT) – as a comprehensive assessment of the probability of success and failure, and magnitude of reward and punishment of different types of physically risky behaviors with a variety of outcome scores. Participants (N = 224) completed APRT in a laboratory setting, half of whom had a 1.5 s delay interposed between button presses. Main effects, two-way interactions among five within-subject factors, and interactions between the within-subject factors and Delay were estimated for four APRT outcome scores using Generalized Estimating Equations. Results indicated that Injury Magnitude and Injury Probability exerted much stronger effects than the other factors. APRT scores correlated with self-reported risk taking in thrilling activities specifically, but only for those who did not have a delay between APRT responses.

Mindfulness research implicitly conceives of mindfulness as an identifiable real ‘thing’ that exists beyond what is directly observed. Recently, a new methodology has been developed which allows mindfulness to be modelled as a complex system or network at the level of self-report. In these models, items become a network’s nodes, and the statistical relations between them, edges. Interpreted causally, nodes are thought to increasingly influence each other via their edges, such that they become increasingly correlated. This study hypothesises that at a cross-sectional level, this may result in differences in overall network connectivity (density) between practitioners and non-practitioners.Mindfulness networks were estimated for practitioners and non-practitioners using the Friedberg Mindfulness Inventory (FMI). A total of 371 regular mindfulness practitioners and 283 non-practitioners (including 59 irregular practitioners) were recruited online from Amazon Mechanical Turk.Comparisons of practitioners’ and non-practitioners’ networks indicated that network density did not significantly differ, whereas evidence was found in support of a significant difference in network structure. An exploratory analysis revealed substantive group differences in how items (practices) were connected. In particular, the practice of Acceptance appeared more central to the practitioners’ network relative to the non-practitioners network, indicating Acceptance may be particularly useful for engagement in mindfulness practices.The study supports investigating mindfulness as a complex network at the level of self-report, with implications for how the development of mindfulness is conceptualised. The lack of difference in network density indicates that research is needed to examine network dynamics in the context of regular mindfulness practice.

We present a new measure of the Dark Tetrad (narcissism, Machiavellianism, psychopathy, sadism). The measure is brief (28 items), making it useful in identifying socially aversive individuals in an economical fashion.

Adults with autism spectrum disorder (ASD) are at elevated risk for social anxiety disorder (SAD). Limited information exists on the appropriateness of using social anxiety instruments with these adults. This study examines psychometric properties of self-report social anxiety instruments in autistic adults without intellectual disability, compared to adults with SAD. Additionally, we compared instrument scores between autistic adults with a dual diagnosis of SAD and adults with SAD only. Adults diagnosed with SAD (N=316) or ASD (N=102) were recruited from the Brain and Mind Centre in New South Wales, Australia. Sixty autistic participants received a diagnosis of SAD. Participants completed the Liebowitz Social Anxiety Scale-self-report, the Social Interaction Anxiety Scale, the Social Phobia Scale, and the Brief Fear of Negative Evaluation Scale. All instruments showed excellent internal consistency in autistic adults. Summary scores across instruments were strongly correlated within both groups, and the strength of these correlations were equivalent between ASD and SAD. For all instruments, the performance of autistic adults with a dual diagnosis of SAD was comparable to adults diagnosed with SAD only. Findings suggest that these instruments display adequate psychometric properties in autistic adults without intellectual disability, supporting the use of these instruments in clinical services.

What is wisdom? How can we measure wisdom in a questionnaire? Ongoing debate about definitions and best practices are central to the wisdom literature. With different theoretical orientations come different measurement tools; one measure differentiates itself as a theory-independent, empirically constructed scale: The Brief Wisdom Screening Scale (BWSS). Very little research has focused on validation or subscale analysis of the BWSS. To this end, we present four studies, collecting 720 participants’ BWSS responses. In the first study (n = 389), we apply exploratory factor analysis, uncovering a two-factor structure with “Poise” relating to the internalized, stable, equanimous component of wisdom and “Attunement” relating to the learning from life, optimization-orientation of wisdom. We subsequently pre-registered a confirmatory factor analysis using a new sample (n = 331). Results indicate that the two-factor model provides consistently superior fit-indices, but that the item-loadings may not be robust to differing sample characteristics. Exploratory correlational analyses using several additional questionnaires provides good evidence for convergent and divergent construct validity. We discuss the mixed findings and point to future use-cases where Poise and Attunement could be researched using targeted interventions, such as meditation training. The BWSS was conceived as a fast, empirically derived scale for use in studies where wisdom is not the main measure of interest and our findings support continued use and validation of this scale. Amid disagreement about how wisdom should be defined and measured, this empirically constructed scale may provide novel, data-driven insights that could ultimately inform theoretical perspectives from the bottom up.

Dance has become an important for research in empirical aesthetics, social and motor cognition, and as an intervention for neurodegenerative and neurodevelopmental disorders. Despite this growing scientific interest in dance, no standardised psychometric instrument exists to assess people’s dance experience. Here, we introduce the Goldsmiths Dance Sophistication Index (Gold-DSI), a 26-item questionnaire to measure individual differences in active and passive dance experience on a continuous scale. The Gold-DSI was developed in three stages. In the first stage, a set of 76 items was generated; a) by adapting questions from the Goldsmiths Musical Sophistication Index (Müllensiefen et al., 2014), and b) through a stakeholder workshop using a grounded theory approach. The second stage focused on item reduction. Using a large-scale online survey (N=424) , hierarchical factor analysis was used to fit a model comprising of one general and six secondary factors (28 items in total). In stage three, six new items were added to specifically capture individual differences in dance spectatorship. We then collected data from two samples for final model estimation (N=127) and evaluation (N=190). The final version of the Gold-DSI comprises 26 items; 20 items relate to one general factor that captures Active Dance Experience. This includes four secondary factors: Body Awareness, Social Dancing, Urge to Dance, and Dance Training. A further six items separately measure dance spectatorship, thus capturing Passive Dance Experience. In sum, the Gold-DSI provides a brief, standardised and continuous assessment of both active and passive dance experience.

Background: The prevalence of anxiety is three times greater in patients with chronic diseases than in the general population. There has recently been growing interest in the roles of inflammation in contributing to the development of anxiety in people with Immune-mediated inflammatory diseases (IMID). Patient-reported outcome measures can facilitate the assessment of physical and psychological functioning. In recent years various patient outcome measures (PROMs) have been implemented. The National Institutes of Health (NIH)'s Patient-Reported Outcomes Measurement Information System (PROMIS®) is a set of publicly available and standardized Patient-Reported Outcomes (PROs) that cover physical appearance, mental health, and social health. The PROMIS® has been built through an Item Response Theory approach (IRT), a model-based measurement in which trait level estimates depend on both persons' responses and on the properties of the items that were administered. The aim of this study is to test the psychometric properties of the original anxiety items of the PROMIS® in a cohort of outpatients with IMIDs.Methods: We translated the original eight anxiety items from the PROMIS into Italian and administered them to consecutive outpatients affected by Inflammatory Bowel disease (n = 246), rheumatological (n = 100) and dermatological (n = 43) diseases, and healthy volunteers (n = 280). We analyzed the data through an Item Response Theory (IRT) analysis to evaluate the psychometric properties of the Italian adaptation of the PROMIS® anxiety short form.Results: Taken together, Confirmatory Factor Analysis and Exploratory Factor analysis suggested that the unidimensionality assumption if the instrument holds. The instrument has excellent reliability from a Classical Theory of Test (CTT) standpoint (Cronbach's ⍺ = .94, McDonald's ω = .95). The 1PL Graded Response Model (GRM) model provided a good balance between the goodness of fit and parsimony (BIC = 9099.82) as compared to the 2PL GRM model (BIC = 9111.02). We did not find signs of Differential Item Functioning (DIF), but we found that the local independence assumption was violated for some items. The analysis of the test reliability curve suggested that the instrument is most reliable for higher levels of the latent trait of anxiety.Discussion: The Italian adaptation of the PROMIS® anxiety short form shows acceptable psychometric properties both from a CTT and an IRT standpoint. Its compatibility with the 1 parameter logistic (1PL) model implies that its total score is a sufficient statistic for the latent anxiety trait. The TRC shows that this instrument is mostly informative for people with higher levels of anxiety, making it particularly suitable for clinical populations such as IMID patients.

Response time information has recently attracted a significant amount of attention in the literature as it may provide meaningful information about item preknowledge. The methods that propose the use of response time information in identifying examinees with potential item preknowledge make an implicit assumption that the examinees with item preknowledge differ in their response time patterns compared to other examinees without item preknowledge. In this study, we analyzed the differences in response time of examinees with potential item preknowledge and examinees without item preknowledge based on a real experimental dataset provided by Toton and Maynes (2019). A multiple-group extension of van der Linden’s Lognormal Response Model with a gating mechanism was used to capture the differences in latent speed for control and experimental groups on disclosed and undisclosed items. The model used in the study and estimated parameters from this experimental dataset may inform future simulation studies in this area of research to simulate realistic datasets with item preknowledge behavior.

Response time (RT) information has recently attracted a significant amount of attention in the literature as it may provide meaningful information about item preknowledge. In this study, a Deterministic Gated Lognormal Response Time (DG-LNRT) model is proposed to identify examinees with potential item preknowledge using RT information. The proposed model is applied to a real experimental dataset provided by Toton and Maynes (2019) in which item preknowledge was manipulated, and its performance is demonstrated. Then, the performance of the DG-LNRT model is investigated through a simulation study. The model is estimated using the Bayesian framework via Stan. The results indicate that the proposed model is viable and has the potential to be useful in detecting cheating by using response time differences between compromised and uncompromised items.

In many forensic psychiatric hospitals, patients’ mental health is monitored at regular intervals. Typically, clinicians score patients using a Likert scale on multiple criteria including hostility. Having an overview of patients’ scores benefits staff members in at least three ways. First, the scores may help adjust treatment to the individual patient; second, the change in scores over time allows an assessment of treatment effectiveness; third, the scores may warn staff that particular patients are at high risk of turning violent, either before or after release. Practical importance notwithstanding, current practices for the analysis of mental health scores are suboptimal: evaluations from different clinicians are averaged (as if the Likert scale were linear and the clinicians identical), and patients are analyzed in isolation (as if they were independent). Uncertainty estimates of the resulting score are often ignored. Here we outline a quantitative program for the analysis of mental health scores using cultural consensus theory (CCT; Anders & Batchelder, 2015). CCT models take into account the ordinal nature of the Likert scale, the individual differences among clinicians, and the possible commonalities between patients. In a simulation, we compare the predictive performance of the CCT model to the current practice of aggregating raw observations and, as an alternative, against often-used machine learning toolboxes. In addition, we outline the substantive conclusions afforded by the application of the CCT model. We end with recommendations for clinical practitioners who wish to apply CCT in their own work.

Network theory has emerged as a popular framework for conceptualizing psychological constructs and mental disorders. Initially, network analysis was motivated in part by the thought that it can be used for hypothesis generation. Although the customary approach for network modeling is inherently exploratory, we argue that there is untapped potential for confirmatory hypothesis testing. In this work, we bring to fruition the potential of Gaussian graphical models for generating testable hypotheses. This is accomplished by merging exploratory and confirmatory analyses into a cohesive framework built around Bayesian hypothesis testing of partial correlations. We first present a motivating example based on a customary, exploratory analysis, where it is made clear how information encoded by the conditional (in)dependence structure can be used to formulate hypotheses. Building upon this foundation, we then provide several empirical examples that unify exploratory and confirmatory testing in psychopathology symptom networks. In particular, we (1) estimate exploratory graphs; (2) derive hypotheses based on the most central structures; and (3) test those hypotheses in a confirmatory setting. Our confirmatory results uncovered an intricate web of relations, including an order to edge weights within comorbidity networks. This illuminates the rich and informative inferences that can be drawn with the proposed approach. We conclude with recommendations for applied researchers, in addition to discussing how our methodology answers recent calls to begin developing formal models related to the conditional (in)dependence structure of psychological networks.

Person fit statistics are frequently used to detect aberrant behavior when assuming an item response model generated the data. A common statistic, [Formula: see text], has been shown in previous studies to perform well under a myriad of conditions. However, it is well-known that [Formula: see text] does not follow a standard normal distribution when using an estimated latent trait. As a result, corrections of [Formula: see text], called [Formula: see text], have been proposed in the literature for specific item response models. We propose a more general correction that is applicable to many types of data, namely survey or tests with multiple item types and underlying latent constructs, which subsumes previous work done by others. In addition, we provide corrections for multiple estimators of [Formula: see text], the latent trait, including MLE, MAP and WLE. We provide analytical derivations that justifies our proposed correction, as well as simulation studies to examine the performance of the proposed correction with finite test lengths. An applied example is also provided to demonstrate proof of concept. We conclude with recommendations for practitioners when the asymptotic correction works well under different conditions and also future directions.

In social sciences, the study of group differences concerning latent constructs is ubiquitous. These constructs are generally measured by means of scales composed of ordinal items. In order to compare these constructs across groups, one crucial requirement is that they are measured equivalently or, in technical jargon, that measurement invariance holds across the groups. This study compared the performance of multiple group categorical confirmatory factor analysis (MG-CCFA) and multiple group item response theory (MG-IRT) in testing measurement invariance with ordinal data. A simulation study was conducted to compare the true positive rate (TPR) and false positive rate (FPR) both at the scale and at the item level for these two approaches under an invariance and a non-invariance scenario. The results of the simulation studies showed that the performance, in terms of the TPR, of MG-CCFA- and MG-IRT-based approaches mostly depends on the scale length. In fact, for long scales, the likelihood ratio test (LRT) approach, for MG-IRT, outperformed the other approaches, while, for short scales, MG-CCFA seemed to be generally preferable. In addition, the performance of MG-CCFAs fit measures, such as RMSEA and CFI, seemed to depend largely on the length of the scale, especially when MI was tested at the item level. General caution is recommended when using these measures, especially when MI is tested for each item individually. A decision flowchart, based on the results of the simulation studies, is provided to help summarizing the results and providing indications on which approach performed best and in which setting.

When validating psychological surveys, researchers tend to concentrate on analyzing item responses instead of the processes that participants use to generate these responses. When screening for invalid responses, researchers have generally focused on participants who manipulate their answers for personal gain or respond carelessly. In this paper, we introduce a new invalid response process, discordant responding, that arises when participants disagree with the use of the survey and discuss similarities and differences between this response style and protective responding. Results show that nearly all participants reflect on the intended uses of an assessment when responding to items and most decline to respond or modify their responses if they are not comfortable with the way the results will be used. Incidentally, we also find that participants may misread survey instructions if they are not interactive. We introduce a short screener to detect invalid responses with dishonest responder identifiers (DRI), providing researchers with more confidence in the validity of their inferences. Finally, we provide recommendations about how researchers may use these findings to design surveys that reduce this response manipulation in the first place.

A common way to form scores from multiple-item scales is to sum responses of all items. Though sum scoring is often contrasted with factor analysis as a competing method, we review how factor analysis and sum scoring both fall under the larger umbrella of latent variable models, with sum scoring being a constrained version of a factor analysis. Despite similarities, reporting of psychometric properties for sum scored or factor analyzed scales are quite different. Further, if researchers use factor analysis to validate a scale but subsequently sum score the scale, this employs a model that differs from validation model. By framing sum scoring within a latent variable framework, our goal is to raise awareness that (a) sum scoring requires rather strict constraints, (b) imposing these constraints requires the same type of justification as any other latent variable model, and (c) sum scoring corresponds to a statistical model and is not a model-free arithmetic calculation. We discuss how unjustified sum scoring can have adverse effects on validity, reliability, and qualitative classification from sum score cut-offs. We also discuss considerations for how to use scale scores in subsequent analyses and how these choices can alter conclusions. The general goal is to encourage researchers to more critically evaluate how they obtain, justify, and use multiple-item scale scores.

This paper explores the subject of generative adaptive testing using the digit span test as an example. A large-sample study of computer-generated and administered digit-span items on Navy recruits showed an almost perfect correlation (.98-.99) between digit span length and IRT difficulty. Predicted IRT parameters can be used for adaptive testing using items generated in real-time. Our results suggest that the best research strategy for developing generative adaptive tests may be to start with the most elementary cognitive tasks, and then build toward more complete psychometric models of complex mental tasks. The results of this study are sufficiently encouraging so that the same research approach should be tried with other forms of memory span tests and more complex working memory tests, including tests for figures, colors, and words. The paper advances the conjecture that the test information function of a generative CAT system has a mathematical relationship to the model fit and the distribution of the model-specified item parameters, independent of the content domain of the test.

The feasibility of generating items in real-time for computerized adaptive testing is explored, using forward digit span as an exemplar. A sample of 531 recruits at the Naval Training Center in San Diego were administered 36 computer-generated forward digit span items of varying lengths. Calibrations showed that the item difficulty was a simple linear function of the number of digits in the item, thus making the difficulty of newly generated items predictable. Simulations of Computerized adaptive testing with the approach were conducted with favorable results.