Search

Multidimensional item response models are known to be difficult to estimate, with a variety of estimation and modeling strategies being proposed to handle the difficulties. While some previous studies have considered the performance of these estimation methods, they typically include only one or two methods, or a small number of factors. In this paper, we report on a large simulation study of between-item multidimensional IRT estimation methods, considering five different methods, a variety of sample sizes, and up to eight factors. This study provides a comprehensive picture of the methods’ relative performance, as well as each individual method’s strengths and weaknesses. The study results lead us to make recommendations for applied research, related to which estimation methods should be used under various scenarios.

Latent variable models (LVMs) are incredibly flexible tools that allow users to address research questions they might otherwise never be able to answer (McDonald, 2013). However, one major limitation of LVMs is evaluating model fit. There is no universal consensus about how to evaluate model fit, either globally or locally. Part of the reason evaluating these models is difficult is because fit is typically reduced to a handful of statistics that may or may not reflect the model’s adequacy and/or assumptions. In this paper we argue that proper evaluation of model fit must include visualizing both the raw data and the model-implied fit. Visuals reveal, at a glance, the fit of the model and whether the model’s assumptions have been met. Unfortunately, tools for visualizing LVMs have historically been limited. In this paper, we introduce new plots and reframe existing plots that provide necessary resources for evaluating LVMs. These plots are available in a new open- source R package called flexplavaan, which combines the model plotting capabilities of flexplot with the latent variable modeling capabilities oflavaan.

In this article, we apply Vuong's (1989) likelihood ratio tests of nonnested models to the comparison of nonnested structural equation models (SEMs). Similar tests have been previously applied in SEM contexts (especially to mixture models), though the nonstandard output required to conduct the tests has limited their use and study. We review the theory underlying the tests and show how they can be used to construct interval estimates for differences in nonnested information criteria. Through both simulation and application, we then study the tests' performance in nonmixture SEMs and describe their general implementation via free R packages. The tests offer researchers a useful tool for nonnested SEM comparison, with barriers to test implementation now removed. (PsycINFO Database Record

Strictly proper scoring rules, including the Brier score and the logarithmic score, are standard metrics by which probability forecasters are assessed and compared. Researchers often find that one's choice of strictly proper scoring rule has minimal impact on one's conclusions, but this conclusion is typically drawn from a small set of popular rules. In the context of forecasting world events, we use a recently proposed family of proper scoring rules to study the properties of a wide variety of strictly proper rules. The results indicate that conclusions vary greatly across different scoring rules, so that one's choice of scoring rule should be informed by the forecasting domain. We then describe strategies for choosing a scoring rule that meets the needs of the forecast consumer, considering three unique families of proper scoring rules.

We propose the use of signal detection theory (SDT) to evaluate the performance of both probabilistic forecasting systems and individual forecasters. The main advantage of SDT is that it provides a principled way to distinguish the response from system diagnosticity, which is defined as the ability to distinguish events that occur from those that do not. There are two challenges in applying SDT to probabilistic forecasts. First, the SDT model must handle judged probabilities rather than the conventional binary decisions. Second, the model must be able to operate in the presence of sparse data generated within the context of human forecasting systems. Our approach is to specify a model of how individual forecasts are generated from underlying representations and use Bayesian inference to estimate the underlying latent parameters. Given our estimate of the underlying representations, features of the classic SDT model, such as the receiver operating characteristic (ROC) curve and the area under the ROC curve (AUC), follow immediately. We show how our approach allows ROC curves and AUCs to be applied to individuals within a group of forecasters, estimated as a function of time, and extended to measure differences in forecastability across different domains. Among the advantages of this method is that it depends only on the ordinal properties of the probabilistic forecasts. We conclude with a brief discussion of how this approach might facilitate decision making.

A growing number of researchers have advocated for the advancement of cognitive neuroscience by blending cognitive models with neurophysiology. The recently proposed joint modeling framework is one way to bridge the gap between the abstractions assumed by cognitive models and the neurophysiology obtained by modern methods in neuroscience. Despite this advancement, the current method for linking the two domains is hindered by the dimensionality of the neural data. In this article, we present a new linking function based on factor analysis that allows joint models to grow linearly in complexity with increases in the number of neural features. The new linking function is then evaluated in two simulation studies. The first simulation study shows how the model parameters can be accurately recovered when there are many neural features, that mimics real-world applications. The second simulation shows how the new linking function can (1) properly recover a representation of the data generating model, even in the case of model misspecification, and (2) outperform the previous linking function in a cross-validation test. We close by applying a model equipped with the new linking function to real-world data from a perceptual decision making task. The model allows us to understand how differences in the model parameters emerge as a function of differences in brain function across speed and accuracy instruction.

In this paper, we address the use of Bayesian factor analysis and structural equation models to draw inferences from experimental psychology data. While such application is non-standard, the models are generally useful for the unified analysis of multivariate data that stem from, e.g., subjects' responses to multiple experimental stimuli. We first review the models and the parameter identification issues inherent in the models. We then provide details on model estimation via JAGS and on Bayes factor estimation. Finally, we use the models to re-analyze experimental data on risky choice, comparing the approach to simpler, alternative methods.

Imputation methods are popular for the handling of missing data in psychology. The methods generally consist of predicting missing data based on observed data, yielding a complete data set that is amiable to standard statistical analyses. In the context of Bayesian factor analysis, this article compares imputation under an unrestricted multivariate normal model (Multiple Imputation [MI]) to imputation under the statistical model of interest (Data Augmentation [DA]). The former method is popular in applied research, but the latter method is more straightforward from a Bayesian perspective. Simulations demonstrate that DA yields less-biased parameter estimates for moderate sample sizes and high missingness proportions. MI, however, yields less-biased parameter estimates for large sample sizes with misspecified models. The incorporation of auxiliary variables in DA is also addressed, and BUGS code is provided.

Choice confidence is a central measure in psychological decision research, often being reported on a probabilistic scale. Simple mechanisms that describe the psychological processes underlying choice confidence, including those based on error and confirmation biases, have typically received support via fits to data averaged over subjects. While averaged data ease model development, they can also destroy important aspects of the confidence data distribution. In this paper, we develop a hierarchical model of raw confidence judgments using the beta distribution, and we implement two simple confidence mechanisms within it. We use Bayesian methods to fit the hierarchical model to data from a two-alternative confidence experiment, and we use a variety of Bayesian tools to diagnose shortcomings of the simple mechanisms that are overlooked when applied to averaged data. Bugs code for estimating the models is also supplied.

Binary recursive partitioning (BRP) is a computationally intensive statistical method that can be used in situations where linear models are often used. Instead of imposing many assumptions to arrive at a tractable statistical model, BRP simply seeks to accurately predict a response variable based on values of predictor variables. The method outputs a decision tree depicting the predictor variables that were related to the response variable, along with the nature of the variables' relationships. No significance tests are involved, and the tree's 'goodness' is judged based on its predictive accuracy. In this paper, we describe BRP methods in a detailed manner and illustrate their use in psychological research. We also provide R code for carrying out the methods.

In this paper, we consider a family of recently-proposed measurement invariance tests that are based on the scores of a fitted model. This family can be used to test for measurement invariance w.r.t. a continuous auxiliary variable, without pre-specification of subgroups. Moreover, the family can be used when one wishes to test for measurement invariance w.r.t. an ordinal auxiliary variable, yielding test statistics that are sensitive to violations that are monotonically related to the ordinal variable (and less sensitive to non-monotonic violations). The paper is specifically aimed at potential users of the tests who may wish to know (i) how the tests can be employed for their data, and (ii) whether the tests can accurately identify specific models parameters that violate measurement invariance (possibly in the presence of model misspecification). After providing an overview of the tests, we illustrate their general use via the R packages lavaan and strucchange. We then describe two novel simulations that provide evidence of the tests' practical abilities. As a whole, the paper provides researchers with the tools and knowledge needed to apply these tests to general measurement invariance scenarios.

In this paper, we address the use of Bayesian factor analysis and structural equation models to draw inferences from experimental psychology data. While such application is non-standard, the models are generally useful for the unified analysis of multivariate data that stem from, e.g., subjects’ responses to multiple experimental stimuli. We first review the models and the parameter identification issues inherent in the models. We then provide details on model estimation via JAGS and on Bayes factor estimation. Finally, we use the models to re-analyze experimental data on risky choice, highlighting advantages of the approach over traditional ANOVA.

In this paper, we develop and illustrate a psychologically-motivated model for aggregating judgments of magnitude across experts. The model assumes that experts' judgments are perturbed from the truth by both systematic biases and random error, and it provides aggregated estimates that are implicitly based on the application of nonlinear weights to individual judgments. The model is also easily extended to situations where experts report multiple quantile judgments. We apply the model to expert judgments concerning flange leaks in a chemical plant, illustrating its use and comparing it to baseline measures.