Search

The basic random effects meta-analytic model is overwhelmingly dominant in psychological research. Indeed, it is typically employed even when more complex multilevel multivariate meta-analytic models are warranted. In this paper, we aim to help overcome challenges so that multilevel multivariate meta-analytic models will be more often employed in practice. We do so by introducing MLMVmeta-an easy-to-use web application that implements multilevel multivariate meta-analytic methodology that is both specially tailored to contemporary psychological research and easily estimable, interpretable, and parsimonious-and illustrating it across three case studies. The three case studies demonstrate the more accurate and extensive results that can be obtained via multilevel multivariate meta-analytic models. Further, they sequentially build in complexity featuring increasing numbers of experimental factors and conditions, dependent variables, and levels; this in turn necessitates increasingly complex model specifications that also sequentially build upon one another., (© 2022. The Psychonomic Society, Inc.)

In high stakes assessments of personality and similar attributes, test takers may engage in impression management (aka faking). This article proposes to consider responses of every test taker as a potential mixture of "real" (or retrieved) answers to questions, and "ideal" answers intended to create a desired impression, with each type of response characterized by its own distribution and factor structure. Depending on the particular mix of response types in the test taker profile, grades of membership in the "real" and "ideal" profiles are defined. This approach overcomes the limitation of existing psychometric models that assume faking behavior to be consistent across test items. To estimate the proposed faking-as-grade-of-membership (F-GoM) model, two-level factor mixture analysis is used, with two latent classes at the response (within) level, allowing grade of membership in "real" and "ideal" profiles, each underpinned by its own factor structure, at the person (between) level. For collected data, units of analysis can be item or scale scores, with the latter enabling analysis of questionnaires with many measured scales. The performance of the F-GoM model is evaluated in a simulation study, and compared against existing methods for statistical control of faking in an empirical application using archival recruitment data, which supported the validity of latent factors and classes assumed by the model using multiple control variables. The proposed approach is particularly useful for high-stakes assessment data and can be implemented with standard software packages. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Ideally, survey respondents read and understand survey instructions, questions, and response scales, and provide answers that carefully reflect their beliefs, attitudes, or knowledge. However, respondents may also arrive at their responses using cues or heuristics that facilitate the production of a response, but diminish the targeted information content. We use eye-tracking data as covariates in a Bayesian switching-mixture model to identify different response behaviors at the item-respondent level. The model distinguishes response behaviors that are predominantly influenced either positively or negatively by the previous response, and responses that reflect respondents' preexisting knowledge and experiences of interest. We find that controlling for multiple types of adaptive response behaviors allows for a more informative analysis of survey data and respondents., (© 2021. The Psychometric Society.)

This paper presents a systematic investigation of how affirmative and polar-opposite items presented either jointly or separately affect yea-saying tendencies. We measure these yea-saying tendencies with item response models that estimate a respondent's tendency to give a "yea"-response that may be unrelated to the target trait. In a re-analysis of the Zhang et al. (PLoS ONE, 11:1-15, 2016) data, we find that yea-saying tendencies depend on whether items are presented as part of a scale that contains affirmative and/or polar-opposite items. Yea-saying tendencies are stronger for affirmative than for polar-opposite items. Moreover, presenting polar-opposite items together with affirmative items creates lower yea-saying tendencies for polar-opposite items than when presented in isolation. IRT models that do not account for these yea-saying effects arrive at a two-dimensional representation of the target trait. These findings demonstrate that the contextual information provided by an item scale can serve as a determinant of differential item functioning.

Recent applications of item response tree models demonstrate that this model class is well suited to detect midpoint and extremity response style effects in both attitudinal and personality measurements. This paper proposes an extension of this approach that goes beyond measuring response styles and allows us to examine item-feature effects. In a reanalysis of three published data sets, it is shown that the proposed extension captures item-feature effects across affirmative and reverse-worded items in a psychological test. These effects are found to affect directional responses but not midpoint and extremity preferences. Moreover, accounting for item-feature effects substantially improves model fit and interpretation of the construct measurement. The proposed extension can be implemented readily with current software programs that facilitate maximum likelihood estimation of item response models with missing data., (© 2019 The British Psychological Society.)

We introduce multilevel multivariate meta-analysis methodology designed to account for the complexity of contemporary psychological research data. Our methodology directly models the observations from a set of studies in a manner that accounts for the variation and covariation induced by the facts that observations differ in their dependent measures and moderators and are nested within, for example, papers, studies, groups of subjects, and study conditions. Our methodology is motivated by data from papers and studies of the choice overload hypothesis. It more fully accounts for the complexity of choice overload data relative to two prior meta-analyses and thus provides richer insight. In particular, it shows that choice overload varies substantially as a function of the six dependent measures and four moderators examined in the domain and that there are potentially interesting and theoretically important interactions among them. It also shows that the various dependent measures have differing levels of variation and that levels up to and including the highest (i.e., the fifth, or paper, level) are necessary to capture the variation and covariation induced by the nesting structure. Our results have substantial implications for future studies of choice overload.

The recently proposed class of item response tree models provides a flexible framework for modeling multiple response processes. This feature is particularly attractive for understanding how response styles may affect answers to attitudinal questions. Facilitating the disassociation of response styles and attitudinal traits, item response tree models can provide powerful process tests of how different response formats may affect the measurement of substantive traits. In an empirical study, 3 response formats were used to measure the 2-dimensional Personal Need for Structure traits. Different item response tree models are proposed to capture the response styles for each of the response formats. These models show that the response formats give rise to similar trait measures but different response-style effects. (PsycINFO Database Record, ((c) 2017 APA, all rights reserved).)

Two different item response theory model frameworks have been proposed for the assessment and control of response styles in rating data. According to one framework, response styles can be assessed by analysing threshold parameters in Rasch models for ordinal data and in mixture-distribution extensions of such models. A different framework is provided by multi-process item response tree models, which can be used to disentangle response processes that are related to the substantive traits and response tendencies elicited by the response scale. In this tutorial, the two approaches are reviewed, illustrated with an empirical data set of the two-dimensional 'Personal Need for Structure' construct, and compared in terms of multiple criteria. Mplus is used as a software framework for (mixed) polytomous Rasch models and item response tree models as well as for demonstrating how parsimonious model variants can be specified to test assumptions on the structure of response styles and attitude strength. Although both frameworks are shown to account for response styles, they differ on the quantitative criteria of model selection, practical aspects of model estimation, and conceptual issues of representing response styles as continuous and multidimensional sources of individual differences in psychological assessment., (© 2017 The British Psychological Society.)

We review and evaluate selection methods, a prominent class of techniques first proposed by Hedges (1984) that assess and adjust for publication bias in meta-analysis, via an extensive simulation study. Our simulation covers both restrictive settings as well as more realistic settings and proceeds across multiple metrics that assess different aspects of model performance. This evaluation is timely in light of two recently proposed approaches, the so-called p-curve and p-uniform approaches, that can be viewed as alternative implementations of the original Hedges selection method approach. We find that the p-curve and p-uniform approaches perform reasonably well but not as well as the original Hedges approach in the restrictive setting for which all three were designed. We also find they perform poorly in more realistic settings, whereas variants of the Hedges approach perform well. We conclude by urging caution in the application of selection methods: Given the idealistic model assumptions underlying selection methods and the sensitivity of population average effect size estimates to them, we advocate that selection methods should be used less for obtaining a single estimate that purports to adjust for publication bias ex post and more for sensitivity analysis-that is, exploring the range of estimates that result from assuming different forms of and severity of publication bias., (© The Author(s) 2016.)

Statistical power and thus the sample size required to achieve some desired level of power depend on the size of the effect of interest. However, effect sizes are seldom known exactly in psychological research. Instead, researchers often possess an estimate of an effect size as well as a measure of its uncertainty (e.g., a standard error or confidence interval). Previous proposals for planning sample sizes either ignore this uncertainty thereby resulting in sample sizes that are too small and thus power that is lower than the desired level or overstate the impact of this uncertainty thereby resulting in sample sizes that are too large and thus power that is higher than the desired level. We propose a power-calibrated effect size (PCES) approach to sample size planning that accounts for the uncertainty associated with an effect size estimate in a properly calibrated manner: sample sizes determined on the basis of the PCES are neither too small nor too large and thus provide the desired level of power. We derive the PCES for comparisons of independent and dependent means, comparisons of independent and dependent proportions, and tests of correlation coefficients. We also provide a tutorial on setting sample sizes for a replication study using data from prior studies and discuss an easy-to-use website and code that implement our PCES approach to sample size planning., ((c) 2016 APA, all rights reserved).)

The conventional randomized response design is unidimensional in the sense that it measures a single dimension of a sensitive attribute, like its prevalence, frequency, magnitude, or duration. This paper introduces a multidimensional design characterized by categorical questions that each measure a different aspect of the same sensitive attribute. The benefits of the multidimensional design are (i) a substantial gain in power and efficiency, and the potential to (ii) evaluate the goodness-of-fit of the model, and (iii) test hypotheses about evasive response biases in case of a misfit. The method is illustrated for a two-dimensional design measuring both the prevalence and the magnitude of social security fraud.

Statistical power depends on the size of the effect of interest. However, effect sizes are rarely fixed in psychological research: Study design choices, such as the operationalization of the dependent variable or the treatment manipulation, the social context, the subject pool, or the time of day, typically cause systematic variation in the effect size. Ignoring this between-study variation, as standard power formulae do, results in assessments of power that are too optimistic. Consequently, when researchers attempting replication set sample sizes using these formulae, their studies will be underpowered and will thus fail at a greater than expected rate. We illustrate this with both hypothetical examples and data on several well-studied phenomena in psychology. We provide formulae that account for between-study variation and suggest that researchers set sample sizes with respect to our generally more conservative formulae. Our formulae generalize to settings in which there are multiple effects of interest. We also introduce an easy-to-use website that implements our approach to setting sample sizes. Finally, we conclude with recommendations for quantifying between-study variation., (© The Author(s) 2014.)

Asking sensitive or personal questions in surveys or experimental studies can both lower response rates and increase item non-response and misreports. Although non-response is easily diagnosed, misreports are not. However, misreports cannot be ignored because they give rise to systematic bias. The purpose of this paper is to present a modeling approach that identifies misreports and corrects for them. Misreports are conceptualized as a motivated process under which respondents edit their answers before they report them. For example, systematic bias introduced by overreports of socially desirable behaviors or underreports of less socially desirable ones can be modeled, leading to more-valid inferences. The proposed approach is applied to a large-scale experimental study and shows that respondents who feel powerful tend to overclaim their knowledge.

In this article, I show how item response models can be used to capture multiple response processes in psychological applications. Intuitive and analytical responses, agree-disagree answers, response refusals, socially desirable responding, differential item functioning, and choices among multiple options are considered. In each of these cases, I show that the response processes can be measured via pseudoitems derived from the observed responses. The estimation of these models via standard software programs that allow for missing data is also discussed. The article concludes with two detailed applications that illustrate the prevalence of multiple response processes., (PsycINFO Database Record (c) 2013 APA, all rights reserved.)

Loss aversion and reference dependence are 2 keystones of behavioral theories of choice, but little is known about their underlying cognitive processes. We suggest an additional account for loss aversion that supplements the current account of the value encoding of attributes as gains or losses relative to a reference point, introducing a value construction account. Value construction suggests that loss aversion results from biased evaluations during information search and comparison processes. We develop hypotheses that identify the influence of both accounts and examine process-tracing data for evidence. Our data suggest that loss aversion is the result of the initial direct encoding of losses that leads to the subsequent process of directional comparisons distorting attribute valuations and the final choice.

Randomized response is a misclassification design to estimate the prevalence of sensitive behaviour. Respondents who do not follow the instructions of the design are considered to be cheating. A mixture model is proposed to estimate the prevalence of sensitive behaviour and cheating in the case of a dual sampling scheme with direct questioning and randomized response. The mixing weight is the probability of cheating, where cheating is modelled separately for direct questioning and randomized response. For Bayesian inference, Markov chain Monte Carlo sampling is applied to sample parameter values from the posterior. The model makes it possible to analyse dual sample scheme data in a unified way and to assess cheating for direct questions as well as for randomized response questions. The research is illustrated with randomized response data concerning violations of regulations for social benefit.

Current psychometric models of choice behavior are strongly influenced by Thurstone's (1927, 1931) experimental and statistical work on measuring and scaling preferences. Aided by advances in computational techniques, choice models can now accommodate a wide range of different data types and sources of preference variability among respondents induced by such diverse factors as person-specific choice sets or different functional forms for the underlying utility representations. At the same time, these models are increasingly challenged by behavioral work demonstrating the prevalence of choice behavior that is not consistent with the underlying assumptions of these models. I discuss new modeling avenues that can account for such seemingly inconsistent choice behavior and conclude by emphasizing the interdisciplinary frontiers in the study of choice behavior and the resulting challenges for psychometricians.

L. L. Thurstone's (1927) model provides a powerful framework for modeling individual differences in choice behavior. An overview of Thurstonian models for comparative data is provided, including the classical Case V and Case III models as well as more general choice models with unrestricted and factor-analytic covariance structures. A flow chart summarizes the model selection process. The authors show how to embed these models within a more familiar structural equation modeling (SEM) framework. The different special cases of Thurstone's model can be estimated with a popular SEM statistical package, including factor analysis models for paired comparisons and rankings. Only minor modifications are needed to accommodate both types of data. As a result, complex models for comparative judgments can be both estimated and tested efficiently., (Copyright 2005 APA, all rights reserved.)

Markov models provide a general framework for analyzing and interpreting time dependencies in psychological applications. Recent work extended Markov models to the case of latent states because frequently psychological states are not directly observable and subject to measurement error. This article presents a further generalization of latent Markov models to allow for the analysis of rating data that are collected at arbitrary points in time. This extension offers new ways of investigating change processes by focusing explicitly on the durations that are spent in latent states. In an experience sampling application the author shows that such duration analyses can provide valuable insights about chronometric features of emotions., (Copyright 2005 APA, all rights reserved.)

Although comparative judgment methods have a number of distinct advantages over ratings, they share one common problem: On the basis of comparative judgments, it is not possible to recover the origin of item evaluations. One item may be judged more positively than another, but this result does not allow any conclusions about whether either of the items are attractive or unattractive. This article discusses the implications of this limitation for the interpretation of individual differences in comparative judgments. It also presents 3 different methods that may allow determination of the scale origin using a nested model comparison approach. An application illustrates the proposed approach as well as the benefits of determining the scale origin in understanding value judgments., (((c) 2004 APA, all rights reserved).)

Multivariate count data are commonly analysed by using Poisson distributions with varying intensity parameters, resulting in a random-effects model. In the analysis of a data set on the frequency of different emotion experiences we find that a Poisson model with a single random effect does not yield an adequate fit. An alternative model that requires as many random effects as emotion categories requires high-dimensional integration and the estimation of a large number of parameters. As a solution to these computational problems, we propose a factor-analytic Poisson model and show that a two-dimensional factor model fits the reported data very well. Moreover, it yields a substantively satisfactory solution: one factor describing the degree of pleasantness and unpleasantness of emotions and the other factor describing the activation levels of the emotions. We discuss the incorporation of covariates to facilitate rigorous tests of the random-effects structure. Marginal maximum likelihood methods lead to straight-forward estimation of the model, for which goodness-of-fit tests are also presented.

Ratings of affect words are the most commonly used method to assess pleasant affect (PA) and unpleasant affect (UA). The reliance on self-reports would be problematic if affect ratings were heavily influenced by response styles. Several recent publications have indeed suggested (a) that the influence of response styles on affect ratings is pervasive, (b) that this influence can be controlled by variations of the response format using multitrait-multimethod models, and (c) the discriminant validity of PA and UA is spurious. In this article, we examined the evidence for these claims. We demonstrate that (a) response styles have a negligible effect on affect ratings, (b) multiple response formats produce the same results as a single response format, and (c) the discriminant validity of PA and UA is not a method artifact. Rather, evidence against discriminant validity is due to the use of inappropriate response formats that respondents interpreted as bipolar scales.

Thurstonian models provide a flexible framework for the analysis of multiple paired comparison judgments because they allow a wide range of hypotheses about the judgments' mean and covariance structures to be tested. However, applications have been limited to a large extent by the computational intractability involved in fitting this class of models. This paper demonstrates that the Monte Carlo EM algorithm facilitates maximum likelihood estimation of Thurstonian paired comparison models even when the number of items is large. A paired comparison study is presented in detail to illustrate the estimation approach.

The method of paired comparisons belongs to a small group of techniques that provide explicit information about the consistency of individual and aggregated choices. This article investigates the link between the individual- and group-level judgments by extending R. D. Luce's (1959) model, which was originally developed for individual choice behavior, to a mixed-effects paired comparison model. It is shown that standard multilevel software for binary data can be used to estimate the model. The interpretation of the paired comparison parameters and statistical model tests are discussed in detail. An extensive analysis of an experimental study illustrates the usefulness of a hierarchical approach in modeling multiple pairwise judgments.

Although few would dispute the usefulness of looking at behavioral change from a stage-sequential perspective, until recently the lack of appropriate modeling techniques has hampered rigorous empirical tests of stage theories. In particular, for behavioral measurements that are ordinal, there is a need for methods that represent the underlying change processes in the form of qualitative and discontinuous shifts. This article introduces a stage-sequential ordinal model by postulating that at any point in time there are a finite number of latent stages. Panel members may shift among these stages over time. The authors show that the stage-sequential model provides a general approach for both the analysis of ordinal time-dependent data and tests of various competing theories and hypotheses about psychological change processes. An analysis of a 5-year study concerning attitudes toward alcohol consumption by teenagers is presented to illustrate the modeling approach.