Search

I use a path/confirmatory factor analytic framework to present an under‐the‐hood introduction to (a) the effects of measurement error and method variance on observed measures, (b) the effects of common latent Trait and Method factors on relationships between observed measures in the presence of measurement error and (c) examine implications of developments presented here for the possible reinterpretation of Campbell and Fiske's original criteria for convergent and discrimainat validity and presence of common method bias. I use results from one previously published and often‐cited primary study and four meta‐analyses of multitrait–multimethod studies to provide empirical examples to illustrate points made here. I show analytically that (a) some common interpretations of method effects are correct, (b) others are incorrect and (c) still others are more nuanced than is usually recognised. Finally, I offer some suggested directions for future research. [ABSTRACT FROM AUTHOR]

Anecdotal evidence has suggested that rater-based measures (e.g., parent report) may have strong across-trait/within-individual covariance that detracts from trait-specific measurement precision; rater measurement-related bias may help explain poor correlation within Autism Spectrum Disorder (ASD) samples between rater-based and performance-based measures of the same trait. We used a multi-trait, multi-method approach to examine method-associated bias within an ASD sample (n = 83). We examined performance/rater-instrument pairs for attention, inhibition, working memory, motor coordination, and core ASD features. Rater-based scores showed an overall greater methodology bias (57% of variance in score explained by method), while performance-based scores showed a weaker methodology bias (22%). The degree of inter-individual variance explained by method alone substantiates an anecdotal concern associated with the use of rater measures in ASD. [ABSTRACT FROM AUTHOR]

Masstige marketing is a strategic word for market penetration for premium but reachable products based on brand equity, trying to develop brand awareness, likability, affection and attachment. Hence, masstige scale may allow firms to measure brand equity to derive insights into the popularity of their brands. However, there is no empirical evidence available to test whether these scales are related measures of brand equity and, at the same time, independent measures, respectively. This study investigates whether the masstige scale and multidimensional consumer‐based brand equity scale measure the same constructs. A total of 493 participants evaluated four different athletic shoe brands. The multi‐trait, multimethod and confirmatory factor analyses suggested that the masstige scale may be a viable alternative to consumer‐based brand equity and masstige value. We discuss the implications and provide directions for future research derived from our findings. [ABSTRACT FROM AUTHOR]

Income redistribution is determined, to some extent, by how fair citizens judge the income distribution in their societies to be. Nonetheless, there seems to be a contradiction between what people declare as a fair income distribution, and the increase in inequalities across countries. An unexplored methodological reason for that is that survey instruments do not perfectly capture individual's perception of income fairness, biasing results. Therefore, in this paper we use data from a Multitrait-Multimethod experiment conducted in wave 6 of the probability-based CROss-National Online Survey (CRONOS) panel in Great Britain, Estonia and Slovenia. Specifically, we explore the effect of three different scales on how fair people report an income to be, and the subsequent measurement quality of these answers. Overall, we find that survey scales do have an impact on what people report as a fair income, and the quality of these answers. Specifically, we find that the use of branching scales increases participants' likelihood of considering an income distribution as "Fair", while using partial-labelling and visual clues to separate (fair/unfair) dimensions increases the likelihood of considering these as "extremely unfair high/low." In addition, our results suggest that using a 9-point fully labelled unfolding scale without visual clues yields the best measurement quality across all countries, being preferred over the other tested methods (9-point partially labelled unfolding scale with visual clues; 9-point fully labelled branching scale without visual clues). [ABSTRACT FROM AUTHOR]

A focus on implementing social and emotional (SE) learning into curricula continues to gain popularity in K-12 educational contexts at the policy and practitioner levels. As it continues to be elevated in educational discourse, it becomes increasingly clear that it is important to have reliable, validated measures of students' SE skills. Here we argue that framework and design are additional important considerations for the development and selection of SE skill assessments. We report the reliability and validity evidence for The Mosaic™ by ACT® Social Emotional Learning Assessment, an assessment designed to measure SE skills in middle and high school students that makes use of a research-based framework (the Big Five) and a multi-method approach (three item types including Likert, forced choice, and situational judgment tests). Here, we provide the results from data collected from more than 33,000 students who completed the assessment and for whom we have data on various outcome measures. We examined the validity evidence for the individual item types and the aggregate scores based on those three. Our findings support the contribution of multi-method assessment and an aggregate score. We discuss the ways the field can benefit from this or similarly designed assessments and discuss how the assessment results can be used by practitioners to promote programs aimed at stimulating students' personal growth. [ABSTRACT FROM AUTHOR]

Self-report and monomethod bias threaten the validity of positive work and organizations research. The current study used multitrait-multimethod analyses to examine the relationship between PERMA+4 and work role performance (adaptivity, proactivity, and proficiency) beyond self-report and monomethod bias. Findings from 221 coworker pairs demonstrated convergence between self-reported and collateral-reported PERMA+4 (r >.85) and work role performance (r >.85). A multitrait-multimethod matrix showed that monomethod trait interrelationships were systematically higher than heteromethod trait interrelationships with inflated correlations ranging from.07 to.20. Monomethod parameter estimates and coefficients of determination were generally higher than in bias corrected self-reports, knowledgeable collateral-reports, and heteromethod parameter estimates. The results provide support for the validity of the relationship between PERMA+4 and work role performance, but also suggest the importance of including procedural design and statistical control methods in positive work and organizations surveys to correct for self-report and monomethod bias. [ABSTRACT FROM AUTHOR]

The development of improved methods for genome‐wide association studies (GWAS) for genetics of quantitative traits has been an active area of research during the last 25 years. This activity initially started with the use of mixed linear model (MLM), which was variously modified. During the last decade, however, with the availability of high throughput next generation sequencing (NGS) technology, development and use of pangenomes and novel markers including structural variations (SVs) and k‐mers for GWAS has taken over as a new thrust area of research. Pangenomes and SVs are now available in humans, livestock, and a number of plant species, so that these resources along with k‐mers are being used in GWAS for exploring additional genetic variation that was hitherto not available for analysis. These developments have resulted in significant improvement in GWAS methodology for detection of marker‐trait associations (MTAs) that are relevant to human healthcare and crop improvement. [ABSTRACT FROM AUTHOR]

Survey data is used in many social science studies. The measurement quality of these data is crucial as it determines the accuracy of the information on which these studies are based. Besides, since these studies are used to provide insights to key political and social actors, it also determines the accuracy of the information on which crucial decisions are based. In this paper, we estimated the measurement quality (proportion of the variance of the observed survey responses explained by the latent trait of interest) of 67 common social sciences questions that were part of Multitrait-Multimethod experiments in the seven first rounds of the European Social Survey. These questions were asked using response scales with different characteristics and in up to 41 country-language groups. Our results show that measurement errors are omnipresent: the average measurement quality across all questions is 0.65. Thus, overall, on average 35% of the variance in the observed survey answers can be attributed to measurement errors. Furthermore, the size of errors varies across questions as well as across country-language groups. The questions’ average measurement quality across all country-language groups ranges from 0.25 to 0.88, depending on the response scale and topic, and the country-language groups’ average measurement quality across questions ranges from 0.52 to 0.76. Thus, the impact of measurement errors on applied research can be different depending on the exact question formulation and response scale used as well as on the country and language of interest. Consequently, in each study, researchers should consider assessing the size of the measurement errors of their variables and how this affects their results.

A focus on implementing social and emotional (SE) learning into curricula continues to gain popularity in K-12 educational contexts at the policy and practitioner levels. As it continues to be elevated in educational discourse, it becomes increasingly clear that it is important to have reliable, validated measures of students’ SE skills. Here we argue that framework and design are additional important considerations for the development and selection of SE skill assessments. We report the reliability and validity evidence for The Mosaic™ by ACT® Social Emotional Learning Assessment, an assessment designed to measure SE skills in middle and high school students that makes use of a research-based framework (the Big Five) and a multi-method approach (three item types including Likert, forced choice, and situational judgment tests). Here, we provide the results from data collected from more than 33,000 students who completed the assessment and for whom we have data on various outcome measures. We examined the validity evidence for the individual item types and the aggregate scores based on those three. Our findings support the contribution of multi-method assessment and an aggregate score. We discuss the ways the field can benefit from this or similarly designed assessments and discuss how the assessment results can be used by practitioners to promote programs aimed at stimulating students’ personal growth.

Although the approach to correct for measurement errors in research has been known since the early 1970s, most researchers in the social sciences appear to ignore these recommendations. One possible reason is that the models with latent variables that were originally suggested may be too difficult to apply in practice. We suggest an alternative but simpler procedure to correct for measurement errors. If one knows the sizes of the random errors and method effects for the different measures, the correlation and covariance matrices for the observed variables can be corrected. This approach has already been shown for simple concepts measured by a single measure. In this paper, we show that this approach can be generalized to studies that use a mixture of simple and complex concepts measured by several questions. Using these corrected matrices as the basis for the estimation, the parameters of the models are automatically corrected for measurement errors. The conditions for the quality of these procedures will also be discussed. [ABSTRACT FROM AUTHOR]

Emotional and behavioral problems among children and adolescents may be studied using the Strengths and Difficulties Questionnaire, containing five subscales, based on ratings by parents, teachers, or adolescents themselves. We investigate two measurement issues using data from a longitudinal sample of 8,806 participants aged 7 to 9 years and 11 to 13 years from the Bergen Child Study in Bergen, Norway. First, convergent validity of parent and teacher ratings is studied using a multitrait–multimethod approach. Second, longitudinal measurement equivalence is studied using confirmatory factor analysis, which requires us to deal with the considerable attrition. The multitrait–multimethod indicates not only good convergent validity but also considerable method variance for parents and teachers. The reliability and validity of some subscales are relatively low. Attrition analysis indicates that attrition is not missing completely at random, but estimation assuming missing at random makes no real difference. We conclude that assuming missing completely at random is acceptable. Comparing ratings by parents and teachers results in partial scalar equivalence. In addition, all subscales exhibit (partial) longitudinal scalar measurement equivalence. We recommend using latent variable modeling and not summated scales for longitudinal modeling using the Strengths and Difficulties Questionnaire. [ABSTRACT FROM AUTHOR]

Students’ school well-being, social inclusion, and academic self-concept are considered important outcome variables of schools. In the present study, these three variables were examined from teachers’ and students’ perspective (grades 5–9). The aim of the study was to investigate the construct validity (convergent and discriminant validity) of the teacher’s version of the Perception of Inclusion Questionnaire (PIQ). Further, we investigated whether or not it is meaningful to include the perspective of a second teacher. The dataset consists of PIQ ratings of 151 students as well as ratings from two main subject teachers. The results for psychometric properties show that the students’ as well as the teachers’ version of the PIQ is suitable for secondary school students. The confirmatory factor analyses demonstrated good model fit for the three-dimensional factorial structure. By excluding one teacher’s rating from the model, the multitrait-multimethod analysis provided indicators for the PIQ’s construct validity (convergent and discriminant validity) of the traits and discriminant validity of the methods.

One of the central objectives of inclusive education, and education in general, is not only to support every students' academic learning, but also their social and emotional development. It therefore is important to identify difficulties in a child's socio-emotional development at school. The current study investigates students' emotional inclusion and social inclusion, as well as students' academic self-concept from four different perspectives using the Perceptions of Inclusion Questionnaire (PIQ). In particular, we analyzed the degree of agreement between teacher, mother, and father ratings with students' self-reports. Moreover, we tested if students' gender and special educational needs (SEN) are predictors for possible bias in parent and teacher reports. Survey participants included 721 Austrian, Grade 4 students from 48 classes. In addition, data from 46 teachers, 466 mother reports, and 375 father reports were included. We assessed the consistency (i.e., agreement) between the different raters by means of multitrait-multimethod analyses, or more precisely, a correlated trait–correlated method minus one (CT-C[M-1]) model. Results of the CT-C(M-1) analyses indicated a rather strong rater bias (i.e., method effects) for all three dimensions of inclusion. However, the consistency for academic self-concept was higher than for emotional and social inclusion. Furthermore, gender and SEN status affected rater bias, particularly for teacher reports. Results suggested that it matters who reports students' emotional inclusion, social inclusion, and academic self-concept, which has methodological and practical implications. [ABSTRACT FROM AUTHOR]

Uniqueness is a fundamental aspect of individual identity, yet commonly used conceptualizations of uniqueness are based on the contrast between an individual and other people, an understanding that is not congruent with person-centred definitions from humanistic approaches. This study is based on the idea that uniqueness is concerned with the acceptance of one's existence and uses Şimsek and Yalınçetin's (Personality and Individual Differences, 49, 576–581, 2010) conceptualization, namely, a personal sense of uniqueness. Relying on both self- and observer reports, we examined the mediating role of authenticity in the relationship between a personal sense of uniqueness and happiness. This study also provides an extension of previous research by furthering the understanding of how dimensions of authenticity are linked to well-being. In line with our hypotheses, we found that a personal sense of uniqueness was positively related to authentic living and negatively related to self-alienation. Our results also showed a negative correlation between self-alienation and happiness and a positive correlation between authentic living and happiness. Self-alienation, a core dimension of authenticity, mediated the relationship between a personal sense of uniqueness and happiness. [ABSTRACT FROM AUTHOR]

Students' school well-being, social inclusion, and academic self-concept are considered important outcome variables of schools. In the present study, these three variables were examined from teachers' and students' perspective (grades 5–9). The aim of the study was to investigate the construct validity (convergent and discriminant validity) of the teacher's version of the Perception of Inclusion Questionnaire (PIQ). Further, we investigated whether or not it is meaningful to include the perspective of a second teacher. The dataset consists of PIQ ratings of 151 students as well as ratings from two main subject teachers. The results for psychometric properties show that the students' as well as the teachers' version of the PIQ is suitable for secondary school students. The confirmatory factor analyses demonstrated good model fit for the three-dimensional factorial structure. By excluding one teacher's rating from the model, the multitrait-multimethod analysis provided indicators for the PIQ's construct validity (convergent and discriminant validity) of the traits and discriminant validity of the methods. [ABSTRACT FROM AUTHOR]

Self-compassion is natural, trainable and multi-faceted human capacity. To date there has been little research into the role of culture in influencing the conceptual structure of the underlying construct, the relative importance of different facets of self-compassion, nor its relationships to cultural values. This study employed a cross-cultural design, with 4,124 participants from 11 purposively sampled datasets drawn from different countries. We aimed to assess the relevance of positive and negative items when building the self-compassion construct, the convergence among the self-compassion components, and the possible influence of cultural values. Each dataset comprised undergraduate students who completed the “Self-Compassion Scale” (SCS). We used a confirmatory factor analysis (CFA) approach to the multitrait-multimethod (MTMM) model, separating the variability into self-compassion components (self-kindness, common humanity, mindfulness), method (positive and negative valence), and error (uniqueness). The normative scores of the Values Survey Module (VSM) in each country, according to the cultural dimensions of individualism, masculinity, power distance, long-term orientation, uncertainty avoidance, and indulgence, were considered. We used Spearman coefficients (rs) to assess the degree of association between the cultural values and the variance coming from the positive and negative items to explain self-compassion traits, as well as the variance shared among the self-compassion traits, after removing the method effects produced by the item valence. The CFA applied to the MTMM model provided acceptable fit in all the samples. Positive items made a greater contribution to capturing the traits comprising self-compassion when the long-term orientation cultural value was higher (rs = 0.62; p = 0.042). Negative items did not make significant contributions to building the construct when the individualism cultural value was higher, but moderate effects were found (rs = 0.40; p = 0.228). The level of common variance among the self-compassion trait factors was inversely related to the indulgence cultural value (rs = -0.65; p = 0.030). The extent to which the positive and negative items contribute to explain self-compassion, and that different self-compassion facets might be regarded as reflecting a broader construct, might differ across cultural backgrounds.

Abstract It remains unresolved whether a dark core "D" underlies the dark triad that exceeds the unique meaning of its individual sub-dimensions (Machiavellianism, psychopathy, narcissism). Based on life history theory, we argue in favor of the existence of "D". According to life history theory, individuals high in "D" pursue a fast life history strategy which tends to provide them with short-term reproductive benefits. In a German sample (N = 395), we use a multitrait-multimethod approach integrating standard measures of the dark triad (Mach IV, SRP-III, NPI) with the Short Dark Triad scale and the Dirty Dozen. We compare a three-dimensional measurement model to a bifactor model and find that accounting for "D" significantly improves model fit. Moreover, we extend the dark triad's nomological network onto core self-evaluations and also include self-esteem and life satisfaction. Results show that "D" relates negatively to life satisfaction and core self-evaluations (even after accounting for the dark triad's sub-dimensions) but not to self-esteem. Our findings are consistent with life history theory, considering that distal outcomes such as life satisfaction suffer most from fast life history strategies. We conclude that the dark triad has a dark core. Highlights • Model comparisons suggest that the dark triad has a dark core "D". • Standard measures, Short Dark Triad, and Dirty Dozen all load on "D" (λs > 0.39). • "D" relates negatively to life satisfaction and core self-evaluations. • Our models test dark triad's internal structure at the construct level. [ABSTRACT FROM AUTHOR]

The authors' purpose was to evaluate the multiple-gate Preschool Behavior Screening System (PBSS) for identifying children's social, emotional, and behavioral (SEB) difficulties. The sample included 122 triads of preschool children between 3 and 5 years old and their parents and teachers. The teacher and parent samples were predominantly women and European American with a substantial African American representation. The design involved numerous indicators of the reliability of scores and the validity of ensuing inferences. Scores from the Behavior Assessment System for Children, Second Edition were used as criterion measures for scores from the PBSS. The PBSS scores were found reliable across scales with Cronbach's alpha values ranging from.85 to.97. Agreement between teachers and parents was consistent with findings from previous research. Analyzed within a Multitrait-Multimethod Matrix, correlational evidence supported inferences about externalizing difficulties, prosocial behavior, and total problems, as well as internalizing difficulties to a lesser degree. Receiver-operating characteristic curve analyses supported the PBSS across a range of scales and cut scores. Findings support the use of the PBSS as one method for identifying preschool children who may be at risk for SEB difficulties and underscore the importance of using multiple informants in the assessment of SEB difficulties. [ABSTRACT FROM AUTHOR]

This paper proposes a method to simultaneously estimate both measurement and nonresponse errors for attitudinal and behavioural questions. The method uses a Multi-Trait Multi-Method (MTMM) approach, which is commonly used to estimate the reliability and validity of survey questions. The classic MTMM model is in this paper extended to include the effects of measurement bias and longitudinal nonresponse. Measurement and nonresponse errors are expressed on a common metric in this model, so that their relative sizes can be assessed over the course of a panel study. Using an example about political trust from the Dutch LISS panel, we show that measurement problems lead to both small errors and small biases, that dropout in the panel study does not lead to errors or bias, and that therefore, measurement is a more important source of both error and bias than nonresponse.

School-level contextual factors have been found to influence reports of school climate. The purpose of the current study was to evaluate the extent to which these associations are related to the school climate traits being measured or the methods (i.e., informants) used to obtain them. Data from a multilevel multitrait-multimethod (MTMM) design in which structurally different and interchangeable students (n = 45,641) and teachers (n = 12,808), residing within 302 high schools, responded to items measuring four dimensions of school climate were evaluated through a multilevel correlated trait-correlated method (CT-CM) latent analysis that allowed for the estimation of both school-level trait and informant-based method factors. The resulting trait and method factors were regressed on several school-level contextual variables. Results indicated that the percentage of students receiving free and reduced-price meals (FRPM) in schools was associated with both school climate traits and informant-based method factors, school size and the percentage of minority students in schools were associated with some traits, and school size was associated with student method effects. Findings support the use of controlling for school-level contextual factors in school climate research. [ABSTRACT FROM AUTHOR]

Reliable and valid data form the foundation for evidence-based practices, yet surprisingly few studies on school-based behavioral assessments have been conducted which implemented one of the most fundamental approaches to construct validation, the multitrait-multimethod matrix (MTMM). To this end, the current study examined the reliability and validity of data derived from three commonly utilized school-based behavioral assessment methods: Direct Behavior Rating – Single Item Scales, systematic direct observations, and behavior rating scales on three common constructs of interest: academically engaged, disruptive, and respectful behavior. Further, this study included data from different sources including student self-report, teacher report, and external observers. A total of 831 students in grades 3–8 and 129 teachers served as participants. Data were analyzed using bivariate correlations of the MTMM, as well as single and multi-level structural equation modeling. Results suggested the presence of strong methods effects for all the assessment methods utilized, as well as significant relations between constructs of interest. Implications for practice and future research are discussed. [ABSTRACT FROM AUTHOR]

The correlated trait–correlated method (CTCM) model for the analysis of multitrait–multimethod (MTMM) data is known to suffer convergence and admissibility (C&A) problems. We describe a little known and seldom applied reparameterized version of this model (CTCM-R) based on Rindskopf’s reparameterization of the simpler confirmatory factor analysis model. In a Monte Carlo study, we compare the CTCM, CTCM-R, and the correlated trait–correlated uniqueness (CTCU) models in terms of C&A, model fit, and parameter estimation bias. The CTCM-R model largely avoided C&A problems associated with the more traditional CTCM model, producing C&A solutions nearly as often as the CTCU model, but also avoiding parameter estimation biases known to plague the CTCU model. As such, the CTCM-R model is an attractive alternative for the analysis of MTMM data. [ABSTRACT FROM AUTHOR]

This article compares maximum likelihood and Bayesian estimation of the correlated trait–correlated method (CT–CM) confirmatory factor model for multitrait–multimethod (MTMM) data. In particular, Bayesian estimation with minimally informative prior distributions—that is, prior distributions that prescribe equal probability across the known mathematical range of a parameter—are investigated as a source of information to aid convergence. Results from a simulation study indicate that Bayesian estimation with minimally informative priors produces admissible solutions more often maximum likelihood estimation (100.00% for Bayesian estimation, 49.82% for maximum likelihood). Extra convergence does not come at the cost of parameter accuracy; Bayesian parameter estimates showed comparable bias and better efficiency compared to maximum likelihood estimates. The results are echoed via 2 empirical examples. Hence, Bayesian estimation with minimally informative priors outperforms enables admissible solutions of the CT–CM model for MTMM data. [ABSTRACT FROM PUBLISHER]

It is increasingly acknowledged that creative potential involves partly a generalized ability, partly a set of domain-specific abilities, and partly a set of task-specific abilities. We extend and illustrate this view in a study of 482 children and adolescents, exploring the extent to which the scores variance of the Evaluation of Potential Creativity (EPoC)'s eight subtests can be decomposed by five variance components: thinking-process general, thinking-process specific, domain-specific, task-specific, and measurement error. A structural equation model derived from an extension of the multi-trait multi-method matrix analysis revealed that (1) the contribution of each variance component depends greatly on the task under consideration, and that (2) the contribution of a general creative thinking-process factor is overall limited. This study outlines the multidimensional and hierarchical structure of creative potential and the need to measure it with comprehensive test batteries sampling a range of creative tasks, domains and creative thinking-modes. [ABSTRACT FROM AUTHOR]

The paper explores the consumer value of materialism in Indian socio-cultural milieu. The construct validity of the Richins & Dawson (1992) scale of materialistic values is established and relationship of materialism with the constructs of happiness, life-satisfaction, and religiosity is assessed using data collected from 252 respondents through a self administered questionnaire. The research methods used for analysis included MTMM approach for construct validation and correlation analysis for assessment of relationships. The analysis revealed that the Richins & Dawson (1992) scale has acceptable reliability, convergent validity and discriminant validity leading to construct validation. Further, while materialistic tendencies do not influence happiness and religiosity, it actually enhances the life-satisfaction level. The results have utilitarian value in terms of target market profiling, product positioning and designing of advertising appeal.

We compared six different analytic models for multitrait–multimethod (MTMM) data in terms of convergence, admissibility, and model fit to 258 samples of previously reported data. Two well-known models, the correlated trait–correlated method (CTCM) and the correlated trait–correlated uniqueness (CTCU) models, were fit for reference purposes in comparison to four other under- or unstudied models, including (a) Rindskopf’s reparameterization of the CTCM (CTCM-R) model, (b) a correlated trait–constrained uncorrelated method model and two of its more general cases, (c) a correlated trait–constrained correlated method model, and (d) a correlated trait–uncorrelated method model. Results show that (a) the CTCM-R model often solved convergence and admissibility problems with the CTCM model at rates equivalent to the CTCU model and (b) constrained models often provided convergent and admissible solutions but significantly worse model fit, indicating that they are often not plausible when analyzing real data. A follow-up simulation study showed that the CTCM-R model also provided the most accurate estimates of the full range of parameters relevant to a confirmatory factor analytic model of MTMM data. [ABSTRACT FROM AUTHOR]

Multiple ratings are becoming increasingly popular for the assessment of a wide range of behaviors. Teaching evaluation designs and 360-degree feedback often use multiple raters alongside self-ratings. Eid and colleagues (2008) proposed a multilevel structural equation model for the analysis of data stemming from such designs that assumes that raters stem from independent populations of raters. However, it is quite common for raters to rate multiple targets, thus implying a cross-classified data structure. A simulation study was conducted to assess the effects of this rater nonindependence on parameter and standard error estimates in multilevel structural equation models. Results show parameter estimation biases to be small, whereas standard errors pertaining to the Level 1 covariance matrices as well as the mean structure on Level 2 are underestimated. [ABSTRACT FROM AUTHOR]

Each research domain carries the burden of examining the effects of common method variance (CMV) on published research within the domain. To focus on this concern in the context of the theory of planned behavior (TPB), this research empirically compares several methods of detecting the presence of and estimating the level of CMV in the TPB domain. These methods include various implementations of the marker variable technique and versions of the multitrait-multimethod (MTMM) technique. The results show that the marker variable technique provides estimates of CMV and CMV-corrected correlations and paths that are consistent with those produced using the other methods. Next, one implementation of the marker variable technique method is implemented post hoc on a large data set of published TPB studies. This analysis provides strong confirmatory evidence that the effects of CMV do not alter the substantive inferences of study results in prior research. Overall, these findings support putting to rest concerns about the adverse influence of CMV in the TPB domain. [ABSTRACT FROM AUTHOR]

Mixed-mode questionnaires are increasingly used in research. Psychological measures, developed for paper-and-pencil (paper) administration require measurement equivalence testing when administered in an alternative mode. Here, Icelandic translations of the NEO-FFI personality measure and Adult Reading History Questionnaire (ARHQ) were tested for equivalence of measurement and data quality between paper and web mode. Perceived sensitivity of data and preference for survey mode were also assessed. One hundred adults were recruited to answer both modes in a randomized, crossover design. Eighty-eight participants completed both administrations with an average of 63.8 days ( SD = 2.2) between them. Within-subjects comparisons of means between modes demonstrated measurement equivalence for both measures. However, differing invalidity coefficients by mode observed by multi-trait multi method (MTMM) analysis suggested systematic effects not captured by traditional psychometric evaluation. Of note was a greater tendency for acquiescence responding to the NEO-FFI observed in web mode. Neither personality traits nor ARHQ reading difficulty scores were associated with preference for survey mode. However, 36% of participants who considered their personality data moderately to highly sensitive scored higher in neuroticism and lower in agreeableness compared to those rating their personality data of low sensitivity. While the Icelandic NEO-FFI and ARHQ have demonstrated measurement equivalence in paper and web mode, mode equivalence of psychometric measures may require ongoing evaluation as perceptions of web privacy continue to evolve. [ABSTRACT FROM AUTHOR]

Although Objective Personality Tests (OPTs) have a long history in psychology and the field of psychological assessment, their validity, and reliability have not yet been sufficiently studied. In this study, we examined the convergent and discriminant validity of objective (personality) tests, Implicit Association Tests (IATs), and self-report measures for the assessment of conscientiousness and intelligence. Moreover, the convergent and discriminant validity of these measures was assessed on the trait (stable) and occasion specific (momentary) level by using the multimethod latent state-trait (MM-LST) model proposed by Courvoisier, Nussbeck, Eid, and Cole (2008) which allows for the decomposition of different sources of variance. Data from 367 students assessed on three different measurement occasions was incorporated. Results indicate generally low convergence of OPTs with data gained by other approaches. Additional analyses revealed that the OPTs used assess stable rather than momentary components of the constructs. Reliabilities of different tests ranged from .54 to .95. Furthermore, a substantial amount of trait method specificity revealed that different methods assess trait components that are not shared between OPTs and other measures. Data on the criterion validity of the objective conscientiousness test revealed that it is related to the punctuality of test takers in the laboratory. [ABSTRACT FROM AUTHOR]

Although agree–disagree (AD) rating scales suffer from acquiescence response bias, entail enhanced cognitive burden, and yield data of lower quality, these scales remain popular with researchers due to practical considerations (e.g., ease of item preparation, speed of administration, and reduced administration costs). This article shows that if researchers want to use AD scales, they should offer 5 answer categories rather than 7 or 11, because the latter yield data of lower quality. This is shown using data from four multitrait-multimethod experiments implemented in the third round of the European Social Survey. The quality of items with different rating scale lengths were computed and compared. [ABSTRACT FROM AUTHOR]

When designing questionnaires, many choices have to be made. Because the consequences of these choices for the quality of the questions are largely unknown, it has often been said that designing a questionnaire is an art. To make it a more scientific activity we need to know more about the consequences of these choices. In order to further such an approach we have: 1. made an inventory of the choices to be made when designing survey questions and created a code book to transform these question characteristics into the independent variables for explaining quality of survey questions; 2. assembled a large set of studies that use Multi-trait Multi-method experiments to estimate the reliability and validity of questions 3. carried out a meta-analysis that relates these question characteristics to the reliability and validity estimates of the questions. On the basis of the results of these efforts we have constructed a database. This data base contains at present 1023 measurement instruments based on 87 experiments conducted on random samples from sometimes regional but mostly national samples of 300 to 2000 respondents. The database contains information on studies of reliability and validity of survey questions formulated in three different languages: English, German and Dutch. The purpose of this study was to generate cross national generalizations of the findings published so far drawn from national studies. This analysis provides a quantitative estimate of the effects of the different choices on the reliability, validity and the method effects.

Highlights: [•] Six major critiques of the GFP are reviewed. [•] The major focus is the social desirability hypothesis. [•] We consider factor saturation, blended variable models, genetic dominance, and correlation with g. [•] In addition we delineate problems with current MTMM models. [•] We present evidence and argument which show reasonable doubt with respect to each critique. [Copyright &y& Elsevier]

We examined the psychometric properties of an experience-sampling measure of affect (PANAS) using data from self- and peer reports. A multivariate multilevel model was used to assess the reliability of the latent PANAS scales at the within- and between-person level. Findings suggest satisfying internal consistencies for self- and peer reports of affective experiences at both levels of analysis. Convergent and discriminant validity of the two affect scales were examined by means of a multilevel multitrait-multimethod approach (MLM-MTMM) indicating distinct findings at the within- and between-person level. These findings provide further insights into the structural relations between the two PANAS scales: Whereas positive and negative affect were unrelated at the between-person level; they were negatively correlated at the within-person level. Copyright © 2010 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

The aim of this study was to examine three different instruments which have been used in research conducted in the physical domain to measure mastery, performance-approach, and performance-avoidance goals. The construct validity of the assessment tools, including a determination of their convergent and discriminant validity as well as a method effect, was tested via several confirmatory factor analyses. Four hundred and fifty athletes from different sport clubs were involved in the present study. Participants were asked to fill out three different multiple achievement goal instruments as well as two external criteria scales. The results based on CFA showed that all of the three instruments measured the same latent construct of multiple achievement goals. The convergent validity as well as the discriminant validity was supported. Evidence for a limited method effect in terms of the different measures was also provided. [ABSTRACT FROM AUTHOR]

In the present study we applied a validation strategy for implicit measures like the IAT, which complements multitrait-multi-method (MTMM) analyses. As the measurement method (implicit vs. explicit) and underlying representation format (associative vs. propositional) are often confounded, the validation of implicit measures has to go beyond MTMM analysis and requires substantive theoretical models. In the present study (N = 133), we employed such a model (Hofmann, Gschwendner, Nosek, & Schmin, 2005) and investigated two moderator constructs in the realm of anxiety: specificity similarity and content similarity. In the first session, different general and specific anxiety measures were administered, among them an Implicit Association Test (IAT) general anxiety, an IAT-spider anxiety, and an IAT that assesses speech anxiety. In the second session, participants had to deliver a speech and behavioral indicators of speech anxiety were measured. Results showed that (a) implicit and explicit anxiety measures correlated significantly only on the same specification level and if they measured the same content, and (b) specific anxiety measures best predicted concrete anxious behavior. These results are discussed regarding the validation of implicit measures. [ABSTRACT FROM AUTHOR]

Abstract: The present study investigates the relationship between personality and positive affects (contentment, love, interest, and joy). Using multitrait-multimethod analysis and structural equation modeling procedures with self- and peer-report, the authors demonstrated the importance of differentiating between the positive affects. A multidimensional model fitted better than a general-factor model. Concerning the personality–affect relationship, the strongest relations were found between extraversion and both joy and love, agreeableness and love, openness to experiences and interest, and neuroticism and contentment. Thus, results suggest that it is insufficient to concentrate solely on extraversion when investigating the personality–positive affect relationship. [Copyright &y& Elsevier]

The analysis of convergent and discriminant validity is an integral part of the construct validation process. Models for analyzing the convergent and discriminant validity have typically been developed for cross-sectional data. There exist, however, only a few approaches for longitudinal data that can be applied for analyzing the construct validity of fluctuating states. In this article, the authors show how models of latent state—trait theory can be combined with models of multitrait—multimethod analysis to develop a model that allows for analyzing convergent and discriminant validity in time: the multimethod latent state—trait model. The model allows for identifying different sources of variance (trait consistency, trait-method specificity, occasion-specific consistency, occasion-specific method specificity, and unreliability). It is applied to the repeated measurement of depression and anxiety in children, which was assessed by self and teacher reports (N = 375). The application shows that the proposed models fit the data well and allow a deeper understanding of method effects in clinical assessment. [ABSTRACT FROM AUTHOR]

Abstract: Measuring single motives with multiple methods often yields low intercorrelations. This raises questions about the meaningfulness of the motive constructs and the validity of the methods. Seeking to explain these findings, theorists distinguish implicit from explicit methods and suggest that they tap somewhat different constructs. We propose an approach that argues for the coherence of the motive constructs across methods, while offering a refined conception of the implicit–explicit methodological distinction. Employing facet theory, we treat motives and methods as two facets that are crossed. Implicit and explicit methods serve as poles of a continuum on the methods facet. Based on this approach, we generate and test hypotheses concerning the structure of relations among motive indicators in multidimensional space. We test these hypotheses with ordinal MDS, reanalyzing multitrait–multimethod matrixes from five studies that examined affiliation, achievement, and power motives. Analyses reveal a correspondence between different indicators of the same motive, independent of assessment method. They also show that the implicit–explicit distinction is best viewed as ordering methods along a definable continuum. [Copyright &y& Elsevier]

Customer value and customer satisfaction are pivotal but at the same time elusive concepts in services marketing theory. This paper focuses on discussing the relationship between these two concepts. We propose operationalization by developing and testing scales, especially operational indicators, for important dimensions and drivers of the services-value construct. A multitrait-multimethod design is used to test the robustness of the operationalization. Furthermore, a cross-cultural data set is used to explore country influences using confirmatory factor analysis and structural equation models. Results indicate that the measurement construct is robust and useful in country-comparative studies. [ABSTRACT FROM AUTHOR]

The study investigated the relations of the five-factor model of personality with a faceted, a hierarchical, and a shortened measurement model for fluid and crystallized intelligence. The faceted model for fluid and crystallized intelligence contained a content facet comprising a verbal, a numerical, and a figural factor. The hierarchical model contained primary factors for verbal, numerical, and figural fluid intelligence and for verbal, numerical, and figural crystallized intelligence as well as second-order factors for fluid and crystallized intelligence. The shortened model contained a fluid-intelligence factor based on only figural reasoning tasks, and a crystallized-intelligence factor based on only verbal knowledge tasks. A total of 789 German participants ranging from 16 to 21 years of age completed the NEO Five Factors Inventory and the Intelligence Structure Test 2000 R. Substantial correlations of fluid intelligence with openness to experience were only found for the faceted model. No correlations of fluid and crystallized intelligence with personality factors occurred for the remaining measurement models. [ABSTRACT FROM AUTHOR]

Three major reviews of the assessment center (AC) construct-validity literature have disagreed as to the most appropriate analytic model for AC postexercise dimension ratings. We report a Monte Carlo study addressing the following questions: (a) To what extent does the "true" model (i.e., the model that generated the data) actually appear to fit the data well? (b) To what extent can a model appear to fit the data well even though it is the wrong model? and (c) Is model fit actually a useful empirical criterion for judging which model is most likely the pop- ulation model? Results suggest that "true" models may not always appear as the best fitting models, whereas "false" models sometimes appear to offer better fit than the true models. Implications for the study of AC construct validity are discussed. [ABSTRACT FROM AUTHOR]