Search

The effects of different response option orders on survey responses have been studied extensively. The typical research design involves examining the differences in response characteristics between conditions with the same item stems and response option orders that differ in valence--either incrementally arranged (e.g., strongly disagree to strongly agree) or decrementally arranged (e.g., strongly agree to strongly disagree). The present study added two additional experimental conditions--randomly incremental or decremental and completely randomized. All items were presented in an item-by-item format. We also extended previous studies by including an examination of response option order effects on: careless responding, correlations between focal predictors and criteria, and participant reactions, all the while controlling for false discovery rate and focusing on the size of effects. In a sample of 1,198 university students, we found little to no response option order effects on a recognized personality assessment vis-à-vis measurement equivalence, scale mean differences, item-level distributions, or participant reactions. However, the completely randomized response option order condition differed on several careless responding indices suggesting avenues for future research.

Self-report assessments are the standard for personality measurement, but motivated respondents are able to manipulate or fake their responses to typical Likert scale self-report. Although progress has been made in research seeking to reduce faking, most of it has focused on normative personality traits such as those measured by the five factor model. The measurement of socially aversive personality (e.g., the Dark Triad) is less well-researched. The negative aspects of socially aversive traits increase the opportunity and motivation of respondents to fake typical single-stimulus self-report assessments underscoring the need for faking resistant response formats. A possible way to reduce faking that has been explored in basic personality research is the use of the forced-choice response format. This study applied this method to socially aversive traits and illustrated best practices to create new multidimensional forced-choice and single-stimulus measures of socially aversive personality traits. Results indicated that participants were able to artificially alter their scores when asked to respond like an ideal job applicant, and counter to expectations, the forced-choice format did not decrease faking. Our results indicate that even when best practices are followed, forced-choice format is not a panacea for respondent faking. [ABSTRACT FROM AUTHOR]

While perceived overqualification (POQ) has received increased research attention in recent years, the identification of variables that moderate POQ-outcome relationships is critical to our understanding of how the construct affects career outcomes. This study, involving 170 full-time primary and secondary school educators in a suburban mid-Atlantic school system, found that POQ is negatively related to job satisfaction and affective commitment while positively related to turnover intentions and desire to turnover. While POQ was not significantly related to performance or continuance organizational commitment, the relationships between POQ and both performance and continuance organizational commitment were significantly moderated by the experience of career calling orientation. Generally, the relationship between POQ and performance was stronger, and the relationship between POQ and continuance organizational commitment was weaker, for those with high calling. Additionally, the effects of career calling were considerably stronger than those of POQ for all criteria. The implications surrounding these results, and opportunities for future investigation, are discussed.

When data are collected via anonymous Internet surveys, particularly under conditions of obligatory participation (such as with student samples), data quality can be a concern. However, little guidance exists in the published literature regarding techniques for detecting careless responses. Previously several potential approaches have been suggested for identifying careless respondents via indices computed from the data, yet almost no prior work has examined the relationships among these indicators or the types of data patterns identified by each. In 2 studies, we examined several methods for identifying careless responses, including (a) special items designed to detect careless response, (b) response consistency indices formed from responses to typical survey items, (c) multivariate outlier analysis, (d) response time, and (e) self-reported diligence. Results indicated that there are two distinct patterns of careless response (random and nonrandom) and that different indices are needed to identify these different response patterns. We also found that approximately 10%-12% of undergraduates completing a lengthy survey for course credit were identified as careless responders. In Study 2, we simulated data with known random response patterns to determine the efficacy of several indicators of careless response. We found that the nature of the data strongly influenced the efficacy of the indices to identify careless responses. Recommendations include using identified rather than anonymous responses, incorporating instructed response items before data collection, as well as computing consistency indices and multivariate outlier analysis to ensure high-quality data. (Contains 14 tables, 1 figure, and 3 footnotes.)

Confirmatory factor analytic tests of measurement invariance (MI) require a referent indicator (RI) for model identification. Although the assumption that the RI is perfectly invariant across groups is acknowledged as problematic, the literature provides relatively little guidance for researchers to identify the conditions under which the practice is appropriate. Using simulated data, this study examined the effects of RI selection on both scale- and item-level MI tests. Results indicated that while inappropriate RI selection has little effect on the accuracy of conclusions drawn from scale-level tests of metric invariance, poor RI choice can produce very misleading results for item-level tests. As a result, group comparisons under conditions of partial invariance are highly susceptible to problems associated with poor RI choice. (Contains 5 figures, 4 tables and 2 footnotes.)

The current study demonstrates the use of item response theory (IRT) to conduct measurement invariance analyses in careers research. A self-report survey was used to assess the importance 1,363 fourth-year medical students placed on opportunities to provide comprehensive patient care when choosing a career specialty. IRT analyses supported measurement invariance across gender. Additional analyses indicated that compared with men, women placed significantly greater importance on opportunities to provide comprehensive patient care. This in turn predicted career choice, with women being more likely than men to pursue primary care specialties. This study extends the careers literature both methodologically and substantively. Methodologically, this study exemplifies how and why to use IRT to assess measurement invariance prior to comparing groups on career-related attitudes. Substantively, this study is the first to demonstrate that the importance placed on comprehensive patient care mediates the effect of gender on intentions to pursue primary care specialties. (Contains 3 figures and 3 tables.)

This study investigates the effects of sample size, factor overdetermination, and communality on the precision of factor loading estimates and the power of the likelihood ratio test of factorial invariance in multigroup confirmatory factor analysis. Although sample sizes are typically thought to be the primary determinant of precision and power, the degree of factor overdetermination and the level of indicator communalities also play important roles. Based on these findings, no single rule of thumb regarding the ratio of sample size to number of indicators can ensure adequate power to detect a lack of measurement invariance. (Contains 8 tables, 2 figures and 6 footnotes.)

This article highlights issues associated with the use of the differential functioning of items and tests (DFIT) methodology for assessing measurement invariance (or differential functioning) with Likert-type data. Monte Carlo analyses indicate relatively low sensitivity of the DFIT methodology for identifying differential item functioning (DIF) under some conditions of differential functioning with previously recommended significance values. The differential test functioning index was extremely insensitive to differential functioning under all study conditions. The authors recommend alternative noncompensatory DIF cutoff values used to evaluate the significance of DIF for different DIF effect sizes. Additionally, contrasts between polytomous and dichotomous data are drawn, and problems with determining measurement invariance at the scale, rather than item, level for Likert scale data are highlighted. (Contains 6 tables, 6 figures and 9 notes.)

This study examined the gender-related differential predictive validity of five subscales of the Institutional Integration Scale (IIS) with regard to college student withdrawal. Differential functioning of the IIS across genders was assessed using an item response theory (IRT)-based framework of differential item and test functioning. The results confirmed the absence of differential functioning and supported the predictive validity of two of the five subscales for student withdrawal. IRT analyses revealed that a number of the items did not adequately reflect the construct and should be revised or removed from the measure. A discussion of these results and the implications for higher education institutions focused on preventing student withdrawal are presented. (Contains 5 tables.)

If measurement invariance does not hold over 2 or more measurement occasions, differences in observed scores are not directly interpretable. Golembiewski, Billingsley, and Yeager (1976) identified 2 types of psychometric differences over time as beta change and gamma change. Gamma change is a fundamental change in thinking about the nature of a construct over time. Beta change can be described as respondents' change in calibration of the response scale over time. Recently, researchers have had considerable success establishing measurement invariance using confirmatory factor analytic (CFA) techniques. However, the use of item response theory (IRT) techniques for assessing item parameter drift can provide additional useful information regarding the psychometric equivalence of a measure over time that is not attainable with traditional CFA techniques. This article marries the terminology commonly used in CFA and IRT techniques and illustrates real advantages for identifying beta change over time with IRT methods rather than typical CFA methods, utilizing a longitudinal assessment of job satisfaction as an example.

In recent years, confirmatory factor analytic (CFA) techniques have become the most common method of testing for measurement equivalence/invariance (ME/I). However, no study has simulated data with known differences to determine how well these CFA techniques perform. This study utilizes data with a variety of known simulated differences in factor loadings to determine how well traditional tests of ME/I can detect these specific simulated differences. Results show that traditional CFA tests of ME/I perform well under ideal situations but that large sample sizes, a sufficient number of manifest indicators, and at least moderate communalities are crucial for assurance that ME/I conditions exist.

Much progress has been made in the past 2 decades with respect to methods of identifying measurement invariance or a lack thereof. Until now, the focus of these efforts has been to establish criteria for statistical significance in items and scales that function differently across samples. The power associated with tests of differential functioning, as with all significance tests, is affected by sample size and other considerations. Additionally, statistical significance need not imply practical importance. There is a strong need as such for meaningful effect size indicators to describe the extent to which items and scales function differently. Recently developed effect size measures show promise for providing a metric to describe the amount of differential functioning present between groups. Expanding upon recent developments, this article presents a taxonomy of potential differential functioning effect sizes; several new indices of item and scale differential functioning effect size are proposed and illustrated with 2 data samples. Software created for computing these indices and graphing item- and scale-level differential functioning is described. Keywords: measurement invariance, differential functioning, DIF, invariance testing DOI: 10.1037/a0018966

To link to full-text access for this article, visit this link: http://dx.doi.org/10.1016/j.chb.2008.05.005 Byline: Phillip W. Braddy (a), Adam W. Meade (b), Christina M. Kroustalis (b) Keywords: Internet recruiting; Online recruiting; Web recruiting; Website usability; Website attractiveness; Signaling theory Abstract: Previous research on Internet recruitment has made the implicit assumption that recruitment websites influence viewers' opinions of recruiting organizations. This study tested this assumption using a pretest/posttest design. Findings revealed that participants' organizational favorability, image as employer, and organizational attractiveness perceptions were affected by their viewing of organizational recruitment websites. Greater increases in favorable organizational evaluations from the pretest measures to the posttest measures occurred with organizations maintaining websites that were easy to navigate and/or that were appealing. Contrary to predictions made by signaling theory, recruitment websites had similar effects on the organizational impressions of all individuals, regardless of their familiarity with the organizations maintaining the recruitment websites that they viewed. Author Affiliation: (a) Center for Creative Leadership, One Leadership Place, P.O. Box 26300, Greensboro, NC 27438-6300, United States (b) Department of Psychology, North Carolina State University, United States Article Note: (footnote) [star] Portions of this study were completed while the first author was affiliated with the Department of Psychology at North Carolina State University. Also, an earlier version of this manuscript was presented at the 20th Annual Conference of the Society for Industrial and Organizational Psychology, Los Angeles, CA, April, 2005.

To purchase or authenticate to the full-text of this article, please visit this link: http://dx.doi.org/10.1111/j.1559-1816.2008.00372.x Byline: Lisa E. Baranik (a2), Adam W. Meade (b), Chad E. Lakey (a), Charles E. Lance (a), Changya Hu (c), Wei Hua (d), Alex Michalos (e) Abstract: We examined the differential item functioning (DIF) of Rosenberg's (1965) Self-Esteem Scale (RSES) and compared scores from U.S. participants with those from 7 other countries: Canada, Germany, New Zealand, Kenya, South Africa, Singapore, and Taiwan. Results indicate that DIF was present in all comparisons. Moreover, controlling for latent self-esteem, participants from individualistic countries had an easier time reporting high self-esteem on self-competence-related items, whereas participants from communal countries had an easier time reporting high self-esteem on self-liking items (Tafarodi & Milne, 2002). After adjusting for DIF, we found larger mean self-esteem differences between the countries than observed scores initially indicated. The suitability of the RSES, and the importance of examining DIF, for cross-cultural research are discussed. Author Affiliation: (a)The University of Georgia (b)North Carolina State University (c)National Taiwan University of Science and Technology (d)Singapore Management University (e)University of Northern British Columbia Article note: (2) Lisa Baranik, Department of Psychology, The University of Georgia, Athens, GA 30602-3013. E-mail: baranik@uga.edu

Confirmatory factor analytic tests of measurement invariance (MI) based on the chi-square statistic are known to be highly sensitive to sample size. For this reason, G. W. Cheung and R. B. Rensvold (2002) recommended using alternative fit indices (AFIs) in MI investigations. In this article, the authors investigated the performance of AFIs with simulated data known to not be invariant. The results indicate that AFIs are much less sensitive to sample size and are more sensitive to a lack of invariance than chi-square-based tests of MI. The authors suggest reporting differences in comparative fit index (CFI) and R. P. McDonald's (1989) noncentrality index (NCI) to evaluate whether MI exists. Although a general value of change in CFI (.002) seemed to perform well in the analyses, condition specific change in McDonald's NCI values exhibited better performance than a single change in McDonald's NCI value. Tables of these values are provided as are recommendations for best practices in MI testing. Keywords: measurement invariance, measurement equivalence, fit indices

Data are described as ipsative if a given set of responses always sum to the same total. However, there are many properties of data collection that can give rise to different types of ipsative data. In this study, the most common type of ipsative data used in employee selection (forced-choice ipsative data; FCID) is discussed as a special case of other types of ipsative data. Although all ipsative data contains constraints on covariance matrices (covariance-level interdependence), FCID contains additional item-level interdependencies as well. The psychological processes that give rise to FCID and the resultant psychometric properties are discussed. In addition, data from which both normative and ipsative responses were provided by job applicants illustrate very different patterns of correlations as well as very different selection decisions between normative, FCID and ipsatized measures., Mathematically speaking, data are ipsative if a given set of responses always sum to the same total. Problems associated with ipsative data were originally recognized some time ago by Cattell [...]

While rating-scale-based assessments have been shown to be useful for measuring a variety of workplace-relevant constructs, assessment length and response distortion present practical limitations on their use. We describe a new type of measurement method termed rapid response measurement (RRM) in which stimuli are presented on a computer screen one at a time in rapid succession and respondents are asked to quickly provide a dichotomous response. Two personality assessments using RRM were developed and reliability and validity evidence across four independent samples were evaluated. Both RRM assessments showed adequate reliability, even at short test lengths, with acceptable levels of convergent and discriminant validity with traditional survey-based measures. Analyses based on a within-participants design indicated that the RRM was significantly more difficult to fake when instructed than was a survey-based measure of personality. The second RRM was related to several aspects of job performance. While initial results show promise, further research is needed to establish the validity and viability of the RRM for organizational and psychological measurement. [ABSTRACT FROM AUTHOR]

A lack of human interaction and environmental control in Internet‐based data collection have been suggested as possible antecedents of careless responding, which occurs when participants respond to survey items without regard for item content. To address these possible antecedents, this study investigated whether survey proctoring deterred careless response in an undergraduate sample by reducing environmental distractions. The study randomly assigned respondents to one of three proctoring conditions: remote online un‐proctored, remote online virtually proctored, and in‐person classroom proctored. Data quality was examined via nine careless response indicators. Analyses indicated that proctor presence had effects on a small number of careless response indicators. Virtually proctored participants performed better than un‐proctored participants on one of nine careless response indicators, and in‐person proctored participants performed better on two careless response indicators compared to un‐proctored participants. Environmental distraction fully mediated the relationship between in‐person proctor presence and self‐reported diligence. Implications for survey administration are discussed. [ABSTRACT FROM AUTHOR]