Search

This study examined differential item functioning (DIF) in the Center for Epidemiologic Studies Depression Scale (CES-D) between Chinese and White adolescents (aged 13 to 17 years) living in Canada. A series of ordinal logistic regressions were used to test for uniform and non-uniform DIF on items in the CES-D. The DIF analyses identified non-uniform DIF for Item 7 ("I felt that everything I did was an effort"). Controlling for gender and strength of ethnic identity in the DIF analyses did not alter the DIF results. The results of this study suggest that CES-D Item 7 does not appear to discriminate at higher levels of depression in Chinese adolescents. The results of this study hold notable implications for the use of the CES-D given that Chinese adolescents demonstrated a systematic difference in expression/experience of depression.

Methods to generate random correlation matrices have been proposed in the literature, but very few instances exist where these correlation matrices are structured or where the statistical p...

Veenhoven (Happiness in nations, subjective appreciation of life in 56 nations, 1946–1992 (Studies in social-cultural transformation, 2), Erasmus University Rotterdam, Risbo, 2003, Quality of life in the new millennium: ‘Advances in quality-of-life studies, theory and research’, Part 2: Refining concepts and measurement to assess cross-cultural quality-of-life, Springer, Dordrecht, 2008) has called for continued statistical and survey research to support pooling survey response data from different surveys, with an eye to eventual comparative studies or research synthesis using meta-analysis. Data pooling across various social surveys is made complicated because there is no universally agreed-upon response format (response rating scales) for commonly used survey questions. Two widely used solutions to this problem are transforming the survey response data using either the Linear Stretch or Percentage of Scale Maximum transformations. In line with Veenhoven and his colleagues (de Jonge et al., Diversity in survey questions on the same topic: Techniques for improving comparability, Springer, New York, 2017) and others, we are critical of these two commonly used transformation methods. Using mathematical analysis, we show how these two most commonly used transformation methods are inadequate because they are not well defined; therefore their unique interpretation comes from social convention. Ironically, if one needs to rely on social convention to make these transformations interpretable, it would have been more reasonable to agree to use the same response format from the start and save ourselves all this trouble. The chapter closes with a description of an alternative approach based on a family of promising multivariate statistical and psychometric methods, of which the statistical methods described in de Jonge, Veenhoven, and Kalmijn, are an instantiation.

The purpose of this research was to develop a questionnaire to assess the multidimensional construct of teamwork in sport and to examine various aspects of validity related to that instrument. A preliminary questionnaire was first created, and feedback on this instrument was then obtained from a sample of team-sport athletes (n = 30) and experts in sport psychology (n = 8). A modified version of the questionnaire was then completed by 607 athletes from 48 teams, and 5 multilevel confirmatory factor analyses were conducted to examine the structural properties of data derived from this instrument. Evidence of adequate model-data fit along with measurement reliability was obtained for each of the 5 models. Taken together, the results from this research provide support for the content, substantive, and structural aspects of construct validity for data derived from the 66-item Multidimensional Assessment of Teamwork in Sport.

The purpose of this paper is to highlight the importance of a population model in guiding the design and interpretation of simulation studies used to investigate the Spearman rank correlation. The Spearman rank correlation has been known for over a hundred years to applied researchers and methodologists alike and is one of the most widely used non-parametric statistics. Still, certain misconceptions can be found, either explicitly or implicitly, in the published literature because a population definition for this statistic is rarely discussed within the social and behavioural sciences. By relying on copula distribution theory, a population model is presented for the Spearman rank correlation, and its properties are explored both theoretically and in a simulation study. Through the use of the Iman-Conover algorithm (which allows the user to specify the rank correlation as a population parameter), simulation studies from previously published articles are explored, and it is found that many of the conclusions purported in them regarding the nature of the Spearman correlation would change if the data-generation mechanism better matched the simulation design. More specifically, issues such as small sample bias and lack of power of the t-test and r-to-z Fisher transformation disappear when the rank correlation is calculated from data sampled where the rank correlation is the population parameter. A proof for the consistency of the sample estimate of the rank correlation is shown as well as the flexibility of the copula model to encompass results previously published in the mathematical literature.

This article explores the potential for ethnographic observations to inform the analysis of test item performance. In 2010, a standardized, large-scale adult literacy assessment took place in Mongolia as part of the United Nations Educational, Scientific and Cultural Organization Literacy Assessment and Monitoring Programme (LAMP). In a novel form of interdisciplinary collaboration, an ethnographer worked closely with psychometric researchers to investigate the sources and explanations of differential item functioning and test item performance. The research involved detailed ethnographic observations of literacy assessment events. The results illustrate the potential for ethnography to provide insights into testing situations, group and test item performance, and to combine with psychometric analysis in cross-cultural assessment.

Although anxiety is frequently reported in children with autism spectrum disorder (ASD), existing anxiety scales are often psychometrically inappropriate for this population. This study examined the internal structure, reliability, convergent and discriminant validity of the Spence Children's Anxiety Scale-Parent Report (SCAS-P; Spence 1999) in 238 school-aged children with ASD. While confirmatory factor analysis did not support the original six-correlated-factor structure, structural support as well as acceptable internal consistency and convergent validity was found for Generalized Anxiety, Separation Anxiety, Panic, and Agoraphobia subscales. Use of the SCAS-P in its original form for assessment in children with ASD was not supported. However, four subscales showed viability, and may benefit re-analyses of existing SCAS-P data and future scale adaptations for research and clinical purposes.

As a convenient data source from computerized tests, response time could also be very informative evidence for the validity of test scores, offering an opportunity for insights into parts of the test that test takers linger over and other parts of the test where they glide through the material. Given these hopes for, and expectations of, response time data, we should critically evaluate how such data are studied and more importantly, to what extent this type of data lives up to its promise. We begin this chapter by defining response time and briefly discussing its use as validity evidence. We then describe the typical uses of response time data and review the different approaches to studying response time data in various research. The chapter closes with an evaluation of response time data as validity evidence and suggests the conditions that would facilitate better use of response data for validation purposes.

The results of large-scale student assessments are increasingly being used to rank nations, states, and schools and to inform policy decisions. These uses often rely on aggregated student test score data, and imply inferences about multilevel constructs. Validating uses and interpretations about these multilevel constructs requires appropriate multilevel validation techniques. This chapter combines multilevel data analysis techniques with an explanatory view of validity to develop explanations of score variation that can be used to evaluate multilevel measurement inferences. We use country-level mathematics scores from the Trends in International Mathematics and Science Study (TIMSS) to illustrate the integration of these techniques. The explanation focused view of validity accompanied by the ecological model of item responding situates conventional response process research in a multilevel construct setting and moves response process studies beyond the traditional focus on individual test-takers’ behaviors.

A critical step in the development and use of tests of physical fitness for employment purposes (e.g., fitness for duty) is to establish 1 or more cut points, dividing the test score range into 2 or more ordered categories reflecting, for example, fail/pass decisions. Over the last 3 decades elaborated theories and methods have evolved focusing on the process of establishing 1 or more cut-scores on a test. This elaborated process is widely referred to as “standard-setting”. As such, the validity of the test score interpretation hinges on the standard-setting, which embodies the purpose and rules according to which the test results are interpreted. The purpose of this paper is to provide an overview of standard-setting methodology. The essential features, key definitions and concepts, and various novel methods of informing standard-setting will be described. The focus is on foundational issues with an eye toward informing best practices with new methodology. Throughout, a case is made that in terms of best practices, establishing a test standard involves, in good part, setting a cut-score and can be conceptualized as evidence/data-based policy making that is essentially tied to test validity and an evidential trail.

The primary purpose of this study was to examine the underlying factor structure of the Parenting Stress Index-Short Form (PSI-SF) in a large cohort of parents of young children with autism spectrum disorder (ASD). A secondary goal was to examine relationships between PSI-SF factors and autism severity, child behavior problems, and parental mental health variables that have been shown to be related to parental stress in previous research. A confirmatory factor analysis (CFA) was used to examine the three-factor structure described in the PSI-SF manual [Abidin, 1995]: parental distress, parent-child dysfunctional interaction, and difficult child. Results of the CFA indicated that the three-factor structure was unacceptable when applied to the study sample. Thus, an exploratory factor analysis was conducted and suggested a six-factor model as the best alternative for the PSI-SF index. Spearman's correlations revealed significant positive correlations with moderate to large effect sizes between the revised PSI-SF factors and autism severity, externalizing and internalizing child behaviors, and an index of parent mental health. The revised factors represent more narrowly defined aspects of the three original subscales of the PSI-SF and might prove to be advantageous in both research and clinical applications. Autism Res 2011,4:336-346. © 2011 International Society for Autism Research, Wiley Periodicals, Inc.

Whenever one uses a composite scale score from item responses, one is tacitly assuming that the scale is dominantly unidimensional. Investigating the unidimensionality of item response data is an essential component of construct validity. Yet, there is no universally accepted technique or set of rules to determine the number of factors to retain when assessing the dimensionality of item response data. Typically factor analysis is used with the eigenvalues-greater-than-one rule, the ratio of first-to-second eigenvalues, parallel analysis, root-mean-square-error-of-approximation, or hypothesis testing approaches involving chi-square tests from Maximum Likelihood or Generalized Least Squares estimation. The purpose of this study was to investigate how these various procedures perform individually, and in combination, when assessing the unidimensionality of item response data via a computer simulated design. Conditions such as sample size, magnitude of communality, distribution of item responses, proportion of communality on second factor, and the number of items with non-zero loadings on the second factor were varied. Results indicate that there was no one individual decision-making method that identified unidimensionality under all conditions manipulated. Given the low communalities, all individual decision-making methods failed to detect unidimensionality for the combination where sample size was small, magnitude of communality was low, and item distributions were skewed. A set of guidelines and a new statistical methodology are provided for researchers.

Background: The Parenting Stress Index-Short Form (PSI-SF) is one of the most widely used instruments for measuring parenting stress in families of children with autism spectrum disorders (ASD). However, no research to date has examined the psychometric properties of the PSI-SF in a sample of parents of young children with ASD. In this regard, item response theory (IRT) can be used to estimate how much information or discrimination each item of a scale offers across the entire range of the latent variable being measured, by creating individual item information curves or profiles. The purpose of this study was to use IRT to examine the discriminability of PSI-SF items in a sample of parents of young children with ASD who experience varying levels of parental stress. Methods: The study involved the parents of 141 children with autism spectrum disorders (91.4% mothers; mean age 36.2 years) who completed the PSI-SF following diagnosis. Item characteristic curves were constructed for each of the PSI-SF items and examined with regard to item functioning. Results: Results indicated that, for the most part, changes in parental distress severity were reflected in changes on item scores. However, several items on the subscales measuring parent–child dysfunctional interactions and child behavior difficulty functioned poorly to discriminate parents across a range of total stress severity. Conclusions: The parent–child dysfunctional interaction and difficult child subscales of the PSI-SF scale should be used with caution with parents of young children with ASD. More research is required to examine PSI-SF content validity, at least among parents of children with ASD and perhaps parents of children with other disabilities as well.

In a recent Monte Carlo simulation study, Liu and Zumbo showed that outliers can severely inflate the estimates of Cronbach’s coefficient alpha for continuous item response data—visual analogue response format. Little, however, is known about the effect of outliers for ordinal item response data—also commonly referred to as Likert, Likert-type, ordered categorical, or ordinal/rating scale item responses. Building on the work of Liu and Zumbo, the authors investigated the effects of outlier contamination for binary and ordinal response scales. Their results showed that coefficient alpha estimates were severely inflated with the presence of outliers, and like the earlier findings, the effects of outliers were reduced with increasing theoretical reliability. The efficiency of coefficient alpha estimates (i.e., sample-to-sample variation) was inflated as well and affected by the number of scale points. It is worth noting that when there were no outliers, the alpha estimates were downward biased because of the ordinal scaling. However, the alpha estimates were, in general, inflated in the presence of outliers leading to positive bias.

This study introduces the Satisfaction with Life Scale adapted for Children (SWLS-C) and presents psychometric findings regarding its validation. The SWLS-C was adapted from the Satisfaction with Life Scale (SWLS; Diener et al. 1985), which is one of the most commonly used measures to assess satisfaction with life in adults. Three subject matter experts adapted the SWLS by changing the wording of the item stem and response format in order to make it more understandable for children. A stratified random sample of 1,233 students (48% girls) in grades 4–7 (mean age 11 years and 7 months) provided data on the SWLS-C and measures of optimism, self-concept, self-efficacy, depression, emphatic concern, and perspective taking. The SWLS-C demonstrated a unidimensional factor structure and high internal consistency. Furthermore, differential item functioning and differential scale functioning analyses indicated that the SWLS-C measures satisfaction with life in the same way for different groups of children (i.e., with regard to gender, first language learned at home—English vs. other language(s) than English—and across different grades) at the item and at the scale level. Associations between scores on the SWLS-C and demographic variables were statistically non-significant or of small effect size. In addition, the SWLS-C showed evidence of convergent and discriminant validity in relation to the other measures. Our results indicate that the SWLS-C is a psychometrically sound instrument that demonstrated evidence of construct validity for this age group. Limitations and future directions are discussed.

The satisfaction with life scale (SWLS) is a widely accepted and widely used tool for measuring well-being. Although its potential as a cross-cultural index is recognized, an introduction and systematic validation of the Hebrew version is needed. Thus, the purpose of this study is: (1) to describe the process of developing the Hebrew version of the SWLS, and (2) to examine its construct validity as well as its internal consistency. Four hundred and eighty seven working adults completed the following self reported Hebrew language versions of the: (1) SWLS, (2) positive affect and negative affect scales (PANAS), and (3) the self-rated health (SRH) scale. In addition, as way of gathering additional evidence of validity, the SWLS was completed by proxy (i.e., each participant’s life partner or significant other). Confirmatory factor analysis supported a single-factor structure with significant correlations between the SWLS and the rest of the measures—PANAS scores, the SRH scores as well as the SWLS scores as measured by proxy. In addition, item-analysis supports the internal consistency of the scale. The Hebrew version of the SWLS is a valid and reliable scale and can be utilized in the Israeli context.

The purpose of this study was to examine the influence of mode of administration (internet-based, web survey format versus pencil-and-paper format) on responses to the Dimensions of Tobacco Dependence Scale (DTDS). Responses from 1,484 adolescents that reported using tobacco (mean age 16 years) were examined; 354 (23.9%) participants completed a web-based version and 1,130 (76.1%) completed a paper-based version of the survey. Both surveys were completed in supervised classroom environments. Use of the web-based format was associated with significantly shorter completion times and a small but statistically significant increase in the number of missing responses. Tests of measurement invariance indicated that using a web-based mode of administration did not influence the psychometric functioning of the DTDS. There were no significant differences between the web- and paper-based groups' ratings of the survey's length, their question comprehension, and their response accuracy. Overall, the results of the study support the equivalence of scores obtained from web- and paper-based versions of the DTDS in secondary school settings.