Search

The purposes of this paper are to highlight the foundations of multilevel construct validation, describe two methodological approaches and associated analytic techniques, and then apply these approaches and techniques to the multilevel construct validation of a widely-used school readiness measure called the Early Development Instrument (EDI; Janus and Offord 2007). Validation evidence is presented regarding the multilevel covariance structure of the EDI, the appropriateness of aggregation to classroom and neighbourhood levels, and the effects of teacher and classroom characteristics on these structural patterns. The results are then discussed in the context of the theoretical framework of the EDI, with suggestions for future validation work.

The vast majority of measures have, at their core, a purpose of personal and social change. If test developers and users want measures to have personal and social consequences and impact, then it is critical to consider the consequences and side effects of measurement in the validation process itself. The consequential basis of test interpretation and use, as introduced in Messick's (Educational measurement, Macmillan, New York, pp. 13-103, 1989) progressive matrix model of unified validity theory, has been misunderstood by many measurement experts, test developers, researchers, and practitioners. The purposes of this paper were to (a) review Messick's unified view of validity and clarify his consequential basis of test interpretation and use, (b) discuss the kinds of questions evoked by value implications and social consequences and their role in construct validity and score meaning, (c) present a reframing of Messick's model and a new model of unified validity and validation, (d) bring the concept of multilevel measures under the same validation umbrella as individual differences measures, and (e) offer some thoughts and directions for more explicit consideration of value implications, intended social consequences, and unintended side effects of legitimate test interpretation and use. This paper has implications for the interpretation, use, and validation of both individual differences and multilevel measures in education, psychology, and health contexts.

This paper delineates general validity and research questions that are underlying an ongoing program of research pertaining to the Early Development Instrument (EDI, Janus and Offord 2007), a population-level measure, on which teachers rate kindergarten children's developmental outcomes in the social, emotional, physical, cognitive, and communicative domains. It describes the large-scale research projects that are using the EDI to measure children's developmental outcomes for entire populations (e.g., provinces, cities). Given the uniqueness of the EDI's use as monitoring tool, or social indicator, for children's developmental health at an early age, we spell out the opportunities and challenges that these projects provide with regard to validation research. The article sets the stage for the special issue, as it is dedicated to showcase the different aspects of the conceptual, theoretical, and empirical validation research as well as the collaborative community-based projects that are currently being undertaken with respect to the EDI.

Whenever one uses a composite scale score from item responses, one is tacitly assuming that the scale is dominantly unidimensional. Investigating the unidimensionality of item response data is an essential component of construct validity. Yet, there is no universally accepted technique or set of rules to determine the number of factors to retain when assessing the dimensionality of item response data. Typically factor analysis is used with the eigenvalues-greater-than-one rule, the ratio of first-to-second eigenvalues, parallel analysis, root-mean-square-error-of-approximation, or hypothesis testing approaches involving chi-square tests from Maximum Likelihood or Generalized Least Squares estimation. The purpose of this study was to investigate how these various procedures perform individually, and in combination, when assessing the unidimensionality of item response data via a computer simulated design. Conditions such as sample size, magnitude of communality, distribution of item responses, proportion of communality on second factor, and the number of items with non-zero loadings on the second factor were varied. Results indicate that there was no one individual decision-making method that identified unidimensionality under all conditions manipulated. Given the low communalities, all individual decision-making methods failed to detect unidimensionality for the combination where sample size was small, magnitude of communality was low, and item distributions were skewed. A set of guidelines and a new statistical methodology are provided for researchers.

The objective of this study was to investigate the substantive aspect of construct validity of the Satisfaction with Life Scale adapted for Children (SWLS-C; Gadermann et al. in Soc Indic Res 96:229-247, "2010"). Specifically, the study examined the cognitive processes of children when responding to the items of the SWLS-C to find out how they interpret and respond to the items. Think-aloud protocol interviews were conducted with 55 students in grades 4-7 (58% girls, mean age of 11 years, ranging from 8.8 to 12.8 years) and content analysis was used to analyze the data. The findings indicate that most children had no difficulty in understanding the items, and used mainly two strategies for responding to the items: (1) an absolute strategy, in which children used absolute statements to indicate the presence or absence of something that is important for their judgment of their satisfaction with life, and (2) a relative strategy, in which children used relative or comparative statements. In the absolute statements, children primarily referred to social relationships, personal characteristics, time use, and possessions. In the relative statements, the children primarily compared what they have to (a) what they want (b) what they had in the past, (c) what other people have, and (d) what they feel they need. Furthermore, most children considered it important that information on their life satisfaction is obtained. The results are discussed with regard to multiple discrepancies theory (Michalos in Soc Indic Res 16:347-413, "1985") and previous empirical findings. The results provide insights into children's cognitive processes when responding to items on life satisfaction and provide validity evidence that the SWLS-C is an appropriate measure to assess life satisfaction in children of this age.

This paper described the versatility of the multiple-indicator multilevel (MIML) model in helping to resolve four common challenges in studying growth using longitudinal data. These challenges are (1) how to deal with changes in measurement over time and investigate temporal measurement invariance, (2) how to model residual dependence due to the nested nature of longitudinal data, (3) how to model observed trajectories that do not follow well-known functions commonly discussed in the methodology literature (e.g., a linear or quadratic curve), and (4) how to decide which predictors are relatively more important in explaining individuals' change over time. With an example of psychological well-being from the Wisconsin Longitudinal Study, we illustrated how the four methodological challenges can be resolved using the 3-phase MIML procedures and the Pratt's importance measures.

The satisfaction with life scale (SWLS) is a widely accepted and widely used tool for measuring well-being. Although its potential as a cross-cultural index is recognized, an introduction and systematic validation of the Hebrew version is needed. Thus, the purpose of this study is: (1) to describe the process of developing the Hebrew version of the SWLS, and (2) to examine its construct validity as well as its internal consistency. Four hundred and eighty seven working adults completed the following self reported Hebrew language versions of the: (1) SWLS, (2) positive affect and negative affect scales (PANAS), and (3) the self-rated health (SRH) scale. In addition, as way of gathering additional evidence of validity, the SWLS was completed by proxy (i.e., each participant's life partner or significant other). Confirmatory factor analysis supported a single-factor structure with significant correlations between the SWLS and the rest of the measures--PANAS scores, the SRH scores as well as the SWLS scores as measured by proxy. In addition, item-analysis supports the internal consistency of the scale. The Hebrew version of the SWLS is a valid and reliable scale and can be utilized in the Israeli context.

This study introduces the Satisfaction with Life Scale adapted for Children (SWLS-C) and presents psychometric findings regarding its validation. The SWLS-C was adapted from the Satisfaction with Life Scale (SWLS; Diener et al. 1985), which is one of the most commonly used measures to assess satisfaction with life in adults. Three subject matter experts adapted the SWLS by changing the wording of the item stem and response format in order to make it more understandable for children. A stratified random sample of 1,233 students (48% girls) in grades 4-7 (mean age 11 years and 7 months) provided data on the SWLS-C and measures of optimism, self-concept, self-efficacy, depression, emphatic concern, and perspective taking. The SWLS-C demonstrated a unidimensional factor structure and high internal consistency. Furthermore, differential item functioning and differential scale functioning analyses indicated that the SWLS-C measures satisfaction with life in the same way for different groups of children (i.e., with regard to gender, first language learned at home--English vs. other language(s) than English--and across different grades) at the item and at the scale level. Associations between scores on the SWLS-C and demographic variables were statistically non-significant or of small effect size. In addition, the SWLS-C showed evidence of convergent and discriminant validity in relation to the other measures. Our results indicate that the SWLS-C is a psychometrically sound instrument that demonstrated evidence of construct validity for this age group. Limitations and future directions are discussed.

Several measurement assumptions were examined with the goal of assessing the validity of the Multidimensional Students' Life Satisfaction Scale (MSLSS), a measure of adolescents' satisfaction with their family, friends, living environment, school, self, and general quality of life. The data were obtained via a cross-sectional survey of 8,225 adolescents in British Columbia, Canada. Confirmatory factor and factor mixture analyses of ordinal data were used to examine the measurement assumptions. The adolescents did not respond to all the MSLSS items in a psychometrically equivalent manner. A correlated five-factor model for an abridged version of the MSLSS resulted in good fit when all negatively worded items and several positively worded items (the least invariant) were excluded. The abridged 18-item version of the MSLSS provides a promising alternative for the measurement of five life domains that are pertinent to adolescents' quality of life.

A case is made that measures used in quality of life and happiness research will be essentially unidimensional: inherently tapping minor dimensions. This is illustrated using Diener's Satisfaction with Life Scale (SWLS). It is shown that the SWLS does not meet the standard of strict unidimensionality, but that the interpretation of the total scale score is not compromised because the additional dimensions are relatively minor. In the context of the example, a multi-step strategy is described that allows researchers to test for essential unidimensionality. Throughout the article, essential unidimensionality is contrasted with the received view of strict unidimensionality and confirmatory factor analysis methods.

Mediation and moderation are two theories for refining and understanding a causal relationship. Empirical investigation of mediators and moderators requires an integrated research design rather than the data analyses driven approach often seen in the literature. This paper described the conceptual foundation, research design, data analysis, as well as inferences involved in a mediation and/or moderation investigation in both experimental and non-experimental (i.e., correlational) contexts. The essential distinctions between the investigation of mediators and moderators were summarized and juxtaposed in an example of a causal relationship between test difficulty and test anxiety. In addition, the more elaborate models, moderated mediation and mediated moderation, the use of structural equation models, and the problems with model misspecification were discussed conceptually.

The present study explored the short-term temporal characteristics of subjective well-being (SWB) and its potential correlates. Specifically, we examined the intra-individual variability and the trajectories of SWB and its components life satisfaction, positive, and negative affect. Over a two-week period, 27 participants (78% university students, 85% females, mean age of 29 years) provided SWB data every other day. Before and after this period, participants completed several questionnaires assessing personality and environmental variables, as well as several open-ended questions. The results provide evidence that participants experienced a significant amount of intra-individual variability in their SWB and its components. Furthermore, inter-individual differences in the intra-individual variability were related to several of the personality and environmental variables, as well as to participant's age. Results of multiple regression analyses indicated that deliberation (one facet of the Big Five dimension conscientiousness), social support, and age accounted for 58% of the total variance of intra-individual variability in SWB. In contrast, the slopes of the individual trajectories of SWB were on average statistically non-significantly different from zero. These results emphasize that information is gained by looking at both the patterns and the intra-individual variability over time: The individual trajectories of the participants on average appeared to be stable, but intra-individual variability of SWB was meaningfully related to several of the variables.

The present study evaluated subjective importance weighting using data collected with the Injection Drug User Quality of Life Scale (IDUQOL). Weighted and unweighted IDUQOL scores from 241 adults were correlated with convergent, discriminant, and criterion measures. Regression analysis was used to examine the contribution of importance ratings to scores on a global measure of life satisfaction and the corrected weighted IDUQOL total scores. Overall, the results showed that weighted scores did not perform better than unweighted scores in measuring quality of life. However, the mean satisfaction ratings for important domains correlated significantly higher with convergent measures than did the mean satisfaction ratings for unimportant domains. This finding suggests further attention needs to be paid to the meaning and measurement of subjective importance and how it may be incorporated more effectively into measures of quality of life.

The aim of this investigation is to obtain some baseline self-reported data of the health status and overall quality of life of all residents of the Bella Coola Valley of British Columbia aged 17 years or older, and to measure the impact of a set of designated health determinants on their health and quality of life. In the period from August to November 2001, a variety of procedures were used to ensure that all eligible residents of the Valley received a copy of our questionnaire, and 687 useable questionnaires were obtained for our working dataset. Health status was measured by SF-36 and the U.S. Centers for Disease Control healthy days items. Thirty-one items were used to measure the Provincial Health Officer's designated determinants of health in four clusters, namely, biological, social and economic, health behaviours and health services determinants. Quality of life was measured by satisfaction levels in 13 specific domains of life (e.g., family, financial security), four global items (e.g., happiness, life satisfaction) and one global Subjective Well-Being Index. Besides obtaining baseline figures on all our measures for the Valley, we made some comparisons among our figures and those from other areas, e.g., Prince George, BC. Most of the measures indicated that the health status and quality of life of Bella Coola Valley residents were lower than those of Prince George residents. For the sample as a whole, SF-36 scores on the eight dimensions ran from 82.3 (physical functioning) to 50.0 (social functioning), with a mean of 62.7. Residents in the Valley averaged 6.5 days in the past 30 in which their health was physically not good, 5.5 days when it was mentally not good and 4.1 days when their health limited their usual daily activities. Eleven percent of respondents described their general health as "excellent" and another 27% said it was "very good". On a 7-point scale from 1 = very dissatisfied to 7 = very satisfied, respondents had average life satisfaction and satisfaction with the overall quality of life scores of 5.5. For specific domains of life, the lowest mean level of satisfaction was reported for federal and provincial government officials (3.3) and the highest was reported for living partners and personal safety around home (5.8). Regarding bivariate relations, each of the eight dimensions of SF-36 was significantly correlated with a single item measure of general health, and five of the eight were significantly correlated with the number of good health days. Happiness and the Subjective Well-Being Index were positively but moderately correlated with six of the eight dimensions, and life satisfaction was positively correlated with five. Age was negatively related to general health, but positively related to life satisfaction. Not being of aboriginal descent was positively related to all of the four global health indicators and to the Subjective Well-Being Index. Education was positively related to the four global health measures but not to the three global quality of life measures. The Social Support and Good Family Indexes were positively related to all seven global measures. There was a positive correlation between six of the seven global measures and the frequency with which respondents participated in activities sponsored by voluntary organizations. Frequency of smoking was negatively associated with every global dependent variable except the Physical Health Index. Frequency of skipping meals was negatively associated and average hours of sleep per night was positively associated with all seven global measures. Turning to multivariate relationships, the four clusters of health determinants explained from 12% (SF-36 Mental Health Index) to 24% (general health) of the variance in the dependent global health variables, and from 20% (happiness) to 26% (Subjective Well-Being Index) of the variance in the dependent global quality of life variables. Adding domain satisfaction scores to the total set of predictors allowed us to explain from 20% (SF-36 Mental health Index) to 29% (general health) of the variance in the dependent global health variables, and from 39% (happiness) to 62% (life satisfaction) in the dependent global quality of life variables. By including measures of social support and good family relationships in our set of health determinants, we practically guaranteed that the latter would be relatively strongly predictive of global quality of life.