Search

The Cross-Classified Mixed Effects Model (CCMEM) has been demonstrated to be a flexible framework for evaluating reliability by measurement specialists. Reliability can be estimated based on the variance components of the test scores. Built upon their accomplishment, this study extends the CCMEM to be used for evaluating validity evidence. Validity is viewed as the coherence among the elements of a measurement system. As such, validity can be evaluated by the user-reasoned desired or undesired fixed and random effects. Based on the data of ePIRLS 2016 Reading Assessment, we demonstrate how to obtain evidence for reliability and validity by CCMEM. We conclude with a discussion on the practicality and benefits of this validation method.

A variety of differential item functioning (DIF) methods have been proposed and used for ensuring that a test is fair to all test takers in a target population in the situations of, for example, a test being translated to other languages. However, once a method flags an item as DIF, it is difficult to conclude that the grouping variable (e.g., test language) is responsible for the DIF result because there may exist many confounding variables that lead to the DIF result. The present study aims to (i) demonstrate the application of propensity score methods in psychometric research on DIF for day-to-day researchers, and (ii) describe conditional logistic regression for matched data in a DIF context. Propensity score methods can help to achieve the comparability between different populations or groups with respect to participants' pre-test differences, which can assist in examining the validity of making a causal claim with regard to DIF.

Answering the call for response process-based validation, this study shows how researchers can evaluate validity evidence of test scores based on the mental "processes" test-takers use, rather than based on correlations with other "outcome" measures. The proposed methods for process-based validation studies are demonstrated using a sample of 189 adults who took two listening comprehension tasks. Immediately after completing each task, the test-takers filled out a 10-item survey to reflect on the mental processes involved in reaching their answers. These 10 process variables attempted to capture five desired and five undesired response processes in answering multiple-choice listening comprehension questions. We investigated the relationships between these process variables and the binary outcome of item score (correct vs. incorrect) using mixed-effects logistic regression models, and showed how the results could provide validity evidence (or lack thereof). By doing so, we offer an alternative approach to study response process and test performance and encourage more process-based validation studies.

Item response tree (IRTree) models are recently introduced as an approach to modeling response data from Likert-type rating scales. IRTree models are particularly useful to capture a variety of individuals' behaviors involving in item responding. This study employed IRTree models to investigate response styles, which are individuals' tendencies to prefer or avoid certain response categories in a rating scale. Specifically, we introduced two types of IRTree models, descriptive and explanatory models, perceived under a larger modeling framework, called explanatory item response models, proposed by De Boeck and Wilson. This extends the typical application of IRTree models for studying response styles. As a demonstration, we applied the descriptive and explanatory IRTree models to examine acquiescence and extreme response styles in Rosenberg's Self-Esteem Scale. Our findings suggested the presence of two distinct extreme response styles and acquiescence response style in the scale.

This paper examined whether year round schooling (YRS) in California had an effect upon the outcome and growth of schools' Academic Performance Index (API) scores. While many previous studies had examined the connection between YRS and academic achievement, most had lacked the statistical rigour required to provide reliable interpretations. As a response, this study used data collected from 4,569 schools over six years and two integrated and more sophisticated statistical techniques--mixed analysis of covariance and latent growth model. Results showed that YRS did not affect either the outcome or the growth of API scores. (Contains 5 tables, 2 footnotes, and 4 figures.)

Since at least the 1980s, motivated, in part, by findings from international comparisons of students' mathematics achievement, some American (U.S.) educators and policy makers have initiated educational reform that focuses on improving teaching practices and curriculum designs by advocating for the adoption of Asian educational models. The implicit assumption behind such reform seems to overlook the possible roles of outside of school factors in superior academic performance in the Asian countries. The purpose of this study was to investigate how outside of school factors were related to students' TIMSS mathematics achievement for the U.S and five top-performing Asian nations or regions, and to examine whether the relative role of school-associated and outside of school factors was consistent across these five Asian countries. The results indicated that students' achievement was highly related to outside of school factors across all six countries. For Singapore and Hong Kong, school-associated factors had somewhat more salient roles than the outside of school factors. However, for Korea, Taiwan, and Japan, school-associated factors played lessor roles in students' achievement performance. As for the U.S., school-associated factors were related to students' performance, however outside of school factors clearly played more salient roles than the school-associated factors. The findings suggest that students' mathematics achievement covaries substantially with outside of school factors. Hence, students' superior mathematics achievement does not necessarily reflect only a superior public school system but also outside of school factors. (Contains 10 tables and 2 figures.)

This article investigates how test-takers change their strategies to handle increased test difficulty. An adult sample reported their test-taking strategies immediately after completing the tasks in a reading test. Data were analyzed using structural equation modeling specifying a measurement-invariant, ability-moderated, latent transition analysis in M"plus" (Muthén & Asparouhov, 2011). It was found that almost half of the test-takers (47%) changed their strategies when encountering increased task-difficulty. The changes were characterized by augmenting comprehending-meaning strategies with score-maximizing and test-wiseness strategies. Moreover, test-takers' ability was the driving influence that facilitated and/or buffered the changes. The test outcomes, when reviewed in light of adjusted test-taking strategies, demonstrated a form of process-based validity evidence.

This article explores an approach for test score validation that examines test takers' strategies for taking a reading comprehension test. The authors formulated three working hypotheses about score validity pertaining to three types of test-taking strategy (comprehending meaning, test management, and test-wiseness). These hypotheses were formulated in terms of the use of three types of test-taking strategy and their relationships with performance on specific task types (testlets) and overall test performances. We illustrated the proposed method for validation using example data from the Canadian English Language Proficiency Index Program-General (CELPIP-General) reading pilot test. The findings were that (a) test takers were engaging more in processing the texts for comprehending meaning, less in test-management skills, and least in test-wiseness; (b) at the task level, task characteristics (e.g., difficulty) had implications on test takers' engagement with different types of strategies, which, in turn, led to differences in predicting task performances; and (c) at the test level, higher engagement in comprehending meaning led to higher test performance, engagement in test management showed a small negative association with test performance, and higher engagement in test-wiseness led to poorer performance. The high congruence between the working hypotheses and the empirical results offered plausible evidence that supported the validity of CELPIP-General reading scores. Revisions to both hypotheses and research design that might improve the proposed validation method are reviewed in the "Discussion" section.

This article proposes and demonstrates a methodology for test score validation through abductive reasoning. It describes how abductive reasoning can be utilized in support of the claims made about test score validity. This methodology is demonstrated with a real data example of the Canadian English Language Proficiency Index Program (CELPIP)-General test--a program assessing functional English language ability in the community and workplace. Abductive reasoning seeks the enabling conditions through which a claim about a person's ability makes sense. For example, it makes sense that a person has strong functional language proficiency if he or she has been regularly using English to write emails and meet with colleagues at work. A valid test score should be affected by the extent of a person's engagement with such enabling conditions. Empirical evidence that warrants such an abductively reasoned claim is illustrated through a latent class analysis within a structural equation model. Evidence is examined to investigate whether certain classes of test takers who have been differentially engaging in the enabling conditions do, in fact, predict a person's CELPIP-General performance. The steps of the methodology are summarized in the closing section.

Methods for detecting differential item functioning (DIF) and item bias are typically used in the process of item analysis when developing new measures; adapting existing measures for different populations, languages, or cultures; or more generally validating test score inferences. In 2007 in "Language Assessment Quarterly," Zumbo introduced the concept of Third Generation DIF. In the current article we introduce a new methodology, latent class logistic regression, for Zumbo's Third Generation DIF, whose foundation is a novel ecological model of item responding. The ecological model and the new statistical methodology are introduced, and a proof-of-concept is provided, in the context of an example of an international reading test focusing on DIF due to testing language. The new DIF framework is described and contrasted with other methods, "Mplus" code is provided, and the new method is shown to have potential for application in assessment.

This article describes a method based on Pratt's measures and demonstrates its use in exploratory factor analyses. The article discusses the interpretational complexities due to factor correlations and how Pratt's measures resolve these interpretational problems. Two real data examples demonstrate the calculation of what we call the "D matrix," of which the elements are Pratt's measures. Focusing on the rows of the D matrix allows one to compare the importance of the factors to the communality of each observed indicator ("horizontal interpretation"); whereas a focus on the columns of the D matrix allows one to compare the contribution of the indicators to the common variance extracted by each factor ("vertical interpretation"). The application showed that the method based on Pratt's measures is a very simple but useful technique for EFA, in particular, for behavioral and developmental constructs, which are often multidimensional and mutually correlated.

Previous studies have rarely examined the impact of outliers on the decisions about the number of factors to extract in an exploratory factor analysis. The few studies that have investigated this issue have arrived at contradictory conclusions regarding whether outliers inflated or deflated the number of factors extracted. By systematically inducing outliers as well as computer simulations based on real data, the present study demonstrated how outliers affected the decisions about the number of factors to extract using four commonly used and/or recommended decision methods. The studies revealed that both inflation and deflation of the number of factors were found, but the effect depended on (a) the decision methods used and (b) the magnitude and amount of outliers, hence resolving the apparent contradictory conclusions in the previous literature. (Contains 1 note, 6 tables and 3 figures.)

This paper described the versatility of the multiple-indicator multilevel (MIML) model in helping to resolve four common challenges in studying growth using longitudinal data. These challenges are (1) how to deal with changes in measurement over time and investigate temporal measurement invariance, (2) how to model residual dependence due to the nested nature of longitudinal data, (3) how to model observed trajectories that do not follow well-known functions commonly discussed in the methodology literature (e.g., a linear or quadratic curve), and (4) how to decide which predictors are relatively more important in explaining individuals' change over time. With an example of psychological well-being from the Wisconsin Longitudinal Study, we illustrated how the four methodological challenges can be resolved using the 3-phase MIML procedures and the Pratt's importance measures.

In a recent Monte Carlo simulation study, Liu and Zumbo showed that outliers can severely inflate the estimates of Cronbach's coefficient alpha for continuous item response data--visual analogue response format. Little, however, is known about the effect of outliers for ordinal item response data--also commonly referred to as Likert, Likert-type, ordered categorical, or ordinal/rating scale item responses. Building on the work of Liu and Zumbo, the authors investigated the effects of outlier contamination for binary and ordinal response scales. Their results showed that coefficient alpha estimates were severely inflated with the presence of outliers, and like the earlier findings, the effects of outliers were reduced with increasing theoretical reliability. The efficiency of coefficient alpha estimates (i.e., sample-to-sample variation) was inflated as well and affected by the number of scale points. It is worth noting that when there were no outliers, the alpha estimates were downward biased because of the ordinal scaling. However, the alpha estimates were, in general, inflated in the presence of outliers leading to positive bias. (Contains 1 note, 2 tables, and 3 figures.)

Mediation and moderation are two theories for refining and understanding a causal relationship. Empirical investigation of mediators and moderators requires an integrated research design rather than the data analyses driven approach often seen in the literature. This paper described the conceptual foundation, research design, data analysis, as well as inferences involved in a mediation and/or moderation investigation in both experimental and non-experimental (i.e., correlational) contexts. The essential distinctions between the investigation of mediators and moderators were summarized and juxtaposed in an example of a causal relationship between test difficulty and test anxiety. In addition, the more elaborate models, moderated mediation and mediated moderation, the use of structural equation models, and the problems with model misspecification were discussed conceptually.

Identifying the sources of differential item functioning (DIF) in international assessments is very challenging, because such sources are often nebulous and intertwined. Even though researchers frequently focus on test translation and content area, few actually go beyond these factors to investigate other cultural sources of DIF. This article introduces the multiple-variable matching method using logistic regression analysis to identify sources of DIF. A case study demonstrates how this methodology identified Extra Lesson Hours After School (ELHAS) as a potential source of DIF between Taiwan and the United States in the Third International Mathematics and Science Study (TIMSS) 1999. DIF is not a fixed character of any test item, nor is a cultural factor an inherent source of DIF. The legitimacy of a source of DIF relies on the specific context and purpose for the cross-country comparison.

This article describes a factor analytically derived, self-report instrument - the Schedule of Compulsions, Obsessions, and Pathological Impulses (SCOPI) - using data from college students, adults, psychiatric outpatients, and patients with obsessive-compulsive disorder (OCD). The five SCOPI scales all are internally consistent (with coefficient alphas of .80 and higher) and are strongly stable across a 2-month interval (with retest correlations ranging from .79 to .82). They also show good convergent and adequate discriminant validity (a) when correlated with other OCD measures and (b) in analyses of self-ratings versus spouse ratings. Additional analyses indicate that three scales - Obsessive Checking, Obsessive Cleanliness, and Compulsive Rituals - assess core symptoms of OCD. The two remaining scales - Hoarding and Pathological Impulses - appear to tap different (but related) constructs.