Search

As the theme of the 2022 annual meeting of the American Education Research Association, cultivating equitable education systems has gained renewed attention amid an increasingly diverse society. However, systemic inequalities persist for traditionally underserved student populations. As a way to better address diverse students' needs, it is of critical importance to understand different subgroups' performances. In the educational measurement field, evaluating the differences among multiple groups is an important consideration in addressing fairness issues for diverse groups of students. This article offers one technique to do so. It demonstrates how commonly-used multiple regression analysis can be applied to evaluate the equivalence of predictive structure across multiple groups in place of the factor analytic approach that requires a relatively large sample size per subgroup and strong assumptions. The technique is utilized in examining the relationship between English language proficiency and academic performance of English learners in one state when the subgroups are categorized by home language. The results showed statistically significant group differences between the reference group (Spanish-speaking ELs) and other focal groups (different home-language ELs) in various levels of comparisons (model fit, model structure, and individual predictor weights). The strengths and limitations of a proposed multiple group regression (MGR) approach are discussed in the educational research context.

The "GRE"® Psychology test is an achievement test that measures core knowledge in 12 content domains that represent the courses commonly offered at the undergraduate level. Currently, a total score and 2 subscores, experimental and social, are reported to test takers as well as graduate institutions. However, the American Psychological Association (APA) Board of Educational Affairs and the Council of University Directors of Clinical Psychology (CUDCP) have expressed a need for reporting new subscores. To meet such a demand, Educational Testing Service (ETS) content experts, in collaboration with the APA and CUDCP, proposed a new mapping of 12 content domains onto 6 subtest areas: biological; cognitive; social; developmental; clinical; and measurement, methodology, and other. In this study, we investigated the latent structure of the test and evaluated the statistical properties of the 6 subscores. Factor analyses showed that the test was essentially unidimensional. The disattenuated correlations among the 6 raw subscores were relatively high. However, when augmented with the total score, the subscores displayed low to moderate value added. Further, profile analyses showed that the augmented subscores displayed noticeably distinct profiles among test takers with the same total score. No evidence suggested that the content mapping of the 6 subscores was unreasonable. Additional validity research is needed to support the use of the 6 subscores, once their intended use is made clearer.

In this study, we assessed the invariance in the factor structure underlying English-language proficiency for two groups of adolescent learners in Japan: students in middle school (ages 13-15 years) and students in high school (ages 16-18 years). Language proficiency was measured using the "TOEFL Junior"® Comprehensive test, an assessment designed to measure the English skills of adolescent English learners in non-English-speaking countries. The study results indicate that a correlated 4-factor model corresponding to the 4 language abilities of reading, listening, speaking, and writing best represents the nature of language proficiency in the 2 groups of adolescent English learners. Moreover, the factor structure is invariant across these 2 groups. However, there is a statistically significant difference in performance on the reading construct. The results are consistent across 2 random samples, thus providing confirmatory evidence of model invariance.This study provides empirical support for the current score-reporting practices for the TOEFL Junior Comprehensive test and suggests that the test scores have the same meaning across test takers from middle school and high school.

Implementing a randomized controlled trial design, the present study investigated the effects of two types of accommodations, linguistic modification and a glossary, for English learners (ELs) taking a computer-based mathematics assessment. Process data including response time and clicks on glossary words were also examined to better interpret students' interaction with the accommodations in the testing conditions. Regression and ANOVA analyses were performed with data from 513 students (189 ELs and 324 non-ELs) in Grade 9. No statistically significant accommodation effects were detected in this study. Process data revealed possible explanations (i.e., student engagement and glossary usage) for the nonsignificant results. Implications for future research on test accommodations for EL students are discussed.

This study examined the heterogeneity in the English-as-a-second-language (ESL) test population by modeling the relationship between test-taker background characteristics and test performance as measured by the "TOEFL iBT"® using a confirmatory factor analysis (CFA) with covariate approach. The background characteristics studied included: (a) main reason for taking the TOEFL iBT test; (b) time spent studying English; (c) time spent attending a school, college, or university in which content classes were taught in English; and (d) lived in a country where English is the main language. The results indicated that at most levels of the background characteristics studied, there were statistically significant differences in the means of the four underlying latent factors (reading, listening, speaking, and writing) representing English-language proficiency (ELP). Overall, the effect size differences on the reading, listening, speaking, and writing latent factors among the levels for each of the background variables studied ranged from small to medium. The results of this study provide empirical evidence of the association and possible influence of test-taker background characteristics on the four underlying latent factors representing ELP and, thus, on test performance.

The purpose of this inquiry was to investigate the effectiveness of item response theory (IRT) proficiency estimators in terms of estimation bias and error under multistage testing (MST). We chose a 2-stage MST design in which 1 adaptation to the examinees' ability levels takes place. It includes 4 modules (1 at Stage 1, 3 at Stage 2) and 3 paths (low, middle, and high). When creating 2-stage MST panels (i.e., forms), we manipulated 2 assembly conditions in each module, such as difficulty level and module length, to see if any interaction existed between IRT estimation methods and MST panel designs. For each panel, we compared the accuracy of examinees' proficiency levels derived from 7 IRT proficiency estimators. We found that the choice of Bayesian (prior) and non-Bayesian (no prior) estimators was of more practical significance than the choice of number-correct versus item-pattern scoring. For the extreme proficiency levels, the decrease in standard error compensated for the increase in bias in the Bayesian estimates, resulting in smaller total error. Possible score changes caused by the use of different proficiency estimators would be nonnegligible, particularly for the extreme proficiency level examinees. The impact of misrouting at Stage 1 was minimal under the MST design used in this study.

This study illustrates the use of score equity assessment (SEA) for evaluating the fairness of reported test scores from assessments intended for test takers from diverse cultural, linguistic, and educational backgrounds, using a workplace English proficiency test. Subgroups were defined by test-taker background characteristics that research has shown to be associated with performance on language tests. The characteristics studied included gender, age, educational background, language exposure, and previous experience with the assessment. Overall, the empirical results indicated that the statistical and psychometric methods used in producing test scores were not strongly influenced by the subgroups of test takers from which the scores were derived. This result provides evidence in support of the comparability and meaning of test scores across the various test-taker groups studied. This example may encourage language testing programs to incorporate SEA analyses to provide evidence to inform the validity and fairness of reported scores for all groups of test takers.

This study assessed the factor structure of the Test of English for International Communication (TOEIC®) Listening and Reading test, and its invariance across subgroups of test-takers. The subgroups were defined by (a) gender, (b) age, (c) employment status, (d) time spent studying English, and (e) having lived in a country where English is the main language. The study results indicated that a correlated two-factor model corresponding to the two language abilities of listening and reading best accounted for the factor structure of the test. In addition, the underlying construct had the same structure across the test-taker subgroups studied. There were, however, significant differences in the means of the latent construct across the subgroups. This study provides empirical support for the current score reporting practice for the TOEIC test, suggests that the test scores have the same meaning across studied test-taker subgroups, and identifies possible test-taker background characteristics that affect English language abilities as measured by the TOEIC test.

This inquiry is an investigation of item response theory (IRT) proficiency estimators' accuracy under multistage testing (MST). We chose a two-stage MST design that includes four modules (one at Stage 1, three at Stage 2) and three difficulty paths (low, middle, high). We assembled various two-stage MST panels (i.e., forms) by manipulating two assembly conditions in each module, such as difficulty level and module length. For each panel, we investigated the accuracy of examinees' proficiency levels derived from seven IRT proficiency estimators. The choice of Bayesian (prior) versus non-Bayesian (no prior) estimators was of more practical significance than the choice of number-correct versus item-pattern scoring estimators. The Bayesian estimators were slightly more efficient than the non-Bayesian estimators, resulting in smaller overall error. Possible score changes caused by the use of different proficiency estimators would be nonnegligible, particularly for low- and high-performing examinees.

Item analysis is an integral part of operational test development and is typically conducted within two popular statistical frameworks: classical test theory (CTT) and item response theory (IRT). In this digital ITEMS module, Hanwook Yoo and Ronald K. Hambleton provide an accessible overview of operational item analysis approaches within these frameworks. They review the different stages of test development and associated item analyses to identify poorly performing items and effective item selection. Moreover, they walk through the computational and interpretational steps for CTT- and IRT-based evaluation statistics using simulated data examples and review various graphical displays such as distractor response curves, item characteristic curves, and item information curves. The digital module contains sample data, Excel sheets with various templates and examples, diagnostic quiz questions, data-based activities, curated resources, and a glossary.