Search

In the fall of 2017, ACT began providing a limited number of supports (also known as accommodations) to English Learner (EL) students in the US taking the ACT® test. The goal of the supports is to remove construct-irrelevant variance in students' scores related to limited English proficiency and allow students to more accurately demonstrate their true proficiency on the construct being measured. This paper presents a summary of previous research on the impact of providing supports to EL students and a summary of a preliminary analysis conducted by ACT based on data from states who administered the ACT statewide, captured students' EL status, and provided accommodations to EL students. ACT's future research agenda for providing validity evidence for this initiative is also discussed.

The term "equating" refers to a statistical procedure that adjusts test scores on different forms of the same examination so that scores can be interpreted interchangeably. This study examines the impact of equating with fewer items than originally planned when items have been removed from the equating set for a variety of reasons. A real data set from a licensure/certification examination was used for the study. Three linear equating methods and three test forms were involved. The sample size for each of the test forms was at least 10,000 examinees. Item sets were manipulated to discard different numbers of items. The scale scores computed using the decreased sets of equators were either unchanged or one point higher or lower than the scale scores computed using the original set of equators. The scale score fluctuations in nearly all of the decreased equating sets affected the passing scores of approximately two percent of the examinees, either increasing or decreasing the required scale score by one point. For a set in which two scale scores varied rather than just one, approximately four percent of the examinees were affected. Findings show that the equating plan designed for this examination was highly robust. (SLD)

Item pools supporting computer-based tests are not always completely calibrated. Occasionally, only a small subset of the items in the pool may have actual calibrations, while the remainder of the items may only have classical item statistics, (e.g., "p"-values, point-biserial correlation coefficients, or biserial correlation coefficients). Transformations can be applied to the classical statistics to obtain rough estimates of the item parameters from a three-parameter logistic item response model. These estimates, in turn, can be improved by linking them to items with actual calibrations from a program such as BILOG. The resulting item parameter estimates can then be used in a computerized classification test (CCT). An evaluation of the results of using such estimated parameters in simulated CCTs is presented in this paper. Seven simulated item pools were used. Results lead to the suggestion that when considering use of such augmented parameter estimates as those produced by the linking procedures in this paper, consideration be given to the sources of possible error and their effects on the outcomes of the test. (Contains 12 tables and 6 figures.) (SLD)

This study examined the reliability of three methods for detecting differential item functioning (DIF) (i.e., the Mantel-Haenszel method, the standardization method, and the logistic regression method) applied to achievement test data. In addition, the study examined the influences of different sources of error variance, including examinee, occasion, and curriculum sampling on the magnitude of the reliability of the different DIF detection methods. Three datasets were assembled from the 1992 spring and fall standardization administration of the Iowa Tests of Basic Skills, and these were manipulated to control for error variance sources. Results indicated that the Mantel-Haenszel and standardization methods were more reliable in detecting DIF than the logistic regression method. The data also indicated that controlling the error variance of curriculum sampling slightly increased the reliability of DIF detection while controlling for error variance due to examinee sampling gives confusing results. (Contains 13 tables, 8 figures, and 12 references.) (Author/SLD)

The primary concern or focus of a certification or licensure test is to obtain valid criterion-referenced information regarding a candidate's competence to practice. When the test is administered by computer, a valid pass/fail decision can be made with fewer items than an equivalent paper/pencil test by targeting items at the passing score and using a likelihood ratio test such as the one used in the sequential probability ratio test (SPRT). When administered on the computer, the SPRT is frequently referred to as a computerized classification test (CCT) to distinguish it from the usual computerized adaptive test. If the CCT is item response theory-based, an assumption of unidimensionality is usually required, and the concern is when the item pool is not essentially unidimensional. This study investigated the effects that a multidimensional item pool and latent ability space have on the accuracy of the decisions made using CCT. An item pool was created from the multidimensional calibration of six forms of the ACT Assessment mathematics test using the computer program NOHARM. The results with this set of 360 items show that the procedure may be fairly robust to such assumption violations. (Contains three tables and nine references.) (Author/SLD)

Generalizability theory is used to examine the sources of variability present in a teacher and course evaluation instrument. Two studies were conducted. In the first study, four different forms commonly used by one specific college of a large midwestern university were examined using responses of 915 students. The analysis of variance performed on each form separately indicated that one form did not generalize well across students in comparison with the other three. In the second study, the performance of a five-item form across three levels of courses (734 students in undergraduate, intermediate, and graduate courses) in one specific college were examined. The course evaluations from graduate level students were found to be more reliable. In this case, evaluations of undergraduate level courses should not be considered as reliable and generalizable as those collected in higher level courses. (Contains nine tables and nine references.) (Author/SLD)