Search

Tests of English Language Proficiency are often designed such that each section of the test measures a single latent ability. For instance an English Proficiency Assessment might consist of sections measuring Speaking, Listening, and Reading ability. However, Overall English Proficiency and composite abilities are naturally multidimensional. This multidimensionality may even include the case where each item measures more than one latent ability. Parameters for these examples can be estimated within a one-dimensional or multidimensional item response modeling framework. This study compares two common strategies for composite scoring in Test of English Language Proficiency to composite scoring strategies derived from multidimensional models. An example is taken from a sample of 12,008 elementary school students who participated in a State English Proficiency Assessment. Factor analytic techniques were performed to verify the dimensional structure of the composite domains for the Oral scale. Item Response Models with separate unidimensional calibration, simultaneous unidmensional calibration, and multidimensional calibration were examined. Model-data fit, information functions, ability estimates, and interpretive capability among the three different types of composite scores were examined. (Contains 19 figures and 6 tables.)

Year-to-year rater variation may result in constructed response (CR) parameter changes, making CR items inappropriate to use in anchor sets for linking or equating. This study demonstrates how rater severity affected the writing and reading scores. Rater adjustments were made to statewide results using an item response theory (IRT) methodology based on the work of Tate (1999, 2000). The common item equating design was used to place the second year scores to the first year scores after a re-score of the first year test in order to adjust for rater effects. Two samples of data from contiguous years, designated as Year 1 (n ~ 1,200) and Year 2 (n ~ 2,000), from the writing and reading portions of a statewide assessment were examined. The writing test consisted of 32, 36, and 40 selected-response items for grade 4, 6, and 8 and a single writing prompt scored on a six-point scale (0-5) scored by two raters whose scores are added for a composite. The reading test consists of 75, 93, and 91 selected-response items and 12, 14, and 16 constructed response items for grade 4, 6, and 8, respectively. All the CR items in reading were scored on a three-point scale (0-2.) The resulting item parameters were compared between year one and two, with and without rater adjustment. For writing, there were significant shifts in the parameters after rater adjustment. The p-values and TCCs shifted across years when adjusted for rater effects. The impact of the parameter shifts and TCCs manifested in the changes in the proficiency classification before and after adjustment. The results of the study suggests that raters were not consistently more severe or more lenient between grades or content areas, but the resulting rater error (severity or leniency) affected the scores and thereby produced misleading results if not taken into account. (Contains 6 tables and 6 figures.)

The main purpose of this study was to illustrate a polytomous IRT-based linking procedure that adjusts for rater variations. Test scores from two administrations of a statewide reading assessment were used. An anchor set of Year 1 students' constructed responses were rescored by Year 2 raters. To adjust for year-to-year rater variation in IRT linking, a two-step approach was used. First, the Year 1 item parameters for the constructed response items were re-estimated using the ratings assigned by the Year 2 raters. Through this recalibration process, the Year 1 item parameters for the constructed response items were adjusted for rater variation. Second, the Stocking-Lord equating method was used to place the Year 2 form on the Year 1 scale using all the common items between forms. This method was compared with two alternative methods: traditional IRT linking study that links the test forms using (a) all the common items without the rater adjustment and (b) the common selected response items. The results of the Stocking-Lord procedure was compared among the three methods. The differences in the test characteristic curves and the students' scale score distributions produced by the three methods were also compared. Significant shifts in the parameters after rater adjustment were found for one (grade 8) of the three grades examined. The p-values and TCCs shifted across years when adjusted for rater effects. The impact of the parameter shifts and TCCs manifested in the changes in the proficiency classification before and after adjustment. However, a systematic bias might have been introduced in the equating process while trying to adjust for the rater variation through the equating process. Further studies are needed to address this problem in greater detail. Using only the common SR items seems to produce satisfactory results. Thus, in the case where it is not feasible to integrate rater adjustment in the equating process, using SR item anchors is a better approach than using the mixed-item anchors without adjusting for rater effect. The results of the study suggest that raters were not consistently more severe or more lenient between grades, but the resulting rater error (severity or leniency) affected the scores and thereby produced misleading results if not taken into account. (Contains 10 tables and 9 figures.)

Content integration in performance assessment involves mixing different areas of knowledge in one assessment. In this type of testing situation, assessment tasks are designed to measure the ability of students to solve problems by applying their knowledge and skills in multiple content areas. This study examined the effect of integrated science and reading items on the reliability and construct validity of reading assessment for the Maryland School Performance Assessment Program (MSPAP). Using the MSPAP 1998 reading data, this study demonstrated that the integrated science-reading items provide reliable and valid information about the students reading abilities that are comparable to the nonintegrated items. Results suggest that the integrated science-reading items did not compromise the integrity of reading scales in question. There is evidence that the scales are essentially unidimensional. However, there are minor task-related factors that warrant further analysis on the latent structure of the reading scales. This finding is complemented by the high local item dependency among tasks. (Contains 4 tables and 23 references.) (Author/SLD)

This study evaluated the comparative stability and agreement of three approaches to calculating school effects given both student-level and school-level data. The approaches were hierarchical linear modeling (HLM), ordinary least squares (OLS), and weighted least squares (WLS). Analyses were conducted using data from the 1998 Maryland School Performance Assessment Program for 23,461 third graders and 21,226 fifth graders. A two-level model was used for computing HLM school effects with four student-level predictors used to predict achievement in level one, and the school size used in level two to predict the level-1 intercept. The dependent achievement measure in OLS and WLS analyses was the average student score across the six content areas of the assessment. In OLS five variables were used as predictors, at the school level only, and in WLS the same strategy was used except that the sampling variance of the dependent variable was estimated for each school and used as the weighting variable. For OLS and WLS studentized residuals were used as the school effects measure. Results of the analyses indicate that, from a practical perspective, and all other considerations being equal, the HLM approach should be used for school effects measures on the basis of stability. The use of either of the school-level models appears to be viable in the event that only school-level data are available. Reasons for the greater stability of the HLM approach are discussed. (Contains 5 tables and 10 references.) (SLD)

The latent structure of the reading test of the International Association for the Evaluation of Educational Achievement (IEA) Reading Literacy study was investigated. The use of latent class modeling for investigating the measurement properties of a large-scale reading assessment database is demonstrated. The study focused on response data from the United States, Canada, Hong Kong, and Denmark, for a total sample of 15,304 nine-year-olds. One of the main research questions was whether the type of text (narrative, expository, and documents) elicits different types of reading comprehension. Findings indicate that there is no empirical evidence of three separate text types for any of the four countries, but there is empirical evidence of two separate reading domains. One latent variable represents a general reading domain associated with all three text types, and the other represents a prose reading domain associated only with expository and narrative text types. The latent class modeling approach provides a way to examine the latent structure of the IEA reading test that avoids many problems that may occur when applying factor-analytic methodology. (Contains 3 figures, 12 tables, and 30 references.) (SLD)