Search

In "Understanding Pandemic Learning Loss and Learning Recovery: The Role of Student Growth & Statewide Testing" we show how states, with spring 2021 assessment data together with student growth results derived from it, are positioned to address critical questions regarding academic learning loss. Answers to these questions, together with academic data derived from other sources, will help states, districts and schools form approaches to ameliorating the impact of the pandemic on their students. As of January 2021, the United States is almost completely in the dark as to the pandemic's impact on student learning. Early interim assessment results have given us a glimpse of the magnitude of the impact but have been extremely limited otherwise. We need statewide assessment and growth data to intelligently move forward.

On August 9, 2012, the Institute of Education Sciences (IES) convened a meeting of researchers and U.S. Department of Education (ED) and IES staff to discuss recent advances in measuring teacher effectiveness. The objectives of the meeting were to: (1) Identify and discuss recent advances in the use of value-added models (VAMs) and student growth models (SGMs) as one measure of teacher effectiveness, including their implications, applications, and limitations for practicing educators; and (2) Identify considerations for future research. Participants from the research community were carefully chosen to represent a broad range of academic disciplines, expertise and perspectives. Each participant from the research community was asked to share in advance of the meeting a two-page written summary of their views on the implications of recent advances in value-added and SGM measures. These briefs are presented here. (Individual briefs contain references.) [For the meeting summary, see ED539039.]

In this report, student growth percentiles and percentile growth projections/trajectories are introduced as a means of understanding student growth in both normative and a criterion referenced ways. With these values calculated, the author shows how growth data can be utilized in both a norm- and in a criterion-referenced manner to inform discussion about education quality. The author asserts that the establishment of a normative basis for student growth eliminates a number of the problems of incorporating growth into accountability systems, providing needed insight to various stakeholders by addressing the basic question of how much a student has progressed.

This study was performed to determine whether hierarchical logistic regression models could reduce the sample size requirements of ordinary (nonhierarchical) logistic regression models. Data from courses with varying class size were randomly partitioned into two halves per course. Grades of students in college algebra courses were obtained from 40 colleges. The largest sample size group, Group 4, contained 11 colleges with half counts ranging from 171 to 563 (average 307). Nonhierarchical and hierarchical analyses were performed on each half. Compared to their nonhierarchical counterparts, hierarchically estimated cutoff scores from different halves were closer together in value and predicted course outcomes in the other half more accurately. These differences were most pronounced with small samples. It is concluded that the sample size requirements could be substantially reduced if hierarchical logistic regression were used to estimate cutoff scores. (Contains 2 tables, 7 figures, and 28 references.) (SLD)

This study was commissioned to investigate issues related to the inclusion of the American College Testing Program (ACT) assessment within the Colorado State Assessment Program (CSAP). Analyses were carried out using assessment materials, assessment results, and various data from the ACT and the Colorado State Department of Education. One focus of the investigation was the match between the Colorado Model Content Standards (CMCS) and the ACT Assessment. Data indicate that the ACT does not match well with the CMCS, and the match is especially poor in science. This finding casts doubts on the advisability of requiring all Colorado 11th graders to take the ACT. The poor match could potentially inject incoherence into the CMCS-based accountability program as a whole, and it raises questions about how ACT results might be combined with CSAP performance in the calculation of Student Accountability Reports for high schools. (SLD)

The effect of testing accommodations, such as extra time, oral reading of the assessment, or small group testing, on the participation and performance levels of limited-English-proficient students (LEP) on the Rhode Island Grade 4 Mathematics Performance Assessment was studied. A pilot study was conducted with 22 classes of students to provide preliminary evidence of the relative validity of both the performance assessment and the traditionally used Metropolitan Achievement Test (MAT) for language-minority students in comparison with general education students. The statewide data show a clear increase in the number of LED and special education students participating in the performance assessment compared to those who took the MAT. This was probably a result of the availability of accommodations or the test instructions, which stressed full inclusion. Accommodations consistently raised the relative position of LEP and special education students compared to their position on the MAT in the past. In the operational statewide assessment there was no way to evaluate the validity of the achievement gains associated with the use of accommodations, but for the most part the level of gain appeared reasonable. In the pilot sample, teachers' mathematics grades and teachers' standards-based ratings in mathematics could be used as validity criteria to evaluate the performance assessment and the MAT. Evidence supports the validity of the tests for general education students and for language-minority students, but is not as strong for the validity for LEP students. For a first effort, the inclusion of LEP students in the Rhode Island fourth-grade assessment appears to have been reasonably successful. An appendix discusses method to evaluate differential item functioning. (Contains 13 tables, 17 figures, and 19 references.) (SLD)

For assessments that present problems that require extended responses and substantial amounts of time, there is often a desire to allow students to choose which problem they will respond to among two or more options. Student choice of problem may allow students a better opportunity to demonstrate what they know and are able to do. On the other hand, choice raises questions about the comparability of scores obtained by students who respond to different problems. Questions of comparability and validity of scores obtained when students are given a choice among alternative problems were investigated using data for approximately 30,000 students from the Oregon State Assessment Program for Grade 10 Mathematics Assessment administered in spring 1997. The assessment consisted of a multiple-choice section and a pair of extended-response problems. On each of six alternate forms, two problems were presented, and students were instructed to choose one. Data from the six forms were analyzed to evaluate the comparability of scores obtained from responses to different tasks and the validity of the results. It was found that problems differed in popularity, and that the scores students obtained differed systematically as a function of problem choice. On the other hand, confirmatory factor analysis results across forms for students choosing different problems suggest that there was similar validity for measuring the underlying constructs across problem choice. It is concluded that while choice may be justified, some form of equating adjustments would be needed before making high-stakes decisions based on performance of students on problems where choice is allowed. (Contains 23 tables, 2 figures, and 9 references.) (Author/SLD)

The zeitgeist for reform in education precipitated a number of changes in assessment. Among these are performance assessments, sometimes linked to "high stakes" accountability decisions. In some instances, the trustworthiness of these decisions is based on variance components and error variances derived through generalizability theory. Often overlooked is the fact that these statistics are subject to sampling error. This paper introduces techniques used to determine the accuracy of such statistics. It addresses the shortcomings of overlooking sampling error by presenting a number of results with respect to confidence intervals about linear combinations of expected mean squares appropriate for generalizability theory. Simulation results indicate that these intervals, particularly the two-sided and one-sided lower intervals, are accurate or conservative both in simple and complex designs with varying amounts of difference in degrees of freedom across effects. (Contains 1 figure and 12 references.) (Author/SLD)

This paper is an examination of student assessment data using multi-level analysis methods often referred to as "value-added" models. The analyses performed provide measures of "effectiveness" for both teachers and schools. The purpose of the paper is to examine the residuals derived from the model for teachers and schools (i.e., their "value-added" scores) and to examine their relation to demographic variables at the classroom and school level. Three models are examined: (1) the basic variance components model; (2) the random intercepts model; and (3) a model including demographic covariates. Those interpreting the results should be aware of the likelihood of model misspecification in contexts where value-added measures are derived. These measures are a function of the model and much care must be taken to rule out the possibility of model misspecification. Variable description is appended.