Search

"Learning progressions" are one of the most important assessment design ideas to be introduced in the past decade. In the United States, several committees of the National Research Council (NRC) have argued for the use of learning progressions as a means to foster both deeper mastery of subject-matter content and higher level reasoning abilities. Consideration of learning progressions is especially important in the context of the new Common Core State Standards (CCSS) and Next Generation Science Standards (NGSS) that attend specifically to the sequencing of topics and skills across grades to ensure attainment of college and career expectations by the end of high school. In this paper we address the question: Should more formally developed learning progressions be considered for the future design of the National Assessment of Educational Progress (NAEP)? After a brief overview of the research on learning progressions, we describe the idealized model whereby shared, instructionally grounded learning progressions--once developed--could be used to link classroom-level assessments with large-scale assessments such as NAEP. At the same time, we also consider potential problems. In particular, learning progressions--which require agreed-upon instructional sequences--could be problematic in the context of a national assessment program intended to be curriculum neutral (i.e., not favoring one state's or district's curriculum over another). Finally, we use a sample of NAEP and Balanced Assessment in Mathematics (BAM; Mathematics Assessment Resource Service, 2002, 2003) items to explore the possibility of constructing "quasi learning progressions" that could be used to illuminate the substantive meaning of the NAEP achievement results. An appendix presents: Items in Learning Progressions. [For the main report, "Examining the Content and Context of the Common Core State Standards: A First Look at Implications for the National Assessment of Educational Progress," see ED545237.]

In 2001, the enactment of the No Child Left Behind Act intensified the pressure on school districts to raise test scores, close achievement gaps, and turn around low-performing schools. In response, a large number of school districts have adopted interim or benchmark assessments to be administered periodically throughout the school year in anticipation of annual state tests. This report focuses on middle-school mathematics teachers' uses of interim and benchmark assessment results. We present findings from two-stage interviews with 30 teachers in seven districts across two states. While teachers' uses of assessment information varied, few gained substantive insights about students' mathematical understanding. Instead, teachers most frequently retaught standards or items with the lowest scores and focused on procedural competence. Although many teachers expressed an interest in using assessment results to inform instruction, they reported minimal professional development to this end, and often had a different understanding regarding the intended use of the assessments than did district leaders. Examples of big picture growth and proficiency assessment information are appended. (Contains 1 table, 2 figures and 3 footnotes.)

This study is a validity study of the National Assessment of Educational Progress (NAEP), intended to test the adequacy of NAEP for detecting and monitoring the effects of mathematics education reform. The current study design was intended to support a comparison of the relative effectiveness of three different types of large-scale assessments--"Balanced Assessment in Mathematics" (BAM), NAEP, and state assessments--for measuring the learning gains of students participating in a well-implemented reform mathematics curriculum. To provide a context for assessing student learning where the authors could be reasonably certain of observing substantial learning gains in mathematics over the course of a school year, they selected National Science Foundation's (NSF's) Connected Mathematics Project (CMP). Although the authors had initially hypothesized that BAM, being more closely aligned with the reform curriculum, would reveal larger gains than NAEP, they found that both assessments were equally sensitive to the gains of their sample of students in CMP classrooms, and NAEP appeared better able to detect gains in the algebra classrooms. This was true even though the BAM test required twice as much time to administer as the NAEP test. Three appendices are included: (1) Sample NAEP Items; (2) Sample BAM Task; and (3) Analyses Using Booklet Percent Correct Metric. (Contains 21 tables, 1 figure and 8 footnotes.) [This report is based on work that was jointly supported by NCES (contract ED-01-CO-0026-005) and the National Science Foundation (grant 454755).]

Standards-based education reform has a more than 20-year history. A standards-based vision was enacted in federal law under the Clinton administration with the 1994 reauthorization of the Elementary and Secondary Education Act (ESEA) and carried forward under the Bush administration with the No Child Left Behind Act (NCLB) of 2001. In a recent survey of policy makers, standards were acknowledged as the central framework guiding state education policy. Yet, despite this apparent unanimity about the intuitively appealing idea of standards, there is great confusion about its "operational" meaning: exactly what should the standards be, how should they be set and by whom, and how should they be applied to ensure rigorous and high-quality education for American students are the central questions that challenge policy makers and educators. For example, "content" standards (subject-matter descriptions of what students should know and be able to do) are often confused with "performance" standards (which are more like passing scores on a test), and very different theories of action are used to explain how standards-based reforms are expected to work. Ambitious rhetoric has called for systemic reform and profound changes in curriculum and assessments to enable higher levels of learning. In reality, however, implementation of standards has frequently resulted in a much more familiar policy of test-based accountability, whereby test items often become crude proxies for the standards. This disconnect between rhetoric and reality is one of the reasons for the failure of prior reforms. Standards-based education is still the core idea guiding education policy and education reform. But the foregoing issues need to be addressed if the promises of standards-based education are to be kept. As yet, neither state content standards nor state tests reflect the ambitions of standards-based reform rhetoric, and the link between high expectations for all students and capacity building has been almost forgotten. The intentions of standards-based education--to focus greater attention on student learning, to ensure the participation and success of all students, and to provide guidance for educational improvement--are in the best interest of the country. This paper offers the following recommendations: (1) The federal government should encourage the redesign and clear connection of content and performance standards--and the curricula, teacher training, and high-quality assessments to go with them--with the goal of developing clearly articulated statements of the expected progression of learning. Efforts to develop these components may involve partnerships among states, universities, groups of teachers, scholars, and the private sector; (2) The federal government should support research on accountability system indicators to reflect both the status and growth of students. Performance standards should set ambitious but realistic targets for teaching and learning, and they should communicate to the public, parents, educators, and students themselves what is to be learned. Assessment results should be reported in ways that recognize progress all along the achievement continuum; (3) The federal government should support the redesign and ongoing evaluation of accountability systems to ensure that they contribute to school improvement. Less than satisfactory school performance should trigger closer investigation of school operations before remedies or sanctions are applied, and stellar performances should also be verified. Different investigative approaches, including audit assessments, data-driven analyses, or expert constituted inspectorates, should be considered; and (4) The federal government should support an intensive program of research and development to create the next generation of performance assessments explicitly linked to well-designed content standards and curricula. (Contains 77 notes.)

Using a random sample of 1,000 Colorado teachers, this study surveyed the effects of standards, the Colorado Student Assessment Program (CSAP), and school report cards on instruction and test-related practices. Findings show that standards were perceived to have greater impact on improving instruction than did testing. Teachers said they aligned their curriculum, instruction, and lessons to the Colorado standards by adding important content. Attention to the state standards improved the quality of writing instruction and focused instruction in reading, probability, geometry, and mathematics problem-solving explanations. The reported effects of CSAP testing were more mixed. Attention to CSAP improved writing instruction but shifted instruction away from social studies and science, increased the time spent on test format practice, and lowered faculty morale. Nine appendixes provide additional information, including survey responses. (Contains 12 tables and 26 references.) (Author/SLD)

This paper examines the validity of measures of teacher effectiveness from the Tennessee Value Added Assessment System (TVAAS). It considers the following claims regarding teacher effects: (1) they capture teachers' unique contributions to student learning adequately; (2) they reflect adequate standards of excellence for comparing teachers; (3) they provide useful diagnostic information to guide instructional practice; and (4) student test scores capture desired outcomes of teaching adequately. Previous analyses of the TVAAS model and simulation studies highlight potential weaknesses in the system and identify gaps in the current record of empirical evidence bearing on its validity. (Contains 2 figures and 13 references.) (Author/SLD)

The effect of testing accommodations, such as extra time, oral reading of the assessment, or small group testing, on the participation and performance levels of limited-English-proficient students (LEP) on the Rhode Island Grade 4 Mathematics Performance Assessment was studied. A pilot study was conducted with 22 classes of students to provide preliminary evidence of the relative validity of both the performance assessment and the traditionally used Metropolitan Achievement Test (MAT) for language-minority students in comparison with general education students. The statewide data show a clear increase in the number of LED and special education students participating in the performance assessment compared to those who took the MAT. This was probably a result of the availability of accommodations or the test instructions, which stressed full inclusion. Accommodations consistently raised the relative position of LEP and special education students compared to their position on the MAT in the past. In the operational statewide assessment there was no way to evaluate the validity of the achievement gains associated with the use of accommodations, but for the most part the level of gain appeared reasonable. In the pilot sample, teachers' mathematics grades and teachers' standards-based ratings in mathematics could be used as validity criteria to evaluate the performance assessment and the MAT. Evidence supports the validity of the tests for general education students and for language-minority students, but is not as strong for the validity for LEP students. For a first effort, the inclusion of LEP students in the Rhode Island fourth-grade assessment appears to have been reasonably successful. An appendix discusses method to evaluate differential item functioning. (Contains 13 tables, 17 figures, and 19 references.) (SLD)

This study documented current early childhood assessment policies and practices, focusing on changes since the 1988 nationwide survey and on changes occurring in response to Goal 1 of the National Education Goals, performance assessments, and integrated services for young children. Data were collected through telephone surveys and in-depth interviews with the state-level early childhood or elementary coordinator or testing director, as well as Part H and IDEA coordinators, Title I coordinators, Head Start directors, and the Goals 1 contact person. Documentary evidence of policies or legislative mandates was also obtained. The findings, in descending order of prevalence, follow: (1) most states have made an effort to move away from readiness testing and kindergarten retention, and there is a perceived reduction though not elimination of these practices; (2) almost all state-mandated standardized testing for school accountability has been eliminated for children below grade 3; (3) some states and local districts are moving to new assessment forms in the early grades that are more supportive of instruction; (4) misuse of screening instruments for instructional purposes has decreased since 1988; (5) professional training is needed to understand and use new assessment forms; (6) preschool testing is largely driven by mandates for categorical programs; (7) a few states are collecting data to report on progress toward Goal 1; (8) parent involvement was an issue both for identifying children with special needs and because of parent demands for standardized testing; and (9) lack of collaboration and coordination among agencies serving young children continues to be a problem. (Contains 16 references.) (KB)

The effects of performance assessments on student learning were examined in a year-long project to help teachers in 13 third-grade classrooms begin to use performance assessments as part of regular instruction in reading and mathematics. The essential research question was whether students learned more or developed qualitatively different understandings because performance assessments were introduced. Achievement results for the approximately 335 students were compared to the performance of third-grade students in the same schools the year before and to third-grade performance in matched control schools. Researchers worked with the teachers throughout the year to help them develop performance assessments congruent with their own instructional goals. Standardized achievement tests and some items from the Maryland State Department of Education's performance assessment were used to measure achievement. Overall, the predominant finding was one of no difference or no gains in student learning following the year-long effort to introduce performance assessment. Researchers indicated that they saw qualitative changes in performance, but it must be acknowledged that any demonstrated benefits in achievement were small and ephemeral. (Contains 10 tables, 4 figures, and 18 references.) (SLD)

Parent opinions about standardized tests and performance assessments were examined systematically. Mutually exclusive but randomly equivalent stratified samples from schools participating in a study of performance assessment and control schools were used to measure change in parent opinion over time. Approximately one-third of parents (n=105) completed questionnaires at the beginning of the school year, one-third completed them at the end of the year (similar sample), and the remaining third supplied interview samples (n=33 and n=27, respectively). Results demonstrated that parents' favorable ratings of standardized national tests did not imply a preference for this type of educational assessment over other types of assessment for measuring student or school progress. Parents considered report cards, hearing from the teacher, and seeing graded samples of student work as more informative than standardized tests, and they wanted comparative information to measure their own child's progress. When parents had a chance to look at performance assessments through the year, they endorsed their use for district purposes and preferred them for classroom use. Survey data like the Gallup Poll showing widespread approval of standardized tests should not be taken to mean that parents are opposed to other forms of assessment. Appendixes contain the parent questionnaire and the interview protocol. (Contains 3 figures, 17 tables, and 9 references.) (SLD)

As part of a larger study of implementation of alternative assessment, a survey of parent opinions about standardized tests and performance assessments was conducted in three elementary schools. In the three participating schools, 3rd-grade teachers attended workshops on assessment development and implemented these practices in their classrooms. Samples of 69 parents from participating schools and 36 from three control schools were interviewed after completing questionnaires. Findings suggest that parents' favorable ratings of standardized national tests, supported by a Gallup Poll on the issue, do not imply a preference for such measures over other less formal sources of information for monitoring their children's progress or for judging the quality of education at their local schools. Parents tended to rely on the teacher to tell them how their child was doing relative to others, and they seldom mentioned comparison to external and national norms. Even for accountability purposes, parents preferred talking to the teacher and seeing student work. Most parents endorsed the performance assessment problems they saw, although a few expressed concern over the subjectivity of such measures. Twelve tables present survey and interview findings. An appendix presents excerpts from some parent interviews. (Contains 12 references.) (SLD)

Claims that national tests will improve student learning are explored, asking whether national examinations will ensure high-quality instruction and greater student learning and whether tests developed to meet urgent political deadlines will retain essential features of authentic curriculum-driven assessments. Part I presents research evidence on the negative effects of standardized testing, such as the effects of high stakes testing on scores, the curriculum, and instruction. The National Education Goals Panel's (NEGP's) version of national examinations is presented in Part II, with attention to their proposals intended to forestall the negative effects of traditional tests. Part III identifies curricular and technical problems that must be resolved before the NEGP's vision can be achieved. These include: (1) development of world class rather than lowest common denominator standards; (2) development of incorruptible performance tasks; (3) teacher training in curriculum and instruction; (4) high standards for all students without reinstitution of tracking; and (5) cost. If tests are developed before these problems are resolved, new tests are likely to have the same pernicious effects as the old. There is a 32-item list of references. (SLD)

The place for standardized testing in Chapter 1 evaluation is discussed. There is substantial evidence available on the negative effects of high-stakes standardized testing, and there is a clear link between Chapter 1 requirements and the amount of testing in most school districts. Standardized testing is usually used to identify eligible students, evaluate the Chapter 1 program, and hold individual schools accountable. It is argued that each of these purposes can be served better by other means. Alternative assessments are needed for Chapter 1 use, but any such assessments must be removed from the tyranny of normal curve equivalent gains. Any system that is devised should be subjected to its own cost-benefit evaluation to determine the costs and side effects of program improvement monitoring. Five overhead projection figures used in the presentation are included. (SLD)

The effects of standardized testing on instruction were studied in two school districts with high-stakes testing. The present study was part of a larger research project concerned with the effect of testing on instruction and student learning. A total of 360 teachers in grades 3, 5, and 6 in approximately 100 schools in two districts answered a questionnaire addressing test preparation/coaching practices and the effects of testing on instruction. A limitation of the study was the overall response rate of 42%, suggesting that the respondents were not necessarily representative of all teachers. Teachers reported that they felt pressured to improve test scores by the district administration and the media. Because of the importance of testing, teachers gave greater emphasis to basic skills instruction. They felt that content not tested suffered because of the focus on the standardized tests. Testing further distorted teaching because of the extensive time given to test preparation. While it was agreed that flagrant cheating was rare, practices that would clearly boost test scores, such as rephrasing questions, were considered to occur more frequently. Teachers were aware of extensive use of test results for external purposes such as comparisons of schools or districts. In open-ended questions, teachers could point to many benefits from standardized testing, but they felt that these were outweighed by the drawbacks. Fourteen tables summarize the questionnaire results. A 13-item list of references is included. (SLD)

Beliefs that psychometricians hold about learning were examined through telephone interviews with directors of testing from all 50 states and with a sample of test directors from 50 selected school districts. Interpretations of what these measurement specialists believed were based on reanalyses of the primary narrative interview data. A majority of specialists operated from implicit learning theories that encourage the close alignment of tests with curriculum and judicious teaching of test content. These beliefs, associated with criterion-referenced testing, derive from behaviorist learning theory that requires the sequential mastery of constituent skills and behaviorally explicit testing of each learning step. This sequential facts-before-thinking model of learning is contraindicated by a substantial body of evidence from cognitive psychology. It is asserted that the hidden assumptions about learning should be examined precisely because they are covert. Formal debate among measurement specialists will help ensure that testing plays its desired role in the improvement of education. There is a 32-item list of references. Two appendices present two tables summarizing interview responses and nine figures illustrating models of measurement concepts. (SLD)

It is increasingly recognized, following the lead of J. J. Cannell, that actual gains in educational achievement may be much more modest than dramatic gains reported by many state assessments and many test publishers. An overview is presented of explanations of spurious test score gains. Focus is on determining how test-curriculum alignment and teaching the test influence the meaning of scores. Findings of a survey of state testing directors are summarized, and the question of teaching the test is examined. Some frequently presented explanations refer to norms used; others refer to aspects of teaching the test. Directors of testing from 46 states (four states conduct no state testing) replied to a survey about testing. Forty states clearly had high-stakes testing. The most pervasive source of high-stakes pressure identified by respondents was media coverage. Responses indicate that test-curriculum alignment and teaching the test are distorting instruction. A possible solution is to develop new tests every year, changing the tests rather than the norms. Two tables present explanations for test score inflation and selected survey responses. (SLD)

The Texas Examination of Current Administrators and Teachers (TECAT) is a basic literacy test that was administered to 202,000 educators in Texas in March 1986. An in-depth case study was undertaken to examine the effects of the TECAT. The 2-year project involved 10 separate data-collection studies including structured interviews with key legislators and aides; interviews with educators, personnel directors, and Texas citizens; a compilation of legislative and press materials; an analysis of test results; participant observation in test-preparation workshops; a content analysis of relevant newspaper stories; a survey of study materials; and cost analyses. The political climate surrounding the administration of the test was studied. While it had not been expected that more than 5-10% of teachers would fail, Texas teachers spent a great deal of time preparing for the test, including sponsored workshops. The cost to the state was much greater than expected. Many teachers with poor skills did pass the test, but vocational and special education teachers were overrepresented among those who did not pass the test. Many teachers indicated, in response to an interview survey of 96 educators, that having to take the test was demoralizing, and that publicity about failures and the ease of the test had an adverse impact on the public perception of teachers. There are 13 tables of data and six illustrative figures. A nine-item list of references is included. Seven appendices contain supplemental information about the testing process, including the interview protocol and the teacher and personnel director phone surveys. (SLD)

Reporting the results of statewide assessment looms as a problem as more states pass from the planning to implementation phase in their assessment programs. When energies are focused on the purpose of the assessment, formulating objectives, and instrument construction, reporting takes a back seat because it happens last. There are some general principles to be followed in order to report effectively the results of a large scale assessment program. This paper begins with several recent references on how to report the results of large scale assessment programs. The remainder of this paper is intended to provide specific new thoughts for implementation of old principles. The ultimate success of state assessment programs will depend on how well assessment results are reported to their various audiences. In this paper, the most compelling recommendations for improving reporting practices are plan ahead, develop different reports for different audiences, and field test report formats to determine the language and content that are most meaningful to respective audiences. Reporting should receive the same careful attention as instrument construction with sufficient opportunity for feedback from intended users. (Author/DEP)