Search

The POWERSOURCE[c] intervention is a generalizable and powerful formative assessment-based strategy that can be integrated with any ongoing mathematics curriculum to improve teachers' knowledge and practice and, in turn, student learning. Our primary research objectives are based on our hypotheses that as a result of POWERSOURCE[c], teachers will become more proficient in their subject matter knowledge, more skilled in their formative use of assessment, and better focus their instruction on key ideas. Ultimately, teachers will be more effective in helping students to improve their understanding--as shown by measures of student learning. This report provides a detailed overview of our study results from the 2006-07 school year. We found that a knowledge mapping task and a student response analysis task, could be used as a set, or the knowledge mapping task could be used separately from the other tasks, in order to efficiently investigate both content knowledge of teachers and knowledge of teaching algebra. Appendices include: (1) Additional Item Analysis Results; (2) 6th Grade, Teacher Handbook, Properties of Arithmetic: The Distributive Property; (3) Sample Alternative Professional Development Materials; (4) Examples of Professional Development Website Materials; (5) Transfer Measure Items and Sources; (6) Transfer Measure Alignment: Standards and Focal Points; (7) Teacher Surveys; and (8) Interview and Observation Measures. (Contains 1 footnote, 8 tables, and 4 figures.)

The evaluation of educational interventions requires assessments that consistently (reliably) produce data from which accurate (valid) inferences about the test subjects can be made for some stated purpose. Despite codified definitions of all these terms, there remains vibrant debate about the assessment design process and how measures of technical quality should be determined and reported. More importantly, there seems to be little consistency in the process of developing assessment items that demonstrate consistently high technical quality and that are conceptually aligned with instruction. We present a process, rooted in the work of Baker (1974), in which a theory of learning drives the development of knowledge specifications that then drive the integrated development of assessment items and forms and instructional materials. In this case, the instruction is in the form of an educational video game about adding rational numbers. We discuss the development of the knowledge and item specifications, an initial version of the instructional game based on these specifications, and report the consistently high technical quality (internal consistency and test-retest reliability--Cronbach's alpha of 0.9 to 0.94). We discuss as well the ability of pretest measures to predict subsequent student performance (inferential validity--Spearman's rho of 0.69) in the game and on similarly developed assessment measures. Alignment between the instructional game and assessment items is also discussed. Appendices include: (1) Knowledge Specifications; and (2) Puppetman Screen Shots from the First Version Tested with Students. (Contains 7 tables and 5 footnotes.) [This work was supported under the Educational Research and Development Centers Program, PR/Award Number R305C080015.]

While research suggests that formative assessment can be a powerful tool to support teaching and learning, efforts to jump on the formative assessment bandwagon have been more widespread than those to assure the technical quality of the assessments. This report covers initial analyses of data bearing on the quality of formative assessments in middle school mathematics. Specifically, these data address the question of whether relatively short assessments can provide reliable and useful information on middle school students' understanding of conceptual domains in pre-algebra. Items and test forms were developed and tested in four domains (rational number equivalence, properties of arithmetic, principles for solving equations, and applications of these concepts to other domains), all of which are critical to eventual mastery of algebra. We tested the items with sixth-grade students in classrooms in four districts. We then pared down the items to create eight assessment forms that were further tested alongside instructional support materials and professional development. Results of this study suggest that relatively brief formative assessments focused on key conceptual domains can provide reliable and useful information on students' levels of understanding and possible misunderstandings in the domain. Six appendices are included: (1) Final list of big ideas; (2) Reliability coefficients of the test forms used in the pilot study; (3) IRT Tables; (4) Item (Category) Characteristic Curves; (5) The item characteristic curve of every item in the five domains; and (6) The detailed results of the reliability analysis and component loadings. (Contains 12 figures and 7 tables.)

Based on the results of a generalizability study (G study) of measures of teacher knowledge for teaching mathematics developed at The National Center for Research, on Evaluation, Standards, and Student Testing (CRESST) at the University of California, Los Angeles, this report provides evidence that teachers are better at drawing reasonable inferences about student levels of understanding from assessment information than they are in deciding the next instructional steps. We discuss the implications of the results for effective formative assessment and end with considerations of how teachers can be supported to know what to teach next. (Contains 2 tables, 2 figures, and 1 footnote.)

This report describes a web-based assessment design tool, the Assessment Design and Delivery System (ADDS), that provides teachers both a structure and the resources required to develop and use quality assessments. The tool is applicable across subject domains. The heart of the ADDS is an assessment design workspace that allows teachers to decide the attributes of an assessment, as well as the context and type of responses the students will generate, as part of their assessment design process. Although the tool is very flexible and allows the above steps to be done in any order (or skipped entirely), our goal was to streamline and scaffold the process for teachers by organizing all the materials for them in one place and to provide resources they could use or reuse to create assessments for their students. The tool allows teachers to deliver the assessments to their students either online or on paper. Initial results from our first teacher study suggest that teachers who used the tool developed assessments that were more cognitively demanding of students and addressed the "big ideas" rather than disassociated facts of a domain. (Contains 1 figure.)

The inability of students to become proficient in algebra seems to be widespread in American schools. One of the reasons often cited for this inability is that instruction seldom builds on prior knowledge. Research suggests that teacher effectiveness is the most critical controllable variable in improving student achievement. This report details a process of formative assessment and professional development (called PowerSource), which is designed to improve teacher effectiveness and student achievement. We describe the process we used to develop a model of distribution over addition and subtraction, one of three big ideas developed during the year, and the interactions we had with teachers about teaching distribution in various ways. As a consequence of these interactions, we were able to test whether teaching distribution using the notion of multiplication as repeated addition (a concept which students had learned previously), using array or area models, or teaching it procedurally had the greatest effects on student learning. We found that the repeated addition model was not only less likely to create certain student misconceptions, but also found that students taught using the repeated addition model were more likely to correctly answer questions involving distribution than were their counterparts taught using either of the other methods. Teachers subsequently reported that they preferred teaching distribution as an instance of repeated addition than teaching it using other available methods. (Contains 12 tables, 2 figures, and 1 footnote.)

Assessing whether students can both re-present a corpus of learned knowledge and also demonstrate that they can apply that knowledge to solve problems is key to assessing student understanding. This notion, in turn, impacts our thinking about what we assess, how we author such assessments, and how we interpret assessment results. The diffusion of technology into venues of learning offers new opportunities in the area of student assessment. Specifically, computer-based simulations seem to provide sufficiently rich environments and the tools necessary to allow us to infer accurately how well a student's individual mental model of the world can accommodate, integrate, and be used to exploit concepts from a domain of interest. In this paper then, we first identify the characteristics of simulations that our experience suggests are necessary to make them appropriate for pedagogical and assessment purposes. Next, we discuss the models and frameworks (templates) we have used to ensure these characteristics are considered. Finally, we describe two computerized instantiations (objects) of these frameworks and implications for the follow-on design of simulations. (Contains 3 footnotes.) [The work reported in this document was supported by the Inter-agency Educational Research Initiative. This report was first printed as chapter 16 of "Assessments of Problem Solving Using Simulations" (Vendlinski, Baker, & Niemi, 2008).]

Current educational policy highlights the important role that assessment can play in improving education. State standards and the assessments that are aligned with them establish targets for learning and promote school accountability for helping all students succeed; at the same time, feedback from assessment results is expected to provide districts, schools, and teachers with important information for guiding instructional planning and decision making. Yet even as No Child Left Behind (NCLB) and its requirements for adequate yearly progress put unprecedented emphasis on state tests, educators have discovered that annual state tests are too little and too late to guide teaching and learning. Recognizing the need for more frequent assessments to support student learning, many districts and schools have turned to benchmark testing--periodic assessments through which districts can monitor students' progress, and schools and teachers can refine curriculum and teaching--to help students succeed. We report in this document a collaborative effort of teachers, district administrators, professional developers, and assessment researchers to develop benchmark assessments for elementary school science. In the sections which follow we provide the rationale for our work and its research question, describe our collaborative assessment development process and its results, and present conclusions. (Contains 9 tables.)

The characteristics of expert knowledge--interconnectedness, understanding, and ability to transfer--are inextricably linked, a point that is critically important for educators and constitutes a major theme of this paper. This paper explores how an analysis of the architecture of expert knowledge can inform the development of assessments to help teachers move students toward greater expertise in mathematics, and it presents examples of such assessments. It also reviews student responses and preliminary results from pilot tests of assessments administered in sixth-grade classes in a large urban school district. Preliminary analyses suggest that an assessment strategy based on the structure of mathematical knowledge can reveal deficiencies in student understanding of and ability to apply fundamental concepts of pre-algebra, and has the potential to help teachers remediate those deficiencies. (Contains 7 figures and 10 tables.)

Teachers' knowledge of mathematics is pivotal to their capacity to provide effective mathematics instruction and to their ability to assess student learning (Ball, Hill, & Bass, 2005; Ma, 1999; Schifter, 1999). The National Council for the Teaching of Mathematics (NCTM, 2000) makes it clear that teachers need knowledge of the whole domain as well as knowledge about the important ideas that are central to their grade level. POWERSOURCE is expected, through professional development and job aids, to influence teachers' pedagogical content knowledge and assessment practices. To gauge such effects the authors have developed teacher measures that focus on three key mathematical principles that are central to POWERSOURCE: the distributive property, solving equations, and rational number equivalence. (Contains 2 figures.)