Search

The POWERSOURCE[c] intervention is a generalizable and powerful formative assessment-based strategy that can be integrated with any ongoing mathematics curriculum to improve teachers' knowledge and practice and, in turn, student learning. Our primary research objectives are based on our hypotheses that as a result of POWERSOURCE[c], teachers will become more proficient in their subject matter knowledge, more skilled in their formative use of assessment, and better focus their instruction on key ideas. Ultimately, teachers will be more effective in helping students to improve their understanding--as shown by measures of student learning. This report provides a detailed overview of our study results from the 2006-07 school year. We found that a knowledge mapping task and a student response analysis task, could be used as a set, or the knowledge mapping task could be used separately from the other tasks, in order to efficiently investigate both content knowledge of teachers and knowledge of teaching algebra. Appendices include: (1) Additional Item Analysis Results; (2) 6th Grade, Teacher Handbook, Properties of Arithmetic: The Distributive Property; (3) Sample Alternative Professional Development Materials; (4) Examples of Professional Development Website Materials; (5) Transfer Measure Items and Sources; (6) Transfer Measure Alignment: Standards and Focal Points; (7) Teacher Surveys; and (8) Interview and Observation Measures. (Contains 1 footnote, 8 tables, and 4 figures.)

The evaluation of educational interventions requires assessments that consistently (reliably) produce data from which accurate (valid) inferences about the test subjects can be made for some stated purpose. Despite codified definitions of all these terms, there remains vibrant debate about the assessment design process and how measures of technical quality should be determined and reported. More importantly, there seems to be little consistency in the process of developing assessment items that demonstrate consistently high technical quality and that are conceptually aligned with instruction. We present a process, rooted in the work of Baker (1974), in which a theory of learning drives the development of knowledge specifications that then drive the integrated development of assessment items and forms and instructional materials. In this case, the instruction is in the form of an educational video game about adding rational numbers. We discuss the development of the knowledge and item specifications, an initial version of the instructional game based on these specifications, and report the consistently high technical quality (internal consistency and test-retest reliability--Cronbach's alpha of 0.9 to 0.94). We discuss as well the ability of pretest measures to predict subsequent student performance (inferential validity--Spearman's rho of 0.69) in the game and on similarly developed assessment measures. Alignment between the instructional game and assessment items is also discussed. Appendices include: (1) Knowledge Specifications; and (2) Puppetman Screen Shots from the First Version Tested with Students. (Contains 7 tables and 5 footnotes.) [This work was supported under the Educational Research and Development Centers Program, PR/Award Number R305C080015.]

While research suggests that formative assessment can be a powerful tool to support teaching and learning, efforts to jump on the formative assessment bandwagon have been more widespread than those to assure the technical quality of the assessments. This report covers initial analyses of data bearing on the quality of formative assessments in middle school mathematics. Specifically, these data address the question of whether relatively short assessments can provide reliable and useful information on middle school students' understanding of conceptual domains in pre-algebra. Items and test forms were developed and tested in four domains (rational number equivalence, properties of arithmetic, principles for solving equations, and applications of these concepts to other domains), all of which are critical to eventual mastery of algebra. We tested the items with sixth-grade students in classrooms in four districts. We then pared down the items to create eight assessment forms that were further tested alongside instructional support materials and professional development. Results of this study suggest that relatively brief formative assessments focused on key conceptual domains can provide reliable and useful information on students' levels of understanding and possible misunderstandings in the domain. Six appendices are included: (1) Final list of big ideas; (2) Reliability coefficients of the test forms used in the pilot study; (3) IRT Tables; (4) Item (Category) Characteristic Curves; (5) The item characteristic curve of every item in the five domains; and (6) The detailed results of the reliability analysis and component loadings. (Contains 12 figures and 7 tables.)

The authors' hypothesis is that if teachers (as experts) understand and teach concepts from the position of expertise teacher quality will improve. They believe that focusing on the key ideas will deepen both teacher and student understanding and allow learners to build the concepts necessary to form solid foundations for the application of mathematics both in and out of school. But such methods of instruction often require that teachers learn to reorganize and teach fundamental concepts in more expert-like ways. Because many teachers were taught in a system that stressed procedure rather than conceptual understanding, and the dearth of guidance on how teachers might change their teaching in this manner, such change may be difficult. Building on the expert-novice literature, and the findings by Carpenter, Fenamya, Levi, Franke, and Empson (2000) using cognitively guided instruction, and research on formative assessment, they developed a program of professional development designed to help teachers conceptualize mathematics instruction and improve their ability to effectively use formative assessment data in the 6th grade math classroom. Both the formative assessments and the professional development were designed around a collection of big ideas developed by content experts and math educators. The program of professional development supports a larger formative assessment effort as part of a US Department of Education sponsored program called PowerSource. Their ultimate goal is to demonstrate that the professional development methods they advocate improve student proficiency in algebra. Initially, however, they must demonstrate that their efforts actually improve near-term teacher content knowledge organization and have demonstrable effects on teacher ability to use formative assessment results. Their results suggest that both the way teachers organize their content knowledge and the way they evaluate student work changes in important ways. Even with the positive trends, however, the authors' findings suggest that nine hours of professional development may not be enough to produce the degree of change anticipated. As the literature suggests, producing change in teacher pedagogical practice requires sustained and intense efforts. These results suggest areas where important changes might be made such as a more intense focus on pedagogical methods, replacing in-situ curriculum with more conceptually appropriate instruction, and providing focused problems that teachers can use for sustained practice in their classrooms. In addition, the authors intend to use more powerful Heirarchical Linear Models (HLM) to account for the degree of participation in professional development activities and to observe the degree of actual classroom implementation of PowerSource.

Based on the results of a generalizability study (G study) of measures of teacher knowledge for teaching mathematics developed at The National Center for Research, on Evaluation, Standards, and Student Testing (CRESST) at the University of California, Los Angeles, this report provides evidence that teachers are better at drawing reasonable inferences about student levels of understanding from assessment information than they are in deciding the next instructional steps. We discuss the implications of the results for effective formative assessment and end with considerations of how teachers can be supported to know what to teach next. (Contains 2 tables, 2 figures, and 1 footnote.)

This report describes a web-based assessment design tool, the Assessment Design and Delivery System (ADDS), that provides teachers both a structure and the resources required to develop and use quality assessments. The tool is applicable across subject domains. The heart of the ADDS is an assessment design workspace that allows teachers to decide the attributes of an assessment, as well as the context and type of responses the students will generate, as part of their assessment design process. Although the tool is very flexible and allows the above steps to be done in any order (or skipped entirely), our goal was to streamline and scaffold the process for teachers by organizing all the materials for them in one place and to provide resources they could use or reuse to create assessments for their students. The tool allows teachers to deliver the assessments to their students either online or on paper. Initial results from our first teacher study suggest that teachers who used the tool developed assessments that were more cognitively demanding of students and addressed the "big ideas" rather than disassociated facts of a domain. (Contains 1 figure.)

The inability of students to become proficient in algebra seems to be widespread in American schools. One of the reasons often cited for this inability is that instruction seldom builds on prior knowledge. Research suggests that teacher effectiveness is the most critical controllable variable in improving student achievement. This report details a process of formative assessment and professional development (called PowerSource), which is designed to improve teacher effectiveness and student achievement. We describe the process we used to develop a model of distribution over addition and subtraction, one of three big ideas developed during the year, and the interactions we had with teachers about teaching distribution in various ways. As a consequence of these interactions, we were able to test whether teaching distribution using the notion of multiplication as repeated addition (a concept which students had learned previously), using array or area models, or teaching it procedurally had the greatest effects on student learning. We found that the repeated addition model was not only less likely to create certain student misconceptions, but also found that students taught using the repeated addition model were more likely to correctly answer questions involving distribution than were their counterparts taught using either of the other methods. Teachers subsequently reported that they preferred teaching distribution as an instance of repeated addition than teaching it using other available methods. (Contains 12 tables, 2 figures, and 1 footnote.)

Assessing whether students can both re-present a corpus of learned knowledge and also demonstrate that they can apply that knowledge to solve problems is key to assessing student understanding. This notion, in turn, impacts our thinking about what we assess, how we author such assessments, and how we interpret assessment results. The diffusion of technology into venues of learning offers new opportunities in the area of student assessment. Specifically, computer-based simulations seem to provide sufficiently rich environments and the tools necessary to allow us to infer accurately how well a student's individual mental model of the world can accommodate, integrate, and be used to exploit concepts from a domain of interest. In this paper then, we first identify the characteristics of simulations that our experience suggests are necessary to make them appropriate for pedagogical and assessment purposes. Next, we discuss the models and frameworks (templates) we have used to ensure these characteristics are considered. Finally, we describe two computerized instantiations (objects) of these frameworks and implications for the follow-on design of simulations. (Contains 3 footnotes.) [The work reported in this document was supported by the Inter-agency Educational Research Initiative. This report was first printed as chapter 16 of "Assessments of Problem Solving Using Simulations" (Vendlinski, Baker, & Niemi, 2008).]

Current educational policy highlights the important role that assessment can play in improving education. State standards and the assessments that are aligned with them establish targets for learning and promote school accountability for helping all students succeed; at the same time, feedback from assessment results is expected to provide districts, schools, and teachers with important information for guiding instructional planning and decision making. Yet even as No Child Left Behind (NCLB) and its requirements for adequate yearly progress put unprecedented emphasis on state tests, educators have discovered that annual state tests are too little and too late to guide teaching and learning. Recognizing the need for more frequent assessments to support student learning, many districts and schools have turned to benchmark testing--periodic assessments through which districts can monitor students' progress, and schools and teachers can refine curriculum and teaching--to help students succeed. We report in this document a collaborative effort of teachers, district administrators, professional developers, and assessment researchers to develop benchmark assessments for elementary school science. In the sections which follow we provide the rationale for our work and its research question, describe our collaborative assessment development process and its results, and present conclusions. (Contains 9 tables.)

The characteristics of expert knowledge--interconnectedness, understanding, and ability to transfer--are inextricably linked, a point that is critically important for educators and constitutes a major theme of this paper. This paper explores how an analysis of the architecture of expert knowledge can inform the development of assessments to help teachers move students toward greater expertise in mathematics, and it presents examples of such assessments. It also reviews student responses and preliminary results from pilot tests of assessments administered in sixth-grade classes in a large urban school district. Preliminary analyses suggest that an assessment strategy based on the structure of mathematical knowledge can reveal deficiencies in student understanding of and ability to apply fundamental concepts of pre-algebra, and has the potential to help teachers remediate those deficiencies. (Contains 7 figures and 10 tables.)

Teachers' knowledge of mathematics is pivotal to their capacity to provide effective mathematics instruction and to their ability to assess student learning (Ball, Hill, & Bass, 2005; Ma, 1999; Schifter, 1999). The National Council for the Teaching of Mathematics (NCTM, 2000) makes it clear that teachers need knowledge of the whole domain as well as knowledge about the important ideas that are central to their grade level. POWERSOURCE is expected, through professional development and job aids, to influence teachers' pedagogical content knowledge and assessment practices. To gauge such effects the authors have developed teacher measures that focus on three key mathematical principles that are central to POWERSOURCE: the distributive property, solving equations, and rational number equivalence. (Contains 2 figures.)

While most would probably agree that the ultimate success of an educational system should be graduates who are able to evaluate, synthesize, analyze and apply what they have learned, routinely assessing such understanding in students has proven difficult, for a variety of reasons, using most traditional methods. Recent developments in cognitive science, information technology, and analytical tools, however, have provided educators and researchers a means to overcome many of these barriers. This paper explains how high school teachers and educational researchers worked together to apply these developments to successfully implement and assess case-based problem-solving in a typical high school chemistry curriculum. A quasi-experimental study was conducted with 134 first-year high school chemistry students using the Interactive Multi-Media Exercises (IMMEX[TM]) computer based learning and assessment tool to solve qualitative chemistry problems. The students' teacher constructed the problem for her students using the same tool. Results of this study show a strong and significant correlation between computer-aided measures of student understanding and the teacher's manual evaluation of student understanding using her own rubric. While both measures demonstrated that content knowledge was important, neither proved just a surrogate measure of content knowledge. Neither measure demonstrated gender, ethnic or socioeconomic bias. The results suggest that the data collected by this tool not only allows for valid inferences of a student's content understanding, but allows such assessment to happen in a classroom setting in near real time. (Contains 46 references and 5 tables.) (Author)

This study used a computerized simulation and problem-solving tool along with artificial neural networks (ANN) as pattern recognizers to identify the common types of strategies high school and college undergraduate chemistry students would use to solve qualitative chemistry problems. Participants were 134 high school chemistry students who used the Interactive Multi-Media Exercises (IMMEX) software with a problem set related to identification of hazardous materials. Then, based on the calculated probabilities that students would transition between these strategy types over time, hidden chain Markov analysis allowed two objectives to be accomplished. A model of the capacity of the current curriculum to produce students able to apply chemistry content to real world problems was developed, and then this model was used to suggest pedagogical interventions that might be most effective at promoting better student understanding and as a metric by which to evaluate the potential effectiveness of these and other interventions. (Contains 9 tables and 15 references.) (SLD)

Interactive Multi-Media Exercises (IMMEX) is a technology-based learning and assessment tool designed to integrate curricular content and problem-solving skills into real world scenarios. IMMEX inherits much of its structure from case-based and problem-based learning models and provides teachers with quantifiable, visual feedback on student problem-solving performances so that interventions can be acutely informed and immediate. While most teachers utilize the most basic performance indicators provided by IMMEX for measuring student progress, richer analytic tools have shown performance nuances such as: (1) significant correlation between problem-solving and (some) standardized test scores; (2) evidence that males and females approach problem-solving differently; and (3) strategic trends in problem-solving. The classroom integration of technology is supported by extensive professional development activities including workshops where teams of teachers, students, and university faculty write IMMEX cases. (Contains 25 references, 3 figures, and 2 tables.) (Author/SLD)