Search

In this paper, the authors describe a case of measuring implementation fidelity within an evaluation of Math Recovery (MR), a pullout tutoring program for low-achieving first-graders. They use this case to address two aspects of implementation fidelity studies: (1) their feasibility with respect to unscripted interventions, and (2) their relationship to ongoing program development. The aim of the MR program is to use children's current understandings of number as bases for providing instruction that will support them in constructing increasingly sophisticated strategies. Therefore, assessing fidelity in this case is not as simple as monitoring adherence to a script, but requires assessing the complex practice of delivering mathematics instruction attuned to a child's current understanding and needs. Their intentions were to both measure the extent to which the program was implemented as intended, and link the measures to student outcomes. Determining the extent to which the tutoring is enacted as intended requires an explication of "good" tutoring as defined by the developers and systematically evaluating tutors' practices against that ideal. However, they also go beyond MR's notion of "good" tutoring by looking for instances of "positive infidelity" within tutoring sessions. Thus, they view studies of implementation fidelity as potential sources for refining theory and program design. Appendices are: (1) References; and (2) Tables and Figures.

In this paper, the authors describe a case of measuring implementation fidelity within an evaluation study of Math Recovery (MR), a pullout tutoring program aimed at increasing the mathematics achievement of low-performing first graders, thereby closing the school-entry achievement gap by enabling them to achieve at the level of their higher-performing peers in the regular mathematics classroom. Two research questions guided the conduct and analysis of the larger study: 1) Does participation in MR raise the mathematics achievement of low performing first-grade students? 2) If so, do participating students maintain the gains made in first grade through the end of second grade? The analysis reported in this paper follows from a third question: 3) To what extent does fidelity of implementation influence the effectiveness of MR? The two-year evaluation of Math Recovery was conducted in 20 elementary schools (five urban, ten suburban and five rural), representing five districts in two states. The findings suggest it is possible to create a reliable instrument to measure implementation fidelity for differentiated interventions--an endeavor that has, heretofore, been largely avoided in evaluations of educational interventions. Many potentially high-quality interventions are unscripted, instead relying on teacher knowledge and professional development, requiring considerable differentiation by implementers. As individuals work to rigorously evaluate such programs, they need to develop reliable fidelity measures that are both feasible and true to program components, so that evaluators can adequately link measures of treatment integrity to outcomes, to more accurately determine the relative strength of interventions (Cordray & Pion, 2006). This paper outlines the development and use of one such measure as a case of how such fidelity instruments might be developed and used in the future. Critical aspects of the process included: (1) the identification of the core implementation components of the intervention (Fixsen et al., 2005); (2) close work with program developers to operationalize those components; (3) training of coders in both the program itself and the coding schemes/process; and (4) collaborating with the coding team to further refine operationalizations and coding decisions, to strike a balance of feasibility and adherence to program components.