Assessing fidelity of implementation to a technology‐mediated early intervention using process data.

Background: Process data, data generated by a user's interaction with a web‐based application, is an emerging tool in educational research. The current study explores using process data as a measure of implementation fidelity to a randomized control trial (RCT) of the Read It Again Mobile (RIA‐M) curricular supplement. Objectives: To determine the extent to which teachers implemented RIA‐M and to assess the utility of using process data in the assessment of fidelity. Methods: The RCT involved 30 pre‐kindergarten classrooms with a sample of n = 216 students. RIA‐M provides a curricular supplement which teachers may incorporate into classroom instruction and is delivered via a tablet computer. Pre and post literacy assessments are used to determine treatment effect. Process data, produced from teacher interactions with the tablet, and classroom observations are used to assess fidelity. Results and Conclusions: Our findings indicate no difference between treatment and control students in the RCT. Yet, we find that process data provides unique fidelity information concerning treatment exposure, adherence, and quality of program delivery. Specifically, process data indicated that teachers did not demonstrate the same level of fidelity that was captured in classroom observations. This finding provides some evidence for the absence of an intervention effect. Major Takeaways: The current study improves our understanding of how web‐based interventions may be assessed for implementation fidelity using process data. Further, process data offers a potentially reliable and scalable measure of fidelity for other web‐based educational interventions. Lay Description: What we know about the topic: Mobile technology‐based interventions are on the rise in many educational settings.Fidelity to educational interventions can be difficult to reliably assess.Teachers may be resistant to implementing technology in classrooms. What this paper adds: Examines a pilot study of a mobile technology‐based curricular intervention for teachers designed to improve early reading and literacy.Examines a new method for assessing fidelity of mobile technology‐based interventions.Compares the new method to traditional methods of assessing fidelity to educational interventions. Implications for practice and/or policy: The presence of technology in the classroom does not assure that the technology will be used effectively.Mobile technology‐based interventions may require additional monitoring and coaching to assure use as intended.Process data, generated in the use of technology‐based interventions, may provide a unique way to measure fidelity to the intervention. [ABSTRACT FROM AUTHOR]