Search

• Misuse of coefficient indices for inter-rater reliability of observation instruments in the literature. • Existence of kappa paradox through illustration with empirical data from randomized research. • Gwet's adjusted AC 1 as a robust coefficient index for multi-dimension-multi-response nominal scale. • Intraclass correlation as a robust index for Likert-scale with ordinal data. • A fully-crossed rating design with eight raters rating all randomly-selected video clips. In this study, we address an under-studied challenge related to the accurate use and reporting of coefficient indices for inter-rater reliability (IRR) of classroom observation instruments used to measure fidelity of implementation (FOI) in educational programs serving the needs of English learners. Through empirical data obtained from a randomized controlled trial, we confirmed the existence of kappa paradox. We further determined that Gwet's AC1 was a robust index for a multiple-dimension-multi-response protocol with nominal data and intraclass correlation was most appropriate for a Likert-rating scale with ordinal data and multiple raters. We highlight the misuse of indices in the literature and call for a careful consideration of robust indices critical to ensure accuracy of observational data and FOI. [ABSTRACT FROM AUTHOR]

Federal funding agencies have invested significant resources supporting evidence-based, innovative approaches that address education problems and incorporate rigorous design and evaluation, particularly through randomized controlled trials (RCTs), the gold standard for yielding causal inference in scientific research. In this study, we introduced factors (i.e., evaluation design, attrition, outcome measures, analytic approach, and fidelity of implementation) that are often times required in the Federal Notice for application by the U.S. Department of Education, with an emphasis on What Works Clearinghouse (WWC) standards. We further presented a federally funded research protocol with a multi-year, clustered RCT design to determine the impact of an instructional intervention on students' academic performance in high-needs schools. In the protocol, we elaborated on how our research design, evaluation plan, power analysis, as well as confirmatory research questions and analytical approaches were aligned with the grant requirement and WWC standards. We intend to provide a road map to meeting WWC standards and to increase the likelihood of successful grant applications., Competing Interests: Conflict of interest We have no known conflict of interest to disclose., (Copyright © 2023 The Authors. Published by Elsevier Ltd.. All rights reserved.)