Predicting Observer Training Satisfaction and Certification

The last decade produced numerous studies that show that students learn more from high-quality teachers than they do from lower quality teachers. If instruction is to improve through the use of more rigorous teacher evaluation systems, the implementation of these systems must provide consistent and interpretable information about which aspects of teaching practice need improvement and how those improvements can be accomplished. A primary concern for using observation systems in teacher evaluation is the challenge of training observers to score in valid and reliable ways. In this report the authors will present first-year findings from the "Understanding Consequential Assessment Systems for Teachers" (UCAST) study, which investigates how administrators in a large urban school district learn to use a standardized observation protocol. The relevant research questions are as follows: (1) To what extent do administrator characteristics, beliefs, and expectations predict training satisfaction?; (2) To what extent do administrator characteristics, beliefs, and expectations predict certification success?; and (3) What components of the observation protocol are most challenging for observers to certify, and what accounts for these challenges? This project takes place in the Los Angeles Unified School District (LAUSD). This study is designed to investigate observer thinking and performance as it occurs in practice, and our current analyses draw on quantitative performance and perception data that comes from more than 700 administrators trained by the district during year 1 of the study. To improve the quality of instruction in schools, districts across the country are investing great hopes and resources in the implementation of teacher evaluation systems. Because observation offers a direct measure of instruction and it can point to areas for teacher improvement, additional research on how to implement observation protocols at scale is imperative and will be highly useful to school districts and other education stakeholders. Tables are appended.