Search

Classroom observation is a common approach to teacher evaluation. Yet, concerns about differences in rater judgment are widespread. Despite this concern, few researchers have examined the practical impact of such differences in rater judgments on teachers' judged effectiveness. This study fills that gap. Using data from a large-scale teacher evaluation system, we found substantial differences in principal severity that affected teachers' classification within performance categories. We then demonstrate a technique that researchers and practitioners can use to control for differences in rater severity – thus limiting the degree to which rater severity differences can threaten the fairness of classroom observations. We discuss implications for research and practice. [ABSTRACT FROM AUTHOR]

Teacher evaluation systems commonly rely on observation of teaching practice (OTP) by school principals. However, the value of OTP as evidence of teacher effectiveness depends on its psychometric quality. In this study, we address a key aspect of the psychometric quality of principals’ OTP ratings. Specifically, we investigate the degree to which rating scale categories have a consistent interpretation across teaching episodes and practices. Results suggest that the 1,324 principals’ use of the rating scale categories functioned as intended overall. However, we also found that the midpoint category is underutilized and that rating categories do not always reflect similar levels of teaching effectiveness across teaching episodes and practices. When such discrepancies occur, we cannot assume principals’ ratings reflect a consistent level of teacher effectiveness within and across classrooms. This is a critical component of validity evidence that can inform the interpretation of OTP ratings and point to areas for improvement in both the rubrics and in principals’ training for classroom observations. [ABSTRACT FROM AUTHOR]

Teacher evaluation commonly includes classroom observations conducted by principals. Despite widespread use, little is known about the quality of principal ratings. We investigated 1,324 principals’ rating accuracy of six teaching practices at the conclusion of training within an authentic teacher evaluation system. Data are from a video-based exam of four 10-minute classroom observations. Many-Facet Rasch modeling revealed that (1) overall principals had high accuracy, but individuals varied substantially, and (2) some teaching episodes and practices were easier to rate accurately. For example, promotes critical thinking was rated more accurately than uses formative assessment. Because Many-Facet Rasch modeling estimates individuals’ accuracy patterns across teaching episodes and practices, it is a useful tool for identifying areas that individual principals, or groups, may need additional training (e.g., evaluating formative assessment). Implications for improving training of principals to conduct classroom observations for teacher evaluation are discussed. [ABSTRACT FROM AUTHOR]

• Construct-irrelevant factors often threaten the fairness of classroom observations. • We compared ratings over teacher demographics and similarity with principals. • Principals' ratings varied by gender, experience, and school level. • Principals' demographic similarity to teachers did not affect ratings. Meaningful interpretation of teacher evaluation based on classroom observation depends on the degree to which principals' judgments are free from errors and systematic bias. Previous researchers have identified factors that may influence classroom observation ratings, including characteristics of students in the classrooms being observed. In this study, we explored the influence of teachers' and principals' characteristics on classroom observation ratings. Using data from a statewide teacher evaluation system, we examined whether principals' ratings systematically varied across teachers' characteristics, as well as across shared characteristics between principals and teachers. Our results suggested that principals' ratings varied by teachers' gender, years of experience, and school level, but not by shared demographic characteristics between principals and teachers. Principals rated female, experienced, and elementary teachers more highly than male, novice, and secondary teachers. We discuss implications for research and practice. [ABSTRACT FROM AUTHOR]