What's in a score? Problematizing interpretations of observation scores.

Several recent, large-scale observations of naturally occurring classroom instruction have found consistent patterns in how teaching quality systematically varies across domains of instruction, patterns not found in student survey measures. Teaching quality is highest when examining aspects of classroom management; moderately high when examining aspects of student support; and low when focusing on the quality of the instructional support provided to students. This paper problematizes this interpretation, arguing that observed data patterns could result from dilemmas inherent in measuring teaching quality through observation. In problematizing these conclusions, we highlight the complexity of trying to measure instruction at scale, arguing for the need for caution and consideration of the complexity of measurement when interpreting scores from observation systems. • Several recent observational classroom studies characterized teaching quality. • The relative strengths of instruction were similar across these studies. • Similar patterns are not consistently found using other data sources. • Dilemmas in rubric design may cause the similar findings across these studies. • We discuss dilemmas and challenges in observationally measuring teaching quality. [ABSTRACT FROM AUTHOR]