Can We Identify a Successful Teacher Better, Faster, and Cheaper?

Teacher evaluation has become a national education phenomenon. It is promoted by philanthropists, mandated by federal policies, and debated by educators. Classroom observations are a crucial part of most evaluation systems—only six states do not use them, alone or in combination, to evaluate teachers. Because of their prominence, the stakes associated with observations are often high. For example, when observations contribute to low evaluations, they can be grounds for dismissal in 22 states and the District of Columbia.

Educators do not agree on the rules for observations: how often, whether announced, which instruments or rubrics, and who should observe. In the absence of evidence concerning which observational approaches relate best to student achievement, states are adopting previously developed standards-based observations that have been vetted by researchers, most recently as part of the Bill and Melinda Gates Foundation’s Measures of Effective Teaching (MET) study. The central purpose of these observations is to assess the degree to which teachers meet a set of instructional standards, such as California’s Standards for the Teaching Profession (CSTP). They were not designed with the primary goal of predicting student outcomes, although it is frequently assumed that they will.

That assumption, however, is not well supported by research findings. The correlation of observation measures and student gains reported by the MET study ranged from .09 to .27 in math and .04 to .11 in English Language Arts. These weak findings corroborate earlier ones reported by other researchers.

A growing concern is whether the limited information provided by these observations warrants the time, cost, and effort they demand. They tend to have many scoring criteria, in some cases over 60, in order to describe teaching standards fully, and they may require raters to score three or more full-length lessons. The financial burden they place on school systems is significant. In 2008, Thomas Toch estimated that implementing comprehensive observations for all teachers in the US would cost about $3 billion per year—roughly 20% of national expenditures on teacher professional development.

Although some educational activists have called for innovating observations by making them faster, cheaper, and better, others have responded that doing so might undermine their purpose and place them outside the research evidence. Contributing to this debate, we report in this study the results of seven experimental validation studies of the Rapid Assessment of Teacher Effectiveness (RATE) that suggest that innovation is indeed possible and that it can be evidence based. Using the six-item RATE rubric after four hours of training, observers were able to identify effective teachers from just 20 minutes of one lesson as well or better than observers using other instruments with 10 times the items to score multiple, full-length lessons.

In our seven experimental studies, RATE compared favorably with benchmarks that we established. Compared to baseline experiments we conducted previously, RATE classified teachers into high and low effectiveness groups more accurately. Compared with the instruments reviewed by the MET study, RATE correlated with measures of teacher value added as well or better, separated teachers into high and low quartiles more effectively, and achieved higher levels of inter-rater reliability.

Our work suggests that states and school districts may be able to meet both statutory requirements and the practical demands of managing schools by investing less effort, money, and time. Moreover, educators may be able to observe teachers early in the school year, predict which teachers are likely to struggle, and provide rapid support—before students are negatively affected by weak instruction.

The full study can be found in John Gargani and Michael Strong, Can We Identify a Successful Teacher Better, Faster, and Cheaper? Evidence for Innovating Teacher Observation Systems, Journal of Teacher Education, forthcoming.