Library

Quantitative Analysis of Indicators on the RTOP and ITC Observation Instruments

Abstract

Classroom observation is an important component of mathematics and science teacher
professional development programs and other educational evaluation activities. In this paper, the
authors extend their earlier qualitative examination and comparison of two popular classroom
observation tools (Horizon Research, Inc.'s Inside the Classroom Observation and Analytic Protocol
(ITC) and Sawada et al.'s Reform Teacher Observation Protocol (RTOP)) with a quantitative analysis.
The instruments substantially are based on comparable assumptions, foundational philosophies and
domains of interest, and appear to be used with an expectation that they yield similar results, not to
mention that their domains adhere internally. A single well-trained educator observed 21 teachers
from a Mathematics and Science Partnership over the course of two years. Pearson Correlation
Coefficient analysis was applied to items across and within instruments. Because of the relatively
small number of teachers and the use of one rater, the authors applied a relatively strict interpretation
(.75) of high correlations. While some items correlated as expected within and between instruments
based on domain and item construction, many items lacked matches, including those with an
apparently similar focus. RTOP items showed greater alignment than did ITC items. Among
summative "synthesis" domain ratings in the ITC, none of the four aligned with any of the specific
ITC item indicators within their own domain categories. Subtle differences in wording and implicit
differences in overall focus especially appear to restrict both internal matches and matches between
superficially comparable items across the two instruments. These findings complicate the
interpretation of observational results and challenge assumptions that the instruments are
interchangeable or necessarily internally consistent. Additional research and development of
observation instruments is needed, and users of existing instruments must carefully assess their own
needs and understandings before attempting to draw conclusions about classroom practice based on
them.