Measurement of Classroom Teaching Quality with Item Response Theory

Résumé du livre

Recent policy has charged schools and districts with maintaining highly qualified teachers and differentiating among teachers in terms of their effectiveness (U.S. Department of Education, 2009). This emphasis has driven the development and implementation of teacher quality measures which are increasingly being used to evaluate teachers with important consequences. One increasingly common component of these evaluations is the direct observation of teachers in their classrooms. Classroom observations have been long viewed as a promising way to evaluate and develop teachers because they anchor assessments in specific and observable criteria. In this proposal, the authors investigated the robustness of classroom observation scores to three measurement approaches: classical test theory (CTT), generalizability theory (GT), and item response theory (IRT). They investigated the extent to which choices among these approaches lead to indeterminacies in conclusions regarding teaching quality, the precision with which the authors can index this quality, and the relation of this quality to student achievement. They then provide insights delineating the reasons for the discrepancies among methods and explore the extent to which the observed differences are indicative of true differences among teachers in their teaching quality. This study takes place within the larger Developing Measures of Effective Mathematics Teaching study which focused on developing identifying practices and characteristics that distinguish between more and less effective teachers. The sample includes 250 teachers from 40 schools and 4 districts. Overall the results suggest that construct-irrelevant variance is sizeable in classroom observations and that treatment of this variance had significant implications for the resulting scores. Tables and figures are appended.