High-Dimensional Explanatory Random Item Effects Models for Rater-Mediated Assessments

Résumé du livre

Valid and reliable measurement of unobserved latent variables is essential to understanding and improving education. A common and persistent approach to assessing latent constructs in education is the use of rater inferential judgment. The purpose of this study is to develop high-dimensional explanatory random item effects models designed for rater-mediated assessments. The models are built to address three specific issues. First, an important limitation of the use of cross-classified random item effects models in rater-mediated assessments is that the number of latent dimensions increases quickly with the number of items and facets. Second, an important limitation of previous work is that it has not considered the potential for interactions among facets and between facets and items. Third, although previous research has examined the presence of noninvariance in rater-mediated assessments, it has not examined the extent to which characteristics of each facet systematically explain variability in measurement. To illustrate the proposed method, the authors applied it to a longitudinal study of students' academic engagement in elementary school. Two cohorts of students were rated about up to three times per year for three years across grades kindergarten to second or third to fifth. Across the study there were about 6,000 students measured about 17,000 times. Preliminary analyses indicated the model that best balanced parsimony with fit based on the information criteria was the model that included random effects for items, time points, students, raters, students-by-raters, items-by-raters, and items-by-students. Two tables are appended.