An IRT Mixture Model for Rating Scale Confusion Associated with Negatively Worded Items in Measures of Social-Emotional Learning

This paper illustrates the application of mixture IRT models to evaluate the possibility of respondent confusion due to the negative wording of certain items on a social-emotional learning (SEL) assessment. Using actual student self-report ratings on four social-emotional learning scales collected from students in Grades 3–12 from CORE Districts in the state of California, it also evaluates the consequences of the potential confusion in biasing student- and school-level scores, as well as correlational relationships between SEL and student-level variables. Models of both full and partial confusion are examined. The results suggest that (1) rating scale confusion due to negatively worded items does appear to be present; (2) the confusion is most prevalent at lower grade levels (Grades 3–5); and (3) the occurrence of confusion is positively related to reading proficiency and ELL status, as anticipated, and bias estimates of SEL correlations with these student-level variables. For these reasons, future iterations of the SEL measures should use only positively oriented items. To maintain measurement continuity, bias corrections based on the studied mixture model may be useful, although the precision of such corrections is sensitive to the nature of confusion (e.g., full versus partial).