Working Papers

  • Assessing Survey Satisficing: The Impact of Unmotivated Questionnaire Respondents on Data Quality

    Christine Calderon Vriesema, Hunter Gehlbach. October 2019

    Education researchers use surveys widely. Yet, critics question respondents’ ability to provide high-quality responses. As schools increasingly use student surveys to drive policymaking, respondents’ (lack of) motivation to provide quality responses may threaten the wisdom of using surveys for data-based decision-making. To better understand student satisficing (sub- optimal responding on surveys) and its impact on data quality, we examined the pervasiveness and impact of this practice on a large-scale social-emotional learning survey administered to 409,721 students in grades 2-12. Findings indicated that despite the prevalence of satisficing in our sample, its impact on data quality appeared more modest than anticipated. We conclude by providing an accessible approach for defining and calculating satisficing for researchers, practitioners, and policymakers working with large-scale datasets.

  • Students with Growth Mindset Learn More in School: Evidence from California’s CORE School Districts

    Susana Claro, Susanna Loeb. October 2019

    While the importance of social-emotional learning for student success is well established, educators and researchers have less knowledge and agreement about which social-emotional skills are most important for students and how these skills distribute across student subgroups. Using a rich longitudinal dataset of 221,840 fourth through seventh grade students in California districts, this paper describes growth mindset gaps across student groups, and confirms, at a large scale, the predictive power of growth mindset for achievement gains, even with unusually rich controls for students’ background, previous achievement, and measures of other social-emotional skills. Average annual growth in English language arts and math corresponding to differences between students with fixed and growth mindset in a same school and grade level is 0.07 and 0.05 standard deviations respectively, after adjusting for students’ characteristics and previous achievement. This estimate is equivalent to 48 and 35 additional days of learning.

  • An IRT Mixture Model for Rating Scale Confusion Associated with Negatively Worded Items in Measures of Social-Emotional Learning

    Daniel M. Bolt, Yang Caroline Wang, Robert H. Meyer, Libby Pier. October 2019

    We illustrate the application of mixture IRT models to evaluate the possibility of respondent confusion due to the negative wording of certain items on a social-emotional learning (SEL) assessment. Using actual student self-report ratings on four social-emotional learning scales collected from students in grades 3-12 from CORE districts in the state of California, we also evaluate the consequences of the potential confusion in biasing student- and school-level scores as well as correlational relationships between SEL and student-level variables. Models of both full and partial confusion are examined. Our results suggest that (1) rating scale confusion due to negatively-worded items does appear to be present; (2) the confusion is most prevalent at lower grade levels (3rd-5th); and (3) the occurrence of confusion is positively related to reading proficiency and ELL status, as anticipated, and bias estimates of SEL correlations with these student-level variables. For these reasons, we suggest future iterations of the SEL measures use only positively oriented items. To maintain measurement continuity, we suggest bias corrections based on the studied mixture model may be useful, although the precision of such corrections is sensitive to the nature of confusion (e.g., full versus partial).

  • A Middle School Drop: Consistent Gender Differences in Students’ Self-Efficacy

    Erin M. Fahle, Monica G. Lee, Susanna Loeb.

    Academic self-efficacy is a student’s belief in their ability to perform within a school environment. Prior research shows that students experience a drop in academic self-efficacy during middle school that is particularly steep for female students and results in lower self-efficacy for girls than boys throughout middle and high school. In this brief, we probe whether this pattern is consistent across student groups defined by demographics, achievement level, and school of attendance. We find unusual consistency: while non-white, low-achieving, and poor students show somewhat lower self-efficacy than other students, the differential drop in middle school is essentially universal across student groups. Similarly, while schools vary meaningfully in their students’ level of self-efficacy, they also do not differ much in this trend.

  • Can We Measure Classroom Supports for Social-Emotional Learning?

    Robert H. Meyer, Libby Pier, Jordan Mader, Michal Christian, Andrew B. Rice, Susanna Loeb, Hans Fricke, Heather Hough.

    This brief applies value-added models to student surveys in the CORE Districts to explore whether social-emotional learning (SEL) surveys can be used to measure effective classroom-level supports for SEL. The authors find that classrooms differ in their effect on students’ growth in self-reported SEL—even after accounting for school-level effects. Results suggest that classroom-level effects within schools may be larger than school-level effects. However, the low explanatory power of the SEL models means it is unclear that these are causal effects that have appropriately controlled for student-level characteristics. Finally, there are generally low correlations between classroom-level growth in SEL and classroom-level growth in English language arts (ELA) or math, suggesting the SEL measures may capture growth not measured by academic test scores. Although results are preliminary, they indicate there might be measurable student growth in SEL impacted by the environment of classrooms within schools.

  • How Would Test Opt-out Impact Accountability Measures? Evidence from the CORE Districts and the PACE/USC Rossier Poll

    Edward J. Cremata. Policy Analysis for California Education. September 2019

    The number of students opting out of standardized tests has grown in recent years. This phenomenon poses a potential threat to our ability to accurately measure student achievement in schools and districts. This brief documents the extent to which opting out is observed in the CORE districts and models how higher opt-out levels could affect various accountability measures. More students opting out could significantly impact some accountability measures in use in California, but the CORE districts’ growth measure is largely unaffected, as it reports the impact of schools on individual students’ achievement. In contrast, accountability metrics that track student achievement by cohort are at risk of becoming biased even with relatively low absolute levels of opting out. This brief suggests that districts should consider explicitly adjusting for the characteristics of the students who actually sit for tests when designing school accountability systems.

  • Self-Management Skills and Student Achievement Gains: Evidence from California’s CORE Districts

    Susana Claro, Susanna Loeb. Policy Analysis for California Education. September 2019

    Existing research on self-management skills shows that measures of self- management predict student success. However, these conclusions are based on small samples or narrowly defined self-management measures. Using a rich longitudinal dataset of 221,840 fourth through seventh grade students, this paper describes self-management gaps across student groups, and confirms, at a large scale, the predictive power of self-management for achievement gains, even with unusually rich controls for students’ background, previous achievement, and measures of other social-emotional skills.

  • The Properties of Non-Academic School Performance Measures

    Rachel S. White, Morgan S. Polikoff. Policy Analysis for California Education. May 2019

    Although there is a robust body of literature studying targets for academic indicators within school quality systems few studies explore target setting for non-academic indicators. Focusing on elementary schools within the CORE districts, we investigate how moving performance targets for non-academic indicators affects school quality ratings.

  • Measuring School Contributions to Growth in Social-Emotional Learning

    Hans Fricke, Susanna Loeb, Robert H. Meyer, Andrew B. Rice, Libby Pier, Heather Hough. Policy Analysis for California Education. May 2019

    School value-added models are increasingly used to measure schools’ contributions to student success. At the same time, policymakers and researchers agree that schools should support students’ social-emotional learning (SEL) as well as academic development. Yet, the evidence regarding whether schools can influence SEL and whether statistical growth models can appropriately measure this influence is limited.

  • Charter Competition and District Finances: Evidence from California

    Paul Bruno. Policy Analysis for California Education. May 2019

    Charter schools enroll a growing share of public school students, leading to concerns about the financial implications of charter schools for traditional public schools. Using detailed expenditure data for school districts in California, this paper exploits variation in charter school enrollment across time and between districts to evaluate how district spending and overall financial health change as nearby charter sectors expand.

Twitter

  •  
  • 1 of 908

PACE thanks these funders and sponsors for their financial support