The Impact of Unmotivated Questionnaire Respondents on Data Quality
Publication authors
Published
Summary
Education researchers use surveys widely. Yet, critics question respondents’ ability to provide high-quality responses. As schools increasingly use student surveys to drive policymaking, respondents’ (lack of) motivation to provide quality responses may threaten the wisdom of using surveys for data-based decision-making. To better understand student satisficing (sub-optimal responding on surveys) and its impact on data quality, we examined the pervasiveness and impact of this practice on a large-scale social-emotional learning survey administered to 409,721 students in grades 2-12. Findings...
Evidence from California’s CORE School Districts
Publication authors
Published
Summary
While the importance of social-emotional learning for student success is well established, educators and researchers have less knowledge and agreement about which social-emotional skills are most important for students and how these skills distribute across student subgroups. Using a rich longitudinal dataset of 221,840 fourth through seventh grade students in California districts, this paper describes growth mindset gaps across student groups, and confirms, at a large scale, the predictive power of growth mindset for achievement gains, even with unusually rich controls for students’...
Published
Summary
We illustrate the application of mixture IRT models to evaluate the possibility of respondent confusion due to the negative wording of certain items on a social-emotional learning (SEL) assessment. Using actual student self-report ratings on four social-emotional learning scales collected from students in grades 3-12 from CORE districts in the state of California, we also evaluate the consequences of the potential confusion in biasing student- and school-level scores as well as correlational relationships between SEL and student-level variables. Models of both full and partial confusion are...
Consistent Gender Differences in Students’ Self-Efficacy
Published
Summary
Academic self-efficacy is a student’s belief in their ability to perform within a school environment. Prior research shows that students experience a drop in academic self-efficacy during middle school that is particularly steep for female students and results in lower self-efficacy for girls than boys throughout middle and high school. In this brief, we probe whether this pattern is consistent across student groups defined by demographics, achievement level, and school of attendance. We find unusual consistency: while non-white, low-achieving, and poor students show somewhat lower self...
Published
Summary
This brief applies value-added models to student surveys in the CORE Districts to explore whether social-emotional learning (SEL) surveys can be used to measure effective classroom-level supports for SEL. The authors find that classrooms differ in their effect on students’ growth in self-reported SEL—even after accounting for school-level effects. Results suggest that classroom-level effects within schools may be larger than school-level effects. However, the low explanatory power of the SEL models means it is unclear that these are causal effects that have appropriately controlled for student...
Findings From the First Large-Scale Panel Survey of Students
Published
Summary
Measures of school-level growth in student outcomes are common tools for assessing the impacts of schools. The vast majority of these measures use standardized tests as the outcome of interest, even though emerging evidence demonstrates the importance of social-emotional learning (SEL). This article presents results from using the first large-scale panel surveys of students on SEL to produce school-level, value-added measures by grade for growth mind-set, self-efficacy, self-management, and social awareness. The article finds substantive differences across schools in SEL growth, with...
The Case of Garden Grove Unified School District
Published
Summary
A crucial but challenging requirement of successful continuous improvement involves transforming the system’s culture. This case study explores how Garden Grove Unified School District built a culture that puts kids first; nurtures commitment, drive, and loyalty among teachers and other district personnel; and views both student and adult learning as important. This case examines four structures and processes used by Garden Grove leadership to establish and maintain a culture of improvement that has resulted in rising student achievement. The lessons learned could be implemented in many...
The Case of Long Beach Unified School District
Publication author
Published
Summary
Successful continuous improvement requires educators to have a shared clarity of purpose, integrated systems of support, and a clear vision for instruction across the system with the central goal of improving classroom instruction. This case study examines how Long Beach Unified School District, one of the CORE districts involved in the CORE-PACE research project on continuous improvement, fosters these efforts. It is a portrait of a learning system that emphasizes improvement towards high-quality, rigorous instruction for all students through professional learning and capacity-building. Their...
The Case of Ayer Elementary
Publication authors
Published
Summary
Ayer Elementary School in Fresno is an exemplar of leadership practice necessary for successfully building and maintaining a culture of continuous improvement. This case study examines the leadership practices that teachers say allowed them to undertake the challenging work of using data for evidence-based changes that are steadily improving student outcomes in this ethnically diverse, high-poverty school. The report offers insights into how leaders can foster a culture of risk-taking, teacher agency, and collective efficacy. It also raises questions about how to support more principals in...
Lessons from the CORE Districts
Published
Summary
Continuous improvement has become a leading method of changing the way schools and districts foster better student learning and success. As part of the CORE-PACE Research Partnership, PACE spent a year studying the CORE Districts’ approach to implementing continuous improvement with a focus on two key questions: 1) What do we know about how to support educators in learning continuous improvement? 2) What conditions support continuous improvement in districts and schools? The findings are presented in a report that provides an overview of lessons learned in building a successful continuous...
Evidence from the CORE Districts and the PACE/USC Rossier Poll
Publication author
Published
Summary

The number of students opting out of standardized tests has grown in recent years. This phenomenon poses a potential threat to our ability to accurately measure student achievement in schools and districts. This brief documents the extent to which opting out is observed in the CORE districts and models how higher opt-out levels could affect various accountability measures. More students opting out could significantly impact some accountability measures in use in California, but the CORE districts’ growth measure is largely unaffected, as it reports the impact of schools on individual students’...

Evidence from California’s CORE Districts
Publication authors
Published
Summary
Existing research on self-management skills shows that measures of self-management predict student success. However, these conclusions are based on small samples or narrowly defined self-management measures. Using a rich longitudinal dataset of 221,840 fourth through seventh grade students, this paper describes self-management gaps across student groups, and confirms, at a large scale, the predictive power of self-management for achievement gains, even with unusually rich controls for students’ background, previous achievement, and measures of other social-emotional skills. Self-management is...
A Research Summary and Implications for Practice
Published
Summary
Given the importance of a college degree for both individual and societal economic prosperity, policymakers and educators are focused on strengthening the path to college beyond college entry. In this report, we synthesize the existing literature on four factors key to educational attainment—aspirations and beliefs, academic preparation, knowledge and information, and fortitude and resilience—and the implications of each.
Published
Summary
Continuous improvement is a holistic and research-based approach to education grounded in the belief that every system is designed to achieve the results it gets; therefore, change must be systemwide, not piecemeal. California is a national leader in the continuous improvement movement that is spreading throughout local school districts as well as state and county offices of education. At its annual conference in February 2019, PACE convened a panel of California educators working on the cutting edge of continuous improvement. In this brief, they share their stories and lessons learned.
Published
Summary
Governor Gavin Newsom’s budget proposal for 2019–2020 includes $10 million to develop a statewide longitudinal data system—including early education, K–12, and higher education institutions as well as health and human services agencies—to better track student outcomes and improve alignment of the education system to workforce needs. California’s lack of a coherent education database serves as a substantial barrier to fulfilling the state’s continuous improvement policy goal and ensuring all students have access to robust learning opportunities to enable them to be successful in school and...
Publication authors
Published
Summary
Although there is a robust body of literature studying targets for academic indicators within school quality systems few studies explore target setting for non-academic indicators. Focusing on elementary schools within the CORE districts, we investigate how moving performance targets for non-academic indicators affects school quality ratings. We ask: (1) How does school performance on CORE’s school quality improvement measures vary across schools and over time?; and (2) How does the setting of targets on CORE’s non-academic indicators at various levels impact the number and types of schools...
Publication author
Published
Summary
The CORE districts have been measuring SEL via self-report student surveys since 2015. Any district, state, or school looking to use these surveys can now build from what we have learned in the CORE districts. In this paper we provide benchmarking data, including means and standard deviations by construct, grade level and subgroup, and examples of how to use these data in practice. The data come from nearly half a million students across the 8 CORE districts, in grades 4 through 12, who took the survey in the 2015-16 school year. While not a true national sample, the CORE benchmarking sample...
Published
Summary

School value-added models are increasingly used to measure schools’ contributions to student success. At the same time, policymakers and researchers agree that schools should support students’ social-emotional learning (SEL) as well as academic development. Yet, the evidence regarding whether schools can influence SEL and whether statistical growth models can appropriately measure this influence is limited. Recent work shows meaningful differences across schools in changes in SEL scores by grade, but whether these differences represent the effects of schools is still unclear. The current paper...

Describing Chronically Absent Students, the Schools They Attend, and Implications for Accountability
Publication author
Published
Summary

Student absenteeism has recently entered the national spotlight with its emphasis in the Every Student Succeeds Act, and here in California with its inclusion in the School Dashboard. Yet many questions remain about who chronically absent students are and how they are concentrated within schools. In chapter 1 (of the edited book, Absent from School), the author uses data from the CORE districts—which serve nearly one million students in over 1,000 schools in California–to better understand differences across students and schools, comparing these measures to a broader set of school performance...

Published
Summary
California’s shift towards continuous improvement in education makes understanding how districts and schools can learn to improve a more pressing question than ever. The CORE Improvement Community (CIC), a network of California school districts engaged in learning about improvement together, is an important testing ground to learn about what this work entails.
Learning from the CORE Data Collaborative
Published
Summary

Experts agree that effective data use is critical for continuous improvement. However, there is a lack of understanding statewide about how data use for continuous improvement, with its adaptive and iterative nature, differs from data use for other purposes. In this paper, the authors discuss what data are most useful to inform continuous improvement at all levels of the system and provide a case study of how the CORE data collaborative uses a multiple-measures approach to support decision-making. {"preview_thumbnail":"/sites/default/files/styles/video_embed_wysiwyg_preview/public/video...

Building System Capacity to Learn
Publication authors
Published
Summary

Creating continuously improving education systems could be the antidote to one-off education reforms that come and go with little to show for the effort. The strategy has been picking up steam in recent years, urged on by the federal Every Student Succeeds Act (ESSA); the California Department of Education; and the Bill & Melinda Gates Foundation, which recently announced that it is earmarking 60 percent of its $1.7 billion investment in education during the next five years to support school improvement networks. Continuous improvement in education evolved from decades of similar efforts in...

A Pragmatic Approach to Validity and Reliability
Publication authors
Published
Summary
As educational practitioners and policymakers expand the range of student outcomes they assess, student perception surveys—particularly those targeting social-emotional learning—have grown in popularity. Despite excitement around the potential for measuring a wider array of important student outcomes, concerns about the validity of the inferences that might be drawn from student self-reports persist. One of the most ambitious attempts to incorporate student perception surveys into a larger assessment framework has occurred through CORE—a consortium of school districts in California. Pulling...
Evidence from the CORE Districts
Published
Summary
Mounting evidence demonstrates that social-emotional skills are important for students’ academic and life success, yet there is limited evidence on how these skills develop over time and how this development varies across student subgroups. This study uses the first large-scale panel survey of social-emotional learning (SEL) to describe how four SEL constructs—growth mindset, self-efficacy, self-management, and social awareness—develop from Grade 4 to Grade 12, and how these trends vary by gender, socioeconomic status, and race/ethnicity. The results are based on self-report student surveys...
An IRT Modeling Approach
Published
Summary
With an increased appreciation of students’ social-emotional skills among researchers and policy makers, many states and school districts are moving toward a systematic process to measure Social-Emotional Learning (SEL). In this study, we examine the measurement properties of California's CORE Districts’ SEL survey administered to over 400,000 students in grades 3 to 12 during the 2015-16 school year. We conduct analyses through both classical test theory and item response theory frameworks, applying three different polytomous IRT models on both the full student sample and on separate samples...