COVID-19 and the Educational Equity Crisis

The analysis in this commentary has been updated to include further documentation and additional data through winter 2021. See the report and related infographic COVID-19 Impacts on Student Learning: Evidence From Interim Assessments in California.

From the day California schools closed in March, researchers, policymakers, and educators alike have been concerned about the impact that the pandemic would have on student learning. We are especially worried that our most vulnerable students will experience this so-called “learning loss” more than others. Because of data limitations in California, we have not had any estimates to date of how the pandemic has affected learning in our state—overall or for particular student groups. Given the critical importance of evidence to inform decision-making on school reopening, we are sharing early findings on student learning in Grades 4–10 in 18 school districts that are part of the CORE Data Collaborative.1 While full analyses and documentation are forthcoming, our investigation thus far reveals two key findings that we felt it urgent to share:

1. There has been significant learning loss in both English Language Arts (ELA) and Math, with students in earlier grades most affected.
2. The equity impact is severe—certain student groups, especially low-income students and English language learners (ELLs), are falling behind more compared to others.

What is “learning loss” and how can it be measured?

Year to year, students are expected to learn new content and develop new skills; formative assessments are designed to measure student growth throughout the year towards grade-level standards. Concerns about “learning loss” are concerns that students aren’t learning content and mastering skills at the same rate that they typically would be. Figure 1 shows this hypothesized effect, with the blue line indicating typical learning and the orange line showing learning during the pandemic.

Figure 1. How to think about “learning loss” associated with COVID-19

We know from prior years what academic growth in learning looks like in a typical year. Given the disruptions in schooling since March 2020, we hypothesize that students may learn less over the course of the pandemic. The difference between what they would have learned in a normal year and what they learn during the pandemic is what we refer to as “learning loss.” Formative assessments can be used in the 2020–21 school year to monitor how student learning this year is different from that of prior years, and to measure how quickly (or slowly) students are “catching up” academically to where we would predict they would typically be by the end of a typical (nonpandemic) school year.

Why do we need to be worried about the inequitable impact of the pandemic on student learning?

The pandemic has exacerbated a preexisting equity crisis in our state. Due to inequitable access to healthcare, income inequality, and disproportionate employment in high-risk, “essential” jobs, low-income, Black, and Latinx communities are suffering most from the health and economic impacts of the pandemic. Most California schools have been physically closed since Spring 2020, and students in these groups have been doubly disadvantaged by distance learning. Low-income and rural students may not have reliable access to computers or Wi-Fi, making it difficult to access online instruction. Others do not have appropriate conditions for learning at home and struggle with food or housing insecurity or with limited adult support. Distance learning is harder to implement for students in certain groups, including students in the youngest grades, students with disabilities, and students learning English. While some analyses to date have shown inequity in grades, attendance, and enrollment, our analysis is the first to use student-level demographic data to show differences in learning during the pandemic in California by income and language status.

Finding 1: There has been significant learning loss in both English Language Arts and Math, with students in earlier grades most affected.

To measure the learning loss resulting from COVID-19 and associated school closures, we compared “learning during COVID” (Fall 2019 to Fall 2020) to Fall-to-Fall learning in three prior years.2 Our analysis included 6 districts in California that administer the MAP assessment and 12 districts in California that administer the STAR assessment. We analyzed the data separately by grade (4–10), by assessment (MAP and STAR), and by subject (ELA and Math).

Overall, we found that there has been significant learning loss in both ELA and Math across both assessments in the earlier grades. Figure 2 displays the overall learning change results by grade, subject, and assessment in scale score points on the MAP or STAR assessment scale. A value of zero indicates that there has been no significant learning change associated with the pandemic. Orange bars indicate learning loss, or that there was less growth from Fall 2019 to Fall 2020 on the given assessment for that grade and subject compared to previous years. Green bars indicate learning gain, or that there was more growth from Fall 2019 to Fall 2020 than in previous years.3

Figure 2a. Learning change on the MAP assessment4

Figure 2b. Learning change on the STAR assessment5

To help contextualize the results, we can note what typical growth looks like. For MAP, typical yearly growth is about 10 points in elementary school, 5.5 points in middle school, and 3 points in high school. For STAR, typical yearly growth is about 100 scale score points in ELA but it varies substantially in Math depending on the grade. However, the spread of scores between individual students tends to become wider in later grades, which means that a direct comparison between our learning loss measures and typical growth could be overstated in later grades, especially high school.

It is important to note that the accelerated learning in the upper grades is unexpected and warrants further investigation to understand whether observed growth reveals true learning acceleration or whether it might be related to: increasing comfort with technology; changes in test administration; the fact that fewer students were assessed this year than in prior years; or even student cheating.

Finding 2: The equity impact is severe—certain student groups, especially low-income students and English language learners, are falling behind more compared to others.

Our preliminary analysis shows two clear findings with widespread implications for equity. Specifically, we found significantly more learning loss from Fall 2019 to Fall 2020 compared to previous years for: (a) students from a socioeconomically disadvantaged (SED) background, particularly in ELA, on both assessments across Grades 4–8 and on STAR Math in early grades; and (b) English language learners (ELLs), especially on MAP ELA across Grades 4–9 and on STAR Math in early grades.

Figure 3a. Learning change on the MAP assessment, by socioeconomic disadvantage (SED)

Figure 3b. Learning change on the STAR assessment, by socioeconomic disadvantage (SED)

Note. Socioeconomically disadvantaged students (SED) are identified based on free and reduced-price lunch status, parent education levels, and other personal circumstances, including homelessness or placement in foster care.

Similarly, learning loss has been more dramatic for ELLs than for others. As Figure 4 shows, in nearly every grade in ELA and in early grades for Math, across both assessments, ELLs have lost substantially more learning than other students. In some grades, the impact is quite severe. For example, on the MAP ELA assessment (Figure 4a, left), Grade 5 ELLs’ learning grew almost 3 scale score points slower during the pandemic (roughly 30 percent of a year of typical growth), while other students lost the equivalent of roughly 10 percent of a year of typical growth.

It is important to note that, while overall learning loss is less severe in the upper grades (see Figure 2), there is evidence that ELLs in upper grades have been very negatively affected, likely due to the substantial challenges with supporting students who are learning English in a virtual setting, which persist into the upper grades.

Figure 4a. Learning change on the MAP assessment, by English language learner (ELL) status

Figure 4b. Learning change on the STAR assessment, by English language learner (ELL) status

Conclusions and limitations

The COVID-19 pandemic has introduced an unprecedented disruption to U.S. education. In order to recover from this crisis, we first need to understand its impact. California has a notoriously weak data infrastructure and the cancellation of state tests in Spring 2020 left us with very limited information about student performance. In this analysis, we used data from 18 districts in the CORE Data Collaborative to try to shed light on the inequities in learning that have been introduced as a result of the pandemic. We found that there has been substantial learning loss, especially in the earlier grades, in both ELA and Math. But most importantly, we show that the pandemic and its related disruptions to schooling in California have disproportionately affected low-income students and ELLs. Average learning loss estimates mask the reality that some students in California are suffering much more during this time than are others. Without aggressive and bold actions, these students may never catch up; any funding or support designed to mitigate learning loss must be targeted specifically to the students who need it most. Our analysis to date shows that low-income students and those learning English have fallen behind, and future analyses will further illuminate the impact on students in other demographic groups.

At the same time, it is important to note the limitations of our analysis. First, any research on learning loss is only a partial estimate of the true effect, because schools and districts are only able to measure those students who are present for assessments. In our analysis, like others of its kind, some students were unobserved. In the districts in our sample, we know that fewer students completed assessments in Fall 2020 compared to previous years, with a disproportionate number of students with disabilities missing from the Fall 2020 data. In our full forthcoming analysis, we will present results for additional student groups (including results by disability status, prior achievement levels, and race/ethnicity) as well as a deeper analysis of the sample with thorough investigation into which students are missing and what the effect of their absence from the data might be on learning loss estimates. Such further analysis will shed light on whether these results are in fact underestimating the true equity impacts of the pandemic.

Second, this analysis, of course, does not include every student in the state. We included data from 18 districts with a combined 50,000 students. While our sample is very diverse, it does not perfectly match the student population across the state of California, nor does it represent all of the diversity that can be found across our state’s 1,000 school districts. However, the overrepresentation of some student groups (particularly SED and ELL students in the MAP sample) does have the benefit of allowing us to more precisely measure learning loss among these student groups. Because our interest here is to disaggregate patterns of learning loss for these student groups individually rather than to draw broad conclusions about the state of learning loss (or gain) across all students in California, this sample ensured that we have the sample size and statistical power to examine such disaggregated results.

Third, any analysis aimed at understanding learning patterns across students will be improved by including details about programs and practices. In future analyses, we will include the mode of instruction in the districts during this time period (e.g., in person, virtual, or hybrid), and will also try to include any information that can be gleaned from districts’ Learning Continuity and Attendance Plans.

Finally, analyzing whether and how student learning in ELA and Math has been affected is only one piece of understanding what students have experienced during the pandemic and how school will need to be reorganized to meet their needs. The pandemic has introduced a great deal of hardship into many students’ lives, and leaders at the school, district, and state levels will need tools to assess students’ social-emotional well-being, including physical and mental health needs, social systems of support, and general readiness to learn.7 Ultimately, addressing students’ learning loss will require a student-centered approach that puts family and student relationships first, and a systemic transformation in how schools address the overlapping learning, behavioral, and emotional needs that support effective learning and teaching.

• 1Our analysis included 6 local education agencies (LEAs) in California that administer the MAP assessment and 12 LEAs in California that administer the STAR assessment. In Fall 2020, this included roughly 31,000 students on the MAP assessment in English Language Arts (ELA) and Math; roughly 18,000 students on the STAR assessment in ELA; and roughly 3,500 students on the STAR assessment in Math. The samples used in this analysis include an overrepresentation of some student groups, particularly socioeconomically disadvantaged students, English language learners, and Latinx students. These 18 districts serve over 180,000 overall.
• 2Specifically, we used 4 years of Fall interim assessment data (Fall 2017–Fall 2020) to compute 3 years of academic growth (Fall 2017–18, Fall 2018–19, Fall 2019–20). For a given year of growth (e.g., Fall 2017–18), students needed assessment data for both the pretest (Fall 2017) and the posttest (Fall 2018). We computed models separately for each grade and subject for both MAP and STAR. The growth models calculate a predicted score for a student’s posttest based on that student’s pretest and demographic information (i.e., English language learner, student with a disability, economic disadvantage, and homeless status). We then measured growth as the difference between the student’s predicted posttest score and the student’s actual posttest score. Those growth measures were averaged across students in the same year, grade, subject, and district to produce an overall estimate of growth. We then calculated learning change, which is the difference between growth in the most recent year (i.e., growth from Fall 2019 to Fall 2020) compared with prior years (i.e., the average of growth from Fall 2017 to Fall 2018, and from Fall 2018 to Fall 2019). Learning change is always calculated by comparing growth from year to year for students in the same grade, subject, and district; as a result, these measures compare different cohorts of students. In other words, the learning change measure compares Fall-to-Fall growth between the current cohort of students in a grade and subject with that of previous cohorts of students in that same grade and subject. We average learning change measures across districts to compute learning change across our sample in California. It is important to note that this analysis highlights changes in growth rates as a result of the pandemic, which suggests that existing achievement gaps between some student groups are widening in some grades and subjects.
• 3Specifically, on the MAP assessment, we found that most grades had negative learning change in ELA. Only earlier grades showed negative learning change in Math. In both ELA and Math, Grade 5 students showed the largest learning loss. On the STAR assessment
  we find a similar pattern, with earlier grades showing statistically significant negative learning change in ELA and Math, and Grades 5 and 6 showing the largest losses. The black dotted lines indicate confidence intervals: if the dotted line overlaps with zero, it indicates the result was not statistically significant; in other words, there is not enough evidence to conclude whether there was more or less growth compared to previous years.
• 4How to read this figure: Tested students entering Grade 4 grew 0.5 scale score points slower in ELA from Fall 2019 to Fall 2020 than did tested students entering Grade 4 in previous years from Fall to Fall.
• 5How to read this figure: Tested students entering Grade 4 grew 20.2 scale score points slower in ELA from Fall 2019 to Fall 2020 than did tested students entering Grade 4 in previous years from Fall to Fall.
• 6Specifically, tested SED students entering Grade 4 ELA grew 0.8 scale score points slower from Fall 2019 to Fall 2020 than did tested SED students entering Grade 4 in previous years from Fall to Fall. Tested non-SED students entering Grade 4 ELA grew 0.5 scale score points faster from Fall 2019 to Fall 2020 than did tested non-SED students entering Grade 4 in previous years from Fall to Fall.
• 7This research team is also working to analyze results from the CORE Student Well Being and Learning Conditions Diagnostic Survey, which was designed to help educators support student well-being during the pandemic and assesses student social-emotional well-being, resilience, social networks, remote learning environments, assets, needs, and interests.