Changing Patterns of Growth in Oral Reading Fluency During the COVID-19 Pandemic

Introduction

Since the pandemic physically closed schools in March 2020, researchers have worked to predict and then estimate the impact of extended educational disruptions on student learning. These studies have collectively raised the alarm on the issue of “learning loss”—the idea that students have learned less during the pandemic than in prior years. To this end, emerging research has shown that the pandemic has resulted in significant setbacks in English Language Arts and Math; has exacerbated achievement gaps for low-income students and those learning English; and may result in lost wages and lower life expectancy in the long term.

We build on this existing research by focusing on students in the earliest grades (Grades 1–3), who are often not included in existing learning loss analyses, and by investigating the impact of the pandemic on student learning separately in spring 2020 and fall 2020. We address student learning in those critical early years by using data from an oral reading fluency (ORF) assessment—a rapid assessment taking only a few minutes that measures a fundamental reading skill—to examine COVID-19’s effects on children’s ORF during the pandemic in over 100 U.S. school districts nationwide.

The results in this brief and a related working paper can help shed light on the impact of the pandemic on student literacy development in the early grades. Such evidence is unique given the relative paucity of testing in these early grades and can be used to inform decisions being made now about how resources should be allocated at the federal and state level to address student learning impacts related to the pandemic.

Assessing COVID-19’s Impact in the Early Grades Using Oral Reading Fluency

Reading is perhaps the most important skill developed in K–12 education. ORF—the ability to read aloud quickly and accurately—is an important skill for developing readers. It can be measured relatively rapidly and is highly predictive of comprehension and other important measures of student reading ability. Early literacy has been identified as a critical foundation for students’ future academic and life success.

To measure the effect of the pandemic on student development in this critical area, we use data from an assessment delivered by Literably that measures ORF based on first recording students’ readings of texts presented to them on a device and then using a combination of human transcription and speech recognition to score these recordings. While these assessments are normally recorded in person in a classroom setting, students were able to take these recordings remotely during the COVID-19 pandemic, thus allowing us to study changes in growth of ORF related to disruptions in schooling using data from over 250,000 scores collected in over 100 school districts spread across 22 states.

The Importance of Measuring Learning Changes in Both Spring and Fall

Our analysis offers unique evidence about student growth in ORF throughout the various phases of the pandemic, looking both at spring 2020 and fall 2020 growth patterns to investigate when learning losses have been most acute. This is particularly critical given what we know about educational experiences for children in these separate periods: in spring 2020 and through the summer, most U.S. students were sheltering at home with many receiving very little instruction; this fall, while the mode of instruction was quite varied both within and across states, most school districts were able to provide a higher quality of instruction, whether delivered in person or remotely.

Figure 1 provides a conceptual depiction of our analytic approach to measuring pandemic- related learning losses. Many existing studies compare point-to-point (typically fall-to-fall) changes in student performance during COVID-19 to changes in performance over the same time period prior to COVID-19; that is, they compare changes in endpoints of yellow and blue dotted lines in Figure 1 (Panel A). In contrast, by utilizing Literably’s ongoing assessments, we can observe how the pandemic affected student learning throughout the year (Panel B). The measurable, accumulated learning loss at the end of fall 2020 is the same in Panels A and B, but the approach in Panel B allows for a more nuanced analysis. Specifically, the dotted blue line in Panel A suggests that the pandemic caused a uniform decrease in growth that has persisted across time, whereas the blue curve in Panel B reveals potential differences in learning growth across the year. Looking at the data this way provides more information about when learning losses occurred, and thus points towards more effective solutions.

Figure 1. Conceptual Model: Using Ongoing Assessment to Understand Learning Changes Throughout the Pandemic

Note. Gray regions emphasize time points at which outcome data is collected.

In the data used in our analysis, students in Grades 1–4 take three ORF assessments per year on average. We use change across these measures from before the pandemic to establish a baseline for “typical” growth in ORF. In a normal educational setting (i.e., prior to the pandemic), students develop one additional word of ORF per minute (WPM) roughly every 10 days in Grades 1 and 2; older students tend to develop ORF at a slower rate. Against this baseline level of growth, we can compare growth measured during the COVID-affected period.

Our analysis reveals three key findings: (a) growth in ORF stalled in the spring, with Grades 2 and 3 most affected; (b) growth in ORF picked up in the fall for tested students but was insufficient to make up for losses in the spring; and (c) students at lower achieving schools and those not taking the assessment may be at heightened risk of learning loss.

Findings

Growth in Oral Reading Fluency Stalled in the Spring, With Grades 2 and 3 Most Affected

We illustrate the growth of ORF skills here in two ways. First, we examine the long-run growth of children who were in Grade 1 during 2018–19 (Figure 2).

Figure 2. Estimated Within-Student Change in Oral Reading Fluency Over Several Years for a Single Cohort

Note. Gray columns indicate summer.

For these students, who are in Grade 3 in 2020–21, growth was fairly steady until the onset of the pandemic (demarcated by the vertical red line, at which point they were in second grade). However, growth flattened in the spring of 2020 as the pandemic initially disrupted schooling and remained stagnant through the summer (the gray shaded column); during these periods, children’s reading abilities are not increasing. Relative to the extrapolated trajectory of growth (yellow line), these students have clearly suffered learning loss. Such an analysis is suggestive; we refine it below to better account for age-related changes in ORF growth.

To provide another perspective that can adjust for other factors, such as changes in patterns of growth across the academic year, we also look at comparisons to growth for children in the same grade in unaffected years (Figure 3). In Grades 1–3, we show growth of students starting in the given grade in an unaffected year (2018–19; yellow line) and follow them into the following year. In the affected year (2019–20; blue line), same-grade students who experienced the pandemic showed similar patterns of growth up until the onset of the pandemic (demarcated by vertical red line). Following onset, growth slows during spring and summer 2020 before accelerating in the fall of 2020.

Figure 3. Estimated Within-Student Change in Oral Reading Fluency Over 15 Months, Before and During the Pandemic

Note. Vertical red lines indicate onset of COVID-19 pandemic; gray columns indicate summer.

Growth in Oral Reading Fluency Picked Up in the Fall for Tested Students But Was Insufficient to Make Up for Losses from the Spring

After the initial decline in ORF development in the spring of 2020, growth during the fall of 2020 resumed at levels similar to before the pandemic. This effect can be seen above in Figure 3, where the learning rates (the slope of the blue line) are relatively similar before the pandemic and in fall 2020, but with gaps in learning due to the stagnation of learning in spring 2020. To explore this effect further, we consider growth rates during the fall and compare them across academic years (Figure 4). Yellow bars show growth rates for the first 90 days of 2019–20, prior to the onset of the pandemic. Note that these growth rates are fairly high compared to the year-long growth rates noted earlier (e.g., words per minute / day = 0.1); students typically develop ORF faster at the beginning of the year than at the end. Blue bars show growth rates for the first 90 days of the COVID-affected 2020–21 year. In fall 2020, Grade 1 students seemed to learn at rates comparable to those observed for Grade 1 students prepandemic. In Grades 2–4, however, growth rates were somewhat lower in the fall during the pandemic (although not statistically different). For example, growth was lower in fall 2020 in Grade 2 than it was in past years; we observed a growth of 0.18 WPM per day in fall 2019 and growth of 0.16 WPM per day in fall 2020.

While the fall learning rates are improvements over the stagnant growth from the previous spring, they are insufficient to make up for the lost ground students experienced earlier in the pandemic. The accumulated learning loss for second and third graders would leave them 7.3 and 7.7 WPM behind their expected level respectively (representing 26 percent and 33 percent of the expected yearly gains). Further, we have reason to believe these are underestimates of the true learning loss, as many students were not assessed in spring 2020. If unassessed students were not provided instructional opportunities or were otherwise not engaged in learning, their ORF could have been yet more severely impacted.

Figure 4. Grade-Specific Fall Growth Rates Across Academic Years

New Concerns Are Emerging Around Inequitable Impacts in Fall 2020: Students at Lower Achieving Schools Are Falling Farther Behind and Students Not Taking the Assessment May Be at Heightened Risk of Learning Loss

In analyses of this kind, it is always critical to understand the differential impact on students. We emphasize two points of concern for how the pandemic may have inequitably affected students continuing into the fall. First, students in lower achieving districts are perhaps falling farther behind their peers in higher achieving districts. When we compare fall 2020 growth in districts with higher annual state test scores to growth in lower achieving districts (using the Stanford Education Data Archive from the Educational Opportunity Project site: edopportunity.org), we observe more robust growth in ORF in higher achieving districts. Figure 5 shows projected growth over time for students in districts at the 10th and 90th percentile of the achievement scale. The slopes of the lines are steeper for higher achieving districts; students in a 90th-percentile district would be expected to gain 20.4, 19.1, and 13.4 WPM over the fall; in contrast, students in a 10th-percentile district would be expected to gain 12.5, 10.5, and 8.6 WPM respectively. These gaps in growth between high- and low-achieving districts are not generally observed in previous years, suggesting that they are emerging as a consequence of the COVID-19 pandemic and, if left unchecked, will result in a widening of preexisting achievement gaps.

Figure 5. Comparison of Within-Person Change in Oral Reading Fluency for Districts at 10th and 90th Percentile of Achievement on Annual Standardized Tests

Second, students who are not able to take the assessment in fall 2020 may be falling farther behind. We estimate that approximately 10 percent of the students who were present in schools in 2019–20 were unobserved in fall 2020, but we do not have information about why these students were missing. If they have trouble accessing remote assessment—for example because they do not have reliable access to computers or Wi-Fi—they may also be engaging less with distance learning. There is emerging evidence that low-income students have been absent more often or have simply not reenrolled in school20 during the pandemic, raising equity concerns both about who is assessed and what kinds of instructional support they are receiving during the pandemic. Although our analysis reveals that prior ORF scores do not predict whether a student is assessed in fall 2020—meaning that the students with the lowest levels of prior ORF are roughly as likely as others to be missing—we suspect that students missing from our analysis may have experienced even larger learning losses than those students we did observe.

Conclusion

In this analysis we examine the effect of the COVID-19 pandemic on children’s ORF. We find that, overall, students’ ORF in second and third grade is approximately 30 percent behind what we would expect in a typical year. Furthermore, given that many students were not assessed, these averages may be underestimates of true losses. Our analyses also point to continued reasons for concern: students in lower achieving districts experienced greater declines in learning in the fall that will widen existing achievement gaps if left unaddressed.

Implications for Policy and Practice

This interruption in growth of ORF as a result of the pandemic in primary grades is concerning given the fundamental nature of reading for academic development. A large body of prior research has identified literacy by the third grade as critical for students’ future academic and life success. If students are unable to regain this loss, they may experience delays in other reading-related skill building, making it difficult to access future academic content and learning.

We are particularly concerned about the inequitable impact of the pandemic on student learning. Our analysis reveals that students in lower-achieving districts are falling behind faster relative to others. This demands attention from policymakers and educators, as greater learning loss for students in lower achieving school districts may exacerbate disparities. As leaders at the national and local level ponder policies that address the impact of the pandemic on students and families, we recommend that substantial resources be allocated to support literacy development in the early grades. Even greater resources need to be shared with low- achieving districts, which often serve a disproportionate number of low-income students.

Our research demonstrates that, after the initial shock of the pandemic in the spring, learning has resumed this fall. This provides evidence that educators are finding ways to successfully teach and assess ORF even during the continued disruptions and uncertainty of the pandemic. We can build on this success to make sure that educators have the appropriate resources, especially in schools serving students with the highest need. In addition to targeted funding, it will be important to identify the practices that accelerate learning for students who have fallen behind, and to build policy and support structures that help to enact these practices at scale.

Limitations and Next Steps

This analysis, while providing crucial evidence about the impact of the pandemic on the development of basic skills in the early grades, leaves critical questions unanswered. First, our analysis tracks students across 100 school districts in 22 states but does not currently include information about how students were educated. There is tremendous variation across and within states as to whether schools served students in fall 2020 in person, remotely, or in a hybrid fashion, and plans have continued to shift based on local transmission rates, changing safety protocols, and the political environment. Future analysis will investigate differences in outcomes in fall 2020 based on mode of instructional delivery.

Second, additional research should be done to further understand the pandemic’s impact on students who have been missing from school during the pandemic. Based on our analysis, 70 percent of students who tested in fall 2019 did not test again in spring 2020 and we suspect that 10 percent of students we would typically have observed did not test in fall 2020. Because these students are likely to be those least connected to educational opportunity, it is probable that our estimates of learning losses in both fall and spring 2020 are an underestimate of the true average effect. Even more importantly, if losses for these specific students are even more dramatic than what has been observed, they may have more difficulty catching up and will suffer the consequences well into the future. Educators and researchers alike must make concerted efforts first to understand the circumstances surrounding these students and then to reengage them and employ compensatory instructional practices to help them regain learning growth.

Third, results might not generalize to other subjects. Reading is a focal point of instruction in early education and instructional adjustments related to reading were likely a centerpiece of many district’s plans for instruction during the fall of 2020. Given its centrality, learning losses in reading may be qualitatively different from those in other subject areas.

Finally, the full extent of learning loss during the pandemic will not be understood for months or even years. Gaps in ORF that emerge now may lead to gaps in other subjects over time if problems in students’ ORF interfere with content learning in later grades. And new gaps may emerge: for example, with enrollments down in preschool and kindergarten programs this year, it is possible that incoming students in 2021–22 will start behind. As such, we must continue to examine and monitor academic performance, learning loss, and renewed learning growth as students continue in distance learning and return to school. In our ongoing research, we will continue to monitor ORF as educational policies and practices continue to adjust as we recover from the pandemic.

Read the full policy brief here, endnotes included, and the related working paper here.