Carolyn Heinrich on Whether the Labor Market Offers the Same “Credit” for Online Learning

Chancellor: Hello and thanks for joining us for the Poverty Research and Policy Podcast, from the Institute for Research on Poverty at the University of Wisconsin–Madison. I’m Dave Chancellor. For this podcast, we’re going to be hearing from Carolyn Heinrich about a study that compared the longer-term outcomes of high school students using online courses versus traditional classroom instruction for credit recovery. And of course, online education is very relevant right now given that many students around the country are in some sort of remote or blended learning environment due to the COVID-19 pandemic, but it’s important to note that we did this interview prior to the pandemic. But before we get to learning about her study, I asked Professor Heinrich to introduce herself.

Heinrich: I’m Carolyn Heinrich. I’m a professor of public policy, education, and economics at Vanderbilt University and I study education in a very interdisciplinary way. I look at the environment students are in, the supports for learning they get, and also think about this in terms of our policies and how our policies are guiding educational interventions and I particularly focus on students in school districts where there are large numbers of students who are economically disadvantaged or falling behind in their academic progress.

Chancellor: The origins for this study we’re hearing about here actually trace back to the No Child Left Behind act, which was put in place under the George W. Bush administration. Part of what the No Child Left Behind Act did was to look at gaps in achievement and student success and then mandate interventions for school districts that weren’t making the progress they were expected to. One of the common ways this took place was for schools to have to offer extra help to students outside of the regular school day. Under the law, districts were required to rely on outside providers for this educational assistance. The idea being that if the district was failing students during the school day, how would they do better outside the school day? With a grant from the Institute for Educational Sciences, Heinrich and her colleagues partnered with large school districts across the country to see how the districts were approaching extra help and tutoring for their students. And Heinrich says that over the course of the project, she and her colleagues noticed that the tutoring that students were receiving was delivered more and more through online formats. Then with new funding from the W.T. Grant Foundation and a Texas private donor, Mr. Jaime Davila, they sharpened their focus on the different ways that two large urban school districts were using digital educational tools. In one of those districts, similar to many across the U.S., online course-taking was a rapidly-expanding approach to delivering core educational content.

Heinrich: What we saw was that, over time, those online vendors of instructional materials were increasingly delivering the core content of the courses that the students needed to graduate. In high schools, as you know, they do have a set of minimum core courses that they require students to have, in addition to total number of credits, in order to graduate. And so, in school districts where a large number of students initially fail one of those core courses, it’s relatively expensive, especially if there are large numbers of students failing, for the districts to provide them another opportunity to take the course in the classroom. What the online course-taking systems offer is for the students—often the term that’s used is to “recover” those credits, or retake the course in this different setting and then if the students pass the course, then that course grade replaces their failed course grade and then they move on. And if they don’t have to be in the traditional classroom, we saw school districts using other ways for example. Larger online learning labs. Or the students could also complete them in settings outside of school. We can describe it as anywhere, anytime access to the course instruction.

Chancellor: And this shift in the way that instruction was being delivered introduced a bunch of variables that Heinrich and her team wanted to understand.

Heinrich: For example, in some of the online learning labs that we saw school districts use, you may have larger class sizes. Students are in the lab, and they’re all possibly taking different types of courses, whatever they fail to recover. And then the instructor or the live instructor in that lab, if they’re to help any of those students, they would have to know a lot of different types of content. They also have to know the system to help the students. The students have to learn the system. Lots of things, as we know, technical issues can come up. We saw those happening over time. It changes a number of things, right? It changes the role of the instructor in the classroom, whether they’re more managing student access to the course content, whether they’re rather than actually being an instructor or helping students to understand it. So how much of the different types of roles were instructors in those classrooms actually playing?

Chancellor: Heinrich and her team also paid a lot of attention to the classroom setup and how those dynamics might have mattered when it came to the students’ learning.

Heinrich: You’re taking kids who in various classes had failed them, and you’re grouping all of these kids together in these settings if they’re doing it in school and then expecting these kids with less live instructor help to be able to progress through classes where, essentially what it looks like from our perspective is, there’s a video instruction which is what we call asynchronous or there’s no live interchange, there’s not a live instructor in there, just a video. The students watch the video, they’re given an assessment. There might be modules and then they take quizzes in those modules and then they’ll progress to an end of unit test. And so the assumption of the way it works is that the students are engaging with those instructional videos, learning, the assessment tells them how well they’re doing, and they have to retake it if they fail it, but ultimately they can complete it, and then when they complete the modules for a given course it can complete the course. So that, is in a nutshell, what we were looking at. It’s also important I think to point out that the vendor determines that content. What’s in there, and of course most of the vendors today are selling the content as meeting the Common Core Standards, right, in order for school districts not to have to worry about that.

Chancellor: I asked Professor Heinrich how the students used and responded to these online courses.

Heinrich: I think you have to kind of think about high school students in general. In the classroom settings we saw a lot of signage on the walls, you know, “no cell phones, no talking,” and students, and it’s hard for them to give up their cell phones. So imagine rows of students in front of desk top computers, or sometimes they had laptops, and they were spread around the room and they had headphones or earbuds in so that they can all do their own course and there’s not noise in the room from everybody’s different courses. But what we often saw was that certainly the headphones were on or the earbuds were in, but often the students had them plugged into their cell phone, listening to music, sometimes they would be downloading music online, sometimes students would be watching a football game online. So you put a device in front of the students and expect them to work for hours on it and not surprisingly, they’re off doing other things, just like those of us who work all day long in front of a computer, once in a while you go off and you go to different websites just to kind of refresh.

Chancellor: But Heinrich says that their concern was that students weren’t just occasionally refreshing by looking at something else online, but that they weren’t really listening to the videos or engaging with them at all.

Heinrich: It’s hard to imagine that they’re learning the content and so then when they come to an assessment, what we oftentimes saw—again, they’re in front of a computer—they could go to a website and they could just Google, so they paste in the text of the question, and Google and see what they find. Now, there might still be some work in determining what’s the right answer but most of the tests are multiple choice questions, too, so they’re not writing essays, they’re not in discussions with their fellow students, there’s no group project work. So, some of the other ways that we think about engaging students and actually learning content, to make it interesting—hands-on learning. One student I talked to who was graduating and who had told me that she had taken courses online all four years of her high school and she said specifically that she really missed the opportunity for hands-on learning or more engagement with her peers in the learning process.

Chancellor: Heinrich’s team was part of a research practice partnership—so they were actively sharing their findings with the district, and some schools had policies about trying to make the learning more engaging and incorporating input from a live instructor. But, in practice, a lot of this came down to the individual instructor and how much they were able to encourage students and work with them. Regardless, one of the problems was that a lot of these instructors were working with very large class sizes.

Heinrich: We saw sometimes rally large student to teacher ratios. For example, up to 70 students for one teacher and maybe an aide, sometimes 40, 45. So that’s a big class to be managing and there’s a lot of logistical things a teacher has to do. So, the students are supposed to be watching instructional videos, have a notebook out, taking notes, and then they’re supposed to bring those notes out when they go to the assessment. And so that was the practice that was encouraged. Again, whether or not that was enforced depended a lot on the live instructor, their interest, and capacity. Students were pretty good about when they saw a teacher stand up and start walking around, getting off the website they weren’t supposed to be on, pulling out their notebook, and looking like they’re doing something. And then when the teacher makes the turn …. back to what they were doing. So, part of our work involved literally hundreds of observations of the educational settings. And we had developed a specific research instrument that we used to document the interactions we observed between the teachers and the students, the students and the technology, the student, teacher, and technology interacting. Something we called “student digital citizenship.” So, were the students using the technology in the ways it was intended or in unintended ways? We looked at the classroom environment which could also affect students, so, how much was going on with their peers around them to distract them from learning and sometimes teachers were stricter in the classroom and sometimes teachers let the students do whatever they wanted and so there was a lot less engagement on the part of the students when a lot of your peers are goofing around, talking, doing other things. It’s harder for you to just sit in front of your computer.

Chancellor: The school district and the teachers were well-aware of these issues with distractions, and Heinrich says they were continually trying to develop strategies to help students learn in these settings.

Heinrich: So, for example, setting up goals, asking teachers to meet with students, setting for each student weekly goals or expectations for how much progress they would make in their course. And then for students who aren’t making progress, developing a strategy over time to basically what they called “disable the course,” so that the student couldn’t go on and continue and require the student to re-engage with the teacher in order to develop a plan for how they’re going to improve their work. The other thing we saw over time was that some students simply weren’t suited to taking courses online. Teachers often remarked that students weren’t up to the reading levels that were built into the courses, again, because the district itself wasn’t developing the content, they were relying on the vendor to be setting the content level. The vendor itself says that it requires such and such a reading level but unfortunately, in some of our large urban school districts, we still have some students reading at primary levels when they’re entering high school. That makes it really difficult. So, part of it too was identifying over time the students that really weren’t suited or prepared.

Chancellor: Heinrich and her fellow researchers also found that the structure of some of the lessons in the online settings didn’t fit with the real life needs of some of the English Language Learners.

Heinrich: But there were also things for example that online courses had—supposedly had—accommodations for English Language Learners where the students could get a translation but the problem was that the translation that was provided was a written translation not an audio translation. For example, some students, who were English language learners and maybe they for example hear a particular language at home spoken, they may not ever study the written word of that language, especially if they’re going to school learning English. A written translation for them is not very helpful, they need an audio translation. The system wasn’t accommodating those kinds of things, too. So those were things we learned over time and information we gave back to the school districts so they could understand what were some of the barriers kids were having.

Chancellor: Heinrich and her fellow researchers also worked with some of the teachers and tried to draw lessons from instructors who were successful in their engagement with students.

Heinrich: So, we, for example, identified teachers that we saw using particularly effective practices and created a teacher fellow program so they could share those practices with their fellow teachers and thought of ways like that to improve it. But again, the concern was that these are students, if they’re failing courses, they’re already struggling academically, and they may need more, not less, of those kinds of supports. And content assistance, engagement with an instructor—that’s really hard to do if your student to teacher ratio is really large. Or if you have expectations that a teacher is going to be able to help students in the content for eight different types of courses. So another practice that the district over time was trying to—if possible, and that of course would depend on the number of users in a high school—group students by the type of course they’re recovering online and then have the teacher in the classroom be a teacher with some content expertise there. It’s not going to work perfectly because what the school districts have to do is balance the resources that they can allocate to this versus having the traditional arrangement that they would have if they didn’t have the online course taking option.

Chancellor: So far, we’ve been hearing a lot of descriptive information about what the researchers saw in the classrooms, but for the study itself, the researchers were very interested in the actual learning outcomes for the students.

Heinrich: So, one of the main reasons we wanted to study this is, of course, is because we care about whether students are learning and whether or not they are benefiting, or possibly losing, from this option. So, what are the ways they might benefit? One is opportunity. For some kids, going back into the same classroom, the same environment would just mean another failure for the students and maybe an online option, because it is anytime, anywhere access, would give them a better opportunity to complete the course successfully. This could be a positive thing, right? Or, for example, we saw online course taking in higher proportions in high schools where there were students with particular educational needs. For example, pregnant and parenting teens, students who had had interactions with the criminal justice system and may not be able to sit all day long in a classroom, may have other obligations. Or they’re coming back after being out of school to try to recover the credits and complete, and so there was also the possibility for some of these kids, without this option, if they just had to go back into a traditional environment for all of their classes, they wouldn’t be able to complete the degree. So, it could also be beneficial in helping students who would otherwise not complete high school complete high school. And in the school district we’re studying, it’s a high poverty school district so over 80 percent of the kids qualify for free and reduced-price lunch. It’s also an urban district in an urban environment with a lot of economic challenges. And these are also students, historically underserved students. They are a high proportion of Black students, Hispanic or Latinx minority students. We say minority is a term that we don’t use as much because they are the majority in this school, are more than the majority in this school. So that’s kind of an old term for describing it. So, we worry, these are kids who’ve been historically underserved in our public schools and so we are concerned about this, the fact that these more educationally disadvantaged and economically disadvantaged. We want to make sure it’s helping them.

Chancellor: They look specifically at the subset of kids who have failed courses and then compare those who complete them online with those who do so in the traditional classroom environment.

Heinrich: We compare the kids who have taken courses online with those who haven’t. We adjust for characteristics. We have to use fairly sophisticated econometric tools to adjust for the fact that we would expect those kids to be different from the student population, so that’s why we also limit our sample for our analysis to just those kids who ever failed a course. We tried to create two groups that were as comparable as possible. I think we have pretty good strategies for doing that. And then we look at their outcomes and so, for example, one of the things that we saw early on was that kids are recovering those credits, right? When they complete the class online, it replaces a failed course and they are doing better on progressing on their credit accumulation which is important for graduation. We also saw that the upperclassmen, 11th and 12th graders, used the tool much more effectively than the underclassmen, especially the freshmen weren’t doing so well. So over time we gave those findings to the school district and they used that tool less and less for the underclassmen. The upperclassmen were definitely recovering credits and we did follow them through high school and into the labor market. We also do see from our estimates that more of those kids who take courses online than hadn’t are graduating on time. And this is generally something we see in national statistics so just to be clear, this vendor, this program that we’ve been studying is used in all 50 states in the country, it is used in 8 of the 10 largest school districts, and school districts look a lot like our urban school district that we studied, higher poverty, more economic and educational disadvantage, more students who are English language learners. We do see, and you can see in the general statistics that yes, more kids are completing high school.

Chancellor: Heinrich says that one key finding is that these online credit recovery options are really cost effective. Her team’s analysis found that this kind of intervention is up to 30 times cheaper than other interventions school districts have tried to use to boost graduation rates.

Heinrich: From our look and our research and our estimates and getting cost estimates from the school district on what they spend, this intervention is about 8 to 30 times cheaper than other interventions school districts have tried in the past to raise graduation rates. They can much more cheaply increase the graduation rate through the use of credit recovery. Some district leaders, again these are from across the country, Florida, Nashville, Tennessee where I am from, other places have stated pretty clearly that—Texas, the Texas Education Commissioner, how can we not use a tool that has such a big effect on increasing our graduation rates? So, our concern was that if the kids moved to graduation but they’ve learned less. Another thing we saw in our research that even though the credit accumulation was higher, we were seeing negative relationships between online learning and their test scores.

Chancellor: Or put another way, the more online courses students took, the lower their test scores generally were. And the fact that one vendor was providing materials to a lot of school districts was a problem.

Heinrich: And test scores we think of as to capture what they’re learning, right? And that was a concern. Were we terribly surprised by that? No. What we had seen in the online learning labs were again the kids, the students plugging their headphones into other things, not really listening to lectures, googling for answers. We think over time they got pretty effective, there were actual sites they would go to where the question banks and answers had been posted. Not surprising if it’s used in 50 states. These are tech savvy kids in many ways. So if they’re just using a quick way to complete assessments, they’re probably not going to learn and it’s not surprising that they wouldn’t perform well on standardized tests which are attempting to assess whether or not they have learned particular types of content.

Chancellor: So, they wanted to look beyond high school graduation to see if there would be effects down the road for these students. The JPB Foundation provided resources for them to expand their research to look at students’ earnings after high school.

Heinrich: We had a couple theories about this. One would be that, you know, if there isn’t a lot learned in high school generally whether you’re taking a traditional course or online, if kids are tuning out or just trying to get through, then maybe it doesn’t matter, all the employers will see, or the post-secondary institutions, it doesn’t say on the kids’ transcripts ‘this course was recovered through an online credit recovery course’ they just see the completion of those credits. We did see increases in high school completion and estimated increases in their post-secondary education opportunities, however, in the post-secondary education opportunities, we also saw that they were going to lower quality institutions. More open access, open admissions colleges, as well as in general looking at various markers of selectivity or four year versus two year. They were getting access to postsecondary education but less quality than their peers who failed a course and did not take the course online or students who just weren’t taking the online courses. But again, what will happen later in the labor market, right, whether they finish their college and go on. So we were able, with resources from the JPB foundation, to link student earnings records to their high school records and follow them, right now we’re following them up to 4 to 5 years in the labor markets and so if again it doesn’t matter, they don’t learn much in traditional settings and they don’t learn a lot online or they learn equal amounts then we wouldn’t expect to see any differences down the road in the student earnings. However, if, as possibly suggested by the lack of relationship between or the negative relationship their test score outcomes and online learning, if they really were learning less we would expect over time for that to be reflected in their earnings, right, so maybe the employer initially thinks they’re no different than any other high school graduate. But over time they’re incapable of performing some tasks, or maybe some of the things I was saying before were absent when you’re taking your course in front of a computer, you’re not talking through problems or issues, you’re not in a lab setting working on a lab with your peers, studying things, hands on, measuring things, or you don’t have group projects, you don’t have much in the way of writing assignments that get graded and you get feedback in that way.

Chancellor: Their analysis showed that there may be something to this second storyline, that these learning gaps do start to play out over time when it comes to wages. And they found that these gaps got wider when they measured them two, three, four years out from graduation.

Heinrich: And they’re particularly big for the students who ever enrolled in college, so we were just looking at students, focusing only on that subsample who ever failed a course in high school. Well half of them repeated online, half of them repeated in traditional settings. And then if we look at those groups and say well who enrolled in college and who didn’t, we basically see that if they ever enrolled in college and they ever failed a course, those who completed the course online versus not online, down the line, those who did are thousands of dollars behind per year versus those who didn’t. So those gaps open up. That’s what we kind of expected if there was an issue of whether or not they were learning. If we look just at the kids who didn’t ever enroll in college, there is a gap that still opens up, it’s just not as big. They didn’t have as much earnings potential in general, those kids.

Chancellor: Heinrich says their findings should serve as kind of a caution because, even outside of the pandemic, districts are increasingly using these tools. And it raises questions about how these online instructional tools can be used in a way that still ensures students are learning the content.

Heinrich: Some of the ways that we have worked with the school districts that we’re working with have really been attending to this. So for example, like I was mentioning, trying to think of ways to encourage teachers, prepare teachers for blended learning, not just putting the kids in an online course and letting them work their way through it, but trying to find ways that they could engage with the students and help them learn the content. So that is more work on the part of the teachers and it means that they have to do their own kind of work in constructing supplemental instruction for students online. And right now, I think in most school districts they’re not prepared to build the capacity to do that. We’re, in our recommendations that are coming from the project, we’re encouraging school districts on the front end to think about these capacities that the students will need to really learn with these tools. And what that might mean for their contracts with the vendors. So maybe rather than just buying the equipment or the contract you have with the vendor to allow the access to students, think about maybe ways you could require the vendor to provide supports to teachers, not just at the onset to learn how to operate, ensure kids can get access to the courses, the technical aspects of it, but also to provide support so that instructors learn the ways of personalizing or blending live instruction with online instruction which is the way the vendors say they intend for this to be used but they’re selling to districts that are very resource constrained. It should be a lesson of caution really to even those for example that sometimes at state levels that say, oh, we want to roll out a 1 to 1 digital device initiative. The vendors, the makers of the technology can sell these really cheap today compared to years in the past. But that’s a small part of the cost, I think. I think many fail to understand. Oh great, every kid will have their own laptop or their own device for learning but if they really want them to learn in the ways we intend with those devices, there’s a whole lot more resources and capacity that need to come along with the rollout of those initiatives.

Chancellor: Thinking about the implications of this study, I asked Professor Heinrich how much these lessons apply when we think about the use of online education tools across the country.

Heinrich: I think one thing I would like to just mention is that we have studied this very intensively in a particular school district with a particular vendor. Like I said, this vendor is in all 50 states and 8 of the biggest one, and I feel like from our conversations and attendance for example at conferences where these vendors go, we kind of see who’s in the mix, I feel like it’s fairly representative of what’s out there in these types of programs that are used. At the same time, it is one district that we’re studying. We have heard in presenting our findings that they resonate fairly well with what has been seen in other districts, how they’re using them. I also feel confident about that. But, one of the things that was kind of unique about this project was that the vendor agreed to allow us to take their data which was literally tens of millions of observations because we know in the case, when we’re looking at the data, we can know for a particular student exactly when they logged in how much time they’re productive, what they completed. So that’s kind of a unique aspect of the project. But we actually also feel that it should encourage other school districts to think about that potential for, in the contracting process, for asking vendors to share those data. In the absence of that, what happens is that school districts just get a report from the vendors. When we link it to student records in schools, we can get at questions of which types of students have better opportunities to learn with these tools and which don’t?

Chancellor: Heinrich and her team had support from the JPB Foundation, WT Grant, and Jaime Davila in Texas for doing this work, and she says that their support allowed them to do a lot of work linking data and doing a lot of the evaluation work to get a deep understanding of how these processes were working. And while they were working in one school district with one vendor, she says that they think it’s representative of what’s going on, and they wanted to think about how they could offer this work in a way that’s useful for other districts.

Heinrich: We have this observation instrument that we make freely available to the school districts. We also developed a shorter, what we call walk-through tool so if staff don’t have time to sit through an entire period and complete an observation, what can they learn from walking into the classroom, what should they be looking for to kind of get an assessment of how the tools are being used? So, we’ve tried to recognize that not all school districts are going to have a multiyear partnership with a university bringing in lots of funds to support the learning process. What can other districts take from our work and use to have a more critical or evaluative lens on thinking about how is this working for our students? Are we disadvantaging the exact students we’re trying to help in the way we roll this out? And I think that’s an important equity question all school districts should be asking themselves as they think about who they are targeting for the use of these tools, who they’re grouping together in classrooms. If you put all of the students together in one room who are failing classes, how does it affect how they learn from their peers as well. A lot of important questions, probably giving districts as much in tools and guidance and suggestions, important questions they need to consider as they take on these initiatives.

Chancellor: Thanks to Carolyn Heinrich for taking the time to share this research with us. If you would like to learn more, you can check out her book with Jennifer Darling-Aduana and Annalee Good called Equity and Quality in Digital Learning, that was published by Harvard University Press in September of 2020. Or, take a look at Heinrich, Darling-Aduana, Good, and Huiping Cheng’s December 2019 article in the American Education Research Journal that recently won the Palmer O. Johnson Memorial Award for best article.

This podcast was supported as part of a grant from the U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation but its contents don’t necessarily represent the opinions or policies of that Office, any other agency of the Federal government, or the Institute for Research on Poverty. To catch new episodes of the Poverty Research and Policy Podcast, you can subscribe on Apple, Stitcher, or Google Play Podcasts. You can also find all of our past episodes on the Institute for Research on Poverty website. Our theme music for this episode is “Staring Straight” by Maarten De Boer. Thanks for listening.