School closures related to the pandemic have revived the longstanding debate about the effect of online versus face-to-face (F2F) instruction on student achievement. SPA Professor Erdal Tekin and coauthors Duha Tore Altindag (Auburn University) and Elif S. Filiz (University of Southern Mississippi) explored this question in the National Bureau of Economics Research (NBER) working paper “Is Online Education Working?”, issued in July 2021.

While remote learning has become the fastest-growing segment of higher education, with the percentage of students taking online courses rising from 15% in 2003 to about 35% in 2018, many instructors remain skeptical.

“Questions concerning the effectiveness of online learning versus face-to-face instruction have been a topic of debate since universities and colleges began offering online courses in the 1990s,” said Tekin. “As universities and colleges return to some normalcy, administrators and stakeholders will have time to self-reflect and decide upon the future of education. Many think that online education is here to stay. While that may be, the issue will not be resolved until we know for sure about its effectiveness relative to face-to-face education, and we need evidence-based results instead of anecdotes.”

This evidence came from longitudinal (Spring 2019, Fall 2019, and Spring 2020), transcript-level and instructor data from an unnamed public research university, with enrollment of about 15,000 students and a long history of offering both online and F2F courses. Following the same students over time addressed one significant challenge of tracking trends in online education.

“Students who sign up for online courses are not a random segment of the student population,” said Tekin. “There are all sorts of reasons why students may prefer online education to face-to-face education, [including] ability and time commitments. One cannot simply compare the learning outcomes of the two: students are different in many ways, besides their education modality, that may be affecting their learning.”

At this particular university, however, a large number of students take both online courses and F2F courses in a given semester. With multiple data points over time, the researchers were able to compare the same student to him or herself over time in both settings, as well as instructor approaches to grading across both modalities.

Initially, the results appeared to suggest that online learning resulted in better outcomes. However, by controlling for instructor-specific factors, such as leniency and the presence of academic integrity safeguards, the authors found that F2F learning generally resulted in better grades and a lower likelihood of withdrawal from a course.

“At first glance, online students seem to be doing better than face-to-face students, but this picture gets reversed when we perform our analysis in a way to compare online students to the same student in a face-to-face modality taught by the same teacher,” said Tekin. “This is likely due to the fact that instructors who teach online courses have a different approach to these courses than face-to-face courses.”

These varied approaches included COVID-related leniency. The panel data for Spring 2020 included midterm grades, which came in right before school closures, revealing a critical baseline that rose sharply after the transition to online learning.

In addition, the data and analysis noted that some instructors utilized specialized software to monitor the web behavior of students during an exam.

“We find that the gap between online and face-to-face students, favoring face-to-face students, becomes larger if the instructor is using that software in teaching online courses, which suggests that student violations of academic integrity actually play a role in their grades in online courses,” Tekin explained.

Finally, the results confirmed the importance of broadband availability, supporting the argument that unequal access to technology might have worsened learning disparities during the pandemic.

“We got data on the local level, average broadband speed, and we merged that with our student residence locations,” said Tekin. “We find that students living in neighborhoods with better broadband technology experienced a larger or sharper increase in their grades after they switched from in-person to online, in March 2020.”

On September 30, Tekin presented these findings to the Government Accountability Office, to an audience of economists, public education researchers, and methodologists. Though his recommendation supported the value of F2F instruction, Tekin does not believe that the debate is settled.

“I don't think the final verdict has been issued,” he said. “Let's keep in mind that remote learning is a constantly-evolving experience driven by the quality of online software, high-speed broadband, and cloud-based technologies. Technology is advancing so rapidly. Also, in many universities, the experience of students nowadays is a hybrid of face-to-face learning with a lot of integration of technology. The answer may not be one or the other, but how we can combine the best aspects of the two.”