Meet your new tutor

One of our macroeconomics courses enrolled 140 undergraduate students, mostly in their first or second year of college, divided across four sections.

Students’ course materials, assignments and exams were identical across all four sections. Students were generally not allowed to use AI tools or collaborate with classmates during exams. Students took all tests in person and were not allowed to reference any notes or other materials during the exam.

As a result, exam scores reflected what students understood and could explain on their own – without the help of AI or any other outside source.

After all students took their first exam, we randomly assigned the four class sections to take on a different study format.

We prompted one group of students to work individually, without Macro Buddy; another group of students worked in groups, without Macro Buddy; a third group of students worked individually, with Macro Buddy; and a fourth group of students worked in groups, with Macro Buddy.

We wanted to compare how different study approaches – working alone, working with classmates, using Macro Buddy or combining both – altered how well students did on exams.

Macro Buddy’s skills

We trained Macro Buddy with the help of lecture transcripts, slides and homework questions specifically from this macroeconomics course.

Macro Buddy had internet access turned off, so it relied only on the instructor’s course materials.

Macro Buddy was designed to act like a tutor, not an answer machine. Instead of giving students complete solutions, Macro Buddy asked follow-up questions meant to guide students toward an answer.

For example, if a student asked why lower prices might increase consumers’ spending, Macro Buddy would not offer a quick, full explanation. It might instead ask what happens to people’s purchasing power when prices fall. The student would then have to connect the concepts and explain their reasoning, in their own words, step by step.

This distinction between explaining an idea and receiving a finished answer matters.

An AI tool that simply delivers answers can allow students to skip thinking through a problem. One study found that when college students rely on a chatbot as a crutch, they perform worse when they no longer have access to it. A tool that asks questions requires students to do the work themselves, even while receiving guidance. This is the very process that makes learning stick.

What happened to students’ learning

The one group of students that continued working individually, without AI, served as our control group.

The other three groups changed how they studied: One began working in groups without AI, one worked individually with Macro Buddy, and the last group combined group work with Macro Buddy.

All of the students’ average scores declined when they took their second exam, across all four study groups.

By the third exam, however, differences across sections became clearer.

Students who used both Macro Buddy and group discussion earned the highest average scores. Students who used Macro Buddy alone also scored higher than those who worked alone without Macro Buddy. Students who worked in groups without Macro Buddy showed smaller improvements, when compared to the students in other groups.

The third exam happened several weeks after we introduced the new study formats.

By that point, students in the combined group may have grown more comfortable using Macro Buddy to test their understanding, while also explaining ideas to classmates. Working with peers meant having to articulate reasoning clearly and respond to questions, which can deepen understanding over time.

Why this matters

Some critics of AI worry that students will rely on AI to do the hardest parts of learning for them. This reflects a fear that students may stop practicing the skills that build expertise. Students become experts in their fields while struggling with confusing material, revising explanations and seeing whether they truly understand an idea.

Our experiment suggests erosion of learning when using AI is not inevitable.

We found that when AI is designed as a tutor that asks questions instead of simply giving answers – and when students are also required to explain their reasoning to classmates – the technology can support learning rather than replace it.

Most students today use general-purpose chatbots that are not designed as tutors. They type in a question and receive a response. But our findings suggest that even small design choices, such as building an AI chatbot with guiding questions, can shape how students engage with the material.

Peer discussion also adds something to the learning process that AI cannot provide: social accountability and exposure to alternative reasoning.

Together, these practices encourage students to think through problems more actively.

The evidence from our experiment highlights a practical distinction: AI can be used to replace thinking, or it can be used to support it. The impact may depend less on the technology itself and more on how it is structured and integrated into learning.

Saharnaz Babaei-Balderlou, Teaching Assistant Professor of Economics, University of Wisconsin-La Crosse and Shishir Shakya, Assistant Professor of Economics, Appalachian State University

This article is republished from The Conversation under a Creative Commons license. Read the original article.