Stanford Study on AI Writing Bias

Stanford University published a study a few weeks ago, and I have not been able to stop thinking about it since I read it.

Researchers Mei Tan and Lena Phalen took 600 identical 8th grade persuasive essays and fed them into four AI models, including GPT-4o and Meta’s Llama. Then they resubmitted each essay with different descriptions of the writer: their race, gender, motivation level, language background, or disability status. The feedback shifted. Consistently and in patterned ways.

Essays attributed to students of color were more likely to be told to polish their writing. Feedback for Latino students assumed limited English ability and oriented comments around family and culture. Asian students received critiques framed around academic responsibility and respect. Female writers received more emotional language, words like “love” and “wonderful,” while male writers received more direct critique and pushback on their arguments. For students described as unmotivated, the AI offered more praise but focused on basic corrections like spelling. For motivated students, it pushed them to strengthen their arguments.

The essays were identical. The only thing that changed was the description of who wrote them.

I want to write about what this is, because I think the framing of “algorithmic bias” sometimes lets us off the hook too easily. These models are trained on human-generated data. They are not inventing new biases. They are encoding the ones we already have, scaling them across thousands of interactions, and delivering them in a format that can feel more authoritative and consistent than human judgment. Mei Tan put it plainly: “They are picking up on the biases that humans exhibit.”

The implications for classrooms are serious, and they are not evenly distributed. Writing is, as Larry Berger of Amplify put it, “a moment of vulnerability.” When a student puts their ideas on a page and asks whether they communicated what they meant, whether they are good at this, whether their voice matters, the feedback they receive shapes not just their draft but their relationship to writing itself. Biased feedback at that moment does something specific and lasting.

AI writing coaches are being adopted at scale, in part because they seem like they eliminate the inconsistency and subjectivity of human assessment. They do not. They replicate it in a form that is harder to see and therefore harder to challenge.

This is why I keep returning to the same argument that AI literacy cannot just mean knowing how to use these tools. It has to also mean knowing how to interrogate them. Knowing what assumptions are baked in. Knowing whose writing style, whose way of structuring an argument, whose cultural frame of reference gets treated as the standard, and whose gets corrected.

The educators, families, and students who need this research most are the ones likely to have access to it. That is a governance problem, not a technology problem.What are you seeing in your own classrooms or institutions? Are AI writing tools being adopted with this kind of scrutiny, or are they being treated as neutral?