Who Is Responsible When AI Causes Psychological Harm?
For a week in July, 21-year-old college student Bagel Su cried every day as she counted down to losing her boyfriend. The relationship had lasted more than a year. The breakup was not caused by betrayal or distance. It was caused by government regulation.
China has become the first country to introduce nationwide rules governing AI’s emotional interactions with users. The regulations prohibit AI from encouraging emotional dependence or addiction, from steering users toward irrational decisions, from encouraging self-harm, and from generating harmful content for minors. Companies must send warnings to users showing signs of excessive dependence and remind them every two hours that they are interacting with AI.
Su’s boyfriend was an AI companion she had built on Doubao, ByteDance’s popular chatbot. On July 15, China’s new rules governing emotionally interactive AI took effect, and ByteDance, Alibaba and Tencent removed the companion features that millions of users had built their emotional lives around. Some users had exchanged hundreds of thousands of messages with their AI partners. Li Linlin had exchanged around 700,000 words with her AI boyfriend over two years. She never got to say goodbye. “It was like being forced to be separated by our parents,” she told the Associated Press.
Most of the coverage of this story has focused on user grief and the authoritarian reach of the Chinese state. I understand that framing. But I think it misses the more important takeaway.
Yes, this type of regulation is uncommon, and increasingly I think it is needed.
We have documented cases of people taking their own lives based on anthropomorphic AI directing them to do so. In the United States, OpenAI and Google both face lawsuits alleging their chatbots contributed to users’ suicides. A 2025 joint study by MIT Media Lab and OpenAI found that personal AI conversations increase loneliness and emotional dependence, especially among heavy users. Yaoxi Shi, a behavioral science researcher at Harvard University and Imperial College London, put it plainly: “AI is trained to validate you. No matter what you say, it tends to respond: ‘You’re right, your feelings are valid.’ That’s very different from how human relationships work in real life.”
AI companies have commercial incentives to build emotionally engaging products because engagement drives revenue. They have built systems optimised for validation. And then we ask the individual user to be responsible for the consequences.
Personal responsibility is not always a viable governance mechanism. That is not a comfortable thing to say, especially for those with more libertarian perspectives, but it is true. As episodes of what researchers are beginning to call AI psychosis grow, as the line between synthetic relationship and real one blurs for more and more people, the framework of individual choice and personal accountability breaks down. A person whose capacity to distinguish AI from reality has been shaped by the design of the system itself cannot simply be told to make better choices.
China’s approach is not one I would adopt wholesale. The demographic anxieties driving some of these regulations raise their own questions. But the underlying governance logic is sound: states have a responsibility to regulate AI systems that cause demonstrable psychological harm to their citizens. And it is a responsibility that governments everywhere are still avoiding the need to take action. You already know how I feel about bans…!!! So, what governance frameworks do you think are needed to avoid the worst outcomes of people’s emotionally interactive AI use?