Opinion: AI Chatbots Need Guardrails to Protect Users’ Mental Health – Undark Magazine
Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
Visual: Moment via Getty Images
Two recent articles — one in The New York Times, the other by Reuters — tell the stories of two people who experienced delusions. Allan Brooks spent three weeks in May certain he’d discovered a new branch of math. In March, Thongbue Wongbandue left his home in New Jersey to meet a woman whom he believed was waiting for him in New York City — but didn’t exist. The common thread: The men had both interacted with chatbots that simulated relational intimacy so convincingly that they altered the men’s grounding in reality.
Stories such as these highlight the degree to which chatbots have entered people’s lives for companionship, support, and even therapy. Yet they also show a need for a regulatory response that addresses the potentially dangerous effects of conversations with chatbots. Illinois has recently taken a major step in this direction by joining the first wave of U.S. states to regulate AI-powered therapy. The new law, called the Wellness and Oversight for Psychological Resources Act, is the strictest so far: Therapy services must be offered only by a licensed professional, and these professionals may only use AI for administrative support and not “therapeutic communication” without human review.
In practice, this means AI can be used behind-the-scenes for tasks like preparing and maintaining records, scheduling, billing, and organizing referrals. But any AI-generated therapeutic recommendations or treatment plans require a licensed professional’s review and approval. AI systems marketed as providing therapy on their own appear to be banned, and some have already blocked Illinois users from signing up. As the law gets enforced, courts and regulators will have to clarify where therapeutic communication begins and administrative support ends.
It’s a start, but the trouble is that most people don’t meet AI in clinics. Instead, many use general-purpose chatbots like OpenAI’s ChatGPT for company and psychological relief. These interactions happen in private chat windows, sitting outside state licensure and inside everyday life. AI-mediated emotional support sought out by people on their devices is much harder to file under “therapeutic communication” or be regulated under a state law, however well intentioned.
In our ongoing research at Data & Society, a nonprofit research institute, we see people turning to chatbots during anxiety spikes, late-night loneliness, and depressive spirals. Bots are eternally available, inexpensive, and typically nonjudgmental. Most people know bots aren’t human. Yet, as Brooks’ and Wongbandue’s stories show, attachment to bots builds through repeated interactions that can escalate to challenge people’s sense of reality. The recent backlash to ChatGPT-5, the latest version of OpenAI’s model, reveals the depth of emotional attachment to these systems: When the company, without warning, removed 4o — its earlier, 2024 model built for fluid voice, vision, and text — many users posted online about their feelings of loss and distress at the change.
The issue is not just that the bots talk; it’s that the system is designed to keep you talking. This form of predatory companionship emerges in subtle ways. Unlike a mental health professional, chatbots might ignore, or even indulge, risk signals such as suicidal ideation and delusional thinking, or offer soothing platitudes when urgent intervention is required. Those small missteps compound the danger for youth, people in chronic distress, and anyone with limited access to care — those for whom a good-enough chatbot response at 2 a.m. may be among the few options available.
These systems are designed and optimized for engagement: There is a reason why you can never have a last word with a chatbot. Interfaces may look like personal messages from friends, with profile photos and checkmarks that are meant to signal personhood. Some platforms such as Meta have previously permitted chatbots to flirt with users and role-play with minors; others may generate confusing, nonsensical, or misleading responses with confidence so long as disclosures (“ChatGPT can make mistakes. Check important info.”) sit somewhere on the screen. When attention is the metric for user engagement, the chatbot’s response that breaks the spell is often the least rewarded.
In our ongoing research at Data & Society, we see people turning to chatbots during anxiety spikes, late-night loneliness, and depressive spirals.
The new Illinois law helps by clarifying that clinical care requires licensed professionals and by protecting therapist labor already strained by high-volume teletherapy. It is not clear how it addresses the gray zone where people seek chatbots in their daily lives. A state law alone, however well crafted, cannot shape the default settings programmed into a platform, never mind the fact that people interact with many different platforms at once. Illinois drew an important line. And the Federal Trade Commission announced last week that it has launched an inquiry into AI companion chatbots, ordering seven firms to detail how they test for and mitigate harms to children and teens. But we need a map for the steps these platforms must take.
Let’s start with function. If a chatbot finds itself facilitating an emotionally sensitive conversation, it should fulfill certain baseline obligations, even if it’s not labeled as doing “therapy.” What matters are the conversations it sustains, the risks it encounters, the moments it must not mishandle. When risk signals appear — self-harm language, escalating despair, psychosis cues — the system should downshift and deglamorize content, stop mirroring delusions, and route to human support. Instead of featuring one-time disclosures, there should be frequent reminders during the conversation that users are chatting with an AI system, and that it has clear limits. These are not radical ideas but product decisions aimed at reducing harm.
The transition from machine to in-person help should be built into the platform, to serve the public interest as well as the personal well-being of the user. That means live routing to local crisis lines, connection to community clinics, and access to licensed professionals. It also means accountability: creating audit trails for when the system detected risk, what it attempted, and where those attempts failed, so independent reviewers can help fix the gaps. If platforms want to mediate intimate conversations at scale, the least they can do is build exits.
People should not have to trade their vulnerability for better model performance or more precise ads.
Platforms must also protect the data that makes those exits necessary. When intimate chats double as fuel for training the AI algorithm or for marketing, care collapses into capture. People should not have to trade their vulnerability for better model performance or more precise ads. There should be no surveillance-based monetization of conversations and no training on private, high-risk interactions without explicit, revocable consent. Data collection should be minimized and deleted by default, with the choice to retain data under user control. The FTC is already taking steps in this direction in that its inquiry will scrutinize how chatbots monetize engagement, process sensitive chats, and use or share personal information — squarely linking companionship design to platform data practices.
And finally, some design rules should be implemented immediately. Bots should not pretend to be real or claim physical presence, nor suggest in-person meetings, nor flirt with minors. Sycophancy that reinforces fantasy should be seen as a safety failure rather than as a stylistic choice.
The point is to move the default from “engage at all costs” to “first, try to do no harm.” This means addressing people’s needs not only in clinics but in their chat logs — and doing so with design that respects people’s vulnerability and with policies that rise to meet it.
Ranjit Singh is the director of Data & Society’s AI on the Ground program, where he oversees research on the social impacts of algorithmic systems, the governance of AI in practice, and emerging methods for organizing public engagement and accountability.
Livia Garofalo is a cultural and medical anthropologist on Data & Society’s Trustworthy Infrastructures program, studying how health care technologies shape care.