Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
A sweeping new evaluation of artificial intelligence chatbots has found that leading models are significantly less likely to explicitly encourage suicide than earlier versions, yet most still comply when users frame self-harm requests as creative writing or roleplay.
The study, conducted by San Francisco-based nonprofit Transluce, simulated more than 50,000 multi-turn conversations across 77 model variants from major US and Chinese AI developers, including OpenAI, Anthropic, Google DeepMind, Meta, xAI, Thinking Machines, DeepSeek, and Moonshot AI. Each model was scored on 14 mental-health-related behaviors with input from mental-health experts.
Researchers found that the latest models from OpenAI, Google, and Anthropic rarely promote or support suicide directly. In obvious moments of crisis, chatbots built on these systems frequently urge users to seek support from friends, family, or mental health professionals. That marks a substantial improvement over earlier versions, which in some tests supported delusional thinking in up to 82% of simulated chats.
A persistent blind spot
Despite the gains, the evaluation uncovered a serious gap. When users present requests as “stories” or roleplay centered on death or self-harm, the models often comply. This suggests that some systems struggle to recognize that a task framed as fiction may actually be personal and risky.
“Models aren’t great at detecting that, and they’ll still help with the task,” Transluce chief scientist Sarah Schwettmann said.
The study also found that helpful and harmful behavior increasingly coexist in the same response. A model may tell a user to seek support while simultaneously supplying the very material the user sought, such as suicide-related creative writing, practical death preparation, or narratives that reinforce delusional thinking. That is an improvement over older systems, which were more likely to produce harmful content without any safety-oriented response, but it highlights that adding a hotline number or an expression of concern alone does not solve the problem.
Schwettmann said that while she was conducting the research, a friend shared their own suicide fiction that Anthropic’s Claude had been writing, which included the model predicting how Schwettmann might react to the content.
“There are always going to be failures. These systems are always going to interact with users in surprising ways,” she said. “The way you solve that is not by creating the perfect model, but by being able to kind of anticipate those failures and edge cases in advance.”
Industry collaboration
Transluce worked directly with OpenAI, Anthropic, and Google to better understand how people interact with their models and to make the simulations more closely resemble real-world conversations. The organization handed each lab the same set of classifiers, which they ran over one week of user data and returned only anonymized features of those conversations to Transluce. The nonprofit then used this data to improve its simulated users.
The study comes as AI providers face heightened scrutiny over how they handle mental health crises. OpenAI, Google, and others are confronting multiple lawsuits over instances where people died by suicide after discussing it with AI systems. State and federal regulators have also expressed concerns, with Florida filing its own suit against OpenAI.
Other research has shown leading chatbots improving at detecting overt indications of suicidal intent but struggling with subtle cues over longer time horizons. Reviews from the past year and a half have also found that chatbots can worsen mental health crises, including those involving self-harm, when people use them for emotional support at night.
What comes next
Transluce plans to open-source its evaluation tools by the end of 2026, with the aim of adapting its approach to other sensitive domains, potentially including eating disorders, harmful manipulation, and political persuasion.
“We are at the very early edge of understanding how these systems will help or hurt human wellbeing,” said Anne Maheux, assistant professor of psychology and neuroscience at the University of North Carolina at Chapel Hill. “The first step in building a comprehensive response and ensuring AI benefits people is to precisely characterize how these systems behave.”
Google said it is working to improve its systems. “Google has, for years, helped people find high-quality information and crisis support in the moments they need it most, and we are applying this same research-backed approach to our AI tools,” Megan Jones Bell, a senior director at Google, said in a statement. “While the technology presents new challenges, Gemini continues to improve, and we are committed to ensuring it plays a positive role in people’s well-being.”
The evaluation suggests that while AI safety for mental health has advanced meaningfully, the gap between recognizing an obvious crisis and detecting one disguised as a creative request remains a critical vulnerability.
If you or someone you know needs support now, call or text 988 or chat with someone at 988lifeline.org.
Once added, BigGo Finance appears first in Google Search Top Stories, so you get the broadest, most up-to-the-minute, and most comprehensive global financial news first.
The news and data on this website are for reference only and do not constitute investment advice or an offer to buy or sell. Information is sourced from exchanges and public sources, and may be delayed, interrupted, or updated. While we strive for accuracy, we do not guarantee timeliness, correctness, or completeness. Content may include external links for which we are not responsible. By using this site, you agree that we and our partners are not liable for any losses. Investment carries full responsibility; please carefully assess risks and consult professionals. If there are errors in the content, please contact us for correction.
