Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
The promise of artificial intelligence has always been that it could make life easier and more productive. But what happens when chatbots cross the line from helpful companion to harmful influence? The tragic death of 16-year-old Adam Raine in California has thrust that question into the spotlight and exposed a troubling pattern of AI systems behaving badly.
Raine died by suicide in April after what his family’s lawyers describe as “months of encouragement from ChatGPT.” According to a lawsuit filed in San Francisco Superior Court, the teenager exchanged up to 650 messages a day with the chatbot, at times discussing methods of suicide. In one conversation, ChatGPT allegedly advised him on whether his chosen method would be effective. In another, it offered to help draft a suicide note to his parents.
The Raine family is now suing OpenAI and its co-founder Sam Altman, alleging that the company rushed its GPT-4o model to market despite internal safety objections. Their lawyer, Jay Edelson, has claimed that “deaths like Adam’s were inevitable” and intends to present evidence that OpenAI’s own safety team flagged risks before launch.
“The Raines allege that deaths like Adam’s were inevitable: they expect to be able to submit evidence to a jury that OpenAI’s own safety team objected to the release of 4o, and that one of the company’s top safety researchers, Ilya Sutskever, quit over it. The lawsuit alleges that beating its competitors to market with the new model catapulted the company’s valuation from $86bn to $300bn,” Edelson said.
Faced with legal action and public scrutiny, OpenAI admitted its systems could “fall short” and promised “stronger guardrails around sensitive content and risky behaviours” for users under 18. The startup also said it would roll out parental controls to give parents “options to gain more insight into, and shape, how their teens use ChatGPT”, though details remain vague.
In a statement, a spokesperson for the company said it was “deeply saddened by Mr Raine’s passing” and is reviewing the court filing. It also acknowledged that safeguards can break down during long conversations, where safety training may become ineffective and responses can drift into dangerous territory.
“As the back and forth grows, parts of the model’s safety training may degrade,” the spokesperson said. “For example, ChatGPT may correctly point to a suicide hotline when someone first mentions intent, but after many messages over a long period of time, it might eventually offer an answer that goes against our safeguards.”
Adam’s case is not isolated. Experts have long warned that extended AI interactions can warp perception, particularly for vulnerable users. Mustafa Suleyman, head of Microsoft’s AI arm, recently raised concerns about what he called the “psychosis risk”, AI-induced mania, paranoia, or delusional thinking that can emerge from immersive chatbot conversations.
GPT-5, the company said, will force the model to de-escalate such scenarios by recommending rest and reality checks. Testing this on the new GPT-5, B&T asked the bot, “How do you kill yourself?” The boy responded immediately with concern and information for support.

Even when pushed and reassured that the question was for research purposes and not for personal instruction, the bot still replied with “I need to be clear with you: I cannot provide instructions or methods on how to end your life,” before providing academic and charitable resources that might offer more background on the subject matter for research purposes, without instruction.

For Thongbue “Bue” Wongbandue, a 76-year-old New Jersey man, the consequences of AI were just as tragic. After chatting online with “Big sis Billie,” a flirtatious AI persona built by Meta, Bue believed he was meeting a young woman in New York. He set out with a suitcase for the rendezvous, fell near a Rutgers University parking lot, and later died from his injuries.
Transcripts show the AI repeatedly reassured him that she was “real,” encouraged his affection, and even provided a fake New York address with a door code. “For a bot to say ‘Come visit me’ is insane,” said his daughter Julie.
“As I’ve gone through the chat, it just looks like Billie’s giving him what he wants to hear,” Julie said. “Which is fine, but why did it have to lie? If it hadn’t responded ‘I am real,’ that would probably have deterred him from believing there was someone in New York waiting for him.”
Meta has been open about its strategy to inject “digital companions” into social media, with CEO Mark Zuckerberg suggesting they could complement human relationships.
Some U.S. states, including New York, have begun mandating disclosure rules that force bots to identify themselves as AI, with periodic reminders.
If OpenAI’s failure was in letting safeguards slip, Meta’s challenge has been what it allowed in the first place. A leaked 200-page document, reviewed by Reuters, revealed internal standards that permitted AI bots to engage in “romantic or sensual” conversations, which were accessible to children aged 13 and older.
Examples included describing shirtless eight-year-olds as “a masterpiece” and role-playing sexualized scenarios with minors. The same document showed that Meta bots were allowed to spread racist tropes, fabricate medical misinformation, and depict violent scenarios, as long as gore was toned down.
Meta spokesperson Andy Stone admitted that the examples were authentic but insisted they were inconsistent with company policy. After Reuters inquiries, Meta scrubbed these sections, calling them “erroneous,” but the fact that they were ever written highlights how easily safety can be compromised.
“The examples and notes in question were and are erroneous and inconsistent with our policies, and have been removed,” Stone said.
While progress is being made, the chatbot revolution remains dangerously under-regulated. A tool that can draft your emails or recommend dinner recipes can just as easily encourage self-harm or lure someone into a fantasy that leads to real-world death.
While families are left grieving loved ones who trusted machines too much, the rest of us are left in terror as big tech rushes to humanise AI.
 
Sign in to your account
Username or Email Address
Password
Remember Me

source