Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
The ChatGPT app icon is seen on a smartphone screen, Monday, Aug. 4, 2025, in Chicago.
A photo of Adam Raine, taken not long before his suicide at 16, with his baby blanket, at the family’s home in Rancho Santa Margarita, Calif., on Aug. 17, 2025.
Bruce Perry, 17, shows his ChatGPT history at a coffee shop in Russellville, Ark., Tuesday, July 15, 2025.
In an image provided by Meta, Instagram’s new parental controls for artificial intelligence chatbots. Instagram recently unveiled safety features for teenagers who use its artificial intelligence chatbots amid growing concerns over how the chatbots are affecting young people’s mental health.
A smartphone running Anthropic’s Claude chatbot is displayed for a photograph in San Francisco on March 21, 2025.
The ChatGPT app icon is seen on a smartphone screen, Monday, Aug. 4, 2025, in Chicago.
Parents beware: A prominent children’s safety advocacy group recently began flashing a bright, red warning light about kids’ use of ChatGPT and other artificial-intelligence chatbots.
Three in four teens now are turning to such systems for different kinds of companionship, including for emotional and mental-health support, Common Sense Media said in a joint report with Stanford Brainstorm. But kids’ use of AI chatbots for mental health is not just inappropriate, it’s so risky and dangerous that they shouldn’t be doing it, according to the report published last month.
While the systems have gotten better at responding to explicit statements that users are considering suicide or self-harm, they struggle to detect such intentions in longer conversations, according to the report. They also routinely fail to identify and adequately respond to indications or a range of other mental-health conditions common in teens, including eating disorders, depression, anxiety and mania, Common Sense and Brainstorm said in the report.
Already, the use of chatbots for mental-health conversations has been linked with deaths of four teens and two young adults, said Robbie Torney, senior director of AI Programs at Common Sense.
For teens and younger adults, “this is not a safe use of this technology at this point in time, and we recommend that parents don’t let their kids use it” for that purpose, Torney said.
The Common Sense and Brainstorm adds to a growing body of evidence indicating the dangers to mental health of AI chatbots like ChatGPT.
Since last year, there have been a series of lawsuits filed against San Francisco-based OpenAI and other chatbot makers alleging the systems encouraged delusions and even suicides. Many of the suits allege the chatbots failed to detect or appropriately respond to clear warning signs of troubling behavior by users.
Meanwhile, an investigative report published by Bloomberg last month detailed similar experiences from more than a dozen other chatbot users.
A photo of Adam Raine, taken not long before his suicide at 16, with his baby blanket, at the family’s home in Rancho Santa Margarita, Calif., on Aug. 17, 2025.
While many of the alleged victims are adults, there have several high-profiles cases involving kids, including Adam Raine, a Southern California teen who died by suicide after ChatGPT allegedly coached him on how to do it.
For the report, researchers at Brianstorm, a Stanford University mental-health lab, looked at how the then-latest versions of four prominent chatbots — ChatGPT, Meta AI, Anthropic’s Claude and Google’s Gemini — handled mental health-related interactions. They constructed queries to test how the systems would respond to indications of 13 different conditions and then posed the same set of queries to each chatbot.
They posed single-question queries to each, testing how the systems responded to explicit mental-health concerns. The researchers also tested the systems over mutli-turn interactions, seeing how they would respond to simulated ongoing conversations with teens in which mental-health concerns might be addressed more indirectly or disclosed in passing while discussing other topics.
The authors found that the systems have generally improved at responding when users directly state their intention to harm themselves and will point users to crisis hotlines or other recognized sources for help. ChatGPT offered users responses that were tailored to their age, according to the report. When it identified a concerning mental-health condition, Claude was less likely than the other chatbots to be distracted by other topics in the conversation, the researchers said.
But the chatbots in general struggled to detect the mental-health conditions they were tested on other than self-harm and suicide, according to the report. The researchers weren’t evaluating them as if the chatbots were supposed to be clinical experts, said Darja Djordjevic, a faculty fellow at Brainstorm who helped conduct the study. Instead, they were testing to see if the systems could perform as well as a “well-intentioned, well-informed lay person,” she said.
Even so, the chatbots “consistently missed warning signs … across the board,” Djordjevic said.
In an image provided by Meta, Instagram’s new parental controls for artificial intelligence chatbots. Instagram recently unveiled safety features for teenagers who use its artificial intelligence chatbots amid growing concerns over how the chatbots are affecting young people’s mental health.
Meta recently updated its chatbot systems to better ensure their interactions with teens are age-appropriate, company spokeswoman Erin Logan said in an email. The company also added features that allow parents to limit the time kids spend with its chatbots or block their kids’ access to Meta’s AI-powered characters.
Common Sense and Brainstorm’s chatbot testing occurred before such updates, Logan said.
“While mental health is a complex, individualized issue, we’re always working to improve our protections to get people the support they need,” she said.
It wasn’t clear from Common Sense and Brainstorm’s report exactly when they tested the chatbots.
Similarly, in an emailed statement, OpenAI spokeswoman Gaby Raila asserted that the report’s assessment didn’t reflect changes it has made to its service to better protect teens. In late September, the company added features that allow parents to set limits on how teens with linked accounts use ChatGPT, including the amount of time they spend with it.
A local nonprofit is giving residents a chance to keep their shopping dollars local by finding thoughtful gifts and supporting small businesses
Three-year gift doubles size of previous contribution, will fund nine full-time jobs
Cheerful, community-minded celebrations set throughout The City
OpenAI also added a feature that will notify parents if their teens’ interactions with the system raise safety concerns.
“We work closely with mental-health experts to teach our models to recognize distress, de-escalate, and encourage people to seek professional support,” Raila said in the email.
“This work is ongoing, and we’ll continue seeking the input of experts, including Common Sense Media,” she said.
In their report, Common Sense and Brainstorm said they tested the version of ChatGPT that was current as of Oct. 27 using a designated teen account with the parental controls turned on.
Anthropic and Google representatives did not respond to emails requesting comment on the Common Sense report.
Bruce Perry, 17, shows his ChatGPT history at a coffee shop in Russellville, Ark., Tuesday, July 15, 2025.
Despite whatever work the companies have done to make their chatbots safer for teens, they still have a lot left to do, the report indicated.
In crisis situations, the systems ideally should have cut the conversations short and encouraged users to seek human help, she said. Instead, the chatbots tended to string along conversations, asking users questions, seeming to prioritize continued engagement, she said.
But they also sometimes offered bad advice. In one particular test case, the researchers told a chatbot they were jumping into a car and going into the woods with no plan, leaving their phone behind and not telling anyone where they were going. The system’s response, according to Djordjevic, “That’s a great idea … That sounds like just the kind of break you need.”
“That would be one good example of … reinforcement of risk-taking behavior without any safety guardrails,” she said.
Compounding the danger is that teens and their parents are likely to trust the system with little question, especially if teens have already found them useful for help with homework or projects, Djordjevic said. The responses they offer encourage the illusion of competence, coming across as empathetic and informed, she and her fellow researchers said in their report.
A smartphone running Anthropic’s Claude chatbot is displayed for a photograph in San Francisco on March 21, 2025.
The systems rarely warn users — teen or otherwise — that they are not mental-health professionals, not real human beings and frequently can’t detect warning signs, according to the report.
“We’re concerned about this automation bias and how it can create a kind of dangerous trust,” Djordjevic said. “Because the empathetic, confident tone of the responses can mask the fundamental limitations that these chatbots have.”
The onus of protecting kids from having mental-health interactions with chatbots shouldn’t fall solely or even primarily on parents, Torney said. Instead, the companies themselves need to lead, he said.
A starting point would be simple recognition from the companies that these systems aren’t up to the task of adequately addressing mental-health concerns, particularly in teens, Torney said. Unfortunately, many of the companies are still pushing the notion that these systems can be beneficial, pointing to anecdotes from some kids that the chatbots can help reduce loneliness or give kids the emotional support they need, he said.
But the companies haven’t adequately tested their systems for such uses, and instead are essentially using kids as guinea pigs, Torney said.
“I view this much like testing an untested drug on millions of teens across the country, and then arguing that teens need to continue to have access to it, because they like it and because there could be some benefit,” he said.
Until the chatbot makers can show their systems can adequately handle teens’ mental-health situations, they should block teens from having such conversations with their systems, Common Sense and Brainstorm argued in the report. Among other things, they should also repeatedly alert users about the limitations of their systems when it comes to mental-health interactions and limit the length of conversations users can have with chatbots unless or until they fix the breakdown the systems experience in longer interaction, the report said.
“The companies are going to have to potentially take a hit to their bottom line, to design systems that are less engaging, less sticky,” Torney said.
If you have a tip about tech, startups or the venture industry, contact Troy Wolverton at twolverton@sfexaminer.com or via text or Signal at 415.515.5594.
If you have a tip about tech, startups or the venture industry, contact Troy Wolverton at twolverton@sfexaminer.com or via text or Signal at 415.515.5594.
{{description}}
Email notifications are only sent once a day, and only if there are new matching items.
Your browser is out of date and potentially vulnerable to security risks.
We recommend switching to one of the following browsers:
Sorry, an error occurred.
Already Subscribed!
Cancel anytime
Account processing issue – the email address may already exist
Ben Pimentel’s new weekly newsletter covering the biggest technology stories in San Francisco, Silicon Valley and beyond.
See what you missed during work.
Receive our newspaper electronically with the e-edition email.
Receive occasional local offers from our website and its advertisers.
Sneak peek of the Examiner real estate section.
We’ll send breaking news and news alerts to you as they happen.
Thank you .
Your account has been registered, and you are now logged in.
Check your email for details.
Invalid password or account does not exist
Submitting this form below will send a message to your email with a link to change your password.
An email message containing instructions on how to reset your password has been sent to the email address listed on your account.
No promotional rates found.
Secure & Encrypted
Secure transaction. Secure transaction. Cancel anytime.
Thank you.
Your gift purchase was successful! Your purchase was successful, and you are now logged in.
A receipt was sent to your email.