Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
Researchers argue that AI health advice can harm people and leave no record for doctors or regulators, and they lay out what should change.
After reading about the harms of table salt, a 60-year-old man asked ChatGPT how to cut chloride out of his diet.
He subsequently replaced his salt with sodium bromide and ended up in the hospital with paranoia and hallucinations.
Only after days of treatment did his doctors learn that a chatbot was involved, and they never saw the conversation.
When medical treatment goes wrong, doctors and hospitals have systems for documenting and investigating what happened.
When a chatbot gives harmful medical advice, however, there may be no record available to doctors, regulators or independent researchers.
Without that information, dangerous patterns can go undetected.
Matthew DeVerna of Stanford University led the work, with Harry Yaojun Yan of Texas A&M University, Kai-Cheng Yang, an assistant professor at Binghamton University, and Filippo Menczer of Indiana University.
Their paper is a perspective piece, an argument from published evidence rather than a new experiment.
“We’re getting medical advice from these chatbots, but nobody – literally no one – is looking at this. If there’s an error or an issue, there’s just no way for people to know,” Yang said.
By March 2026, about one in three U.S. adults had used AI for health information or advice, according to a KFF poll.
Among them, 65% said a major reason was wanting an answer right away. Another 36% said they felt more comfortable asking health questions privately.
The authors don’t dismiss those reasons. A chatbot can turn a page written for physicians into plain language, and it answers at any hour.
For someone who speaks little English or has no doctor nearby, it may be the only affordable medical advice available.
The researchers are concerned about the lack of consistent oversight.
AI tools used in hospitals may face medical regulations and institutional safeguards, while general-purpose consumer chatbots often operate without comparable protections.
“At present, a question about chest pain is handled like a question about restaurant hours,” the authors wrote.
Other kinds of medical harm leave records. A drug that hurts people generates an adverse-event report, and those reports accumulate until a pattern emerges.
A chatbot conversation is different in two ways, the authors wrote.
The exchange is visible only to the company that runs the model, and AI companies give outside researchers no way to study those exchanges.
And a fluent, confident answer gives the user no sign that anything went wrong. Someone wrongly told their symptoms are harmless may never connect a late diagnosis to the chat.
The bromide poisoning case illustrates this problem. Because the doctors never saw the original chatbot conversation, they could not determine exactly what advice the man had received.
The authors listed the failures. Chatbots produce fluent statements that are factually false, a failure researchers call hallucination.
And people can plant false claims on websites that a model later absorbs during training, a practice known as data poisoning.
Quality also varies from one chatbot to another, and even with how a question is phrased.
Whether a person gets sound medical advice depends partly on which system they happened to open.
Replies in languages other than English are substantially less accurate and less consistent, and the people who rely on them most have the fewest alternatives.
Grok, the AI built into X, answers users’ questions about posts in public feeds. Its replies about vaccine safety are visible to everyone scrolling past, whether or not they asked.
Some accounts now consist largely of AI-generated text, images or video, and their authoritative-sounding health posts reach millions of people.
“Sometimes I’m not even looking for health information, but just by browsing my social media feeds, it’s there,” Yang said.
“It just shows up, and we believe that could have undesirable outcomes, especially if there’s medical misinformation or state actors trying to manipulate the online discussion.”
Coordinated groups of AI agents can make one health claim look like many independent voices agreeing on it.
The authors pointed to anti-vaccine messaging, which measurably reduced vaccination during the COVID-19 pandemic, as a campaign that can now run at scale on social media.
Researchers can’t measure the exposure, because platforms withdrew the data access that once let them study how content spreads.
First, the authors wrote, people should have a right to see their own health conversations with an AI system.
That way, a harmful exchange could be reconstructed and shown to a clinician. Those exchanges should then be reported and investigated the way adverse drug events are.
Every major AI company, they added, already has the technical means to build such a system. A company reporting on itself decides what counts as harm, though.
So the authors also call for independent researchers to have ongoing access to deployed systems, in every language those systems serve.
Platforms, they wrote, should label AI-generated health content and hold it back until experts have vetted it.
And chatbots should treat health questions as a higher-stakes category, answered only from sources vetted by medical societies.
The final recommendation concerns legal accountability. The authors argue that malpractice-style liability should apply to chatbots offering medical guidance.
They also call for changes to Section 230, a U.S. law that generally protects online platforms from liability for certain third-party content.
“We do not think this is something we should rely on the companies to do, because their incentive is always to make more money,” Yang said.
“Building such a system goes against that incentive. So we have to have some kind of third-party monitoring system, an independent evaluation.”
Today no one can say how often AI health advice is wrong, or whether its failures fall hardest on particular groups of people. Only the companies hold the data that would answer those questions.
Changing the law could take years, but the researchers argue that AI companies and social media platforms could introduce many of these safeguards immediately.
The full study was published in the journal Nature Health.
—–
Like what you read? Subscribe to our newsletter for engaging articles, exclusive content, and the latest updates.
Check us out on EarthSnap, a free app brought to you by Eric Ralls and Earth.com.
—–
