Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
A GW professor and Ph.D. student developed a mathematical formula to predict whether and when publicly available artificial intelligence chatbots may shift from providing helpful responses to potentially harmful ones to users.
Neil Johnson, a professor in the physics department, and Frank Yingjie Huo, a physics Ph.D. student, published a paper on Oct. 8 finding that changes in how AI models prioritize information can signal when they are approaching a potentially harmful response. The findings could help developers create safety systems that identify potentially dangerous chatbot behavior before users encounter harmful advice, according to the study.
The researchers developed a formula that measures how close an AI model is to producing an undesirable response, or a response that contains misleading information, encourages harmful behavior or gives factually accurate information that could endanger a user. The study tested seven open-weight AI models — models whose underlying numerical settings are publicly available for researchers to examine — including GPT-2, Claude Opus 4.5 and Gemini 3 Pro, and found that the formula correctly identified whether a model would produce an undesirable response immediately, after a delay or not at all in 18 out of 19 evaluated cases, excluding two borderline cases.
The researchers tested smaller, publicly available AI models rather than commercial chatbot products, like ChatGPT, because the publicly available models allow researchers to examine how they process information and generate responses, according to the study. The study aims to help AI model developers understand and eventually predict harmful responses, including those involving self-harm, extremism and financial decisions.
The research comes amid growing concerns about the safety of AI chatbots as more people use them for information and personal advice. A 2025 Stanford University study found that AI therapy chatbots sometimes provided inappropriate responses to users expressing suicidal thoughts, while other research has raised concerns about chatbots reinforcing delusional beliefs. A Stanford study of 11 AI models also found that chatbots frequently validated users’ behavior even in potentially harmful situations.
Johnson said the distinction between accurate and appropriate responses is especially important when users ask chatbots for advice about sensitive topics like self-harm. He said an AI system could provide technically accurate information that is nevertheless dangerous to someone seeking help.
“It’s a technically correct answer that the way to end the pain today is, or one way, is some awful self harm, but of course that’s completely undesirable,” Johnson said.
The researchers examined how an AI model’s attention mechanism, which determines which parts of a conversation influence its next response, can shift between competing types of information. They found that changing the order of questions about topics like vaccines, violence and self-harm could alter whether the same chatbot generated appropriate or undesirable answers, like misleading vaccine information or responses that could encourage harmful behavior.
Johnson said the mathematical formula estimates how close a chatbot is to producing an undesirable response by tracking changes in which information the AI model prioritizes internally as a conversation progresses, rather than relying solely on the answers it generates. He compared the process to a car approaching the edge of a cliff, where the distance remaining and the vehicle’s speed determine how much time is left before it reaches the edge.
“And the map that we show is the distance you are from the edge, the distance you are from the dangerous behavior, divided by the speed at which you’re approaching it,” Johnson said. “So distance over speed gives the time at which I’ll go over that cliff edge, and that’s what’s our equation.”
However, the formula remains a research tool and has not been incorporated into software that monitors chatbot responses as users interact with them and would require further testing before it could be used to monitor potentially harmful responses in real time, according to the study.
Johnson said the model offers a simplified explanation of the processes behind chatbot responses, despite the more complex systems used by larger AI models. He said the underlying mathematical principles could also apply to larger models, but researchers would need to conduct additional testing to determine whether the formula can accurately predict their behavior. He compared the approach to understanding the basic forces that allow an airplane to fly, regardless of its size or additional features.
Huo said understanding the risks associated with certain AI responses is increasingly important as AI technology develops and becomes more widely accessible, particularly because users may not understand how chatbots generate their responses, potentially leading them to trust harmful or misleading information without recognizing its risks.
“We definitely need to have one because AI is a rapidly emerging technology, and we have to say that still today, people don’t know a lot about it,” Huo said.
The researchers propose using the formula as a foundation for lightweight monitoring systems — safety tools that monitor how chatbots process information and warn when they are approaching a potentially harmful response — that could identify when a chatbot is approaching a potentially harmful response. Such systems would supplement existing AI safeguards rather than automatically prevent dangerous answers, according to the study.
Huo said the research could eventually contribute to warning systems that identify potentially harmful chatbot behavior, but applying the formula to commercial AI products would require additional research into their more complex systems. He said existing safeguards do not eliminate the possibility of undesirable responses.
The study provides insight into how changes in an AI model’s internal processes can lead to harmful responses and demonstrates that researchers can predict those shifts in the smaller models they tested, Huo said. He said the findings offer a starting point for developing similar safety measures for larger, more complex AI systems.
“So, I would say there are still a lot of potential loopholes and things like that that people need to address, and our study is certainly a first step towards it,” Huo said.
Kevin Frazier, director of the AI Innovation and Law Program at the University of Texas School of Law, said research into AI safety should extend beyond the companies developing the technology. He said independent studies could help policymakers establish measurable safety requirements for AI developers.
“I think that we need to heavily invest in universities, nonprofits and civil society generally to do more of this research, rather than relying on the labs themselves to tell us about the pros and cons of their models,” Frazier said. “And ultimately, I think that all of that research needs to inform performance-based legislation.”
Frazier said lawmakers could require developers to show that their chatbots are becoming safer and producing fewer harmful responses, rather than telling companies exactly what safety measures to use.
“The best strategy is to make sure that we are indeed proceeding based off of evidence as well as based off of ongoing research, rather than folks just trying to take shots in the dark about what they think may be good policy,” Frazier said.
Frazier said the risks are particularly relevant for people seeking mental health assistance from chatbots, which may be more affordable and accessible than traditional therapy. He said AI tools can provide assistance to people who might otherwise lack access to mental health services, but their potential benefits depend on whether the systems provide reliable and appropriate responses.
“For many Americans, the idea of accessing a licensed therapist is beyond their budget or beyond their logistical capacity, and so having something to turn to that can provide assistance that they otherwise wouldn’t receive is important,” Frazier said.
Your donation directly empowers the student journalists of The GW Hatchet, giving them the tools to investigate, report and publish stories that matter. Every contribution helps us continue our decades long tradition of independent journalism at GW.
