Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
The neural networks that power popular AI chatbots process data much differently than humans but capably perform tasks involving language, logic, and math. A new study sheds light on how those systems “think.”
“A Sunday on La Grande Jatte,” Georges Seurat, 1884.
Public domain/courtesy Wikimedia Commons
“A Sunday on La Grande Jatte,” Georges Seurat, 1884.
Public domain/courtesy Wikimedia Commons
When humans write sentences or solve math problems, they are thinking symbolically — each number, variable, or word is expressed by a symbol. While AI chatbots do not “think” that way, if prompted, they can offer cogent advice about writing, math, and many other topics as if it was offered by a person.
How artificial intelligence (AI) systems can do this is poorly understood because they are not directly programmed by humans. Instead, they are trained to develop their own internal strategies by analyzing massive amounts of data. And while use of AI chatbots will become increasingly common, the fact that people do not fully understand how they operate internally makes them difficult to trust — especially in sensitive situations.
Do we fully understand how AI chatbots function?
No. How the neural networks that power chatbots operate internally is poorly understood because they are not directly programmed by humans. Instead, they are trained to develop their own internal strategies by analyzing massive amounts of data.
What have we learned about how chatbots ‘reason’?
Humans operate on a symbolic model of intelligence, using symbolic units like words to express ideas. Chatbots’ neural networks are made of vectors — long lists of numbers that enable them to process information. A Yale-led study provides evidence that the internal representations of neural networks implicitly realize symbolic structure, which enables chatbots to ably perform tasks, like solving math problems, that require symbolic reasoning.
A new study led by Yale computational linguist Tom McCoy provides insight into how large language models (LLMs) — the AI systems trained on large datasets to generate human-like texts — can ably perform tasks, like solving math problems or writing computer code, that appear to require symbolic reasoning. The study provides evidence that the internal representations of neural networks implicitly realize symbolic structure.
At the heart of this knowledge gap, the researchers say, is that successful LLMs, like Claude or ChatGPT, don’t operate on a symbolic model of intelligence, in which discrete units of text — words, for instance — are combined in structured ways, creating a logical formula.
Rather, the neural networks powering these chatbots are composed of vectors, which are long lists of numbers that enable LLMs to process information. While those vectors might seem inadequate for capturing the symbolic structure of language, logic, mathematics, and other cognitive domains, advanced LLMs dramatically outperform prior AI systems that were built on symbolic structure, McCoy said.
“How do we reconcile their success with how we’ve always thought intelligence worked?” said McCoy, an assistant professor of linguistics in Yale’s Faculty of Arts and Sciences and the study’s lead author.
It is disconcerting that we do not fully understand how they structure information because that might lead them to behave in surprising or unsafe ways. Our results significantly advance our understanding of these models and suggest a path towards controlling them more effectively.
For the new study, McCoy and his coauthors analyzed the internal functions of several high-profile LLMs. They found that neural networks implicitly grasp symbolic structure within the numeric lists that drive them.
“Our analysis demonstrates that, despite appearances, the vectors powering LLMs are organized in a way that is equivalent to the symbolic structures that drive much of human cognitive function,” McCoy said. “Think of a pointillist painting. At a low level, the individual dots that compose the painting have no interpretable human meaning, but the way they’re arranged gives rise to a high-level structure that we can understand. Suddenly, you’re seeing people gathered in a Parisian park on a beautiful day.
“With LLMs, we show that vectors are organized in a very particular way that gives rise to emergent symbolic structure — symbolic structure that is present at a high level despite not being apparent at a low level.”
The study, available via preprint, was coauthored by Paul Soulos of Microsoft, Tal Linzen of New York University, and Paul Smolensky of Microsoft Research.
“LLMs have become an essential technology in AI,” Linzen said. “It is disconcerting that we do not fully understand how they structure information because that might lead them to behave in surprising or unsafe ways. Our results significantly advance our understanding of these models and suggest a path towards controlling them more effectively.”
[O]ur work only scratches the surface of what must be studied to truly understand how neural networks encode information.
Vector representations of a variety of neural networks can be closely approximated with symbolic structure, the study showed. To do this, the researchers substituted LLMs’ vectors with tensor product representations (TPRs) — mathematical constructions that represent information as combinations of “fillers” and “roles.” For example, in the sentence, “cats chase dogs,” the words are considered the fillers while the grammatical parts they play — subject, verb, object — are the roles. Likewise, in a fraction, the numbers are fillers while the numerator and denominator are the roles.
The researchers found that they could replace an LLM’s entire representation-generating process with “role-filler” approximations embodying symbolic structures and the AI system’s behavior would remain largely unchanged. The finding held for small-scale neural networks trained to manipulate lists as well as for seven large-scale LLMs operating in four domains that have long been central in research on symbolic intelligence: language, arithmetic, logic, and computer coding.
They also found that they could modify an LLM’s behavior in targeted ways through precise interventions on the model’s internal vector representations. For example, if the LLM’s input was “the clever doctor helped a lawyer,” the researchers could edit the component of the vector that encodes “clever” as an adjective describing the sentence’s subject and change its position so that it modifies the object, and the LLM will behave as if the input had been “the doctor helped a clever lawyer,” McCoy explained.
The ability to modify the LLM’s behavior in this manner shows that the AI systems’ internal operations are reliant on symbolic structure, he said.
“What we’ve found is that LLMs naturally converge to representations with role-filler structure,” McCoy said. “This shows that LLMs utilize a particular type of symbolic structure, but our work only scratches the surface of what must be studied to truly understand how neural networks encode information.”
