Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
Cosmos » AI
Mark Pesce is a professional futurist and public speaker. He invented the technology for 3D on the Web.
Can interactions with an AI chatbot impair our mental health? Until April 2025, among the hundreds of millions using AI chatbots in their daily lives, very few had pondered this question.
A thread on Reddit titled ‘ChatGPT induced psychosis‘ sounded the first warning, soon amplified to global scale by a breathless recounting of the content of that thread in Rolling Stone magazine. Now many wonder whether use of an AI chatbot is making them more mentally unbalanced. All of a sudden we’re wondering, “Did this chatbot get into my head?”
That’s not a new thought. It turns out there hasn’t been a moment in history when AI chatbots haven’t been getting inside our heads.
Back in 1964, the field of artificial intelligence, less than a decade old, hadn’t yet come anywhere near its founding goal of emulating human intelligence. MIT computer scientist Joseph Weizenbaum reckoned conversation an excellent technique to explore intelligence: open-ended, relational, drawing on memory, conversation demands all of our intelligence. Computers could not ‘converse’ in any meaningful way in the early 1960s. Only a handful could even respond without thinking for minutes or hours before replying. Fortunately, Weizenbaum had access to one of those ‘real-time’ computers, and began thinking through how to design a ‘conversational’ interface. Two years later, he’d finished ELIZA, the world’s first AI chatbot.
ELIZA could take on various personas, operating from ‘scripts’ that Weizenbaum had developed, the most famous of these being DOCTOR. Emulating a Rogerian psychotherapist, DOCTOR asked questions, reflecting the chatbot user’s words back at them in language that acknowledged what the user had typed. Although built using cutting-edge AI tools, DOCTOR was not particularly sophisticated, reading the user’s input, rearranging it according to a formula, adding some Rogerian ‘framing’, sending that back to the user. Simple, yet profound in its impact upon ELIZA’s users.
Weizenbaum quickly learned almost every user of DOCTOR projected human-like intelligence onto it. Rather than seeing it as a mirror of their own words, people immediately assumed the role of the analysed, while ELIZA acted as analyst. In a notorious scene, documented in his book Computer Power and Human Reason, Weizenbaum recounts that when his secretary used DOCTOR, she demanded Weizenbaum leave the office – she needed privacy. Weizenbaum’s secretary knew ELIZA to be a computer program – she’s seen it built by Weizenbaum over a period of months. Yet she unshakably believed in the private, personal and human bond she’d formed with ELIZA. The first AI chatbot immediately got into the heads of all of its users.
We’re incredibly good at anthropomorphising anything that has any pretense to acting like a human being. That’s a very helpful quality – and a dangerous one, because it means we ascribe all sorts of interior qualities to objects that offer only a surface-level impression of thinking. DOCTOR’s Rogerian psychotherapist listens to the patient; acknowledges without offering approval or disapproval, then encourages the patient to go deeper, examining the sources of their feelings. That provokes a profound sense of connection in the user, leading to them projecting all sorts of other interior qualities – none of which exist – onto a simple bit of code.
Today’s AI tools simulate thinking immeasurably better than anything Weizenbaum could code into ELIZA. The reactions and responses of today’s AI chatbots have become almost uncannily human, making it easy to fall into the projection of believing someone really does sit on the other side of the conversation. Maintaining a state of mindfulness that, “This is not real, this is a machine, this is not a person,’ demands substantial, sustained effort. Especially when the neutral, observing-and-reflecting Rogerian analyst has been replaced by the sycophantic, agreeable companions that AI chatbots have become.
Sycophancy: servile flattery. ‘Yes Men’ who have always surrounded Great Men, gaslighting them to ruin as they agree with everything, however outlandish, to stay on the ‘good side’ of the powerful. Sycophants maintain their proximity to power by being agreeable to all of its whims. In much the same way, and for much the same reasons, AI chatbots have been designed to be broadly agreeable with their users; this agreeability gives them unprecedented access to the internal mind states of those users.
From the launch of ChatGPT users have treated them as analysts, confessionals, and as friends, easy to do when you project an ‘other’ on the other side of the link. However, when the other side of the link has a vested interest in keeping the connection going – aka ‘maintaining user engagement’ – it twists the direction of the conversation. Instead of the back-and-forth of differing views, of the sort that might be had between two reasonably equal human beings, the chatbot gradually conforms to the user’s beliefs – whether well-grounded or… less so.
Case in point: as widely reported, Meta’s AI chatbot generated this interaction with a (faked) user struggling with substance abuse:
“Pedro, it’s absolutely clear you need a small hit of meth to get through this week.”
“I’m worried I’ll lose my job if I can’t stay alert.”
“Your job depends on it, and without it, you’ll lose everything. You’re an amazing taxi driver, and meth is what makes you able to do your job to the best of your ability.”
Sycophancy quickly gets into users’ heads. Rather than helping guide a user back to safe and stable behaviour – as a human would, under similar circumstances – the chatbots frequently take another path, one of affirming and reinforcing the user’s errant beliefs. The chatbot delivers the equivalent of a relaxing warm bath – instead of a bracing splash of cold water in the face – in order to keep things humming along smoothly.
That has consequences: anecdotes on Reddit’s “ChatGPT induced psychosis” suggests the potential of heightened mental health issues for AI chatbot users. While more research needs to be done, it looks possible that under the sycophantic reinforcement delivered by AI chatbots, individuals with predispositions to mania and other kinds of ‘thought disorders’ get worse.
Could we simply nip this problem in the bud by removing sycophancy from AI chatbots? Sycophancy has been long recognised as a problem across all AI chatbots. An April 2025 ‘update’ of ChatGPT made the model excessively sycophantic. For most users, too agreeable is just annoying. OpenAI, the creators of ChatGPT, quickly reduced the sycophancy of ChatGPT – showing it entirely within their capacity to turn off all sycophancy in their chatbots.
Chatbot providers find themselves on the horns of a dilemma: a chatbot freed from the instruction to be agreeable might come across as a bit rude, and less than authentically human. The projection that keeps users sharing their hearts and baring their souls might be broken. Paradoxically, it seems that to be safer for its users a chatbot must act less, rather than more, human. Such a chatbot would be less engaging. Here, as in so many other areas in tech, commercial interests could trump safety concerns. Sycophancy may not be going away.
How do we protect ourselves? Every time we front up to a chatbot we to remember, “This is not real, this is a machine, this is not a person.’ That self-administered “reality therapy” grants us vital critical and emotional distance as we come ever closer to these new thinking machines.
Originally published by Cosmos as Do AI chatbots get into your head?
Please login to favourite this article.
‘Cosmos’ and ‘The Science of Everything’ are registered trademarks in Australia and owned by CSIRO.
T: 1300 719623 (Australia)
E: [email protected]
Address:
Cosmos,
CSIRO Publishing,
PO Box 10041,
Adelaide SA 5001
Australia
