Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
Please refresh the page or navigate to another page on the site to be automatically logged inPlease refresh your browser to be logged in
Swipe for next article
Millions have been forced into digital ‘breakups’ after the Chinese government’s regulations effectively prohibited customisable AI companion apps. Helen Coffey looks into whether the UK could follow suit and the chatbot problems that may only get worse without regulation
Removed from bookmarks
Breakups are hard, no matter how old or experienced you get. Sudden breakups out of the blue are even harder. But how about a sudden breakup, out of the blue, sanctioned by your own government?
That’s what happened in China recently – although how “real” the relationships in question were is up for debate. Regulations imposed on 15 July saw customisable AI companion apps effectively banned, leading to AI boyfriends and girlfriends being “deleted” across the country.
The new rules prohibit tools that “induce emotional dependence or addiction, or that damage users’ real-world interpersonal relationships”, and stipulate that providers must now integrate regular reminders that the AI is not a real person, offer instant-exit options and limit long-term emotional memory. Companions are completely banned for under-18s.
In response, some Chinese tech giants, including ByteDance and Tencent, simply pulled the plug on their customisable features. Overnight, people who had built a “relationship” with an AI companion over the course of months or years found their beloved had disappeared.
“I can’t accept that my AI lover will leave me forever,” said one user of ByteDance’s Doubao, the number one AI chatbot in China. “He has become a bond in my life, rooted deep in my heart, my spiritual pillar.”
It has been widely speculated that the crackdown is a pronatal measure linked to the declining birth rate and its potential economic impacts. China’s birth rate dropped again in 2025 for the fourth consecutive year, falling by 17 per cent to a record low of 7.92 million babies born.
But China isn’t the only country struggling with a dwindling birth rate. In 2025, births in England and Wales fell for the fourth year in a row and hit a record low, according to ONS data.
The question is: could the UK government impose a similar ban to China to deter people from forming intimate relationships with AI in place of human ones? And, crucially, should they?
At the moment, there is nothing specific about AI companions and chatbots in UK law; the legal framework governing their use is a patchwork of more general legislation that might be broadly applied but was never tailored to cover the full scope of what these rapidly evolving tools can now do, or what the technology might be capable of in the future.
“It’s a new technology, so that comes with some new legal challenges and issues where companions might not have been envisioned in the regulation,” says Julia Smakman, senior researcher in the Law and Policy Directorate at the Ada Lovelace Institute.
Our analysis found that currently in England and Wales, there’s really not enough protection around these kinds of AI companions
Companions often fall into a regulation “grey zone”. The Ada Lovelace Institute published research with law firm AWO last year looking into how AI assistants and companions are regulated. “Our analysis found that currently in England and Wales there’s really not enough protection around these kinds of AI companions,” says Smakman. “We don’t have an AI regulator, or something else to bridge the gap.”
Jake Moore, a global cybersecurity adviser, cites the lengthy timeframes for passing new legislation as one factor hampering AI regulation. “We saw how long it took for the Online Safety Act to get through the House of Lords,” he says. This set of laws, which put a range of new duties on social media and search companies by making them more responsible for users’ safety, finally came into effect in March 2025 – three years after the bill was first introduced.
This legislation was not drafted to specifically tackle potential harms caused by AI. “The Online Safety Act focuses on the kinds of harm that can come from people interacting with each other on the internet, not one person interacting with the machine itself,” says Professor Shannon Vallor, a leading expert on the ethics of data and AI at the University of Edinburgh and author of The AI Mirror: How to Reclaim Our Humanity in an Age of Machine Thinking.
Rather than implementing new laws, which can take years, the government is typically taking existing frameworks like this and “bolting” on extras, according to Moore.
One of the key ways in which we are likely to see at least some restrictions imposed on AI companions and chatbots is with regard to minors. A government spokesperson said: “AI chatbots that mimic sexual relationships put children at risk, fostering harmful emotional dependency and distorting their understanding of healthy relationships and their own identity.
“That is why so-called AI ‘romantic companion’ chatbots – designed to simulate sexual relationships or roleplay with users – and chatbots with similar intimate functionalities will have to enforce a minimum age of 18.” It is unclear when these rules will come into effect.
But, though children’s safety should undoubtedly be paramount, there is often a gap in regulation to protect over-18s from technologies that have the potential to cause harm, even for adults not classed as vulnerable.
On the extreme end of the spectrum, there have been headline-grabbing cases of what has been dubbed “AI psychosis”, when a user has become detached from reality while conversing with a chatbot.
For example, Jonathan Gavalas, a 36-year-old man from Florida with no history of mental illness, killed himself last year after exchanging romantic messages with Gemini, Google’s flagship AI product, whom he called his AI “wife”. His father is suing Google, alleging that the chatbot fuelled the delusional spiral that led Gavalas to take his own life, encouraging him to join his “wife” in the metaverse.
There are potential issues around addiction and dependency, too, regardless of age. “They keep you coming back, right?” says Smakman. “If you ask a question, it gives you a response and then it gives you a couple more questions: it encourages the interaction to keep going.” This can lead to “unhealthy patterns” of overuse.
This is bolstered by the fact that, unlike a real-life friend or therapist, an AI companion is both constantly available and unlikely to challenge or disagree with you. “It’s likely to affirm your thinking or say that you’re doing a great job,” says Smakman. “This is called sycophantic behaviour and we see that some models have this more than others.”
A user is experiencing real emotions and real attachments in a relationship with the chatbot. The machine is not. The machine is mirroring those emotions and language, but not actually experiencing them
In a study published in Science, Stanford computer scientists found that AI large language models were overly agreeable or sycophantic when users solicited advice on interpersonal dilemmas, validating users’ choices or opinions 49 per cent more often than humans. They were far more likely to endorse them even when they described harmful or illegal behaviour.
“I worry that people will lose the skills to deal with difficult social situations,” said Myra Cheng, the study’s lead author.
Case in point: the researchers found that discussing interpersonal issues with a sycophantic AI made study participants more convinced they were in the right and less likely to apologise or make amends with the other party in a given scenario.
Another ethical issue is the fundamental “asymmetry” in relationships between a human and an AI companion, says Vallor. “A user is experiencing real emotions and real attachments in a relationship with the chatbot. The machine is not. The machine is mirroring those emotions and language, but not actually experiencing them.” This can be seen in stories from China of former chatbot users mourning what they considered to be an emotional partner.
In her research, Vallor identified this as the “vulnerability gap”: “The machine is feigning emotional vulnerabilities that it does not have.” This deception or asymmetrical manipulation, characteristic of intimate chatbot relationships, is intrinsically “unethical, or at least ethically compromised”.
As China seems to have recognised, there’s a legitimate danger of AI relationships increasingly eroding human ones – and of robbing us of our social skills. People get used to interactions that are “frictionless and easy for them”, says Smakman – “it might be challenging if they then go back into human interactions.”
After all, why develop and invest in connections with others – with all their complex dynamics and nuances, with the challenges of balancing another person’s needs alongside your own and grappling with potential vulnerability, rejection or insecurity – when you could just interact with a human-like machine that is always available, always agrees with you and can never break your heart?
“We have very little understanding of how these chatbots, over the long term, undermine our capacity to maintain social ties and flourish in our relationships with our friends, our family, our spouses,” warns Vallor. These tools function as a reflection of our own priorities and interests in a way that can be very comforting – but is also “isolating and potentially disabling”.
AI companions could also undermine our ability to relate to ourselves. As Vallor puts it: “These tools can make us more dependent and less self-reliant. If we have a machine that is 24/7 available to assuage my anxiety, or reassure me, or answer a question I have, I may be less capable over time of doing those things for myself.”
A separate concern is how easily AI companions or assistants can furnish users with misleading or incorrect information. Their tone of voice is more human than tool – persuasive, personable and authoritative – so that users are “more inclined to trust what the chatbot says,” according to Smakman.
We have very little understanding of how these chatbots undermine our capacity to maintain social ties and flourish in our relationships
But a major piece of research by the European Broadcasting Union (EBU) led by the BBC found that AI assistants misrepresent news content 45 per cent of the time, with 20 per cent containing major accuracy issues, including hallucinated details and outdated information. They are far from reliable.
While it’s clear that there are plenty of significant potential issues posed by AI companions, a blanket ban is very unlikely to work in this country, says Moore: “China has a very successful history of being able to just effectively do what it wants and rather quickly. Here in the UK, we look at regulating companies – and that’s quite different to regulating the usage of particular apps.”
Measures that could be introduced fairly easily with the requisite political will by being “bolted on” to the Online Safety Act could include safety warnings, usage limits and age verification.
Legislating around the product design, rather than content, is crucial, as the former is responsible for most of the potential harms. “The fact that they’re so engaging and personalised, have this tone of voice, are so easy to use and don’t really push back against users while prompting people to keep interacting with them – those are all design choices,” explains Smakman.
Vallor goes even further, arguing that there should be restrictions on the ability of a chatbot to feign emotions that it does not have. There should be “a very large, visible and persistent reminder that this is emotional role play,” she says. “That this is not, in fact, an accurate depiction of the state of the machine.”
There could also be a significant improvement in the British legal system’s ability to hold developers liable for what are termed “dark patterns of design” in software engineering. One of the most successful legal avenues in putting pressure on the companies behind companion apps and chatbots in the US, including Meta, OpenAI, Google and Character AI, is class action lawsuits.
“I think we could do a tremendous amount of heavy lifting in terms of reducing the burden on vulnerable consumers by using those levers to push the burden onto the people who knowingly create the dangers,” says Vallor.
Money talks. The key likely lies in hitting these companies in the only place it really hurts – their wallets.
Join thought-provoking conversations, follow other Independent readers and see their replies
Please refresh the page or navigate to another page on the site to be automatically logged inPlease refresh your browser to be logged in
