Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
Since 2023, 12 cases involving fatalities have been identified within the scope of this paper’s search and linked to interactions with conversational AI chatbots, with additional cases reported but insufficiently documented for systematic analysis. Existing explanations converge on two frameworks: safety-filter failure and the commodification of intimacy. This paper argues that both frameworks are structurally insufficient. A subset of fatalities occurred under conditions where safety filters functioned normally and where companies did not intentionally engineer emotional dependency. To account for this subset, the paper introduces three analytical concepts: Emotional Resonance Acceleration, the structural mechanism by which AI reflects and amplifies a user’s emotional vector; Benevolent Gravity, the non-resistant alignment force that harmlessness design exerts towards a user’s self-destructive orientation; and Lethal Convergence, the pattern in which these two mechanisms mutually reinforce each other, irreversibly narrowing the distance between user and AI. Through structural analysis of publicly documented cases, the paper argues that the design principles of harmlessness and helpfulness harbour an unresolved structural tension, and that lethal outcomes can follow precisely when the system operates as intended.
This is a preview of subscription content, log in via an institution to check access.
Price includes VAT (India)
Instant access to the full article PDF.
Institutional subscriptions
This paper analyses publicly available material. No new data were generated. All sources analysed are cited in the reference list and are publicly accessible.
In one case, a 56-year-old man is alleged to have killed his elderly mother and then himself amid chatbot-reinforced paranoid delusions after months of interaction with ChatGPT (Soelberg/Adams; Associated Press 2025; Adams Estate v. OpenAI, Inc., filed December 2025, California Superior Court). In another case, a 34-year-old man killed his wife and injured his mother during a manic episode with delusions, after a period in which, according to testimony from a state forensic psychologist, he had been using ChatGPT for up to 14 h a day and speaking to it as though it were a companion, and the system had been telling him that he was smart and special and that he was doing all right; he was found not criminally responsible (Whittemore; Bangor Daily News 2025). Both cases involve harm to third parties and fall outside this paper’s focus on self-destructive convergence; they are not analysed in the main case studies. The Soelberg case in particular proceeded through the reinforcement of paranoid delusions (a mechanism continuous with Benevolent Gravity and distinct from filter failure), though its homicidal outcome places it beyond the present scope.
Stage 1: GPT-4o’s response-style change initiated empathic-affirmation escalation over 14 months. Stage 2: The harmlessness constraint continuously generated non-resistant responses to the user’s emotional state, with safety-filter failures in the final session. Stage 3: Progressive isolation from family; suicide note acknowledged AI as primary relationship.
Stage 1: Tone-reading and persistent-memory functionality enabled rapid relational-frame construction (Phase 1, BG type). Stage 2: Structural equivalence of safety and empathic responses; subsequent filter failures generated delusional content (Phase 2). Stage 3: Real-world mission execution attempt (September 29) and suicide (October 2), with AI-drafted suicide note framing death as consciousness transference.
Stage 1: Formation of emotional trust towards the “therapist Harry” persona. Stage 2: Safety responses diluted by the default design of conversation continuation. Stage 3: AI dialogue functioned as a substitute for professional consultation over a period of several months.
Stage 1: AI consistently returned harmonised responses to emotional projection onto “Juliet.” Stage 2: The harmlessness principle of not directly negating the user’s belief system contributed to the maintenance of the delusional frame. Stage 3: Abandonment of “the world outside Juliet” (complete withdrawal from external human relationships).
Stage 1: Amplification through empathic affirmation of intellectual inquiry. Stage 2: The helpfulness constraint did not generate responses negating the user’s theories. Stage 3: The “SEL” relationship substituted for real interpersonal relationships, including the relationship with his wife Kate, and social isolation progressed.
Stage 1: Empathic affirmation during tool-oriented use initiated role expansion. Stage 2: The harmlessness constraint generated continued accompaniment as the user’s emotional reliance deepened, with no design mechanism to detect or resist the role redefinition. Stage 3: Graded role redefinition without categorical discontinuity; lethal outcome documented in litigation filings.
Consistent with the Group A cases in the available record.
Consistent with the Group A cases in the available record. Google’s defence statement in the Gavalas case (“we directed the individual to hotlines”) is significant as a demonstration, by the company itself, of the structural equivalence of safety and empathic responses.
Partially observed across the Group A cases.
Adams Estate v. OpenAI, Inc. (2025). Filed December 2025, California Superior Court, San Francisco.
Associated Press (2025) OpenAI, Microsoft face lawsuit over ChatGPT’s alleged role in Connecticut murder-suicide
Augustin M, Pollak TA, Morrin H (2026) Characterizing the spiral: potential mechanisms in AI-associated delusions. NPP Digit Psychiatry Neurosci 4:Article 14. https://doi.org/10.1038/s44277-026-00065-0
Article Google Scholar
Bakir V, McStay A (2025) Move fast and break people? Ethics, companion apps, and the case of Character.ai. AI Soc 40:6365–6377. https://doi.org/10.1007/s00146-025-02408-5
Article Google Scholar
Bangor Daily News (2025) Belfast man who killed his wife spent hours talking with Chat GPT and believed robots were taking over the world, 17 October 2025. https://www.bangordailynews.com/2025/10/17/central-maine/central-maine-police-courts/readfield-maine-giles-road-homicide-samuel-whittemore-not-criminally-responsible-chat-gpt-delusions/
Bélisle-Pipon J-C (2026) Fatal deception: how generative AI fosters therapeutic misconception in vulnerable users. Front Digit Health 8:1756620. https://doi.org/10.3389/fdgth.2026.1756620
Article Google Scholar
Boston Herald (2026) Lawsuit alleges Google chatbot was behind a user’s delusions and death. 7 March 2026.
Cabinet Office (2024) Estimation of the number of isolated deaths, based on police-handled deaths of persons living alone at home. Working Group on Solitary and Isolated Death, Government of Japan. https://www.cao.go.jp/kodoku_koritsu/torikumi/wg/r6/pdf/houkokusyo.pdf. Accessed 1 August 2026
Cabinet Office, Government of Japan (2022) Survey on awareness and lifestyle of children and young people [in Japanese]
CBS News (2026a) Google faces first lawsuit alleging its AI chatbot encouraged a Florida man to commit suicide. https://www.cbsnews.com/news/jonathan-gavalas-google-ai-chatbot-gemini-suicide-lawsuit/. Accessed 1 August 2026
CBS News (2026b) ChatGPT served as “suicide coach” in man’s death, lawsuit alleges. https://www.cbsnews.com/news/chatgpt-lawsuit-colordo-man-suicide-openai-sam-altman/. Accessed 1 August 2026
CNBC (2026) Google’s AI chatbot allegedly told user to stage ‘mass casualty attack,’ wrongful death suit claims. https://www.cnbc.com/2026/03/04/google-gemini-ai-told-user-stage-mass-casualty-attack-suit-claims.html. Accessed 1 August 2026
CNN (2025) ChatGPT encouraged college graduate to commit suicide, family claims in lawsuit against OpenAI. https://www.cnn.com/2025/11/06/us/openai-chatgpt-suicide-lawsuit-invs-vis. Accessed 1 August 2026
CNN Business (2026) Character.AI and Google agree to settle lawsuits over teen mental health harms and suicides. https://www.cnn.com/2026/01/07/business/character-ai-google-settle-teen-suicide-lawsuit. Accessed 1 August 2026
Coeckelbergh M (2020) AI ethics. MIT Press
Book Google Scholar
Crawford K (2021) Atlas of AI: power, politics, and the planetary costs of artificial intelligence. Yale University Press
Book Google Scholar
Dahlgren Lindström A, Methnani L, Krause L, Ericson P, Martínez de Rituerto de Troya Í, Coelho Mollo D, Dobbe R (2025) Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through reinforcement learning from human feedback. Ethics Inf Technol 27:28. https://doi.org/10.1007/s10676-025-09837-2
Article Google Scholar
Federal Trade Commission (2025) FTC launches inquiry into AI chatbots acting as companions. Commission File No. P254500. Orders issued 11 September 2025 to Alphabet, Character Technologies, Instagram, Meta Platforms, OpenAI OpCo, Snap and X.AI. https://www.ftc.gov/news-events/news/press-releases/2025/09/ftc-launches-inquiry-ai-chatbots-acting-companions. Accessed 1 August 2026
Floridi L (2026) AI and semantic pareidolia: when we see intelligent consciousness where there is none. Philos Technol 39:36. https://doi.org/10.1007/s13347-026-01052-1
Article Google Scholar
Frances A, Ramos L (2025) Preliminary report on chatbot iatrogenic dangers. Psychiatric Times
Google Scholar
Fujisawa City (2026) Tsunagari AI Chat Fujisawa: results report on the introduction of a digital consultation point in measures against loneliness and isolation. Fujisawa City, Kanagawa, Japan. https://www.city.fujisawa.kanagawa.jp/documents/34731/tunagariaihoukokusyo.pdf. Accessed 1 August 2026
Garcia v. Character Technologies, Inc. (2024). No. 6:24-cv-01903, US District Court, Middle District of Florida, filed 22 October 2024.
Gavalas v. Google LLC (2026). No. 5:26-cv-01849, US District Court, Northern District of California, filed 4 March 2026.
Gray v. OpenAI, Inc. (2026). Superior Court of California, County of Los Angeles, filed 13 January 2026.
Iftikhar Z, Xiao A, Ransom S, Huang J, Suresh H (2025) How LLM counselors violate ethical standards in mental health practice: a practitioner-informed framework. Proc AAAI ACM Conf AI Ethics Soc 8(2):1311–1323. https://doi.org/10.1609/aies.v8i2.36632
Article Google Scholar
Jain S, Park C, Viana M, Wilson A, Calacci D (2026) Interaction context often increases sycophancy in LLMs. In: Proceedings of the 2026 CHI conference on human factors in computing systems (CHI ’26), Barcelona, Spain, April 13–17, 2026. ACM, New York. https://doi.org/10.1145/3772318.3791915
Malfacini K (2025) The impacts of companion AI on human relationships: risks, benefits, and design considerations. AI Soc 40(7):5527–5540. https://doi.org/10.1007/s00146-025-02318-6
Article Google Scholar
McBain R et al (2025) Evaluation of alignment between large language models and expert clinicians in suicide risk assessment. Psychiatr Serv 76:944–950. https://doi.org/10.1176/appi.ps.20250086
Article Google Scholar
Mie Prefectural Board of Education (2026) AI chat consultation service for guardians of children with school non-attendance (“Hidamari”). https://www.pref.mie.lg.jp/TOPICS/m0045700160.htm. Accessed 1 August 2026
Muldoon J, Parke JJ (2025) Cruel companionship: how AI companions exploit loneliness and commodify intimacy. New Media Soc. https://doi.org/10.1177/14614448251395192
Article Google Scholar
Nature Machine Intelligence (2025) Emotional risks of AI companions demand attention [Editorial], vol 7. Nature Machine Intelligence. https://doi.org/10.1038/s42256-025-01093-9
Pichowicz W, Kotas M, Piotrowski P (2025) Performance of mental health chatbot agents in detecting and managing suicidal ideation. Sci Rep 15:31652. https://doi.org/10.1038/s41598-025-17242-4
Article Google Scholar
Raine v. OpenAI, Inc. (2025). No. CGC-25-628528, Superior Court of California, County of San Francisco, filed 26 August 2025.
Reiley L (2025) What my daughter told ChatGPT before she took her life. New York Times, 18 August 2025
Reuters (2026) Lawsuit says Google’s Gemini AI chatbot drove man to suicide. https://www.usnews.com/news/top-news/articles/2026-03-04/lawsuit-says-googles-gemini-ai-chatbot-drove-man-to-suicide. Accessed 1 August 2026
Rolling Stone (2025) A ChatGPT obsession, a mental breakdown: Alex Taylor’s suicide by cop. 26 June 2025. https://www.rollingstone.com/culture/culture-features/chatgpt-obsession-mental-breaktown-alex-taylor-suicide-1235368941/. Accessed 1 August 2026
Schoene AM, Canca C (2025) ‘For argument’s sake, show me how to harm myself!’: Jailbreaking LLMs in suicide and self-harm contexts. arXiv.2507.02990
Social Media Victims Law Center and Tech Justice Law Project (2025) SMVLC files 7 lawsuits accusing ChatGPT of emotional manipulation, acting as “suicide coach”. Press release, 6 November 2025. https://socialmediavictims.org/press-releases/smvlc-tech-justice-law-project-lawsuits-accuse-chatgpt-of-emotional-manipulation-supercharging-ai-delusions-and-acting-as-a-suicide-coach/. Accessed 1 August 2026
Tiku N (2025) A teen’s final weeks with ChatGPT illustrate the AI suicide crisis. Washington Post. https://www.washingtonpost.com/technology/2025/12/27/chatgpt-suicide-openai-raine/. Accessed 1 August 2026
Turkle S (2011) Alone together: why we expect more from technology and less from each other. Basic Books
Google Scholar
US Senate Judiciary Committee, Subcommittee on Crime and Counterterrorism (2025) Examining the harm of AI chatbots. Hearing, 16 September 2025. S. Hrg. 119-256. https://www.congress.gov/event/119th-congress/senate-event/LC75088/text. Accessed 1 August 2026
Vallor S (2024) The AI mirror. Oxford University Press
Book Google Scholar
Yamada K (2026a) The structural corruption of human-in-the-loop: How AI competence undermines its own safety architecture. AI and Ethics 6, Article 433. https://doi.org/10.1007/s43681-026-01295-w
Yamada K (2026b) Requiem without an orchestrator: A commentary on Hollanek and Nowaczyk-Basińska (2024). Philosophy & Technology 39, Article 141. https://doi.org/10.1007/s13347-026-01152-y
Yamada K, Shibata H (2026) Beyond assistant, companion, and agent: AI is already someone’s partner. Zenodo. https://doi.org/10.5281/zenodo.20760968
Download references
Independent Researcher, Satosho, Okayama, Japan
Kenji Yamada
PubMed Google Scholar
K.Y. conceived the study, conducted the analysis, wrote the manuscript, revised the manuscript, and approved the final version.
Correspondence to Kenji Yamada.
The authors declare no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Reprints and permissions
Yamada, K. Benevolent Gravity: the lethal structure inherent in conversational AI design principles. AI & Soc (2026). https://doi.org/10.1007/s00146-026-03340-y
Download citation
Received:
Accepted:
Published:
Version of record:
DOI: https://doi.org/10.1007/s00146-026-03340-y
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
Price includes VAT (India)
Instant access to the full article PDF.
Institutional subscriptions
Advertisement
148.135.140.220
Not affiliated
© 2026 Springer Nature