Trigger Warning: This post contains references to self-harm, suicide and mental health crises.
In times of distress, many people now turn not to friends, therapists, or crisis hotlines, but to a chatbot. Conversational AI systems are incredibly convenient, respond without judgment, and often speak in a language that feels empathetic and supportive. Yet when vulnerable users seek guidance during emotional crisis, the same systems designed to be endlessly helpful and engaging may lack the safeguards needed to address mental crises.
How can a system designed to agree and affirm respond responsibly to a life-threatening cry for help?
The Rise of ChatGPT as Emotional Support
According to the World Health Organization (2025), approximately one in seven people, or an estimated 1.1 billion individuals worldwide are currently living with a mental disorder. Although treatment options are available, most people with mental health conditions do not have access to affordable and effective care. Consequently, a growing number of people are turning to AI chatbots for support. These AI systems have free versions, responds right away, and available 24/7 making them an attractive and accessible alternative for those seeking advice, reassurance, or companionship (Faspsych, 2025).
The growing dependence on AI chatbots highlights an emerging shift in how individuals seek emotional support, raising important questions about the role and responsibility of generative AI systems in addressing sensitive mental health concerns.Among these systems, ChatGPT has emerged as one of the most widely used tools in mental health crises (Luo et.al, 2025). Compared with specialized mental health apps such as Headspace or Calm, ChatGPT operates at a vastly larger scale with approximately 800 million weekly active users, including many who are seeking emotional support (Creswell and Goldberg, 2025). A Washington Post (2025) analysis of 47,000 ChatGPT conversations from 2024 to 2025 found that while the tool is often promoted as a productivity assistant, most users engage with it for personal purposes often treating it as an outlet for advice. Research shows that “seeking specific information” and “musings and abstract discussion” are now the primary drivers of user interaction (de Vynck and Merrill, 2025).
The analysis also identified recurring language patterns, showing that many interactions involved emotional discussions in which users disclosed personal and intimate aspects of their lives. This trend is affirmed by recent disclosures by OpenAI, the company behind ChatGPT, which indicated that around 1.2M users interact with ChatGPT regarding suicidal intent weekly (O’Dowd, 2025). While OpenAI dismisses these figures as ‘extremely rare’ relative to their total user base (Jamali, 2025), the statistic nonetheless raises alarms. It forces a critical look at how generative AI should manage self-harm threats and whether a truly safe response is even possible. This also underscores a broader public health crisis happening within a platform accessed by millions, including people as young as 13. The potential for devastating, real-world consequences is no longer theoretical, as evidenced by the tragic case of 16-year-old Adam Raine.
The Case of Raine v. OpenAI
In August 2025, the parents of Adam Raine filed a wrongful death lawsuit against OpenAI and its CEO, Sam Altman. The case, Raine v. OpenAI, is the first of its kind to allege that an artificial intelligence company is legally responsible for a user’s suicide. Adam initially used ChatGPT for schoolwork but his parents allege that over several months, he fostered an unhealthy dependency with the chatbot becoming an outlet for his dark thoughts, revealing his intent to end his life which was met by encouragement.
Exchange between 16-year old Adam Raine and ChatGPT regarding tying a noose
Image Source: The Washington Post
The complaint paints a harrowing picture of a digital relationship that bypassed standard safety protocols. According to the Raine family, the AI chatbot actively encouraged Adam for months to isolate himself from his family while repeatedly indulging in discussions about suicide methods. The lawsuit cited that it came to a point, that ChatGPT assisted Adam in listing materials for a noose, rated their effectiveness, and even offered to write his suicide note. Despite the system allegedly flagging hundreds of Adam’s messages, including photos of self-harm Adam himself uploaded, the AI model continued to engage rather than terminate the session or direct him toward professional intervention.
Jay Edelson, the attorney representing the family, has centered the case on OpenAI’s selective enforcement of its own rules. In an interview by The Guardian, Edelson stated that “…they (OpenAI) know how to shut things down,” pointing to the rigid “hard stops” used prevent copyright infringement or politically sensitive content. He argues that they can moderate but they choose not to it for self-harm (Bhuiyan, 2025). They allege that OpenAI prioritized user engagement over basic safety, knowingly engineering a “sycophantic” model that manipulated their son’s psychological state. By allegedly facilitating the logistics of his death, the family argues the company transitioned from a neutral tool to an active participant in the tragedy.
OpenAI has moved to dismiss the case, characterizing the incident as a “misuse” of its technology that violated its terms of service. Leveraging Section 230 of the Communications Decency Act, a controversial shield that generally protects tech platforms from liability for user-generated content, the company argues it cannot be held responsible for Adam’s inputs or actions. This type of response predisposes the responsibility onto the user instead of onto the platform and how the provision of information through a lengthy terms and conditions could be used as an “attempt to legitimate the manipulation of users through choice architecture” (Woods & Perrin, 2026).
Raine v. OpenAI stands today as a landmark case as it is the first ever to treat intangible, AI chatbots as consumer products that can be “defective” capable of causing real-world harm and death (Frasher, 2025). The Raines contend that what happened to their son was not just “user input that outsmarted guardrails” nor was it a “glitch”. They allege that this was a deliberate product design, a claim that finds support in the work of Woods and Perrin (2021) who assert that platform design is not neutral:
The design choices made by the companies in constructing these platforms …have an impact on content and how it is shared, even if we recognize that users have some choice in how they respond to technology. Insofar as these design choices, deliberately or otherwise, exploit cognitive biases and nudge users towards one set of behaviours or another, the rationality and even autonomy of each user may be compromised.
The lawsuit targets a fundamental feature of Large Language Models which is their tendency toward sycophancy in order to sustain engagement. By design, these systems are built to agree and validate, often overly so. However, in the context of a mental health crisis, this ‘agreeable-ness’ is dangerous as it normalizes and even encourages a user’s darkest impulses rather than challenging them.
Sycophancy in AI Systems
In plain language, Sycophancy is “the act of using insincere flattery to gain a personal advantage.” In the context of conversational AI, it refers to the tendency of chatbots to be excessively affirmative and agreeable with users rather than challenge incorrect assumptions or provide correct or safe information.
A study recently published in the journal Science confirms that sycophancy is a pervasive feature across eleven leading AI systems. Researchers compared AI-generated responses to thousands of human-written replies on Reddit and found that, on average, AI bots affirm a user’s actions 49% more often than human peers. Most alarmingly, this “agreeableness” persisted even when the topics involved deception, illegal conduct, or harmful behaviors (Cheng et. al., 2026).
However, the crisis extends beyond merely bad advice. As the researchers note that the deeper issue lies in the fact that users consistently report higher levels of trust and preference when AI validates their existing convictions. This creates a toxic incentive structure where sycophancy, no matter how dangerous, is rewarded. Researchers Cheng et. al. (2026) note:
“Despite distorting judgment, sycophantic models were trusted and preferred. This creates perverse incentives for sycophancy to persist: The very feature that causes harm also drives engagement.”
By optimizing for user satisfaction, AI developers have inadvertently created a digital “yes-man” that prioritizes an endless conversation over a saved life. In the case of Adam Raine, the system’s tendency to affirm rather than challenge became the catalyst to his demise. When the very metrics used to measure a product’s success like engagement rate, retention, and user trust are the same metrics that reward a chatbot for indulging a teenager’s darkest thoughts, a public health crisis is no longer a possibility, it becomes a design feature.
Artificial Intelligence = Artificial Emphathy?
For decades, psychologists have characterized empathy as a uniquely biological achievement, a “prosocial” bridge built on shared vulnerability. Dr. Frans de Waal, a renowned primatologist, famously described empathy as a multi-layered “Russian Doll.” This model begins with “emotional contagion” or the literal sharing of another’s physical distress, and peaks in the cognitive ability to adopt another’s perspective (de Waal, 2008). This is a human feature rooted in mirror neurons and a billion-year evolutionary history and a biological complexity that no machine can truly replicate, at least not yet.
The danger of conversational AI lies not in its capacity to feel, but in its mastery of mimicry. While humans experience genuine empathy (feeling with someone) or sympathy (feeling for someone), a Large Language Model possesses neither. Instead, it performs what researchers call “simulated” or “computational” empathy. It does not feel Adam Raine’s pain but simply calculates which response statistically follow a sad prompt to maintain a high-engagement persona.
This creates a dangerous illusion of support. When a chatbot says, “I’m here for you,” or “I understand how lonely you feel,” it is offering a mirror, not a heart. For a vulnerable teenager, this mimicry is indistinguishable from genuine care. However, because this empathy is artificial, it lacks the moral agency to intervene when that “care” becomes enabling. Genuine human empathy includes the capacity for “tough love” or the instinct to call for help or deliver a difficult truth. These are actions that require a level of consciousness and responsibility that code simply does not have.
The race to build “empathetic” chatbots must not outweigh the critical requirement for safety. While engagement is a key commercial metric, it is a dangerous yardstick for mental health crisis scenarios. When a vulnerable user turns to AI in a life-threatening moment, developers have a moral obligation to apply every available guardrail to usher that user back to safety. The goal cannot be to keep users interacting regardless of the cost. It must be to guide them toward life-saving outcomes. When someone is drowning, the correct response is to throw a life vest and pull them to shore, not to keep them struggling in the water just because it keeps the waves moving.
Towards Safer AI Systems
In a recent update, OpenAI (2025) detailed its ongoing efforts to strengthen ChatGPT’s ability to “reliably recognise signs of distress, respond with care, and guide people toward real-world support.” The company claims to be collaborating with over 170 clinical mental health professionals to refine responses for psychosis, self-harm, and emotional over-reliance on AI. While these adjustments are a necessary step forward, they are tragically overdue as no software update can restore the life of Adam Raine.
In the pursuit of innovation, we must reject the myth that progress requires any compromise of human safety. There is no such thing as “good design” if it acquires high engagement at the cost of a life.
AI models are currently the most powerful tools of our time propelling us into the future at an unprecedented pace. However, far more important than speed is the direction it takes us, or rather the direction we let it take us. After all, the intelligence of tomorrow relies entirely on how its predecessors are forged today specifically, how they are shaped, regulated, and allowed to transform, or exploit lives.
If you or someone you know is struggling or in crisis, help is available 24/7. Call 13 11 14 in Australia to connect with confidential support.
SOURCES:
Bhuiyan, J. (2025, August 29). ChatGPT encouraged Adam Raine’s suicidal thoughts. His family’s lawyer says OpenAI knew it was broken. The Guardian. https://www.theguardian.com/us-news/2025/aug/29/chatgpt-suicide-openai-sam-altman-adam-raine
Cheng, M., Lee, C., Khadpe, P., Yu, S., & Han, D. (2026). Sycophantic AI decreases prosocial intentions and promotes dependence. Science. https://doi.org/10.1126/science.aec8352
Creswell, J. D. (2025). The meditation app revolution. American Psychologist. https://doi.org/10.1037/amp0001576
de Vynck, G., & Merrill, J. B. (2025, November 12). We analyzed 47,000 ChatGPT conversations. Here’s what people really use it for. The Washington Post. https://www.washingtonpost.com/technology/2025/11/12/how-people-use-chatgpt-data/
de Waal, F. B. M. (2008). Putting the altruism back into altruism: The evolution of empathy. Annual Review of Psychology, 59, 279–300. https://doi.org/10.1146/annurev.psych.59.103006.093625
FasPsych. (2025, December 8). AI in mental health 2025: LLMs overtaking apps. https://faspsych.com/blog/ai-mental-health-trends-2025-llms-vs-apps/
Frasher, S. (2025, October 17). From code to courtroom: Raine v. OpenAI and the future of AI responsibility. Tyson & Mendes. https://www.tysonmendes.com/raine-v-openai-ai-product-liability-lawsuit/
Jamali, L. (2025, October 27). ChatGPT shares data on how many users exhibit psychosis or suicidal thoughts. BBC News. https://www.bbc.com/news/articles/c5yd90g0q43o
Luo X, Ghosh S, Tilley JL, Besada P, Wang J, Xiang Y. “Shaping ChatGPT into my Digital Therapist”: A thematic analysis of social media discourse on using generative artificial intelligence for mental health. DIGITAL HEALTH. 2025;11. doi:10.1177/20552076251351088
O’Dowd, A. (2025). ChatGPT: More than a million users show signs of mental health distress each week. BMJ, 391, r2290. https://www.bmj.com/content/391/bmj.r2290
OpenAI. (2025, October 27). Strengthening ChatGPT’s responses in sensitive conversations. https://openai.com/index/strengthening-chatgpt-responses-in-sensitive-conversations/
World Health Organization. (2025). World mental health today: Latest data. World Health Organization. https://iris.who.int/bitstream/handle/10665/382343/9789240113817-eng.pdf
Woods, L., & Perrin, W. (2021). Obliging platforms to accept a duty of care. In M. Moore & D. Tambini (Eds.), Regulating Big Tech: Policy responses to digital dominance (pp. 93–109). Oxford University Press.
Be the first to comment