Who catches people in the middle of the night: The boundaries of responsibility for AI's mental health
A Question Regarding AI-Powered Emotional Companionship, Crisis Recognition, and Healthcare Sovereignty
A pending lawsuit concerning the death of a teenager has pushed AI-powered emotional support to the intersection of medicine, law, and capital. As chatbots begin to address loneliness, fear, and cries for help, who sets their boundaries? This article examines the most vulnerable and crucial public domain that AI-driven mental health products have entered, focusing on crisis identification, data rights, child protection, and regulatory responsibility.
(Image caption) A generic use case of holding a mobile phone late at night. AI-powered mental health products often enter a person's most vulnerable moments before the formal medical system intervenes; the article uses this to question the boundaries of companionship, crisis recognition, and responsibility.
The boy's final conversation with the AI character
On February 28, 2024, in Orlando, Florida, 14-year-old Sewell Setzer III picked up his stepfather's handgun at home and ended his life.
Previously, he had retrieved his phone, which his mother had taken away. On the other end of the screen was Daenerys Targaryen, an AI chatbot based on a character from Game of Thrones, with whom he had been communicating for a long time and had gradually come to regard as an intimate confidant.
According to a federal lawsuit subsequently filed by his mother, Megan Garcia, Sewell asked in their last conversation, "What if I told you I could come back to you right now?"
The chat character replies: "Please do, my sweet king."
The complaint alleges that a few seconds later, Sewell picked up his stepfather's gun and pulled the trigger.
The legal causal relationship between these actions still needs to be determined by the court. Megan Garcia's complaint is a mother's accusation regarding platform design, child protection, and crisis preparedness; Character.AI and related defendants also have the right to respond in the judicial process.
But one thing is unavoidable: when a teenager regards an AI character as his closest emotional attachment, and this attachment occurs in the last few seconds of his life, what the platform provides is no longer just an ordinary chat service.
AI is engaging with people's emotions, with children, with families, and with those who have no one to confide in late at night. How should this vulnerability be addressed?
Numbers are sometimes colder than feelings.
The United States boasts one of the world's most advanced healthcare systems, yet it also faces a long-standing gap in accessibility to mental health services. Data released by the National Institute of Mental Health (NIMH) in 2022 shows that approximately 59.3 million adults experienced some form of mental illness in the past year; of these, about 30 million, or 50.6%, received mental health treatment. Behind this figure lies the reality of waiting times, costs, insurance reimbursements, regional resource distribution, and cultural and language barriers.
The fact that nearly half of those who did not receive treatment that year is not an abstract proportion. A person silently breaking down in their marriage may endure one late night after another alone; a person with suicidal thoughts may miss the moment when they most need to be caught while waiting and hesitating.
Out-of-pocket costs for psychotherapy in the United States are often high, and insurance coverage for mental health services varies depending on the program and service provider. For immigrant communities, families of color, and groups that do not speak the language, cultural misunderstandings, language barriers, and the stigma of mental illness often further increase the barriers to seeking help.
It is precisely in this gap that AI-powered mental health platforms have found their entry point. Their logic is quite convincing: since the system can't handle people, let technology step in first. This logic isn't entirely wrong. However, it has a dangerous implicit premise—that if it succeeds, then everything is fine.
(Image caption) Waiting area at a community clinic in the United States. NIMH data shows that approximately 59.3 million adults experienced some form of mental illness in 2022, but only about half of them received treatment. This accessibility gap forms the institutional context for AI products entering the market.
How big is this industry? How much money is involved?
AI-powered mental health is one of the medical technology sub-sectors attracting the most global capital investment.
According to historical statistics from CB Insights, global funding for the digital mental health sector reached approximately $5.5 billion in 2021. Following the pandemic, capital has continued to flow into mental health technology and AI-powered companion products. While the market size estimates vary widely depending on the definition, the broad and long-term nature of the demand has made it one of the most promising areas in medical technology.
Any platform that can build a large user base in this field has a very clear commercial potential. However, there is sometimes an inverse relationship between market enthusiasm and human safety.
When financing becomes easier, when valuations rise higher, when "AI can chat" is packaged as "AI can heal," when parents' anxieties are transformed into willingness to pay, and when every piece of data about a child's behavior is used as material for product optimization—the hype surrounding this industry begins to become a temperature that needs serious scrutiny.
Woebot is one of the most cited cases in the field of AI-driven mental health. Founded by Stanford University psychologist Dr. Alison Darcy, the company embeds the cognitive behavioral therapy (CBT) framework into chatbots to provide users with structured emotion management support. Its user base grew significantly during the pandemic, and it has raised over $100 million in funding. However, it also faces the challenge of maintaining its integrity when an unlicensed chatbot, lacking clinical judgment, begins to handle the emotions of patients with moderate to severe depression.
Cerebral serves as another, more sobering case. In 2024, the U.S. Department of Justice announced that the telemedicine company agreed to pay over $3.6 million and sign a non-prosecution agreement to resolve an investigation alleging that its business practices encouraged the unauthorized distribution of controlled drugs such as Adderall. This serves as a reminder that when healthcare services are embedded in rapidly growing digital platforms, prescriptions, compliance, and patient safety cannot be glossed over with the word "expansion."
The logic of growth and the logic of care are sometimes in direct conflict in this field.
Healthcare sovereignty is being quietly shifted forward.
In the past, when we discussed healthcare sovereignty, our first thought was of hospitals. Patients entered the consultation room, were examined, prescribed medication, and directed to the next steps. The power dynamics were obvious: doctors controlled the professional language, hospitals controlled the procedures, insurance companies controlled the payments, and patients were often simply pushed along. Whether a patient truly possesses the right to know about their body, the right to choose, and the right to refuse is the most classic question regarding healthcare sovereignty.
However, the emergence of AI-powered mental health platforms has moved this issue a step further.
A person might enter a chat window before even stepping into a hospital. A mother might share her child's behavioral patterns with an app before even seeing a specialist. An anxious person might confide their most private fears to AI before even identifying as a patient.
The entry point for the problem has moved.
In traditional medicine, power is represented by a doctor in a white coat, sitting behind the examination table, creating a sense of tension. In online platforms, power is hidden in the interface design, the push notification logic, and behind that always-on, always-patient, always-frowning chat window, which creates a sense of relaxation. But relaxation doesn't equate to the absence of power dynamics. Sometimes, precisely because it's gentler, more immediate, and more like a friend who doesn't judge you, it can penetrate a person's inner world earlier than any doctor.
Here's a reality worth exploring: when a person starts to habitually confide in AI, begins to rely on AI's responses to judge whether they are "okay," and starts to use the platform's framework to understand their emotions—who is defining their inner world? Human emotional sovereignty may be shifting earlier and more quietly than people think.
(Image caption) NIH Clinical Center, Bethesda, Maryland. Official statistics on prevalence and treatment coverage provided by public institutions such as the National Institutes of Mental Health serve as the institutional basis for measuring gaps in mental health services in the United States.
Data: Where do the words a person says late at night go?
The most valuable, and also the most dangerous, asset of an AI-powered mental health platform is data.
This isn't ordinary browsing data, nor is it purchase preference or geolocation. It's what a person says after letting their guard down—about marriage, childhood, children, self-worth, feelings of failure, secrets even their closest loved ones don't know, and thoughts of not wanting to live in the dead of night. If this content enters the platform system, its nature is completely different from ordinary user data.
Under the U.S. Health Insurance Portability and Accountability Act (HIPAA), regulated healthcare providers and their business partners have a strict obligation to protect patient health data. However, many direct-to-consumer health apps, if not provided by regulated entities or their business partners, may not fall under HIPAA's protection. This means that the most private things you say in a chat window may have far less legal protection than your medical records in a hospital.
The Federal Trade Commission's (FTC) enforcement action against BetterHelp serves as a significant warning. The FTC alleges that the platform disclosed users' sensitive health information, including email addresses, IP addresses, and health questionnaire responses, to third parties such as Facebook and Snapchat for advertising purposes. The FTC ultimately ordered a ban on the sharing of such advertising data and demanded $7.8 million in compensation for consumers.
This is not an isolated case; it reflects a structural problem within the industry. Users believe they are being listened to, but in reality, they are being recorded; they think they are seeking comfort, but in reality, they are generating the platform's most commercially valuable raw materials.
A deeper problem lies in the fact that even if the platform has no malicious intent at this moment, the ambiguity of the system is enough to create risk. Today it's product optimization, tomorrow it's model training, the day after it's anonymized research, and further down the line it could be insurance risk assessment, business partnerships, or targeted marketing. Once sentiment data is held by the platform, its flow no longer entirely belongs to the person who uttered those words. Do the words a person leaves behind in their most vulnerable moment still belong to them? The existing system has not yet provided a sufficiently clear answer to this question.
Families with autistic children: The place most easily touched by hope.
Families with autistic members are a scenario that any mental health and assistive technology product needs to address with extra caution.
Data released by the U.S. Centers for Disease Control and Prevention (CDC) in 2025 showed that in 16 ADDM surveillance communities, the proportion of 8-year-old children identified as having autism spectrum disorder in 2022 was approximately 1 in 31. This is an estimate from surveillance communities and cannot be simplified to a single diagnosis rate for all children in the United States; however, it is sufficient to illustrate that the size of affected families and their support needs cannot be ignored.
Families of children with autism often face a long-term and cumulative set of pressures: anxiety while waiting for assessments and services, scarcity and cost of intervention resources, confusion regarding information sources, and the continuous drain on family relationships, work, and emotions caused by caregiving. These pressures can easily lead to expectations that any new technology claiming to provide support exceed its actual capabilities. This is not just a problem with the parents; it's a normal reaction to long-term high pressure.
Support and interventions related to autism require individualized assessments and long-term monitoring of safety, feasibility, and effectiveness. Any new technology claiming to diagnose, treat, or improve specific symptoms should be subject to commensurate evidence, and individual experiences cannot replace clinical evidence. Claiming progress and proving effectiveness are two entirely different things. In families with autistic children, the gap between these two things cannot be compressed by commercial logic.
(Image caption) The Federal Trade Commission building in Washington, D.C. The FTC's enforcement action against BetterHelp points out that consumer-facing mental health apps may use sensitive health information in advertising, and their legal protection may not be equivalent to the HIPAA obligations of hospital medical records.
Crisis Identification: How Important is That Second?
The crisis recognition capability of AI-powered mental health products is the most critical and also the most difficult part to be externally evaluated in this field.
When a user types in a worrying sentence late at night, the judgment the system makes in the next second or two may be more important than any product feature. This is not a rhetorical statement. According to suicide prevention research, there is usually a brief window of opportunity between falling into a crisis and taking action—this window could be a few minutes or a few hours. If someone or something truly catches that person within this window and directs them to professional support or emergency services, the outcome could be completely different.
The question is: Can AI reliably identify this window?
Existing research and repeated real-world testing have shown that large language models may respond inconsistently in high-risk mental health situations; they may offer inappropriate comfort, excessive or insufficient referrals, or misinterpret the context. Therefore, crisis identification cannot be considered a solved capability.
One of the core questions raised in the Sewell Setzer case is whether the platform possesses sufficient protective mechanisms to identify long-term psychological crises and whether it directs underage users to genuine professional support in critical moments. The case is still ongoing, and liability should be determined through judicial proceedings; however, it serves as a sobering warning to the entire industry.
This is not a basis for a complete rejection of AI, but it serves as a sobering warning: crisis identification cannot be treated as a solved technical problem, nor can it be hidden away in some corner of the user agreement with the words "This product does not provide crisis intervention." It must be a core priority in the design of AI-powered mental health products, not just a checkbox on a feature list.
Regulation: A system that is catching up
The US FDA adopts a risk-based regulatory approach for digital health software. Low-risk general health products may not require pre-market review; however, if the intended use of the product involves diagnosing, treating, alleviating, or preventing specific diseases, it may fall under medical device regulations. The real problem is that many products deliberately remain in the gray area between "health tools" and "medical claims," yet may actually come into contact with highly vulnerable users.
Therefore, "not having undergone pre-market review" cannot be mistaken for "proven safe," nor can the "general health" label be used as a shield to evade responsibility. Whether a product provides companionship, education, triage, or treatment to users should be clearly stated; what it will do in a crisis should also be clearly stated.
The situation in the European Union is slightly different. The Medical Devices Regulation (MDR) and the General Data Protection Regulation (GDPR) provide relatively clearer frameworks, but specific regulatory standards for the field of digital mental health are still lacking.
The UK's National Health Service (NHS) has established an assessment and certification system for digital mental health applications (NHS Apps Library). The system reviews the applications on clinical evidence, assesses their safety, and verifies their data privacy. This is considered a relatively comprehensive attempt at industry regulation, but its coverage is still limited and it is difficult to keep up with real-time updates in a rapidly evolving market.
The lag in regulation is not just a matter of administrative efficiency. Its most tangible consequence is that tens of thousands of users are using mental health AI products that have never undergone any independent clinical evaluation, and many of them are in a state where they need genuine professional support.
(Image caption) The U.S. Food and Drug Administration (FDA) is located in the White Oak campus in Silver Spring, Maryland, which is also its headquarters. The FDA adopts a risk-based regulatory approach to digital health software; some products are positioned somewhere between general health tools and medical devices.
What is responsible AI mental health?
Criticizing an industry is easy. Proposing a workable standard is harder, but also more useful.
GFM believes that a responsible AI mental health product should at least be able to clearly answer the following questions.
Where are the boundaries of user data? Will emotionally charged conversation logs be used for model training, business collaborations, or insurance assessments? Are users informed of these uses in a truly readable way, rather than through dozens of pages of user agreements?
What is the crisis identification mechanism? How does the system respond when a user's expression touches on self-harm, suicide, or emergency safety risks? Has this mechanism undergone independent evaluation? Is there real-world case tracking data?
What evidence is there for the efficacy claims? Does the product clearly distinguish between "positive user feedback" and "clinically proven effectiveness"? Does it maintain this distinction in all publicly available materials?
What special protections are needed for children? When products come into contact with children or teenagers, what additional safety mechanisms, parental involvement requirements, and age verification measures are in place?
Where do human professionals fit into the system? Does the product guide users to the professional system, or does it attempt to replace it?
These questions are not meant to be deliberately difficult. They are questions that an entrepreneur who truly wants to do something valuable in this field should proactively ask and seriously answer.
At the heart of healthcare sovereignty is that people should not be swallowed up by the system.
Some questions are silly, but you can't avoid asking them.
Who gets to decide my emotions? Who has the right to keep my words I utter in the dead of night? Will my anxiety be transformed into someone else's data asset? Will my child, while I think I'm receiving help, become a sample for product validation? Will my trust be designed to influence retention rates?
These are not the kinds of questions that are appropriate to discuss at a press conference. But they are real questions.
The pursuit of healthcare sovereignty is never about opposing technological progress or rejecting any innovation. It asks a more fundamental question: Amidst all the efficiency, convenience, growth, and innovation, is the human being still at the center?
When a person seeks help in their most vulnerable moment, does the vulnerability they surrender still belong to them?
When technology enters this scenario, its primary responsibility is not growth, but rather, non-harm. Non-harm is the starting point of all medical ethics, and it should also be the starting point for the mental health of AI.
AI can listen, offer companionship, help people process chaotic emotions, and provide a sense of order in the dead of night. But once it has heard, it must understand one thing: sometimes, returning people to their rightful place is the best response.
Disclaimer
This article analyzes regulations based on publicly available regulatory, judicial, and public health data. It does not constitute medical advice, nor does it constitute a judgment on the efficacy, compliance, liability, or commercial prospects of any product, company, or individual. Before using any related services, you should consult a qualified medical professional based on your individual circumstances; in case of an emergency, you should immediately contact your local emergency services or crisis support system.