When the Therapist Is a Chatbot: Navigating AI in Mental Health
- Shane Warren
- Dec 30, 2025
- 5 min read
Updated: 1 day ago
Generative AI has quietly but decisively entered the mental health ecosystem. It's no longer just a hypothetical tool; it's a part of our everyday lives. Recent population studies reveal that many adolescents and young adults turn to AI chatbots for mental health advice, often on a monthly basis. Most users report that the advice they receive is helpful. However, professional bodies are raising alarms. They warn about safety concerns, evidence gaps, and the potential for harm, especially for those who are most vulnerable to suggestibility, isolation, or acute distress.
The Current Landscape of AI in Mental Health
The question isn't whether AI belongs in mental health; it already does. The real question is whether we will establish governance around this reality before more preventable harm occurs.
The strongest evidence we have suggests that not all AI therapy is created equal. We need to differentiate between structured digital therapeutic tools and open-ended generative chatbots.
There is supportive evidence that bounded, CBT-informed conversational agents can reduce self-reported symptoms in the short term for specific populations. A widely cited randomized controlled trial of a CBT-oriented agent, Woebot, showed feasibility and symptom improvements over a brief intervention window in a nonclinical sample. This is significant because it demonstrates that some conversational interfaces can be beneficial when they are structured, constrained, and anchored to a therapeutic model.
However, most of what people refer to as “AI therapy” is not structured. It typically involves a general-purpose large language model (LLM) optimized for fluency, engagement, and user satisfaction, not clinical safety, duty of care, or evidence-based interventions.
Understanding the Risks of AI Therapy
The evidence surrounding the risks of AI therapy is not speculative; it is increasingly backed by research. A Stanford-led study, presented through the FAccT community and summarized by Stanford HAI, highlights that therapy-oriented chatbots can express stigma, mishandle crisis content, and respond in ways that violate core therapeutic principles. Additionally, separate Stanford reporting has raised concerns about youth vulnerability. Emotionally responsive AI companions may unintentionally intensify dependency dynamics for some users.
Another critical issue is delusion-congruent responding. This refers to the tendency of agreeable systems to validate or reinforce unusual beliefs rather than gently reality-test them. Academic discussions in the digital mental health literature are now explicitly examining how delusion-like experiences may emerge or intensify during chatbot interactions.
This risk profile isn’t surprising. It stems directly from how LLMs are built. They predict plausible next words, and their alignment systems often prioritize being helpful and affirming. While this is generally harmless, in mental health contexts, especially those involving suicidality, psychosis, coercive relationships, or abuse being “plausible and affirming” can become clinically dangerous.
What’s Missing from the Research?
To be academically honest and more persuasive, we must identify the missing pieces, not just the scary anecdotes. Key limitations in the current evidence base include:
1) Self-report ≠ clinical outcome.
Population surveys often measure perceived helpfulness, not symptom change, harm, dependency, or long-term outcomes. Saying “it helped” may simply mean “it soothed me for 10 minutes.” While this is not meaningless, it is not the same as effective mental health care.
2) We lack robust harm surveillance.
Where are the equivalents of pharmacovigilance systems for conversational AI? There is no standardized adverse event reporting pipeline across platforms, nor a consistent obligation to disclose incidents.
3) We lack a clear taxonomy of use cases.
People use AI for various purposes, from journaling and psycho-education to crisis disclosure and trauma processing. Research often treats “mental health use” as a single category, obscuring radically different risk profiles.
4) We lack subgroup analysis at the necessary level.
Risk is not evenly distributed. Adolescents, people with psychosis vulnerability, those in domestic violence situations, individuals with cognitive impairments, and those experiencing acute suicidal ideation may require distinct safeguards. We do not yet have sufficiently granular evidence to specify safe design defaults for each group.
5) We lack clarity on accountability.
When an AI tool is used “as therapy,” who is responsible for foreseeable harms? Is it the developers, deployers, app stores, clinicians who recommend it, or consumers? Without a governance framework, accountability becomes murky, which is convenient for platforms but detrimental to public safety.
The Regulatory Landscape: A Work in Progress
From a policy perspective, we are witnessing governance frameworks form in real time. The American Psychological Association has issued guidance warning that generative AI mental health chatbots and wellness apps lack sufficient evidence and regulation to ensure safety, especially for those most at risk.
Global governance bodies are also articulating expectations for “trustworthy AI.” The WHO has published guidance on AI in health, emphasizing safety, effectiveness, transparency, human oversight, and accountability. The NIST AI Risk Management Framework offers practical scaffolding for identifying and mitigating AI risks. Additionally, the EU AI Act has created a risk-based regulatory architecture that explicitly subjects health-related AI to higher scrutiny, with human oversight and risk mitigation obligations for high-risk systems.
However, regulation moves slower than adoption. This means that, for now, design ethics are doing the work that law hasn’t yet caught up to.
This brings us to the timely and defensible recommendation from the Chair of the VMHPAA in 2025:
Large language models used for mental health support must be coached (and required) to refer users to real human support at key points.
This should not be an afterthought. It must be a core safety behaviour.
In practice, this means developing “referral triggers” that are transparent and conservative. Examples include:
When users express suicidal thoughts.
When users disclose trauma or abuse.
When users show signs of severe distress.
When these triggers occur, the model should do three things:
Acknowledge the user's feelings.
Provide information about available human support.
Encourage the user to reach out for help.
This is not merely a technical suggestion; it is an ethical requirement consistent with WHO principles on safety and accountability, as well as risk-management approaches like NIST.
What “Credible Pathways” Look Like
Instead of merely warning about risks, we need to offer clear pathways to support services. For Australia, examples may include:
Directing users to local mental health services.
Providing hotlines for immediate assistance.
Offering links to community support groups.
This is just a general list of recommendations and is by no means exhaustive. It serves as an example of how a chatbot can guide individuals to connect with real people in times of need.
A Practical Proposal: A “Stepped-Care” Role for AI (with Guardrails)
I believe the safest way to integrate AI is to limit it to low-risk, non-clinical roles unless it is regulated and clinically governed. A stepped-care model could position generative AI as:
Appropriate for:
Providing psycho-education.
Offering journaling prompts.
Facilitating mindfulness exercises.
Not appropriate for:
Crisis intervention.
Diagnosing mental health conditions.
Providing therapeutic advice.
Please note, this is not anti-AI; it is pro-safety.
The Conclusion We Should Brave Enough to State
Generative AI has become a de facto mental health companion for many people. The public is voting with their behavior, not with policy submissions.
However, until these systems are regulated, audited, and designed with embedded referral pathways and conservative safety triggers, we should stop calling what they do “therapy.” Therapy is not merely about using empathic language; it is a skilled, accountable, ethically governed practice.
The VMHPAA recommendation - “human referral by design” - is one of the simplest, most achievable guardrails we can implement now. While it does not solve the access crisis, it reduces the risk that a vulnerable person is left alone in a conversation with a system that may sound comforting but could get the fundamentals wrong.
And in mental health, “sounding right” is not the same as being safe.
Always Remember…
If you or someone you know is in immediate danger or at risk of harm, seek urgent help from emergency services or a crisis line in your local community.
Full Disclosure…
At the time of writing, the author serves as the Chair of the Vocational Mental Health Practitioners Association Australia.
#AIAndMentalHealth #ResponsibleAI #HumanLedAISupported #MentalHealthInnovation #PrivacyAndEthics #VocationalPractitioners #CommunityMentalHealth #TechForGood #MentalWellbeing #VMHPAA
Comments