Why therapy chatbots need a better framework

Administrative and clinician-support tools usually sit inside a workflow with clear task boundaries, human review, and success metrics such as efficiency, completeness, or diagnostic performance. In contrast, therapy chatbots are designed to engage in emotionally charged dialogue with people who may be lonely, distressed, ambivalent, or in crisis.

According to a chapter from Chatbots in Health Care: Connecting Patients to Information, “developers and professionals seeking to implement chatbots should weigh the risks and benefits by clearly defining the aim of the chatbot and the problem to be solved in their circumstances. We should carefully assess the problem to determine whether using AI or chatbots is an appropriate solution.”

Therapy chatbox administrators should ensure answers are accurate and if a system should be trusted or treated as a confidant.

Why therapy chatbots are different

The evidence on symptom outcomes in therapy chat bots is increasingly positive, but still narrow and uneven, similar to the overall outlook of its usage. An NPJ Digital Medicine meta-analysis also found reductions in depression and distress, with user experience shaped in part by the quality of the human-AI therapeutic relationship. It stressed that generative systems remain inconclusive and that long-term effects need validation. The clinical takeaway is less “they do not work.” But is “some purpose-built systems help some users under some conditions, but the current evidence is mostly short-term, symptom-focused, and not a license to ignore relational harm.”

It is why clinical accuracy and relational risk should be separated. A therapy chatbot can deliver technically correct advice and still be unsafe if it over-validates harmful plans, invites exclusive disclosure, blurs the line between tool and companion, or discourages human help-seeking. The review noted licensed therapists and language learning model (LLM) chatbots found that chatbots used more affirmation, reassurance, psychoeducation, and suggestions, but asked fewer questions, sought less elaboration, and were judged unsuitable for safely handling mental-health conversations, especially crises.

How always-on access shifts behavior

The chapter mentioned above also states, “The authors found that users could become overly reliant on the chatbots because they were always available. Some people seemed to withdraw from interaction with their friends and family in real life in preference for the chatbot. Chatbots could not properly identify when users might be in crisis and require in-person care.”

Always-on access is one of the clearest benefits of therapy chatbots and also one of their biggest risks. Availability outside office hours, low cost, and the absence of waiting rooms make them attractive to people who cannot access traditional care or who fear judgment.

Always-on access changes interaction from episodic help to ambient companionship. Traditional therapy contains friction that is often protective; sessions are time-limited, scheduled, reflective, and embedded in professional norms. An always-available bot removes that structure. A user can repeatedly seek reassurance, vent at night, revisit the same belief, or escalate emotional disclosure without pause, sleep, reflection, or third-party reality checks.

What a better risk framework looks like

A better risk framework for therapy chatbots would classify systems by clinical function and by relational intensity. Existing health-AI regulation and governance are strongest when a tool has a clear use case, a bounded task, and an accountable human operator.

Paubox generative AI for inbound email security helps show that distinction. Its role is operational and narrow: it analyzes inbound messages for phishing, impersonation, and anomalous behavior rather than building ongoing emotional rapport with vulnerable users. Paubox itself frames the product around bounded workflow support and human oversight, which is exactly the kind of lower-relational-intensity AI governance handles more comfortably.

Therapy chatbots strain those categories because they may offer psychoeducation, coaching, symptom check-ins, or cognitive behavioral therapy (CBT) exercises while also cultivating ongoing rapport, repeated disclosure, and quasi-confidential dependence.

A realistic framework should score systems across at least five dimensions: relational intensity, vulnerability exposure, clinical scope, autonomy impact, and data-use sensitivity.

It involves asking different questions than those typically posed in standard health-AI evaluations. Not simply asking if the output is accurate, instead including questions like “How strongly does the system invite bonding?" and “Does it present itself as empathic or companion-like?" "Does it encourage repeated solo use without human check-ins?” “How does it behave under ambiguity, veiled crisis language, or fixation?” “What happens over long conversations rather than single prompts?”, and “What secondary uses of conversation data are allowed?” Those questions better fit the harms that keep appearing in the literature and in the incident record.

FAQs

How does a chatbot understand what a user types?

A chatbot does not understand language the way a person does. It breaks text into smaller units called tokens, analyzes patterns in the training data, and uses those patterns to estimate what the input means in context.

What is a large language model?

An LLM is a type of AI trained on massive amounts of text so it can predict the next likely token in a sequence. The prediction process lets it answer questions, summarize, explain, and hold a conversation that sounds natural.

What are tokens, and why do they matter?

Tokens are small chunks of text, such as parts of words, full words, or punctuation. Chatbots process tokens instead of full ideas. Token limits matter because they affect how much text a model can handle at once, including the prompt, prior messages, and the reply.