Building with AI in healthcare: the definitive guide to staying HIPAA compliant

Last updated: 16 July 2026

Healthcare teams are putting AI into the work that touches patients: intake, scheduling, triage, clinical documentation, and the messages that go back out. Almost every one of those steps touches protected health information (PHI), which turns AI adoption into a compliance question before it is a technical one.

This guide is for the builders standing up that AI: the people choosing the tools, wiring them into healthcare workflows, and answering for whether patient data stays protected. It covers what HIPAA requires when AI meets PHI, which providers will sign a business associate agreement (BAA), the patterns that keep PHI safe once it reaches the model, and the email or message your system sends at the end.

One idea runs through all of it. Signing a BAA is not the same as being HIPAA compliant. The contract is the permission slip; the configuration, the access controls, the logging, and every channel PHI travels through are still your responsibility.

Table of contents

• Where PHI meets AI, and what HIPAA requires
• Which AI providers will sign a BAA
• Shadow AI is the exposure you cannot see
• Safe patterns for running LLMs on PHI
• When an AI agent sends PHI
• A vendor-vetting checklist for AI tools
• Where compliant email fits
• The HIPAA Security Rule update is coming
• Frequently asked questions
• Start here

Where PHI meets AI, and what HIPAA requires

Under HIPAA, a covered entity is the provider, plan, or clearinghouse that holds the patient relationship. A business associate is any vendor that handles PHI on the covered entity's behalf. The moment an AI tool receives, stores, or processes PHI, that tool's provider becomes a business associate, and a BAA has to be in place. Sending PHI to a vendor with no BAA is a violation on its own.

PHI in an AI system is broader than the prompt. It includes the text a user types in, the records you pull in for context, anything you store in a vector database, data that your tooling captures, and the model's output when it carries patient details. Each of those is a place PHI may live, and each needs the same protection as the record it came from.

The BAA is the mandatory first step. Every provider that signs one still hands the operational work back to you: minimum necessary access, encryption, audit logging, and breach procedures remain yours to implement. AWS names this the shared responsibility model, and every major provider describes some version of it.

The frontier models most teams reach for by default are not compliant on their own. Compliance is not a property of the model; it is a property of how you deploy it. A frontier model behind a signed BAA and configured correctly can handle PHI, while the same model in a consumer chatbot cannot.

That is why "our AI vendor signed a BAA" and "our application is HIPAA compliant" are two different statements. One is a contract you sign once. The other is an outcome you have to build, configure, and prove on your side. The HHS guidance on business associate contracts spells out what the agreement itself must contain, and it is worth reading before you sign anything. For the foundations of doing this well in email specifically, see HIPAA compliant email: the definitive guide.

Which AI providers will sign a BAA

The good news for healthcare builders: most major AI providers now sign a BAA. The catch is that each one hides it behind a specific plan, setting, or request, and the consumer and mid-tier plans are usually the ones left out.

Here is where the major providers stand as of mid-2026. Terms change often, so confirm the current agreement with each vendor before you rely on it.

A few patterns are worth internalizing. The self-serve surfaces are the API platforms and the cloud providers: OpenAI's API, Amazon Bedrock, Azure OpenAI, and Google Cloud all let you get a BAA without a sales call. The chat products are stricter, and the naming is a trap. ChatGPT Business sounds like a work-ready plan, yet it cannot handle PHI, while ChatGPT Enterprise can. The standalone Gemini API through AI Studio offers no BAA at all, so the same underlying model is off-limits in one place and available through Vertex AI in another.

A note on which models, and where they run. The table above covers the frontier models from the major providers, which is what most teams use. Two other paths change the picture. If you self-host an open-weight model like Llama or Mistral, there is no model vendor to sign a BAA, because you are running the weights yourself, so compliance rests on your own infrastructure and the BAA with your cloud or GPU host. If you use a smaller or emerging hosted provider, do not assume a BAA exists, and vet it with the checklist later in this guide.

For the step-by-step on getting each provider's BAA turned on, and the fine print on which features fall outside coverage, read the companion piece, how to get a BAA for your AI stack.

Shadow AI is the exposure you cannot see

The signed, sanctioned stack is the part you control. The harder problem is the AI you never approved.

In Paubox research, 85% of healthcare IT leaders suspect staff are using unauthorized AI tools, and only 26% have visibility into that use (Paubox, Shadow AI Is Outpacing Healthcare Email Security, 2025). A clinician pasting a patient summary into a personal chatbot account is sending PHI to a vendor with no BAA, no encryption guarantee, and often a data-retention policy that trains on the input.

For a builder, the takeaway is practical. Give people a sanctioned, BAA-backed way to do the thing they are already doing, and make it the path of least resistance. Shadow AI thrives when the compliant option is slower than the shortcut. If your intake team pastes notes into a consumer model because the approved workflow is clunky, the answer is a smoother approved workflow that beats the shortcut on convenience. Another policy memo will not do it. For more on surfacing and managing it, see why most healthcare IT leaders suspect shadow AI.

Safe patterns for running LLMs on PHI

A BAA lets PHI reach the model. These patterns keep it safe once it does.

Send the minimum necessary. HIPAA's minimum necessary standard applies to building with AI in healthcare, too. Strip identifiers the model does not need to complete the task, and pass a record ID your own system can resolve instead of a full patient profile when the task allows it.

Scrub PHI from logs and traces. Prompt logs, LLM tracing (the detailed record of each prompt, response, and tool call that observability tools keep for debugging), and error monitoring quietly capture whatever you send the model. A crash report that includes the failing prompt can ship a patient record to a third-party monitoring tool that never signed a BAA. Redact PHI before it reaches those systems, and confirm any observability vendor in the path is in scope.

Treat your vector store as a PHI store. Embeddings are the numeric representations of text a model produces so software can search by meaning, and a vector database stores them. Embeddings generated from PHI are still PHI, so the vector database that holds them, and any retrieval-augmented-generation index (which pulls stored records into the prompt as context), needs the same BAA, encryption, and access controls as your primary database.

Keep PHI out of the features a BAA does not cover. Provider BAAs are feature-specific. Anthropic's covered-services list, for example, excludes several tools even on a HIPAA-ready org, and consumer Claude plans are out entirely. Before you route PHI through a feature, confirm it is inside the signed scope, not adjacent to it.

Constrain what an agent can do with untrusted input. An agent that reads an incoming message and can also call tools is exposed to prompt injection, where hostile text in the input tries to redirect what the agent does. Limit the tools an agent can reach, scope its data access to the minimum, and check its actions before it touches a system that holds PHI.

Log access. You need an audit trail of who and what touched PHI, including your automated jobs and agents. Build the logging in from the start, because reconstructing it after an incident is far harder than capturing it as you go.

Turn off training, and match retention to the workload. Where the use case allows, disable training on your inputs and turn on zero-data-retention (a mode where the provider does not keep your inputs or outputs after it finishes processing them). Watch the tradeoffs: Anthropic's covered models, for instance, require a 30-day retention window and cannot run in zero-data-retention mode, so the right setting depends on the workload. Read the provider's data-handling documentation and match your configuration to what you committed to in your own privacy notices.

This is the same discipline HIPAA has always demanded, just applied to a new kind of vendor.

When an AI agent sends PHI

Here is the step teams miss. A BAA with your model provider covers the model, not what your application does with the output.

The moment your app or agent emails a patient a result, sends a reminder, or delivers a visit summary, PHI leaves your system through a channel the model's BAA never mentioned. That outbound message is its own compliance question, and it is the one an auditor will ask about.

This is where the Paubox Model Context Protocol (MCP) server fits. It lets Claude and other AI agents send HIPAA compliant email directly via the Paubox Email API. It works with Claude, Claude Desktop, Claude Code, Cursor, Windsurf, and custom agents built on the MCP spec. The server can also reach Paubox Forms when an agent needs to collect data.

The security design is the part that answers the fear most teams have about agents. The API key stays with the MCP server process and is never shared with the model. It does not appear in a tool call or in the model's context. An agent can send a compliant, encrypted message without ever holding the credential that sends it.

In practice the flow is straightforward. Your agent decides a message needs to go out, calls send_secure_email with the recipient and body, and the server signs the request with the stored credential and passes it to the Paubox Email API, which encrypts and delivers it. The model orchestrates the send without seeing the key or handling the transport. For a focused walkthrough of this setup, see how to send PHI from an AI agent without exposing it.

For the deeper argument on why agent access to PHI needs guardrails before the agent acts, see AI agents are reaching into patient data faster than the rules can keep up.

A vendor-vetting checklist for AI tools

Paubox is itself a business associate, which means covered entities vet us the same way you should vet any AI vendor that will touch PHI. Use these questions on every tool in your stack.

• Will the vendor sign a BAA for the exact plan and configuration you intend to deploy, not a different tier?
• What data leaves your environment, and how is it secured in transit and at rest?
• Which specific features are inside the signed scope, and which are excluded?
• Does the vendor retain or train on your data, and can you turn that off?
• Was the infrastructure built for healthcare, with certifications like HITRUST and SOC to show for it?

If a tool cannot or will not sign a BAA, it does not get PHI. The Health Sector Coordinating Council's AI Cybersecurity Governance Framework Implementation Guide includes vendor contractual-language and inventory tools built for exactly this review.

Where compliant email fits

The outbound layer deserves its own attention, because the general-purpose email tools most apps reach for leave a gap in one of two ways.

Some will not sign a BAA at all. Postmark states plainly that it cannot sign one, and Twilio SendGrid says it is not a HIPAA eligible service and offers no encryption for PHI. Others will sign a BAA but still leave encryption up to you. Amazon SES is "HIPAA eligible" under the AWS agreement, yet it does not enforce encryption by default, so a misconfiguration can put PHI out in the wild in plain text. We tested exactly that failure in Amazon SES can send PHI in plaintext even with Require TLS.

Paubox closes both gaps. Paubox signs a BAA with every customer and enforces TLS 1.2+ encryption by default across all of its products, from the Paubox Email Suite to the Paubox Email API. Messages are compliant by default. The infrastructure is HITRUST certified and keeps an audit trail through Paubox Flow Logs.

For the teams building on it, the API ships with official SDKs in ten languages, from Python and Node.js to Go, Java, Ruby, and Rust, plus a command-line tool. If you are already sending through SendGrid, SES, Mailgun, Postmark, or Resend, there are direct migration guides, so moving the transactional layer (the automated email your app sends, like results and reminders) to a compliant one is a small change rather than a rebuild. You can start for free.

The HIPAA Security Rule update is coming

The proposed overhaul of the HIPAA Security Rule, which would make controls like encryption, multi-factor authentication (MFA), and network segmentation explicit requirements, was postponed in 2026, with the final rule now expected in July 2027. The requirements are still coming; the delay only buys time to prepare.

Teams that build to those controls now will not scramble when the timeline firms up. If you are standing up new AI infrastructure this year, treat mandatory encryption, MFA, and segmentation as design requirements rather than future homework. Building on a foundation that already encrypts by default and signs a BAA on every plan means the rule lands as a confirmation, not a fire drill. For the full breakdown of the timeline and what it will require, see the HIPAA Security Rule overhaul is delayed to 2027.

Watch: Paubox hosted an on-demand webinar breaking down these Security Rule changes and what they mean for healthcare teams. Watch it here.

Frequently asked questions

Is a signed BAA enough to be HIPAA compliant?

No. A BAA is a required contract, but compliance also depends on how you configure the service, control access, log activity, and protect PHI everywhere it moves. The vendor's BAA covers the vendor's part; the rest is yours.

Which AI providers will sign a BAA?

Most major API and cloud providers do, under specific plans: Anthropic, OpenAI, Amazon Bedrock, Google Cloud, and Microsoft Azure OpenAI. The standalone Gemini API in AI Studio and ChatGPT Business do not. Confirm the exact plan and current terms with each vendor before you rely on it.

Can I use ChatGPT with PHI?

Only on ChatGPT Enterprise or Edu, or through the OpenAI API with a BAA in place. ChatGPT Business and the free tier are not eligible.

Are embeddings and vector databases considered PHI?

Yes. Embeddings generated from PHI are still PHI, so the vector store and any retrieval index built on patient data need the same BAA, encryption, and access controls as your primary database.

Do I need a separate BAA for the email my application sends?

Yes. Your model provider's BAA covers the model, not the messages your app sends. Any email carrying PHI needs its own BAA-backed, encrypted path, which is what the Paubox Email API and MCP server provide.

Does Paubox de-identify PHI?

No. Paubox does not de-identify patient data, meaning it does not strip out the identifiers that tie a record to a person. It encrypts PHI in transit and at rest, signs a BAA with every customer, and keeps an audit trail, so the data stays protected rather than stripped.

Start here

Getting the BAA in place is the permission slip. Building the rest of the workflow so PHI stays encrypted, logged, and minimized at every step is the actual work, and it pays to get it right before the first patient record moves through an AI system. Done well, that work is a moat, not a tax.

Map every point where PHI enters, moves through, and leaves your AI application. Put a signed BAA behind each vendor, keep PHI inside the covered scope, log the access, and route every outbound message through a channel that encrypts by default. For the email layer, the Paubox Email API and MCP server give an AI application a compliant way to send from day one.