Seven in ten healthcare leaders (70%) say their organization has found an AI-built tool that handled protected health information (PHI) without IT or security knowing, according to a July 2026 Paubox survey of 151 healthcare leaders. Governing AI-assisted development starts with finding those tools, because a tool that handles patient data falls under HIPAA whether or not IT has it on a list.

The findings come from The CISO's guide to healthcare AI governance, Paubox's report on how healthcare organizations govern the software staff build with AI coding assistants.

AI-built tools already handle patient data

The report focuses on AI-assisted development: healthcare staff using AI coding assistants to build tools, scripts, automations, and integrations. Every organization in the survey handles PHI electronically and uses AI coding assistants.

Most healthcare leaders (85%) say their organization's AI-built tools have handled, stored, or transmitted PHI. Only 9% say none have, and 6% don't know.

Many of these tools are also in front of patients. In The AI Build Benchmark, Paubox's report on adoption from the same survey, 80% of healthcare leaders said their organization had deployed patient-facing AI-assisted tools into production.

Once a tool handles PHI, HIPAA applies to it like any other system that handles patient data. The source of the code doesn't change that.

Strong controls still miss unlisted AI-built tools

Healthcare leaders describe solid controls for AI-assisted development. Nearly nine in ten (87%) say a security, privacy, or compliance professional always or usually reviews PHI-handling AI code before it's deployed, and 83% say all or most of their AI coding assistant vendors have business associate agreements (BAAs).

Those controls share a dependency. A pre-deployment review needs someone to submit the tool. A BAA check needs someone to report the outside connection. A vendor BAA covers only the assistants under contract.

So the 70% figure sits beside high review rates in the same organizations. Review can only catch the tools that reach it.

Written policy is the control with the most room to grow. Only 50% of healthcare leaders say their written policy on AI coding assistants is fully implemented. A policy can require staff to list each AI-built tool, name an owner, and get it reviewed before it handles PHI.

AI-generated code has already exposed PHI

Half of healthcare leaders (51%) say AI-generated or AI-assisted code has already contributed to PHI exposure at their organization. The survey didn't ask whether reviewed code caused it, so the data can't tie it to review failures. It does show high review rates and PHI exposure reported by the same organizations.

Related: How to send PHI from an AI agent without exposing it

OCR now wants proof that risks were managed

The HHS Office for Civil Rights (OCR) has expanded its risk analysis enforcement initiative to cover risk management. In an investigation, OCR now looks for evidence that an organization acted on the risks it documented.

An unlisted AI-built tool never makes it into the risk analysis, so no one acts on its risks. OCR Director Paula M. Stannard described the link between the two steps in a December 2025 interview: "...you can't do risk management unless you've done a good risk assessment. If you haven't done a risk assessment, you can't do risk management."

How to build an inventory of AI-built tools

The report's recommendations put the inventory first and build every other control on top of it.

Rick Kuwahara, Chief Operating Officer of Paubox, put it this way: "You can't protect an AI-built tool you don't know exists. Reviews, BAA checks, and policies only work on the tools IT can see, so finding every tool that touches patient data has to come first."

Find every AI-built tool and what it touches

List every AI-built tool, whether it handles PHI, who owns it, and which AI coding assistant built it. OCR's own recommended steps for regulated entities start in the same place, with electronic PHI (ePHI): "Identify where ePHI exists in the organization, including how ePHI enters, flows through, and leaves the organization's information systems."

Put each listed tool behind review and a BAA check

Route every listed tool through a pre-deployment review with a named owner. Then check that each tool's outside services are covered by a BAA. About one in eight healthcare leaders (12%) say their organization verifies a BAA only sometimes, rarely, or never before those connections go live.

Scan each listed tool

Check every tool on the list for the security issues healthcare leaders report most often in AI-generated code, starting with exposed credentials and unencrypted data.

Write the policy around the inventory

Tie the written policy to the list, so every tool has an owner and a review step. Then re-run the inventory on a schedule. Most healthcare leaders in The AI Build Benchmark expect AI-assisted development to increase over the next 12 months, so new tools will keep appearing.

Give AI-built tools a safe way to send PHI

Some of the tools on the list will email patients or partners, and those need an encrypted way to send PHI. For those tools, the Paubox Email API sends HIPAA compliant email with TLS 1.2 or higher encryption by default. A BAA comes with every plan, including the free tier.

For AI agents, the Paubox MCP server lets Claude and other agents send HIPAA compliant email. The API key stays with the MCP server process and never appears in the model's context or in a tool call.

Go deeper: The CISO's guide to healthcare AI governance

FAQs

What is AI-assisted development in healthcare?

AI-assisted development is healthcare staff using AI coding assistants to build tools, scripts, automations, and integrations. When those tools handle PHI, HIPAA applies to them. The AI Build Benchmark covers how widely healthcare organizations have adopted it.

Do AI coding assistant vendors need a business associate agreement?

If PHI is entered into an AI coding assistant, the vendor receives PHI on the organization's behalf, and a business associate agreement should be in place first. In the Paubox survey, 83% say all or most of their AI coding assistant vendors have business associate agreements (BAAs).

Where should healthcare AI governance start?

It starts with an inventory of every AI-built tool, whether it handles PHI, and who owns it. Reviews, BAA checks, and policy all depend on that list. The full set of recommendations is in The CISO's guide to healthcare AI governance.