Paubox surveyed 151 healthcare leaders in July 2026 about software their staff builds with AI coding assistants, tools that write code from plain-language instructions. Every organization in the sample handles protected health information (PHI) and uses these assistants. According to The CISO's guide to healthcare AI governance, published in October 2026, 85% said AI-built tools have handled, stored, or transmitted PHI, and another 6% couldn't say whether theirs had.

A second figure carries most of the report's argument, 70% of respondents have already found an AI-built tool that handled PHI without IT or security knowing. Some of those tools send patient data outside by email or connect to outside services, putting them under the same HIPAA obligations as any other system holding PHI.

IT or security leaders made up 77% of respondents, and 63% worked at hospitals or health systems. The report gives a margin of error of plus or minus 8% at 95% confidence.

Read more: The 2026 benchmark for building with AI in healthcare | Shadow AI is outpacing healthcare email security | Understanding HIPAA violations and breaches

 

Controls already existing

Most respondents described governance measures already in place. Training on safe use of AI coding assistants reached all or some staff at 93% of organizations, and 87% said security, privacy, or compliance teams always or usually review AI-written code that handles PHI before it goes live.

Business associate agreements (BAAs), the contracts committing a vendor to protect patient data, showed similar coverage. At 87% of organizations, staff always or usually verify a BAA before an AI-built tool connects to a third-party service, and 83% said all or most of their AI coding assistant vendors have signed one. Asked about the year ahead, 85% considered themselves prepared or very prepared to govern AI-assisted development.

Written policy lagged behind. Half the organizations have a fully implemented written policy on AI coding assistants, 34% have one partly in place, and 11% are still developing it.

 

How patient data is used

Staff building, testing, or debugging with AI coding assistants enter real patient information at 93% of these organizations at least some of the time. And 28% said it happens frequently and 42% said usually, against 7% who said never. Those figures sit next to the 93% training rate. The report's checklist concludes that training alone hasn't kept real PHI out of coding assistants. It recommends a single rule on synthetic test data, meaning realistic but invented records, enforced inside the tools rather than left to staff judgment.

 

What AI-written code has contributed to

Respondents were asked which security or privacy problems AI-generated code had contributed to at their organizations. Hardcoded credentials or API keys topped the list at 54%. These are passwords or access keys written directly into the code, where anyone who can read the code can also read the keys. Exposure of PHI and insecure data storage each came in at 51%. Missing or inadequate encryption followed at 44%, and software vulnerabilities or coding bugs at 30%.

 

Concerns compared with what was found

HIPAA compliance or audit gaps were the most common worry about AI-assisted development involving PHI, named by 66%. PHI exposure followed at 61%, insecure code at 54%, and lack of governance or oversight at 46%.

Shadow IT ranked near the bottom at 25%. Shadow IT is technology staff use without the IT department's knowledge or approval. It is the same category as the unlisted tools 70% of these organizations had already found.

 

Why the inventory comes first

HIPAA's risk analysis requirement under 45 CFR 164.308(a)(1)(ii)(A) covers every system holding electronic PHI. According to the report, OCR has expanded its risk analysis enforcement initiative to cover risk management, so investigators now look for evidence that an organization acted on the risks its analysis identified.

A tool IT doesn't know about can't appear in that analysis. OCR Director Paula M. Stannard warned in a July 2026 settlement announcement, quoted in the report, that organizations unaware of the threats to their electronic PHI "will often learn the hard way when their systems are hacked." Rick Kuwahara, Paubox's Chief Operating Officer, makes the same point about AI-built tools in the report, noting that reviews, BAA checks, and policies only work on the tools IT can see.

 

The seven-step checklist

The report closes with seven steps, beginning with an inventory of every AI-built tool that touches PHI and the coding assistants behind each one. The OCR's own recommended step for regulated entities is to identify where electronic PHI exists and how it enters, moves through, and leaves the organization's systems.

Every listed tool then goes behind a pre-deployment review gate with a named owner, since review only covers tools that get submitted. BAAs are checked on each inventoried tool and its outside services, a step 12% of organizations take only sometimes, rarely, or never before those connections go live.

Listed tools are scanned for the four most-reported problems, which are hardcoded credentials, PHI exposure, insecure storage, and weak or missing encryption. The fifth step is the synthetic test data rule enforced in the tools themselves.

The written policy is built around the inventory, so every listed tool has an owner and a review step. The inventory is then re-run on a schedule, because 70% of the organizations in Paubox's earlier benchmark expect AI-assisted development to increase over the next 12 months.

 

Where email fits

Some AI-built tools have to email a patient or a partner, and that is where PHI leaves the organization. The Paubox HIPAA email API sends those messages with TLS 1.2 or higher encryption by default, and a BAA comes with every plan, including the free tier. Teams whose tools already use a standard mail library can point it at the Paubox SMTP relay without rebuilding.

The Paubox MCP server lets Claude and other AI agents send HIPAA compliant email while the API key stays with the server process, out of the code the 54% figure describes. For email that staff send themselves, Paubox Email Suite encrypts every message by default in Microsoft 365 and Google Workspace, so nobody has to remember an extra step.

 

What researchers found in code assistant suggestions

Researchers at The Chinese University of Hong Kong and Sun Yat-sen University tested whether code completion tools would reproduce credentials they had seen in their training data. Their tool for probing the models, called the Hard-coded Credential Revealer, was applied to commercial code assistants, open-source models, and chatbots that write code.

Their paper, accepted at the 2024 ACM Conference on the Foundations of Software Engineering, found that these tools can return exact pieces of their training data and also suggest additional secret strings. Two valid credentials turned up during the experiments, which the authors described as a privacy concern for training data behind commercial tools.

 

FAQs

How many healthcare organizations say AI-built tools have handled PHI?

In Paubox's July 2026 survey of 151 healthcare leaders, 85% said AI-built tools had handled, stored, or transmitted PHI, and 6% didn't know.

 

What is the most common problem AI-written code has contributed to?

Hardcoded credentials or API keys, reported by 54% of respondents. PHI exposure and insecure data storage followed at 51% each.

 

Why does an AI-built tool need to be in a HIPAA risk analysis?

The risk analysis must cover every system holding electronic PHI. A tool IT doesn't know about can't be assessed, and OCR now looks for evidence that identified risks were acted on.

 

Does staff training keep patient data out of AI coding assistants?

Not on its own, according to the report, 93% of staff, yet real PHI reaches coding assistants at least some of the time at 93% of organizations.

 

Where should a governance program start?

With an inventory of every AI-built tool that touches PHI and the coding assistants behind it, followed by a review gate, BAA checks, scanning, a synthetic data rule, a written policy, and a scheduled re-run.