Many compliance departments know that a business associate agreement (BAA) needs to be established before protected health information (PHI) is handled by a third party. Where it becomes unclear is what that third party can do with those messages, files, and documents. Whether it's sending a lab value to a patient, a referral to a doctor, or even a billing question to an insurance company, it can be used to train models.

According to research from Paubox, 95% of healthcare organizations state that their employees who have access to PHI are using AI tools within their emails. With 1 in 4 not having approved the use of any AI.

 

What a business associate agreement allows

HHS clarifies that when a business associate handles PHI, they're assisting the covered entity with its healthcare operations, rather than using it for their own benefit or objectives. With the exception of managing and administering their own business.

Within your BAA, you must specify and obligate particular kinds of PHI usage. 45 CFR 164.504(e) can allow for the vendor to use PHI to manage and administer their business or as required by law.

If your group practice was using a vendor to send follow-up emails to patients, they can use those messages to complete that function.

 

Why training is a separate question

Healthcare operations can include quality assessment and improvement activities, developing clinical practice guidelines and conducting research on a population level. There is a caveat to this if it leads to generalizable knowledge from a study. Training a model to help your organization improve its services may qualify as healthcare operations.

However, if a business associate intends to use the data for model training benefiting themselves, this must be explicitly stated within their contractual agreement. The OCR has not addressed training models with PHI, which means organizations will need to make sure their contracts adress training models.

 

How de-identification changes the answer

After data has been de-identified, it's no longer classified as PHI. The HHS de-identification guidance, it notes that de-identification is only allowed within the boundaries set by your BAA. When de-identifying your data, there are two options that the legislation will allow. Both processes, however, carry the potential for leaving behind some personally identifiable information.

De-identified data can be used for training AI models. However, you'll need to confirm your BAA explicitly permits the vendor to de-identify data and detail the exact procedures they follow.

 

What re-identification research measured

A JAMA Network Open study explored the potential to re-identify activity data from the US National Health and Nutrition Examination Survey for the years 2003 through 2006. Once they'd scrubbed all PHI and location details, they employed support vector machines and random forests, successfully pinpointing 20-minute activity patterns for 80% of the children and 95% of the adults, especially when correlated with other demographic factors.

These numbers may not apply to de-identified HIPAA information. Still, you might want to inquire with your vendors about any other datasets they might possess.

 

What model leakage research adds

According to research from the Nature Portfolio, a study examined the different privacy concerns arising from medical AI. They examined membership inference attacks across seven distinct clinical datasets, encompassing medical imaging, electrocardiograms, and EHRs. Their study found that for some patients, they had an almost 100% chance of identifying if that person's data was used to train the model. Even when overall performance was near chance. They stated that "aggregate privacy metrics can severely underestimate individual privacy risk."

It was also observed that larger models presented a higher risk of a successful breach. As well as more successful attacks on minorities. They investigated medical models alongside specific threats in their research. Consequently, the application to language models trained on email remains uncertain.

 

Check the email workflow and the BAA together

The place you're most likely to spot AI being used without permission is in your emails. 85% of healthcare IT Leaders believe their staff is using AI tools that are not authorized by their organization. Only 26% know if their employees are using these tools and 2/3 of organizations have experienced unauthorized use of AI. As we mentioned earlier, it's best to make sure you have a BAA with your AI vendors that will be handling PHI.

An employee could paste an email about a patient's payment plan into a consumer-based chatbot. This action amounts to giving PHI to a service that lacks the required BAA.

Ask yourself the following questions about your BAA outlines:

  • Will PHI and derived data be used to train, tune, or evaluate models
  • Who may de-identify the data, and under which HIPAA method
  • Will subcontractors and model hosts accept the same terms under 164.504(e)(5)
  • How are long prompts, outputs, and logs retained

 

Where Paubox fits in

By employing large learning models, and generative AI, Paubox’s inbound email security system scrutinizes your incoming mail. Both the Email Suite Plus and Premium plans offer this functionality. This helps spot threats that land in your inbox.

Based on research from Paubox, an AI-powered email attack has been experienced by 64% of US healthcare IT leaders. However, only 38% are actually employing AI to its full potential for spotting and tracking threats.

Naturally, all our outgoing emails are encrypted as a standard feature. If you'd like to learn more about our email encryption, reach out to us today.

Related: HIPAA Compliant Email: The Definitive Guide (2026 Update)

 

FAQs

Can a limited data set be treated as de-identified training data?

No, a limited data set remains PHI and requires a data use agreement for permitted research, public health, or healthcare operations uses.

 

Can an AI service process PHI outside the United States?

HIPAA does not categorically prohibit overseas processing, but applicable BAAs, safeguards, and geographic risks must be addressed.

 

Can patients access AI-generated information used in their care?

If AI-generated information forms part of a designated record set, patients generally have a HIPAA right to access it, subject to the rule’s exceptions.