Healthcare AI adoption has moved well past the pilot stage. Physicians now use AI tools for image review, ambient documentation, and clinical decision support, while administrative teams use it for drafting, summarizing, and correspondence. What has not kept pace is governance. Many organizations have no approved AI tool for staff to use, no business associate agreement covering the tools staff use anyway, and no record of which systems are handling patient data.

 

Where adoption actually stands

Doximity's State of AI in Medicine survey found that 63% of US physicians now use AI tools in some capacity, up from 47% only nine months earlier. Separate research on generative AI adoption found that 46% of US healthcare organizations were actively implementing generative AI technologies, a figure that proves institutional rollout rather than individual experimentation alone.

Adoption has outpaced oversight. A physician using an AI tool to draft a note may not know whether that tool retains the information entered into it, and an organization deploying generative AI may not have documented which tools staff are using or whether those tools are covered by a signed agreement.

Read more: What is machine learning? | What is natural language processing?

 

The compliance gap nobody planned for

Shadow AI is the use of generative AI tools outside any formal IT or compliance approval, and it is where the difference between adoption and oversight becomes measurable. Paubox's 2026 Shadow AI report found that 95% of healthcare organizations suspect their staff is already using generative AI for work-related email or content, while 25% have not formally approved any staff use of AI in email at all. 62% of IT leaders reported directly observing employees experimenting with ChatGPT or similar tools despite the tools being unsanctioned.

The HIPAA obligations that follow are specific and measurable. The same report found that only 42% of healthcare organizations have signed a business associate agreement covering an AI tool used in email, while 21% of teams believe a BAA is not required for an AI assistant at all, a misunderstanding that leaves those organizations exposed without anyone realizing it. 75% of IT leaders believe their staff assumes tools like Microsoft Copilot are automatically HIPAA compliant. Acting on that assumption means patient data can be entered into a tool with no BAA in place, which is an impermissible disclosure under HIPAA and a reportable breach the organization has no record of.

A related Paubox report on healthcare IT confidence found the same pattern in email security. 89% of respondents said AI and machine learning were critical to detecting email threats, yet only 44% had AI-based detection deployed. In both cases, organizations rate a capability as essential without having implemented it.

 

Why has healthcare been slower to build guardrails than to adopt the tools

Governance has consistently lagged behind deployment, and 2025 marked the first real attempt at national-level correction. On September 17, 2025, the Joint Commission and the Coalition for Health AI jointly released the first formal guidance from a US accrediting body on responsible AI use in healthcare, covering more than 23,000 accredited organizations. Dr. Jonathan Perlin, president and CEO of the Joint Commission, described the pace of change driving the guidance directly, noting that AI is changing healthcare at a scale he had not previously seen in his career as a leader.

The guidance sets out seven areas organizations are expected to address, including formal AI governance structures, patient privacy and transparency, data security consistent with HIPAA, ongoing quality monitoring, and structured education and training for staff using AI tools. It remains voluntary for now, but the Joint Commission has signaled that a formal AI certification program, built on the same principles, is planned for its accredited organizations in the years ahead.

Federal regulation has been moving in a related direction. The HHS Office for Civil Rights proposed the first major update to the HIPAA Security Rule in over a decade in January 2025, and the proposed rule addresses AI as an emerging technology, expecting that any AI software used to create, receive, maintain, or transmit ePHI would need to be listed as part of an organization's technology asset inventory. The rule remains under review, but the direction it signals is unambiguous, treating AI tools that touch patient data the same way any other system handling ePHI would be treated.

 

What the clinical use cases actually look like

Away from the governance conversation, the clinical applications driving adoption fall into a few consistent categories. Machine learning underpins most of what gets described as clinical AI, applied to reading medical images, flagging risk in electronic health records, and identifying patterns across large volumes of clinical documentation faster than manual review would allow. Natural language processing supports much of the administrative side, extracting relevant information from unstructured notes and now powering the ambient documentation tools that have become one of the most visible AI applications in day-to-day clinical work.

The Joint Commission and CHAI guidance frames the underlying tension clearly, noting that AI tools can meaningfully improve care while also carrying real risk of bias, reduced transparency, and overreliance if organizations do not validate how a given tool performs against the specific population it is being used on. A model trained on one patient population does not automatically generalize to another, and the guidance recommends organizations request validation data from vendors rather than assuming a tool works as advertised out of the box.

 

Where email security fits into the AI conversation

For a security and compliance team, the AI conversation eventually comes back to email, because email remains where most PHI still moves and where most unauthorized AI use actually happens. An employee is far more likely to paste a patient summary into a chatbot while drafting a message than to expose that data through a dedicated clinical AI system that IT already knows about and has vetted.

Pre-delivery filtering and outbound protections address a different part of the picture, the phishing and impersonation attempts that AI has also made easier for attackers to produce at scale, but the shadow AI problem calls for policy, visibility, and vendor accountability rather than filtering alone. Paubox's own AI use policy shows the kind of documented governance the Joint Commission guidance recommends organizations put in place internally, and Paubox Inbound Email Security uses AI to analyze sender behavior and message intent, addressing the inbound side of a threat environment that is changing alongside the same technology healthcare is trying to adopt responsibly.

Learn more: Paubox Inbound Email Security | HIPAA Compliant Email: The Definitive Guide

 

In the news

In April 2026, the American Medical Association announced a new policy framework addressing AI-generated deepfakes that impersonate physicians, warning that synthetic audio and video are being used to mislead patients and steer them toward unproven treatments. AMA CEO John Whyte, M.D., called the trend "a public health and safety crisis," stating that "when bad actors exploit a doctor's identity, they undermine patient trust and can steer people toward harmful, unproven care." The framework, developed by the AMA's Center for Digital Health and AI, outlines seven principles including treating physician identity as a protected right, requiring informed and revocable consent before AI content can use a doctor's likeness, and mandatory labeling of AI-generated medical content. Fierce Healthcare's coverage noted the framework arrives as impersonation videos of doctors, often promoting fake supplements or weight-loss products, have circulated widely on social media, damaging individual reputations and eroding broader trust in medical guidance found online. The same underlying technology used to impersonate physicians publicly is what security researchers warn is now being directed at healthcare organizations directly, through cloned executive voices used to authorize fraudulent payments or extract sensitive information over the phone.

 

FAQs

How does HIPAA apply to AI tools used in healthcare?

Any AI tool that creates, receives, maintains, or transmits protected health information is subject to HIPAA, which means a Business Associate Agreement is required before that tool can be used with patient data. Paubox's research found that only 42% of healthcare organizations have signed a BAA covering an AI tool in email, leaving many exposed without realizing it.

 

What is shadow AI and why does it matter?

Shadow AI refers to employees using generative AI tools without formal IT or compliance approval, often to speed up drafting notes or messages. It matters because sensitive patient data entered into an unapproved tool may be stored or processed outside any HIPAA-compliant agreement, effectively creating a breach that the organization does not know has happened.

 

What does the Joint Commission and CHAI guidance actually require?

The guidance is currently voluntary and outlines seven areas organizations should address, including governance structures, patient privacy, data security, and staff training. It does not carry regulatory force yet, but it signals the direction accreditation and insurance requirements are likely to move in over the next few years.

 

Does the proposed HIPAA Security Rule update address AI?

Yes. The Notice of Proposed Rulemaking published by HHS in January 2025 expects AI software that touches ePHI to be included in an organization's required technology asset inventory, treating it the same as any other system handling patient data. The rule remains under review and has not been finalized.

 

How can a healthcare organization reduce the risk of shadow AI without banning AI outright?

Most guidance points toward the same combination of steps, formal AI usage policies, signed BAAs with any vendor whose tool touches PHI, staff training specific to AI risks, and giving employees an approved AI option so they are less likely to reach for an unvetted one out of convenience.