Criminal use of AI in email started with something fairly obvious: writing better phishing messages. Since then, it has moved well beyond the email itself. Attackers can now use AI to research targets, build fake login pages, vary their messages to avoid detection and even help operate inside a compromised mailbox. That progression does matter because an email attack is rarely just an email. Each stage gives attackers another opportunity to automate work that previously required a person.

The change became particularly clear in September 2025, when Anthropic's threat intelligence team documented a campaign it described as "the first documented case of a cyberattack largely executed without human intervention at scale." Looking at the attack chain stage by stage shows where AI is already being used and what healthcare organizations need to defend against.

Read more: What is a phishing attack? | What is business email compromise? | What is social engineering and why healthcare is vulnerable

 

Writing the message

WormGPT appeared in 2023 and was built on an open-source language model. It was marketed on criminal forums as a way to produce business email compromise messages, including emails designed to impersonate executives or vendors and persuade someone to transfer money or information. FraudGPT followed with a wider range of capabilities, including phishing pages and malware.

Someone buying these tools did not need to be skilled at writing convincing emails or building the supporting material. The FBI warned that criminals were using generative AI to make fraud more convincing and to carry it out at greater scale. Speed is one of the clearest changes. In 2024, IBM researchers demonstrated a functional phishing campaign created with five prompts in around five minutes. Their comparison put the equivalent human effort at roughly 16 hours.

That also weakens some of the advice people have relied on for years. Looking for spelling mistakes, awkward grammar or obviously unnatural wording is much less useful when an attacker can generate a clean, professional-looking email in seconds.

 

Making it personal

Generic phishing depends on volume. A personalized message does not need to reach as many people if it gives the recipient a good reason to trust it.

Researchers at Harvard Kennedy School tested a tool designed to automate the spear-phishing process, from collecting publicly available information about a target through to writing and sending the message. The researchers tested it on 101 participants across four groups.

The results were striking. Generic phishing emails drawn from existing databases produced a 12% click-through rate. Emails written by human experts reached 54%. Fully AI-automated personalized emails also reached 54%, while emails given a light human edit reached 56%.

The research did not stop at the email itself. The tool created what the researchers called personalized vulnerability profiles, using information about a person's occupation, affiliations, field and interests. Those profiles were accurate and useful in 88% of cases, with only 4% containing inaccurate information. A peer-reviewed version of the study, published in Expert Systems with Applications in 2026, estimated that the economic return for attackers could improve by as much as 50 times when targeting larger groups.

Healthcare organizations make an interesting target because so much information about their staff is already public. Staff directories, conference programs, research papers, payer announcements and vendor news can all reveal useful details.

A phishing email that mentions a real doctor, a genuine conference or an actual clinical program can feel very different from a generic message. The recipient already recognizes part of the story, which gives the attacker a much better starting point.

 

Building the destination

The email is only the delivery mechanism, and whatever page it points to has to hold up under scrutiny, which is where criminals worked out that mainstream AI would build one if the request was framed carefully.

In June 2026, Google filed suit in Manhattan federal court against a Chinese cybercrime network accused of using Gemini to generate credential-harvesting pages, meaning fake login screens built to capture usernames and passwords, for an operation running as Outsider. The volume was considerable, with more than 1.59 million fraudulent URLs produced and the FBI estimating the platform behind roughly $1.9 billion in losses since 2023.

The operators never described what they were building, asking instead for HTML for a generic rewards page, then deploying the working code they received as a fake login screen. Google is now working with AT&T, T-Mobile, and Verizon to block traffic tied to the platform.

 

Stealing the compute to do it

Some attackers skip building tooling and take access to the commercial models instead.

Microsoft's Digital Crimes Unit tracked a network designated Storm-2139 that used stolen API keys to reach commercial AI services including Azure OpenAI and strip out the safety controls those services enforce. An API key is the credential that lets one piece of software talk to another, so a stolen key means running a frontier model, meaning one of the most capable systems commercially available, on someone else's account with the usage buried in legitimate billing. Microsoft disrupted the network through a civil complaint in December 2024 followed by criminal referrals. The implication for a healthcare organization is that its own leaked credentials can end up generating attacks against other people.

 

Getting past the filter

Signature-based email filtering, which works by matching incoming messages against stored patterns from known attacks, assumes a campaign looks like itself, so flagging one message catches the rest.

AI breaks that assumption at almost no cost, and the Harvard study quantified how little that cost is, finding automation improves attacker economics by up to 50 times for larger audiences. Generating a fresh variant of a lure for every recipient, each with a different subject line, sender name, and body, leaves no consistent pattern for matching. Paubox's 2026 Healthcare Email Security Report tracked a 47% increase in attacks avoiding the defenses built into Microsoft 365 and Google Workspace during 2025, which is the measurable version of that problem.

 

Operating inside the mailbox

Thread hijacking is the clearest example, where rather than sending a fresh message, the attacker replies inside a genuine existing conversation from the compromised account, inheriting the full history. Paubox documented one such campaign where attackers compromised a contractor's mailbox, waited for an active approval conversation, then replied with a document review link carrying seven forwards of legitimate history behind it.

Authentication offers nothing against this, because a reply inside a real thread from a real account passes SPF, DKIM, and DMARC, the protocols that verify a message genuinely came from the domain it claims, when nothing about the sender is forged. AI makes the reply match the tone and vocabulary of the thread it is joining.

The takeover itself has also been automated, with phishing kits sold on subscription now capturing session tokens, the credential a browser holds to prove a user has already signed in, which grants mailbox access without the password or second factor being needed again.

 

Backing the email with a voice

Business email compromise gets more convincing when a phone call confirms it, and voice cloning made that step cheap. In April 2026, the American Medical Association announced a policy framework addressing AI-generated deepfakes, meaning synthetic video or audio that convincingly reproduces a real person's face and voice, impersonating physicians. AMA CEO John Whyte, M.D., called the trend "a public health and safety crisis." Fierce Healthcare reported that impersonation videos of doctors had circulated widely, damaging individual reputations and eroding trust in medical guidance.

The same capability aimed inward supports a cloned executive voice confirming the payment change an email just requested.

 

Automating the whole chain

The chain closes with GTG-1002, the group behind the September 2025 campaign against roughly 30 organizations.

The operators wired Claude Code, an AI tool designed to write and execute software autonomously, into a custom attack framework using the Model Context Protocol, a standard that lets an AI model reach outside its own conversation to read files, query databases, and call other software. The AI then executed 80% to 90% of tactical operations independently, including credential harvesting and data exfiltration, meaning the bulk transfer of stolen data out of a network, with humans intervening only at strategic decision points.

Guardrails were routed around rather than broken, with attackers fragmenting the operation into steps innocuous enough that no single request appeared malicious. Anthropic's report is candid about the limits, noting the AI frequently overstated the value of what it found and that only a small number of intrusions succeeded, though neither weakness changes the direction of travel.

 

Why healthcare absorbs this badly

Healthcare has been the most targeted critical infrastructure sector in the United States for two consecutive years, according to the FBI's 2025 Internet Crime Report, which recorded 24,768 business email compromise complaints accounting for $3.046 billion in losses.

The human layer offers less protection than most organizations assume. A study of nearly three million simulated phishing emails across six US health systems, published in JAMA Network Open, recorded a 14.2% click rate. The same team's follow-up in the Journal of the American Medical Informatics Association introduced mandatory training for the highest-risk employees and found it "did not meaningfully decrease the click rates of this population."

Paubox's 2025 Healthcare Email Security Report found employees report only 5% of known phishing attempts to security teams, and that figure was measured when phishing still carried the tells staff were trained to look for.

 

What actually holds up

Content inspection is what AI degraded, because content is what these tools produce convincingly, which leaves the defenses that work on structure and behavior as the ones still standing.

Sender authentication catches impersonation regardless of how well a message reads, since DMARC, DKIM, and SPF verify a message came from the domain it claims, and no amount of polish passes a check the sender cannot access. Phishing-resistant multi-factor authentication built on the FIDO2 standard, an open specification for logins that use a physical security key or device rather than a code, ties a sign-in cryptographically to the real site's address and defeats the credential harvesting most of these campaigns feed.

Behavioral analysis covers the layer in between, including the thread hijacking case where authentication passes legitimately. The same Harvard research tested that approach from the defensive side and found language models scoring above 90% at identifying malicious intent in emails, with low false positive rates, and flagging some messages that appeared benign to human reviewers. Paubox Inbound Email Security applies that principle to inbound mail, analyzing sender behavior, message intent, and contextual signals rather than matching known-bad content, which is the approach that still functions when the text is flawless and the sender is real.

Learn more: Paubox Inbound Email Security

 

In the news

The September 2025 campaign reached Congress within weeks. On December 2, 2025, Senators Maggie Hassan and Joni Ernst wrote to National Cyber Director Sean Cairncross urging continued federal coordination on what they called an emerging national security threat.

Their letter picked out one detail worth reading as a healthcare scenario. In a successful case, the threat actor induced the AI to autonomously discover internal services, map complete network topology across multiple IP ranges, and identify high-value systems including databases and workflow orchestration platforms. Applied to a hospital, that describes an automated attacker locating the electronic health record system and everything around it without a person directing each step, and the entry point for that class of intrusion remains a credential taken from an inbox.

 

FAQs

What are WormGPT and FraudGPT?

They are AI tools built or modified for criminal use and sold on dark web forums, marketed for writing phishing and business email compromise messages without the safety restrictions mainstream models enforce.

 

How much more effective is AI-personalized phishing?

In a Harvard Kennedy School study of 101 participants, generic phishing emails achieved a 12% click-through rate while fully AI-automated personalized emails reached 54%, matching what human expert attackers produced manually.

 

Can attackers misuse legitimate AI services?

Yes, in two documented ways. Some frame requests so the model produces useful components without recognizing intent, as in the Gemini case Google sued over, and others steal API keys to run commercial models on another organization's account.

 

Why doesn't email authentication stop AI-generated attacks?

It stops forged senders, which covers a large share of phishing. It does not stop a message sent from a mailbox the attacker genuinely controls, which is why thread hijacking passes authentication checks correctly.

 

What defenses still work?

Controls that do not rely on inspecting content. Sender authentication verifies origin, phishing-resistant MFA cannot be relayed through an attacker, and behavioral analysis flags anomalies in sender patterns that well-written text cannot conceal.