For healthcare providers, AI offers a quicker path to establishing email data loss prevention (DLP) rules. Users can specify their organization's requirements to generate a foundational rule. Paubox’s announcement, DLP Chat + DLP Observe and Report added to Paubox Data Loss Prevention, explains, “With DLP Chat, describe what you want to catch, and Paubox drafts the rule.”

One word in that sentence truly stands out: drafts. Someone still needs to understand the organization’s email traffic patterns and determine whether the rule is appropriate before it becomes final. Human oversight helps ensure the AI-generated rule is reviewed, configured, and implemented as intended. The feature helps ensure protected health information (PHI) stays within your organization's boundaries.

 

What an AI-generated email DLP rule does

The journal article Cloud Digital Forensics: Beyond Tools, Techniques, and Challenges explains, “Data loss prevention (DLP) protects sensitive data at rest, in transit, and on endpoints to mitigate the risk of data loss, data theft, and cyber-attacks.”

Within a healthcare context, DLP would scan emails and apply your organization's policies when sensitive data or an undesirable event is identified. Users can be alerted, the admin notified, or the email can be held in quarantine.

When prompting AI to create a rule, users can specify the data or activity they want it to detect. For example, an organization may want to detect when a patient’s name and diagnosis are being sent outside the organization. The rule can examine the email body, subject line, attachments, and other relevant message elements.

The next step involves deciding the consequence of meeting the rule. A high-risk scenario could arise where the email must be quarantined. Another option is to simply have your rules generate alerts for testing purposes.

 

AI can assist with policy creation, but it cannot own the policy

The cybersecurity study Beyond performance metrics: evaluating the unique value of generative AI in hybrid cybersecurity threat detection concludes, “This study advocates for carefully scoped human-in-the-loop augmentation, not autonomous GenAI detection.”

The main lesson is that we should let the AI help draft the rules but keep the organization in charge of implementing its policy. An AI-generated rule may include conditions or interpretations that were not part of the user’s original intent. The generated rule needs to be reviewed to ensure it serves your intended purpose.

 

A convincing rule can still contain an error

The article Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support states, “Large language models (LLMs) show promise in clinical contexts but can generate false facts (often referred to as ‘hallucinations’).” AI-generated output can contain inaccurate or fabricated information, commonly described as hallucinations. Depending on its intended use, AI could also generate different kinds of errors. Administrators should review and test the generated rule before enforcement.

Ensure the rule comprehensively covers every element you requested it to identify. Reviewing what triggered the rule can help administrators refine the rule and explain what users may need to change before sending a message.

 

Local email traffic can change how a rule performs

The study Establishing a Validation Infrastructure for Imaging-Based AI Algorithms Prior to Clinical Implementation notes, “Nevertheless, AI models can succumb to reduced performance in new clinical settings and can pose challenges to achieving desired population-level outcomes.” Although the study focuses on medical imaging rather than email security, the broader principle of validating systems in their intended environment is relevant.

A rule that performs appropriately in one workflow may behave differently in another. Administrators should therefore test rules against emails that should trigger the rule as well as emails that should not.

Paubox assists with AI-generated rules by enabling detection. DLP Observe and Report lets you see what might set off your rules and offers suggestions, all without blocking or quarantining anything.

 

Human review needs a structure

The systematic review Automation bias: a systematic review of frequency, effect mediators, and mitigators states, “Automation bias appears to be a fairly robust and generic effect across research fields.” There will always be some automation bias that comes from using these systems. A human reviewer should evaluate the rule before it is approved for enforcement.

Other things to consider include:

  • What are you trying to prevent from happening?
  • Who do you want this rule applied to, for example, senders, recipients, and departments?
  • Compare your request to what was generated by the AI.
  • Test out emails that you want to be blocked and ones that you don't want blocked.
  • Where supported, test or observe a rule before using it to block or quarantine messages.
  • Allow others who work with this data to help review what was detected.
  • Keep documentation on who approved the rule.

For some organizations, having more than one person sign off on the rules would be beneficial. Someone is needed to scrutinize the security of your rule, as well as someone who understands your workflow.

 

FAQs

Will all AI generated rules require the same amount of due diligence?

It depends on the sensitivity of the data, the scope of the rule, and whether the rule only generates alerts or actively blocks or quarantines messages.

 

How many emails should you test against your rule?

There is no universal number of emails that must be tested. However, make sure that you have as many samples as possible that cover various departments or exceptions.

 

Can I create smaller versions of rules for specific departments?

Yes, rules can be scoped around specific departments or workflows when their email patterns and data-handling needs differ.