Large learning models (LLMs) can ethically be used by healthcare providers to produce patient notes, treatment plans, and letters, but only in the context of using the technology as a supervised assistive tool. They are attractive because they have the potential to ease documentation burden, improve the comprehensibility of discharge summaries and patient letters, and enable clinicians to dedicate more time to patient care.

The evidence supports cautious, governed adoption rather than unchecked use. As a scoping Cureus review states, “AI has significant potential to reshape clinical documentation” by improving efficiency, accuracy, and patient engagement. But the same review makes the compliance point clear, as AI-generated summaries and progress notes can still contain hallucinations, omissions, and other errors.

Paubox’s report on shadow AI shows a troubling gap between the adoption of AI and compliance with widespread shadow AI use and misperceptions about HIPAA obligations. The ethical question turns on governance. With human oversight, strong data governance, and transparency, LLMs can support ethical documentation when used in secure, regulated environments. Without those safeguards, the risks, including privacy breaches, biased care, and loss of trust, will outweigh the benefits.

 

What are LLMs being used for in healthcare?

LLMs are generative models trained on massive text datasets. They can draft text, summarize information, and answer questions. A BMC Medical Informatics and Decision Making study noted, “Tools can now autonomously draft structured clinical notes by listening to patient-clinician conversations, while other generative AI models are being deployed to draft responses to the surging volume of patient inbox messages.” Other generative models are used to “draft responses to the surging volume of patient inbox messages.”

Applications include:

  1. AI‑generated patient notes: Ambient scribes convert physician–patient conversations into draft progress notes. A JMIR Medical Education study noted Nuance’s Ambient Experience uses ChatGPT 4 to listen and produce a note for physician review.
  2. AI‑generated treatment plans: LLMs integrated into clinical decision support tools propose diagnoses or treatment suggestions based on patient data.
  3. AI‑generated patient letters: Tools summarizing encounters or creating plain‑language discharge summaries translate medical jargon into readable letters.

 

The case for using LLMs in clinical documentation

Properly supervised, LLMs used to write documentation can boost clinician well-being and patient engagement, advocates say. The JMIR study shows doctors want help with note writing and see a role for AI. LLMs’ ability to summarize visit conversations and remove jargon may improve health literacy.

In the study, clinicians preferred the chatbot's responses 80% of the time compared with physician-written replies, because they perceived them as more empathic. Collaborative use of LLMs may enable clinicians to provide personalized and accessible documentation while maintaining a complete clinical narrative.

 

The ethical risks of using LLMs for patient notes, treatment plans, and letters

Privacy and data security

LLMs need large datasets, as a 2024 perspective warns, generative AI systems “pose acute privacy and security risks” because they are trained on sensitive patient data. Without stringent controls, training data or prompts may leak, leading to reidentification or model inversion attacks. In a Healthcare Information Research article on large language models in medicine, data privacy and security are also mentioned as key risks; breaches could lead to identity theft and loss of public trust.

The article mentions that patient data used for model training can cause “data anonymization/pseudonymization, differential privacy, robust cybersecurity, penetration testing, clear informed consent processes, and tiered data access policies” as mitigation techniques. The concerns grow when organizations cut and paste protected health information (PHI) into public LLMs. The Shadow AI report found that many employees are testing out ChatGPT without approval.

 

Hallucinations, bias, and accuracy

LLMs are statistically driven and can generate plausible but incorrect or fabricated information, known as hallucinations. They may omit critical details or contradict source data. A study comparing GPT‑3.5 and GPT‑4 found that while the newer model reduces errors, it still produces omissions and misinterpretations. Ethical risks also include algorithmic bias; the review on LLMs notes that bias in training data can lead to inequitable care. In generative AI notes, research shows the potential for gender and racial biases. Unchecked, these hallucinations and biases could misinform treatment plans or propagate discrimination.

 

Accountability, consent, and regulatory gaps

The legal status of AI in clinical documentation is unclear. Who would be liable for errors in an AI-generated note or care plan is not clear. The scoping review identifies legal uncertainties of liability and the need for regulations. Right now, uploading patient data into general-purpose LLMs like ChatGPT can violate HIPAA.

Even if vendors claim their AI features are ‘HIPAA compliant,’ they are only compliant when the model provider signs a business associate agreement (BAA) and follows all technical safeguards. Paubox’s Shadow AI report reveals 21% of teams wrongly think a BAA is not needed for an AI email assistant, and 94% of organizations are updating policies to tackle generative AI threats. Shadow AI usage bypasses security controls and introduces risk without formal oversight.

 

Impact on clinician–patient relationships

Ethical questions are also related to the relationship between clinician and patient. Automation of documentation could result in the depersonalization of encounters or less clinician attention. The review on LLMs warns that poor integration could disrupt workflows and “impede clinician trust” and that inappropriate use could weaken the therapeutic alliance. Documentation tailored to patients must still preserve clinical precision; dual‑purpose notes may need separate versions to meet professional and patient needs.

 

What to consider as a healthcare organization when using an LLM for patient documentation

Under HIPAA, any AI service that processes protected health information (PHI) must be covered by a BAA. A general-purpose or public LLM should not be used for PHI unless the healthcare organization has verified that the service supports HIPAA aligned use.

An LLM draft is not a final clinical record because it can contain hallucinations, omissions, fabricated information, or misinterpretations, so it must be reviewed and approved by the responsible clinician before it becomes part of the medical record. LLMs can be used to write patient letters in plain language, especially to simplify medical jargon and improve readability, but the final letter still needs clinical review for accuracy, tone, and appropriate PHI disclosure.

LLMs can draft treatment-plan language, summarize options, or support clinical decision-making, but they should not independently determine diagnosis, treatment, or care pathways. Patients should also be told when an LLM materially helped draft their note or letter, especially where transparency supports consent, trust, and understanding of how their information is being used.

The broader implementation standard should be that LLMs augment, rather than replace, clinical judgment. Balancing AI with clinician autonomy is necessary so healthcare providers remain in control of documentation decisions.

Healthcare organizations should look for a platform that provides:

  • Encryption in transit and at rest
  • Access controls
  • Audit logging
  • A BAA
  • Clear data retention terms
  • Defined limits on model training
  • Human review workflows
  • Documented approval and escalation processes

 

FAQs

Are LLMs the same as artificial intelligence?

Artificial intelligence is the broader category. LLMs are one type of AI focused on language.

 

Can an LLM learn from PHI entered into a prompt?

Depending on the tool’s settings and vendor practices, PHI entered into a prompt may be stored, reviewed, or used to improve the model.

 

How should LLM outputs be audited?

They should be audited by logging prompts, source data, generated content, reviewer edits, final approvals, and error patterns so compliance and clinical teams can verify accuracy, bias, and PHI handling.