PHI Handling in AI Workflows and HIPAA Compliance Requirements

Most teams know HIPAA applies to AI systems that touch patient data. What they miss is that it applies separately, at every layer of the workflow, not as one gate to clear before launch. Any LLM workflow run by a covered entity or business associate that can receive, generate, or store PHI falls under the Privacy Rule and the Breach Notification Rule, and under the Security Rule for electronic PHI. That boundary covers prompts, retrieved context passed to the model, vector embeddings, outputs, logs, and fine-tuning data, not just the model itself. An AI system can be technically excellent and still break HIPAA because of where the data moved, not because the model performed poorly. The question to ask about a deployment is never "which model is HIPAA-compliant": compliance is a contractual and configuration state set by what surrounds the model, not a feature the model carries. The rest of this piece walks through each layer in order, because each one calls for a different control.

PHI at Each Layer of an LLM Workflow

PHI appears at every stage of a modern LLM workflow, often without anyone intending it to, and each stage carries its own exposure profile. Take a prior authorization assistant as a working example. A clinician or staff member types a request that names the patient, the diagnosis code, and the requested procedure: that's PHI in the prompt. The system pulls relevant policy documents and prior clinical notes from a vector store and injects them into the model's context: those documents carry the same PHI risk as the records they came from, because retrieval is not a sanitization step. Vector embeddings built from patient data inherit the same protection obligations as the source records, so they need the same encryption and access controls, but RAG pipelines often skip this. The model then drafts a narrative justifying the authorization: that output can reproduce or synthesize PHI pulled from context, and if it feeds a downstream clinical decision, it's in scope for audit regardless of how much it reads like generated prose. Every one of those steps gets logged somewhere, and under the Security Rule, any access to regulated data, AI-assisted or not, has to produce an audit record. Logs that capture full prompt content can themselves turn into unprotected PHI stores if nobody designed them with that risk in mind. Each of these surfaces calls for its own control, which is the subject of the sections that follow.

The minimum necessary standard applied to prompts and retrieved context

The Privacy Rule's minimum necessary standard isn't only about who gets access to a record. It governs what goes into a prompt and what a retrieval pipeline is allowed to surface, not just who triggers the request. Truncating that field before it reaches the model is a compliance requirement, not a nice-to-have. Parameterizing identifiers works the same way: passing a patient encounter ID instead of embedding a name, date of birth, and medical record number directly into a prompt string cuts exposure without costing the workflow anything it actually uses. Template-first drafting, where model inputs are constrained to approved structures rather than open free text, does two jobs at once: it narrows the PHI surface and it reduces hallucination risk, because the model has less room to wander.

The same logic applies on the retrieval side. Policy filters that restrict which document classes a given role can pull are a direct technical implementation of the Privacy Rule, not an optional layer bolted on afterward. Minimum necessary also means that if a workflow can run on de-identified data, it should. The useful part of this standard is that it doesn't trade off against good engineering: tighter prompts and narrower retrieval scopes also mean fewer tokens, faster responses, and lower cost. Compliance and efficient design point the same direction here.

Each layer of an LLM workflow, prompts, retrieved context, outputs, logs, and training data, creates its own exposure point. A single observability layer tracking what enters and leaves the model at each stage is necessary to manage that exposure. An AI gateway like Concentrate AI offers granular per-request logging and Zero Data Retention options, so you can build these constraints into the data flow itself instead of catching problems in an after-the-fact audit.

What the Security Rule requires at the output and logging layers

Compliance obligations don't end once the model finishes generating a response. The Security Rule's audit control standard, at 45 CFR 164.312(b), says any system handling electronic PHI has to record and allow examination of its own activity. For an LLM workflow, that means capturing who prompted what, when, against which model version, and what came back.

Outputs that shape clinical decisions, diagnosis assistance, prior authorization narratives, discharge summaries, fall under the Security Rule's integrity controls. Human review of high-stakes clinical content isn't just a quality check: it's the mechanism that ties the output to a responsible clinician instead of leaving it floating as the product of an opaque process.

Logging is where the real design tension sits. Audit trails need to correlate prompts, retrieved documents, model versions, and downstream actions in one central log platform, but a log that stores full prompt content is itself a PHI store. It needs the same encryption, access control, and retention treatment as the records it reflects. The better design goal is logging that's rich in metadata and lean on content: record who acted, when, against which model, under which policy, and with a reference back to the encounter. A log store holding prompt content that's exposed in a breach is a reportable incident under the Breach Notification Rule. Log design sets the size of that breach surface before anything goes wrong, not after it does.

Fine-tuning and training data as a separate and often overlooked PHI risk

Fine-tuning data is the layer most compliance checklists leave out: PHI used to train or fine-tune a model isn't consumed or erased by the training process.

Part of why this gets missed is organizational. The vendor relationship also looks different here than it does for inference. A company that fine-tunes a model on patient data is a Business Associate and needs a signed BAA, and that agreement has to spell out training data retention, where model artifacts live, and what happens to the data if the model is retired or the vendor relationship ends.

The safer pattern for improving a model is to use de-identified data, qualified under Safe Harbor or expert determination, for any fine-tuning work. Keep re-identification keys stored separately and secured, and validate the de-identification process before any data enters the training pipeline, not after.

Business Associate Agreements and ZDR

Zero Data Retention and a Business Associate Agreement cover different ground, and neither one stands in for the other. Treating them as interchangeable is one of the most common mistakes in AI vendor selection for healthcare work.

A BAA establishes a vendor's legal obligations around PHI across the entire data chain, before any PHI reaches the vendor's infrastructure: permissible uses, security safeguards, breach notification, and how those obligations flow down to subprocessors. ZDR is narrower. It's a specific, requestable configuration, and it stops a model provider from retaining prompt and output content once a request finishes. It governs retention after processing, not the vendor's broader legal or security obligations.

Having a BAA in place doesn't mean a deployment is compliant. The agreement alone doesn't configure the service into a compliant state. ZDR terms also vary by provider. AWS Bedrock documents ZDR as the default for most models, but guaranteed ZDR requires explicitly setting data_retention_mode to none, and certain models, Claude Fable 5 among them, require retention regardless and are excluded from ZDR without special approval. Anthropic defaults to retention and requires sales-team approval along with a signed ZDR agreement, and any qualified Commercial organization can get one, not just enterprise contracts. OpenAI grants ZDR on prior approval for qualifying eligible API customers, and it doesn't extend to free-tier or standard pay-as-you-go accounts or to ChatGPT browser sessions.

HIPAA readiness is the broader state that these two pieces sit inside: encryption, access controls, audit logging, and breach notification capability across the full data lifecycle. Data can be retained under those safeguards rather than deleted immediately, which is what readiness looks like for a covered entity that needs retention for audit log purposes. An AI developer or vendor that doesn't meet HIPAA's definition of a Business Associate, because it isn't creating, receiving, maintaining, or transmitting PHI on behalf of a covered entity, may sit entirely outside HIPAA's reach even if PHI reaches its systems. Mapping the vendor relationship matters as much as mapping the data flow.

Access control and identity management requirements for LLM systems handling PHI

HIPAA's access control requirements apply to LLM systems the same way they apply to any system touching electronic PHI: access has to tie to an individual, attributable identity. Shared API keys that can't be traced to a specific person don't meet that bar.

Role-based access has to map to actual job functions and PHI exposure levels. Patient-context checks need to confirm that a user or service can only reach records appropriate to their role and the specific encounter, not just the right model endpoint in general.

Break-glass flows, the emergency escalated access paths used when normal permissions block something urgent, have to trigger immediate notification and generate an enhanced audit trail. MFA is also shifting status: under the proposed HIPAA Security Rule update published in December 2024, MFA for systems handling electronic PHI moves from an "addressable" control to one that's unambiguously required. Teams that have treated MFA as optional for internal AI tooling need to revisit that assumption.

This connects directly to the logging layer covered earlier. Identity needs to flow from the organization's SSO provider all the way through to individual request-level attribution in the audit log, rather than stopping at a shared service account once it hits the gateway. Access control and audit logging work as one system here, not two separate items on a checklist: an audit log is only as useful as the identity behind each entry in it.

How a centralized LLM gateway enforces controls across all workflow layers

As the number of LLM integrations in an organization grows, so does the chance that one of them misses a required control, and a single missed integration is enough to constitute a HIPAA violation. Rebuilding redaction, key management, and audit logging separately for every new application and every new provider connection turns compliance into a per-integration guessing game.

A centralized gateway closes that gap because it enforces the same controls everywhere requests pass through, so no team has to remember them on its own. When PHI and PII redaction applies consistently to prompts before they reach a provider, it removes the dependence on individual developers implementing it correctly each time. Provider API keys held in a central vault, with application-layer tokens scoped to team, model, and budget, mean a leaked application credential can't reach a PHI-handling model endpoint without passing the gateway's own policy checks. Per-request audit logging that captures identity, model version, policy applied, and timestamp happens automatically in structured form, instead of being pieced together by hand after something goes wrong. ZDR and BAA-eligible provider routing enforced by policy stop a team from accidentally sending PHI workloads to a provider that lacks the right contractual coverage, and model allowlisting for PHI workloads makes sure only contractually covered endpoints receive that data, regardless of what a developer happens to hardcode.

Individual developer or service keys shared across a team can't be traced to specific users in an audit log; a gateway issuing scoped, identity-linked tokens solves that without requiring every application to change its own code. Concentrate AI, as one option in this space, provides a single API across more than 160 models from 21 providers, including OpenAI, Anthropic, and Google, with centralized key custody, per-request audit logging, PII redaction, and real-time spend visibility by team and project. So a team can apply the same PHI controls across every layer of the workflow without rebuilding compliance infrastructure each time a new provider or workflow shows up. The underlying point stands regardless of which gateway a team picks: the infrastructure layer, how data moves through prompts, logs, and provider calls, carries the actual compliance burden, and a gateway that acts as a transparent, enforced checkpoint rather than a passive pass-through is what lets a team show an auditor exactly where PHI surfaces and what governs it at each point.

What an auditor will examine in an LLM deployment and how to be ready

Showing an auditor that an AI deployment is HIPAA-compliant means producing documentation that ties each workflow layer to the specific control applied there, not simply stating that controls exist somewhere in the system. A data flow diagram should show every component that might touch PHI, prompts, retrieved context, embeddings, cache, logs, and outputs, along with the control sitting at each point. Signed BAAs need to be on file with every vendor in the PHI data chain, spelling out training data use, retention timelines, and subprocessor obligations. Access control records should show individual identity attribution, role assignments, and a record of any break-glass events along with their notifications. You need evidence that the minimum necessary standard holds in actual prompt construction and retrieval configuration, not just on a policy page nobody checks against. The proposed Security Rule update removes the old "addressable" versus "required" distinction for encryption at rest, MFA, vulnerability scanning, and asset inventories, so these belong in any new AI deployment from the start. A team that can point to the specific control governing each layer where PHI surfaces, prompts, retrieved context, outputs, logs, and training data, is a team that can demonstrate compliance to an auditor.