What Counts as PHI in an LLM Prompt and What Must Be Redacted Before a Model Provider Sees It
Organizations must redact patient data from LLM prompts before sending them to external providers.
A HIPAA-compliant AI gateway for teams that handle PHI must enforce redaction before data is sent, limit which providers it can route to, and keep audit evidence for each request. Whether an LLM prompt contains PHI depends on individual identifiability plus health relatedness. A defensible pipeline applies that test to the entire prompt and combines technical enforcement with provider agreements covering the services in use.
Why sending a prompt to a cloud model endpoint is a PHI disclosure under HIPAA
Sending a PHI-bearing prompt to an external cloud model endpoint is a disclosure. OpenAI, Anthropic, Google, and Mistral operate external inference services. The disclosure occurs when the information reaches that service, regardless of whether the application labels the operation drafting, retrieval, or evaluation. Developer intent does not change the boundary.
45 CFR § 164.502 permits uses and disclosures of PHI only as permitted or required by the Privacy Rule. Patient authorization and permitted-purpose provisions provide different legal grounds, while disclosures to business associates have additional requirements. The application must therefore check whether the assembled prompt contains PHI before it sends it. An inspection after the provider receives it cannot prevent that disclosure.
What makes health information "protected" — individual identifiability plus health relatedness, applied to prompt content
“You are assisting Dr. Smith with patient MRN 4821903, admitted 2026-10-14 for CHF exacerbation” satisfies both parts of the PHI test. The medical record number identifies the patient, while the admission and diagnosis describe health care and a health condition. The test has two independent parts. Engineers can treat them as two columns, one asking whether the content identifies someone and the other asking whether it relates to health. Both must be present.
Health relatedness includes past, present, or future physical or mental health conditions, the provision of care, and payment for care. Identifiability includes direct identification and a reasonable basis to believe the information could identify an individual. A name on its own, without health-related context, does not meet both parts of the test. Conversely, removing a name does not necessarily remove identifiability, because a rare condition, specific date, and small geographic area may identify someone together. Proper de-identification removes PHI status, but deleting the most obvious field does not establish that result.
The 18 HIPAA Safe Harbor identifiers and how each one surfaces in a real prompt
45 CFR § 164.514(b)(2) establishes the Safe Harbor identifier-removal standard and requires that the covered entity have no actual knowledge that the remaining information could identify an individual. Removal alone is therefore not sufficient. The categories provide a detection floor across prompt content.
Names appear in user messages, retrieved documents, and patient-scoped system instructions, including names of patients, family members, and providers. Geographic subdivisions smaller than a state show up in EHR material as street addresses, cities, ZIP codes, and counties. Dates appear in details about admission and in clinical histories. Telephone numbers appear in contact-lookup results, and fax numbers can appear in the same contact material. Contact data is easy to overlook.
Email addresses show up in the patient portal context that the model uses to draft replies. Social Security numbers show up in prompts about billing and eligibility, while medical record numbers are often used as retrieval keys. Health plan beneficiary numbers and account numbers come in with records of coverage and billing. Certificate or license numbers can still be in the documents that retrieval brings back. These can all reach the provider without a prompt field set aside for them.
Vehicle identifiers and serial numbers, including license plates, may be in the source material. Device identifiers and serial numbers are especially relevant to connected devices and remote monitoring. Web URLs can encode patient IDs in portal links, while IP addresses appear in access logs submitted to diagnostic agents. Biometric identifiers include finger and voice prints. Full-face photographs and comparable images must be checked along with the text.
The final category covers any other unique identifying number, characteristic, or code. That review must include internal patient IDs, encounter IDs, and application-specific tokens when they uniquely identify a person. This category defeats narrow pattern matching. An encounter ID that looks like a UUID can still be an identifying code. A regex library that recognizes telephone numbers but ignores application-assigned identifiers therefore cannot establish Safe Harbor de-identification.
Where identifiers actually appear across prompt anatomy — system prompts, user turns, RAG context, and agent tool calls
Consider a care coordination agent that calls a FHIR API, retrieves a patient record, and drafts a discharge message. Its system prompt may already include patient-scoped instructions with an MRN, admission date, and diagnosis. Configuration is still transmitted content in this workflow. The user turn may then add a clinical note, treatment history, or discharge request. Reviewing only that turn leaves the initial context untouched.
Retrieval-augmented generation gives identifiers another way in when chunks of patient records become model context. Those chunks may have addresses, record numbers, and details that identify someone in free text. Tool results create the next handoff. A FHIR, scheduling, or billing response becomes prompt content when the agent inserts it into a subsequent model request, even if the original user message passed inspection.
The inspection boundary also extends to model responses. Outputs can contain PHI inferred from the supplied context, so outbound prompt filtering alone does not cover the full interaction. Responses must go through the redaction layer before they reach the application. Together, these surfaces explain why a prompt cannot be treated as one user-controlled string.
Why application-layer redaction fails structurally — the three production failure modes
Unregistered PHI fields create the first failure mode. A redactor that covers enumerated fields does not automatically cover a new free-text clinical note introduced by another feature.
Debug paths to production create the second failure mode. A toggle that sends raw prompts for evaluation can bypass the normal redaction path if promoted into production. The sender retains the ability to evade its own safeguard.
Response PHI creates the third failure mode. Request-only filtering leaves identifying information returned by the model outside enforcement. Fixing one does not remove the others. The component enforcing disclosure policy must be separate from the component requesting inference.
What a HIPAA-compliant AI gateway for teams handling PHI looks like
A defensible architecture places an independent gateway between every application and every model endpoint. It inspects system prompts, user turns, RAG chunks, and tool results before transmission, then applies response inspection on the return path. Enforcement follows a versioned policy. Detection leads to redaction of the identified field, blocking of the request, or allowance of explicitly de-identified or non-PHI content.
The policy decision must be consistent even where detection is difficult. Entity-detection classifiers can find common PII types in clean English text, but their recall drops when they face medical-specific entities, mixed languages, or OCR input. High recall takes priority because missed PHI can create an impermissible disclosure, while false positives reduce output quality in a recoverable way. The gateway must also recognize application-specific codes that fall outside common entity patterns. The gateway centralizes that coverage rather than leaving each application to define it independently.
Routing is part of enforcement. AWS Bedrock’s HIPAA eligibility is model-specific, so an approved service tier is not enough to authorize every model route. Zero Data Retention must also be configured at the provider level, with no-training provisions covering the relevant processing relationship. Training on PHI creates exposure that cannot be undone by deleting the original prompt.
LiteLLM is a legitimate self-hosted option for teams prepared to deploy and maintain the infrastructure. Its paid Enterprise tier includes SSO beyond 5 users, audit logs, and RBAC. Concentrate AI offers a fully managed gateway with built-in PII and PHI redaction, Zero Data Retention options, and a BAA available on the paid plan. SSO, RBAC, and audit logs are free for all customers, while formal SOC 2 Type II report review and extended audit retention are listed under Enterprise. The operational distinction is ownership of the proxy infrastructure, which is significant for a five-person healthcare AI startup without a dedicated platform team.
The BAA covers the provider relationship — it does not replace the disclosure-recording obligation
A BAA establishes the provider’s contractual business-associate obligations for covered services. HIPAA must still permit the use or disclosure. The agreement does not replace the covered entity’s technical safeguards or the operational evidence needed to explain its handling of PHI.
Scope is the decisive contractual boundary. A BAA does not automatically cover services, endpoints, or model tiers outside the agreement. For Bedrock, model-specific eligibility makes that distinction particularly important. For a managed gateway, the gateway’s BAA does not by itself establish coverage for every downstream provider relationship. Even with Concentrate AI’s BAA available on the paid plan, teams must still check which route and services it covers.
Self-hosting open-weight models inside a HIPAA-scoped VPC changes the relationship with the model provider because the data never goes to an external model vendor. There is then no need for a BAA at the model layer, though an agreement for the hosting infrastructure may still apply. Documentation remains a separate responsibility. A signed agreement cannot explain which request was sent, what it contained, or which policy authorized the action.
What the audit log must contain per request — and what it must never contain
Suppose an HHS OCR investigator asks for records of PHI disclosures to AI services over the past 18 months. A policy document and a folder of BAAs would not reconstruct those events. The evidence must come from actual requests. It must establish what the system detected, what it did, and whose identity or role was associated with the action. It must also connect any authorized disclosure to the receiving business associate and the relevant authority.
The per-request record includes a timestamp, workforce member or role identity, detected entity types, redaction or block decision, and governing policy version. Tamper-evident hashes of the prompt and response support the record without duplicating their raw content. Authorized disclosures may require a patient identifier in the appropriately protected record. That exception does not give grounds to log whole clinical payloads.
Entity types and offsets show what was detected without keeping the sensitive values themselves. A record can identify PATIENT_NAME, MRN, and DOB detections rather than copying the name, record number, and birth date. Raw prompt and response bodies do not belong in the metadata audit log. Any PHI kept for authorized processing must stay within the BAA-covered path, with the right protections in place.
HIPAA documentation retention requires applicable compliance documentation to be kept for at least six years. The evidence architecture must support that obligation, with RBAC, SSO, MFA, and break-glass procedures protecting access. Logs need protection of their own. A gateway covering every model route can produce a consistent record under one policy, whereas application logging can miss the same newly introduced or bypassing paths as application redaction.
Applying the framework — a practical checklist for teams building or auditing a PHI-in-prompts pipeline
For teams building or auditing a pipeline, the practical review begins with its full data path. Map system prompts, user turns, RAG context, tool results, and model responses, then confirm that enforcement covers each surface. Check for the Safe Harbor identifier categories at every point on that map. Include a catch-all scan for encounter IDs, internal patient tokens, and other application-specific codes that need custom recognition.
Verify BAA coverage for every PHI-touching provider and the exact endpoints and models in use. Repeat that review after a model or endpoint change. Confirm provider-level Zero Data Retention in the provider documentation and configuration. Enforce RBAC and SSO on gateway access. These checks establish both the permitted route and who can use it.
Test the audit record by simulating an OCR evidence request over a representative window. Confirm that every request has its timestamp, identity, entity types, action, policy version, and tamper-evident hashes without raw sensitive values. Review the breach notification chain as well. The provider agreement must cover notification, and the incident response runbook must cover PHI leaks at model endpoints as well as traditional data breaches.
Finally, assess who will operate the enforcement layer. Teams able to maintain a self-hosted gateway can choose that ownership model. Teams without a dedicated platform function should evaluate a managed option covering redaction, routing, retention controls, audit logging, and BAA availability. The selection criterion is verifiable coverage across the complete request path.
