Routing LLM Traffic Through a HIPAA Compliant Gateway Layer Without Rebuilding Your Stack

A HIPAA-compliant AI gateway puts PHI and PII redaction between applications and model providers, stripping sensitive information before requests leave the gateway. It centralizes zero data retention, audit logs, access controls, and BAA-covered routing without rebuilding existing integrations. For healthcare teams, the gateway supplies a governed request path, while compliance still depends on the organization’s safeguards, risk analysis, and agreements.

Why direct LLM integrations stop working when healthcare teams go to production

A healthcare engineering team connects an application to a model provider, then needs another model, another service, and a fallback for production. Each provider has its own SDK, rate limits, risks of downtime, and terms for how it handles data. Controls that were easy to manage in one service are now spread across applications, so developers must keep routing and data handling consistent.

That fragmentation creates more than an operational problem. A direct request that bypasses redaction and audit controls can disclose PHI without producing the record needed to investigate it. Meanwhile, a provider outage puts pressure on engineers to restore service, even when another available endpoint has not been approved for health data. Clinical workflows make that decision especially consequential.

Without a shared layer for attributing usage as requests happen, teams reconstruct spending from end-of-month invoices. Finance cannot easily link costs to approved projects, and security has no shared view of which workflows are sending traffic. Approval for production thus needs engineering, finance, and security to work together as well as check that the model responds correctly.

What HIPAA actually requires of any system that sends PHI to a language model

The HIPAA Security Rule groups technical safeguards into access control, audit controls, integrity, person or entity authentication, and transmission security. Those safeguards must cover the whole path of a model request. Authenticating a user at the application boundary does not, by itself, establish that the downstream destination is authorized or that the request is auditable.

HHS does not recognize private certifications of Security Rule compliance. A vendor’s claim to be “HIPAA certified” does not replace a covered entity’s safeguards, risk analysis, or BAAs. The proposed Security Rule NPRM, previously targeted for May 2026 and moved to Long-Term Actions with a July 2027 target, would eliminate the addressable versus required distinction. If finalized, it would make encryption at rest, multi-factor authentication, vulnerability scanning, and asset inventories unambiguously required for systems handling ePHI, including AI gateways.

Provider agreements also set which routes a team can use. OpenAI signs a Business Associate and Healthcare Addendum for its API. Anthropic offers a BAA for its first-party API, requiring signing and contact with sales to enable it, and for Claude Enterprise with HIPAA settings enabled by the Primary Owner. AWS lists Amazon Bedrock as HIPAA eligible under its BAA, while Google lists Gemini Enterprise Agent Platform under its Cloud BAA. Microsoft includes its BAA in the Product Terms for in-scope Azure services. Consumer ChatGPT and Claude.ai do not offer BAAs, so consumer access cannot substitute for a covered production endpoint.

How does a gateway strip PHI before it reaches a model provider?

An LLM gateway sits between applications and providers and has one API through which requests pass. That position makes redaction a shared policy across applications. Requests are scanned before leaving the gateway, and PHI and PII redaction is applied before the payload reaches a provider. Enforcement moves out of individual code reviews. The same chokepoint restricts PHI workloads to registered, BAA-covered endpoints, preventing an unreviewed provider from becoming another production route.

The safeguard categories explain the remaining controls. For access control, virtual keys set which providers, models, and tools a caller may reach. For audit controls, per-request logs can record inputs, outputs, tokens, cost, and latency across providers, with content handling governed by the retention policy. For integrity, cryptographically signed audit entries make alterations detectable.

Authentication connects access to the corporate identity provider through OIDC SSO and SCIM provisioning, rather than leaving identity scattered across API keys. Where providers support zero-data-retention headers, those controls keep request content out of provider logs. Retention and transmission security are distinct concerns. ZDR governs what gets stored, and the system must still meet the Security Rule’s transmission-security obligation.

Why “drop-in” is the right description: swapping a base URL is the only application change required

An OpenAI-compatible gateway exposes the interface that an existing compatible client already expects, making a base URL change sufficient to redirect its requests. No SDK swap or application refactor is needed for that path. The pattern applies to the OpenAI SDK, LangChain, and other OpenAI-compatible clients, including integrations reaching other providers through that interface. Retry logic, fallback chains, PHI redaction policies, budget limits, model allowlists, and audit-log routing move into gateway configuration rather than being implemented separately in every service. The gateway is therefore additive infrastructure, although the base-URL-only claim depends on the existing integration using a compatible interface.

How BAA-covered routing and automatic fallback keep a HIPAA-compliant system online when a provider goes down

An approved primary endpoint is only part of a production routing policy. When that provider goes down, the fallback destination must meet the same requirements as the original route. Availability does not establish permission to process PHI. For a clinical workflow, restoring service through an uncovered provider creates a compliance violation at the moment the outage appears to be resolved.

The gateway must therefore apply its BAA-covered allowlist across the entire fallback chain. An alternative endpoint is eligible only when it is covered and permitted for the workload. Engineers do not have to choose between an improvised destination and the routing policy already in force.

Automatic fallback can then preserve service when an approved alternative remains available. It cannot make every outage disappear. Its contribution is narrower and more useful than that promise, because it lets the application recover without opening the request path to an unauthorized provider.

Spend visibility and access control as compliance requirements, not just engineering conveniences

For a CISO, a shared provider key is not sufficient evidence of authorized access. The organization needs to demonstrate who accessed what PHI, when, and under which authorization. Scoped virtual keys connect usage to teams, projects, and models, while RBAC and SSO tie permissions to corporate identity. Role changes must affect endpoint access as well. With provisioning and revocation tied to identity, access no longer depends on someone remembering to rotate a credential left in a separate service when an employee leaves or changes roles.

Finance needs attribution from the same request stream. Real-time tracking by team, project, key, model, and provider connects expenditure to approved AI work, while unexpected token volume can signal an agent loop outside its authorized scope. Spend monitoring supports governance rather than constituting a standalone HIPAA safeguard. It helps with compliance by linking unusual usage to PHI processing that may not be authorized.

Retained, exportable audit logs complete that organizational view. They give the covered entity documentation for an OCR inquiry or its annual risk analysis, while spend reporting gives finance a basis for budget approval. These stakeholders need more than a dashboard of totals. They need records that connect access, authorization, activity, and cost without requiring engineering to reconstruct each question from separate provider accounts.

What to look for in a HIPAA-compliant gateway: the criteria that determine approval

A regulated team should first check the gateway vendor’s own BAA and which parts of its infrastructure the agreement covers. A fully managed deployment removes the burden of operating proxy infrastructure, but the vendor’s environment still needs to be HIPAA-eligible and covered by the agreement. The routing policy must also restrict PHI workloads to covered provider endpoints.

The gateway should enforce request-level PHI and PII redaction before forwarding payloads, retain no copies of request content under its ZDR policy, and apply provider ZDR headers where supported. Audit logs should be per-request, tamper-evident, exportable, and retained for the period established by the covered entity’s risk analysis. Logging and retention policies must fit together. A promise of content-free retention needs to be assessed against what the audit system records.

Access controls should support team, project, and model scopes, corporate SSO, and revocation without manual key rotation. Provider coverage should include the BAA-covered services the team actually needs, such as OpenAI API, Anthropic API, AWS Bedrock, Azure OpenAI, and Google Gemini Enterprise. Broad catalog access is not enough. The gateway must be able to exclude uncovered destinations and apply that restriction to both automatic fallbacks and normal routing.

How Concentrate satisfies these criteria as a managed HIPAA-compliant gateway

Concentrate is a fully managed LLM gateway that provides access to models from 20+ providers through one endpoint, without a proxy for the team to deploy or maintain. Its HIPAA Compliant AI Gateway offers a BAA on the paid plan, without requiring an enterprise procurement cycle. That addresses the contractual starting point for evaluation. Provider coverage and the configured routes still need to match the team’s PHI workloads.

PHI and PII redaction runs at the gateway before requests reach model providers, with policies enforced across traffic rather than instrumented separately in application code. With built-in zero data retention, the gateway does not store request content and also uses provider ZDR headers where supported. Audit logs, SSO, and RBAC are available from day one without an enterprise contract. Governance is not reserved for larger organizations. A five-person healthcare startup can use these controls as it establishes its production access model.

Real-time spend tracking attributes usage by team, project, key, model, and provider. Automatic fallbacks operate across providers serving the same model, and routing can be restricted to BAA-covered endpoints for PHI workloads. Concentrate charges no markup, platform fee, card fee, or BYOK fee, with customers paying for tokens. That pricing structure is relevant to infrastructure budget approval, without establishing that every model is cheaper than every alternative.

Getting from direct provider calls to governed, HIPAA-compliant LLM traffic without a migration project

A team should first sign the gateway vendor’s BAA and check that every provider it plans to use for PHI workloads is covered. It should then configure the covered-provider allowlist and redaction policies before routing production requests. Moving provider keys from application environment variables into the gateway’s key-management layer establishes the new control point, while the compatible application integration changes its base URL.

A non-PHI workload should pass through that path first. This gives the team a way to validate latency, logging, and fallback behavior without introducing PHI during the initial evaluation. PHI workloads should follow only after audit logs and redaction work as configured. Finance and security should then get access to spend and audit dashboards so they can keep watch without asking engineering to answer every question about how the system is running.

In a direct-integration architecture, the team would have to build and maintain redaction, retention handling, audit routing, access controls, and governed recovery across separate services. With the gateway in place, those policies sit at a shared boundary while the existing compatible application code keeps doing the same work. The organization remains responsible for compliance, but it no longer needs each integration to become its own compliance system.