Blocking Patient Data From Reaching OpenAI and Anthropic at the Network Layer

Blocking patient data from reaching model providers requires a shared gateway that intercepts and redacts requests before forwarding them to those providers. A HIPAA-oriented AI gateway must also have BAA-covered routing, controls on who has access, and PHI-safe audit logs. A BAA defines legal obligations, but only a technical control in every request path can enforce redaction across applications and agents.

Why a BAA with OpenAI or Anthropic does not stop PHI from reaching their servers

Consider a request that includes a patient’s name, date of birth, and medical record number and is sent to a consumer tier with no audit log. A signed BAA elsewhere in the organization cannot intercept that request or establish what happened afterward. The contract allocates liability without inspecting, blocking, or redacting the payload, and its coverage does not automatically extend to every product tier.

If the organization’s requirement is that no third-party system sees PHI in plaintext, encryption in transit is insufficient because it protects transmission rather than removing identifiers from the receiving system’s input. TLS does not make those two claims equivalent. BAA scope also varies by contract, feature, and integration, so an active agreement cannot be treated as blanket coverage.

A BAA with the primary provider does not cover a fallback provider that has no agreement in place. When traffic changes destination, the applicable coverage can change with it. Preventing that exposure requires a control that governs the request itself.

Where PHI actually enters an LLM call — and why "guard the prompt box" is not enough

Direct user input is only one entry point. Retrieved documents, patient records, and prior conversation history can bring PHI into a request after prompt validation has finished. Tool-call arguments and results add another path through JSON payloads exchanged in agentic workflows. A check on typed text therefore says little about the complete request eventually sent to a provider.

Request logs and exports can preserve PHI for later access by an unauthorized party, while fallback routing can move a request to an uncovered provider without any application code change. These paths sit outside the prompt box. When input validation is treated as the whole defense, retrieval, tools, logging, and routing each depend on their own assumptions.

Agentic and MCP workloads widen that gap further. Desktop agents with access to the file-system, clipboard, and network can come across sensitive content that a standard chat interface never sees. Tools that inspect only plain message text can miss PHI inside tool-call JSON, MCP payloads, and memory. Each team would otherwise need to implement the right checks at every integration point, with one missed retrieval step or tool response sufficient to expose PHI. This is a structural enforcement problem.

What a gateway-layer interception point actually does to a PHI-bearing request

A gateway places inspection in the shared request path. The sequence runs from the application to the gateway, then to the model provider, and back through the gateway to the application. With a compatible endpoint, the application changes its base URL and retains its SDK integration. The gateway inspects the complete payload before forwarding, including prompts, system messages, retrieved context, and tool-call arguments.

Detected identifiers are replaced with typed tokens such as PERSON_001 and MRN_002, or masked according to policy. The provider receives the sanitized payload instead. Gravitee’s documentation describes placeholder redaction before forwarding and blocking when requests exceed a configured sensitivity threshold. Its policies can apply to incoming prompts and returning responses, allowing output scanning to catch PHI that the model regenerated or inferred.

The audit record then identifies the caller, model, provider, timestamp, and redaction decisions without storing original PHI values. Provider allowlists constrain where the request can go, including during fallback. Zero Data Retention complements this interception but serves a different purpose. Redaction limits what reaches the provider, while ZDR addresses retention of prompts, completions, and derived signals for model improvement. Its availability depends on provider, contract tier, and feature, so routing and retention settings must agree.

What the redaction layer must actually detect — and why regex alone falls short

A shared gateway is only as effective as its detector. If detection misses a patient identifier, central routing simply forwards that identifier consistently. HIPAA’s Safe Harbor method defines 18 identifier categories, covering names, geographic data, dates, contact details, government and healthcare identifiers, account and license numbers, device identifiers, URLs, IP addresses, biometrics, full-face photographs, and other unique identifying numbers or codes.

Regex is useful for recognizable structured strings. Basic templates commonly address phone numbers, social security numbers, credit-card-like strings, and IP addresses, but they do not reliably cover free-text names, variable medical record numbers, or dates of service. Contextual detection and named-entity recognition can identify a patient name in a clinical note or a facility address in prose. They still need to be tested against the languages, document types, and PHI patterns that the deployment uses.

Different detection approaches address different operating needs. Limina, formerly Private AI, covers 50+ entity types across PII, PHI, and payment data in 52 languages and supports a redact-call-re-identify loop. Skyflow stores sensitive values in a privacy vault, sends tokens to the model, and restores values in authorized responses, with coverage for MCP and agentic workloads. Neither approach removes the need to inspect structured payloads. Procurement should establish which entities and payload types are covered, how policies are configured, and how coverage is tested. Gateway redaction reduces exposure, but it is not itself a Safe Harbor de-identification determination, which requires legal review for the specific use case.

Why enforcing redaction once at the gateway is structurally stronger than enforcing it per application

One application may mask emails correctly while an agent using another framework sends them in plaintext because its check is not in the shared path. A gateway policy instead applies to every caller routed through it, whether a LangGraph agent, a direct API call, or a later integration. The guarantee depends on complete routing coverage.

A request-level record gives security and compliance teams a common format for examining callers, destinations, and redaction decisions, removing the need to reconcile separate application logs. The strongest objection is that the gateway becomes a single point of failure. A misconfiguration or outage can affect every application, making availability an explicit operational requirement.

Distributed checks carry operational risk because they can fail silently and selectively, while a gateway configuration is testable and correctable in one place. With a managed deployment, the application team no longer has to keep the gateway available. Redaction enforced by policy at one chokepoint is auditable and defensible, while reliance on developer discipline across separate applications supplies neither uniform enforcement nor a common reviewable record.

What a production-ready HIPAA-oriented gateway must provide beyond redaction

Redaction alone does not make a gateway production-ready. A gateway that removes identifiers before forwarding but writes them into plaintext logs has moved the exposure to those logs. Deployment boundaries therefore belong in the evaluation. Self-hosting in a VPC, on-premises network, or air-gapped environment keeps gateway processing within the organization’s infrastructure, while a managed gateway requires scrutiny of its own data handling and a BAA with its vendor.

Auditability requires more than a searchable request history. The gateway should provide immutable, request-level records of callers, models, providers, and redaction decisions, with retention planned for the HIPAA six-year window. Request records show how callers used the model, while signed audit logs show who changed which control. Both are needed, and neither should keep original PHI values. ONC HTI-1 also requires transparency and source-attribute disclosure for certain Predictive Decision Support Interventions, which are separate from request logging.

Access governance should follow the organization’s existing identity process. Per-team and per-application virtual keys, RBAC, SSO, and per-key budgets and rate limits connect gateway access to joiner-mover-leaver controls. Provider allowlists must also restrict fallback destinations to the covered set, and ZDR must be established for each provider, tier, and feature in use. For smaller healthcare teams, procurement should establish whether BAA access, SSO, audit logs, and SOC 2 Type II assurance require enterprise contracting. These controls must be in place when the gateway is deployed, before traffic starts flowing.

How Concentrate and self-hosted alternatives fit the gateway-layer model

Concentrate offers a managed route to this architecture. It requires no proxy to deploy or maintain and connects applications to models from 20+ providers through a base URL change. PII and PHI redaction runs before requests reach model providers, and a BAA is available on the paid plan. Its stated controls include ZDR, SSO, RBAC, and audit logs, with provider allowlists governing automatic fallbacks across providers serving the same model.

Concentrate’s compliance materials describe SOC 2 Type II, but its pricing page places that assurance under Enterprise, so buyers should not assume every control is included on every paid plan. Its pricing states no markup, platform fee, card fee, or BYOK fee, with customers paying for tokens. Real-time spend tracking covers team, project, key, model, and provider. Buyers should also establish where its infrastructure runs, what its BAA covers, and how ZDR applies to gateway logs as well as provider retention.

LiteLLM is a self-hosted option for teams that need to keep gateway processing and provider keys within their own infrastructure. That choice transfers operations to the healthcare team. SCIM, audit logs, advanced access controls, and some guardrail integrations require its commercial enterprise tier, while SSO is free for up to 5 users and requires Enterprise beyond that. The March 2026 supply-chain compromise, CVE-2026-33634, affecting versions 1.82.7 and 1.82.8 illustrates the responsibility for packages, images, dependencies, vulnerability scanning, and incident response. Ownership costs also include compute, Redis, Postgres, deployment automation, and engineering time. The decision is therefore between operating the enforcement layer internally and entrusting it to a vendor whose contractual and technical boundaries still require review.

Putting the interception layer into practice — what the implementation actually requires

For a team implementing this architecture, the starting task is to map every model-provider request path. That inventory must include applications, agents, integrations, retrieval pipelines, tool traffic, logs and exports, and fallback routes. Existing applications can then use the gateway endpoint through a base URL change without SDK rewrites. Every outbound model request must take that path.

Next, configure the detector’s entity coverage against the 18 Safe Harbor identifier categories as the evaluation floor, and select token replacement or masking. Turn on output scanning to check for PHI that the model returns. Provider allowlists and fallback rules should restrict PHI-bearing traffic to BAA-covered destinations, with provider-level ZDR established for each route. Issue per-team and per-application virtual keys, apply RBAC, connect SSO to the identity provider, and configure per-key rate limits and budgets.

Validation must exercise clinical content. Tests that reflect real use should include names in narrative notes, non-standard dates of service, medical record numbers that look like generic alphanumeric strings, and PHI inside tool-call JSON arguments. Audit validation should check that the record for each request shows its caller, model, provider, and redaction decisions without keeping original identifiers.

The gateway remains a technical safeguard alongside the required provider BAAs, organizational risk analysis, and legal review of de-identification. A managed deployment can reduce the infrastructure work for a small healthcare team, but it does not remove those obligations. The operational objective is a complete shared path. That chokepoint is where PHI interception can be enforced reliably, reviewably, and uniformly across every application and agent the organization runs.