Beyond Pattern Matching How Entity Recognition Models Change the PII Detection Accuracy Equation at the Gateway

Sensitive data redaction before model-provider submission belongs at a gateway that combines regex, entropy analysis, and named entity recognition, or NER, before forwarding requests. NER changes the accuracy equation by detecting contextual entities that pattern matching cannot recognize, but each method leaves different gaps. Layered detection reduces what leaves the perimeter, while provider zero data retention governs what happens to data that detection misses.

Why the detection method determines what actually reaches a model provider

A support agent quotes a customer's SSN from a retrieved ticket note. The user's message never contained the SSN, so the input guardrail never saw it. This failure follows from where detection runs. A detector that checks only user input cannot see information added later, no matter how well it classifies the text it gets. Retrieval-layer and tool-result PII are especially dangerous because no one at request time consciously chose to include them.

PII enters a model pipeline through four distinct points, including user input, retrieved context, tool results, and model output. Most teams guard only the first. Each point needs detection suited to where it sits, because checking the first message does not protect retrieved context, and checking a response does not stop data from being disclosed earlier. The detection method then determines which categories of sensitive information each check can recognize.

A gateway provides a single chokepoint for enforcing one policy across services and producing a central audit trail. Per-application redaction drifts, gets skipped in new microservices, and leaves no unified record. Placement is therefore an architectural decision. The gateway is also the last opportunity to redact an assembled prompt before it crosses the perimeter. Once that prompt reaches a provider's endpoint, favorable retention terms do not change the fact that the provider has received it.

What regex can and cannot detect, stated precisely

Regex is fast, deterministic, and reliable for structured PII. SSNs, credit card numbers, and email addresses have recognizable forms that pattern matching can detect, with validation checksums where applicable. These checks run in sub-millisecond time. Their strength comes from testing explicit rules rather than interpreting the meaning of surrounding prose, making regex well suited to routine inline inspection.

PII falls into three detectability classes, namely structured, semi-structured, and unstructured. Semi-structured secrets such as API keys and tokens call for entropy analysis plus pattern matching, while unstructured names, addresses, and freeform location strings require NER or an LLM. No regex can answer the general question of whether arbitrary text is a person's name. A name in a sentence has no checksum, fixed length, or required delimiter.

Regex can match a known representation of unstructured information, but that is not semantic recognition. Changing the rules does not get past this limit. Freeform PII, partial identifiers, and values outside an expected format can escape because the detector tests form. Homoglyph substitutions can also break an expected pattern, such as a Cyrillic character substituted inside an email address or SSN.

The scope of inspection introduces another limit. Regex that checks one message at a time cannot join a street address in one conversation turn with a ZIP code in another when each piece falls below its detection threshold. The detector lacks the combined input needed to recognize the disclosure.

What NER adds and why context is the operative mechanism

NER classifies tokens by their role in surrounding text. In “call John at extension 4,” the surrounding words establish “John” as a name. Context supplies the signal that regex lacks. This lets NER find names, addresses, and organizations in prose even when the way each entity is written varies.

Purpose-built small encoder models have expanded that coverage. GLiNER2-PII spans 42 entity types across seven categories, namely personal, contact, governmental, financial, digital identity, credential, and calendar date. The Ettin-Encoder family covers 55 PII entity types, including medical record numbers, health plan beneficiary numbers, biometric identifiers, and quasi-identifiers such as race/ethnicity and political views. This extends detection beyond ordinary contact information. It also makes the model's entity taxonomy a concrete criterion for assessing whether its coverage fits a workload.

All three fine-tuned Ettin-Encoder models outperform prompt-based general-purpose models on PII entity extraction by a substantial margin, with the largest reaching F1 96.27%. That result supports using a dedicated extraction model rather than assuming a general-purpose model is the stronger detector. It does not establish universal production accuracy. The distinction is important because recognizing an entity class in an evaluation does not mean recognizing every representation of it in live traffic.

Latency is the immediate architectural cost of contextual detection. NER runs in tens of milliseconds, which adds directly to time-to-first-token when the guardrail runs synchronously. External PII APIs add hundreds of milliseconds. Small encoder models running at the gateway therefore offer a practical way to gain contextual coverage without making remote redaction the default path.

How benchmark F1 scores hide the failure modes that matter in production

An aggregate F1 score brings precision and recall into one number without showing which mistakes led to it. It is a reassurance metric. A detector can score well overall while missing the entity types most likely to create regulatory exposure. A detector that seems strong can also block valid requests so often that it harms the product it protects.

The consequences of those errors are asymmetric. A false negative lets an SSN pass, potentially leaving it in a trace store until a SOC 2 audit exposes the failure. A false positive can mask a routine billing question and make the guardrail intolerable to the product team. Those outcomes can coexist with identical aggregate F1 scores. The number alone does not distinguish a system that leaks sensitive identifiers from one that over-redacts benign language.

Accuracy also depends on entity type and domain. Person names are especially difficult because they have no fixed format and collide with street names, brands, and product identifiers. SSNs can resemble employee IDs or product SKUs, so changing detection thresholds shifts precision and recall differently across workloads. Clean benchmark text does not reproduce those collisions reliably.

Mind the Gap found substantial degradation under distribution shifts across encoder-based NER, rule-based hybrid detection, and generative LLM extraction. Their failure modes differed. Encoder models struggled with unseen surface forms and entity boundaries, rule-based systems with non-standard formats, and generative models with entity-type confusion and generation instability. Rigorous evaluation therefore scores each entity type separately on an adversarial set probing homoglyphs, format-bent identifiers, and code-mixed text, and on a benign set probing legitimate mentions. Calibration by locale and regulation turns those measurements into a meaningful release gate.

The failure modes NER handles poorly regardless of how well it is evaluated

Evaluation can expose a weakness without making it trainable. A structural gap occurs when the detector's input is missing the signal it needs to work, which goes beyond an example it has not seen before. This separates errors in classifying text by its context, which training can address, from missing-input problems that require a change in what gets inspected or which detection method is used.

Domain-specific identifiers illustrate the distinction. An account number, policy ID, internal employee code, or provider-specific medical record number can appear as an arbitrary string with no contextual indication of sensitivity. NER may recognize medical record numbers as an entity class yet miss such a value when neither its form nor the words around it show what it is. Coverage of a label is not coverage of every encoding.

A name established in turn one may reappear only through a pronoun or reference in turn five, leaving a per-message detector unable to connect the disclosure. Quasi-identifiers create a related problem of composition. Age, gender, ZIP code, race/ethnicity, and occupation may be individually non-sensitive yet collectively re-identifying under HIPAA's Expert Determination method. Labeling each field does not show how risky the fields are when put together.

Obfuscation adds cases where both regex and NER may fail, including character substitutions in names or references such as “the nine-digit number on file.” A detector cannot extract the underlying SSN from that phrase alone. Agentic pipelines have a separate gap in timing, because tool calls with sensitive tool-call arguments can run before an output-layer guardrail sees them, leaving no audit trail by default. No improvement in output classification can prevent an action that has already occurred.

The tiered detection stack that covers what each method misses alone

The tiered architecture chains methods in cost order, using the cheapest and most reliable checks first and adding expensive checks where the compliance bar justifies them. Regex runs inline on every request for structured SSNs, credit cards, and email addresses. Entropy analysis follows for API keys and credential strings with high information density but no fixed pattern. NER then supplies contextual coverage for names, addresses, and freeform entities, with small encoder models able to run on CPU at competitive throughput.

An external PII API belongs at the final tier for workloads that justify its latency and exposure. Sending text elsewhere for PII detection exposes that text to a remote endpoint before redaction. The choice therefore involves more than detector accuracy or response time.

Mutation mode rewrites PII and lets a request continue, which keeps the input clean before the model call. Validation mode detects PII and blocks delivery, serving output compliance after the model responds. Logging alone does not enforce a policy. Data that reaches its destination in breach of policy has still been disclosed, even if the audit trail records the violation correctly.

The replacement method determines what utility and obligations remain. Masking irreversibly replaces a value with a placeholder, while tokenization uses a reversible swap backed by a vault-stored mapping. Tokenization preserves useful responses for authorized users on the return path, but it is GDPR pseudonymization and remains regulated personal data. Masking can keep sensitive values out of logs and analytics. However, masking individual fields does not establish GDPR anonymization unless re-identification by any reasonably likely means is impossible across the whole dataset.

The gateway is where these decisions become a uniform policy. A managed, fully hosted implementation can centralize the detection stack without requiring each application team to rebuild it or a platform team to deploy and maintain it. Its control criteria include coverage of all four pipeline insertion points, policy-enforced PII and PHI redaction, and per-request reviewability across models and teams. Centralization makes the chosen controls consistent and their enforcement inspectable.

What the gateway still cannot guarantee and why ZDR is the complementary control

Redaction reduces what the provider receives, while ZDR governs what happens to received data, including sensitive information detection misses. Neither substitutes for the other control. Training opt-out is separate again, because inputs can still be stored even when they cannot be used for training.

Provider policy must therefore be assessed at the endpoint level. OpenAI requires sales approval for ZDR and limits it to eligible endpoints, including stateless ones such as chat/completions, responses, embeddings, and audio. Stateful endpoints such as Assistants, Threads, and Vector Stores are excluded. AWS Bedrock does not use customer data to train models, but default retention depends on the model and can reach 30 days for some. Zero data retention must be explicitly configured.

The compliance posture combines a Business Associate Agreement with the provider, PHI redaction at the gateway, ZDR at the endpoint, and audit logging of PHI-touching requests. Those records hold the user identity, timestamp, model, and details of access. ZDR alone does not fulfill HIPAA requirements. Nor does successful redaction remove the need to govern requests that touch protected information.

The strongest architecture keeps NER-based scrubbing inside the gateway boundary so unredacted PHI is not deliberately sent to a public endpoint for inspection. A stateless gateway that never writes to disk does not add another store that keeps content. Metadata-only audit logs record policy enforcement outcomes without retaining request content, supporting auditable ZDR controls for SOC 2 Type II. The defensible boundary is layered redaction before submission, limited retention after receipt, and evidence of enforcement without a new archive of sensitive text.