Benchmarking LLM Provider Latency for Production Routing Decisions

Benchmarking LLM provider latency for routing decisions is not about finding the fastest model. Latency is not one number, and providers do not fail one at a time. Every section that follows builds toward a routing policy that survives contact with both of those facts.

Why latency benchmarking for LLM routing is hard

Most teams start the same way. They pick a provider off a benchmark leaderboard, wire a hardcoded API call into the product, and ship it. Weeks later, the numbers stop matching what the leaderboard promised, because the benchmark measured something different from the workload actually running in production. It may have clocked a different latency metric. Maybe it ran at a load level nowhere near peak traffic. The hardware tier behind the benchmark may not be the hardware tier serving the account. Any one of those gaps is enough to make a leaderboard ranking useless for a routing decision.

The deeper reason this keeps happening is structural. LLM inference latency breaks into at least three separate components: time-to-first-token, inter-token latency, and total end-to-end time, and each one governs a different kind of workload. A single number labeled "latency" in a benchmark table is almost always just one of those three, with no indication of which. That's a vocabulary problem before it's a measurement problem: if a team can't say which of the three it needs, no benchmark, however well-run, will answer the right question.

Then there's the part no static benchmark can capture at all: providers are not stable baselines. On September 3, 2026, at least three major AI providers, ChatGPT, Claude, and Grok, went down within the same few hours, with Gemini also reporting disruptions. A latency number measured the day before that window was worthless during it. Benchmarking for routing, then, means decomposing "latency" into its real parts and treating every provider's number as a snapshot that can expire without warning.

The three latency metrics and the workloads each one drives

Time-to-first-token is the delay between sending a request and seeing the first token come back. It matters most in streaming interfaces and interactive chat, where a long TTFT feels like a hang even when the rest of the response streams quickly once it starts. It's also the metric most sensitive to the provider's hardware and API stack: Gemini 2.5 Flash-Lite leads published TTFT figures at 0.35 seconds, a result tied to Google's own API infrastructure, and there's no public evidence that general-purpose GPU clusters can't reach similar numbers given the right setup.

Inter-token latency, often reported as tokens per second, tells a different story. It's the rate at which tokens keep arriving once the response has started, and it decides whether a stream feels smooth or stutters. Users of coding assistants and long document generators track this far more closely than TTFT, because once the wait is over, a choppy stream is its own kind of frustrating. Qwen3.7 Max from Alibaba illustrates the tradeoff well: it produces a high token-generation rate paired with a relatively long TTFT, a profile where the initial wait is long but the stream itself is fast once it begins. That makes it a fit for batch-style generation, not for a live chat window where the user is staring at a blank screen. Gemini 2.5 Flash-Lite, by contrast, reaches 213.5 tokens per second, which combined with its 0.35-second TTFT makes it unusual: fast to start and fast to finish.

End-to-end P95 latency covers the full wall-clock time for a complete response, counting queuing, retries, and whatever variability the provider is experiencing at that moment. It's the number that matters most for SLA budgeting and for agentic pipelines, where a slow tail on any single step compounds across the whole chain. P95 figures run well above P50 benchmark numbers during traffic spikes or provider degradation, so a provider with a strong median can still blow through an SLA at the tail. A gateway sitting in front of multiple providers is the only layer positioned to watch P95 across all of them in real time. A direct API client talking to a single provider has no cross-provider signal to compare against.

Laying the three metrics against workload type produces a clear pattern. Interactive chat and voice interfaces live and die by TTFT, since total latency matters far less once streaming has started. Code completion and long document generation should optimize for tokens per second, because users will tolerate a longer wait if the output flows once it begins. Agentic pipelines and multi-step tool chains need P95 end-to-end as the primary number, since tail latency compounds silently until a deadline is missed. Batch classification and background jobs care about total throughput and cost per token, and TTFT barely matters at all. Get this mapping wrong, point a chat interface's routing policy at throughput instead of TTFT, or point an agentic pipeline at TTFT instead of P95, and the routing decision actively works against the workload it's supposed to serve. The three metrics aren't academic distinctions. Picking the wrong one for the job makes the routing logic built on top of it reliably worse.

Where benchmark numbers diverge from production

Published benchmarks are a reasonable place to start narrowing down providers, but they answer a narrower question than most teams assume. Hardware-tier differences in latency are large and structurally stable. Independently measured benchmarks put OpenAI and Anthropic flagship models at roughly 800 milliseconds to 1.2 seconds p50 TTFT, against roughly 120 to 180 milliseconds for wafer-scale and LPU inference providers. That gap persists consistently across measurement runs.

What benchmarks routinely leave out is everything about how your own traffic behaves. Published p50 figures are usually measured at moderate, controlled concurrency, and the p95 your users experience under a real traffic spike can look nothing like that same provider's benchmark number. Prompt length matters too: TTFT scales with how long the prompt is to prefill, and a benchmark built on short, fixed test prompts will understate the latency a team sees once its prompts are padded out with retrieved context or long tool-call histories. Gemini 2.5 Flash posted a 1.1-second median response time across 38 real tasks in one benchmark, a result that differs from synthetic token-generation tests precisely because it includes prefill time, queuing, and response formatting that synthetic tests skip. Provider throughput also shifts with global demand, so a benchmark run in one timezone's morning can misread what a different user base sees in its own afternoon.

Reasoning models introduce hidden chain-of-thought tokens worth tracking closely. They emit hidden chain-of-thought tokens that get billed as output even though the user never sees them, so a model with a fast raw token-generation rate can still produce a slow wall-clock response if its reasoning trace runs long. Benchmark numbers that count only visible output tokens leave that cost untracked.

None of this makes published benchmarks useless. It makes them a shortlisting tool for narrowing providers before measurement against real traffic begins. A routing policy has to be built on measurements taken against a team's own traffic, at its own prompt lengths, at its own concurrency levels, and the only data source positioned to produce that is the gateway's own telemetry, sitting in front of real requests as they happen. Some teams build this from a self-hosted router; others route through a managed model router like Concentrate AI, an AI gateway that handles provider connections, automatic fallbacks, and spend tracking through a single API so the telemetry is already there without infrastructure to maintain.

Live latency signals in routing engine decisions

Latency-aware routing is a live policy, evaluated fresh on every request, built on top of whatever the gateway is observing right now. LLM routing, in architectural terms, is the behavior inside an AI gateway that picks one upstream model from a pool of candidates for each request, based on a policy that reads live provider state such as latency, error rate, and cost per token, then records the choice as an OpenTelemetry GenAI span attribute so the trace remembers which provider actually served it.

What the engine reads at the moment of decision is concrete and specific. It pulls rolling P95 latency from its own telemetry, continuously updated from recent spans rather than from any external benchmark. It checks rate-limit headroom on the API key in use, because a provider can be objectively fast and still be the wrong choice if that particular key is near its limit. It looks at the shape of the request itself, prompt length, whether tool calls are present, whether the response needs to stream, because those details determine which of the three latency metrics actually matters for this specific call.

The span attribute is what closes the loop. Every request tagged with the model requested, the model that actually responded, the routing strategy applied, and any fallback metadata turns the router into something auditable instead of a black box. That audit trail is also what lets the routing policy improve from real outcomes rather than from someone manually editing a config file after the fact.

This is also the clearest way to see why an LLM router isn't a repackaged load balancer. A network load balancer spreads identical traffic across identical backends. An LLM router spreads non-identical traffic across non-identical backends, on a policy that has to read the request itself. A hard reasoning prompt and a one-shot classification call carry completely different latency profiles, and a router that treats them the same is giving up most of the value latency-aware routing is supposed to provide.

The five routing strategies production teams compose

No single routing strategy covers every latency dimension on its own; production teams generally run two or three strategies layered together by workload rather than picking one and calling it done. Five strategies have settled into common use across production gateways, and each answers a different question.

Round-robin spreads requests sequentially across a pool of providers. It solves rate-limit exhaustion, since no single key gets hammered, but it ignores latency differences between providers entirely, which makes it a reasonable fit only when every provider in the pool already has a similar latency profile. Weighted distribution assigns fixed traffic fractions to each provider by configuration, which is useful during a gradual migration between providers or when a team wants to reserve a fast but expensive provider for a defined slice of traffic without sending everything there. Latency-aware routing sends each request to whichever provider currently shows the lowest rolling latency, optimizing directly for TTFT or P95 depending on what the gateway is measuring, and it's the strategy that does the most direct work against the problem this piece is built around. Cost-aware routing sends traffic to the cheapest provider that still clears a minimum latency bar, which requires the gateway to track cost per token in real time and to enforce a latency floor so that chasing savings doesn't quietly wreck the user experience. A fifth strategy, quality-aware override, routes flagged or high-stakes request types to a specific model regardless of cost or speed, protecting accuracy on the requests where a wrong answer costs more than a slow one.

In practice, these strategies stack rather than compete. The question production teams face is rarely which single strategy to run, and almost always which two or three apply to which workload. A common pattern narrows the field to a cost-aware shortlist first, then applies latency-aware selection within that shortlist, with a quality-aware override sitting on top for any request type flagged as high-stakes. OpenRouter's load balancer shows one version of this layering in production: it first moves providers with significant outages in the last 30 seconds to the back of the line as fallbacks, then weights the remaining stable candidates by the inverse square of their price. Availability gets filtered before cost or latency ever enter the calculation, which reflects a simple fact: a provider that's fast on paper but down right now contributes nothing.

Mapping strategy choice back to workload type, the same pattern from the earlier metrics discussion holds. Interactive chat should run latency-aware routing on TTFT as the primary signal, with cost-aware selection applied only as a secondary filter within the set of providers that are already fast enough. Agentic pipelines need latency-aware routing on P95, paired with a hard SLA floor and an explicit fallback chain, since tail latency compounds across every step in the chain. Batch jobs flip the priority entirely, running cost-aware as the primary strategy with latency treated as a soft constraint rather than a hard one. The strategy a team picks should follow directly from which latency metric that workload actually depends on.

Why fallback chains are a latency problem

A fallback chain built only to answer "is something, anything, still responding" solves the wrong problem. A chain designed purely around availability, switching to whatever provider is up, can produce latency and cost outcomes far worse than the primary route it was meant to protect, since that design never checks whether the fallback provider is fast or cheap enough to justify the switch.

The September 2026 outage window makes the case concretely. When several major model providers all went down within the same few hours on September 3, 2026, with another also reporting disruptions, a fallback chain built on availability alone would have had nowhere good to send traffic. A chain that only asks "is this provider up" treats every surviving provider as an equally acceptable landing spot, when in reality a surviving provider under that kind of simultaneous load spike is likely to post P95 latency far above its normal baseline, turning a reliability save into a latency disaster for every agentic pipeline that lands there.

Building a fallback chain with the same rigor as the primary routing policy means applying the same latency metrics, decided per workload, to every link in the chain, not just the first one. A fallback for a chat interface needs to be checked for TTFT, not assumed fast because it's merely responding, since TTFT is one of the distinct latency components that matters for its own class of workload. Treating availability and latency as one problem, rather than solving for the first and hoping the second takes care of itself, is what separates a fallback chain that protects the user experience from one that only protects uptime.