Load Balancing Across Multiple LLM Providers in Production

Load balancing across LLM providers is a category of techniques. It is a stack of decisions, layered on top of each other, and each layer solves a different failure mode. Get the stack right and a production AI system stays cheap and reliable even when a provider goes down or traffic spikes. Get it wrong and every outage or price hike becomes a fire drill for the engineering team.

Latency-aware routing and what it actually measures in production

Latency for an LLM call doesn't mean the same thing as latency for a normal HTTP request. Two numbers matter: time-to-first-token, which sets how responsive the app feels to a person watching text appear, and tokens-per-second, which sets how fast a long completion finishes once it's started. Both shift by provider, by model size, and by how loaded that provider is right this second.

Latency-based routing doesn't pick "the fastest provider" as some fixed label. It picks the fastest provider right now, based on what the gateway has actually measured in the last few seconds or minutes. That distinction matters because provider speed moves around with time of day, region, and upstream capacity. A provider that wins a benchmark run at 9am can lag badly during a 2pm traffic peak. Static rankings, built once and left alone, go stale fast.

So the gateway needs live health signals: rolling latency windows tracked per provider and per model, updated continuously, not a historical average pulled from last month's testing. Streaming adds another wrinkle. A streamed response can hold a connection open for seconds or minutes, so routing logic has to separate time-to-first-token from total response time. Users notice the first word appearing, not the last one.

Latency routing sits inside a narrower decision than people assume. It only picks among models that have already cleared the capability and cost bar for that request. It's a tiebreaker among acceptable options, never an override that lets a fast-but-wrong model win.

A provider with a good median latency can still have ugly tail latency, meaning its p99 requests, the slowest one percent, spike hard. That kind of instability can hurt user experience more than a provider that's a bit slower but consistent every time. Track percentile distributions, not averages, or the routing logic will keep picking a provider that occasionally stalls for ten seconds.

How weighted and cost-based routing reduce spend without degrading output quality

Enterprise spending on LLM API calls has climbed fast, jumping from roughly $3.5 billion in late 2024 to $8.4 billion by mid-2025. A large share of finance and procurement leaders report costs blowing past forecasts once they scale. That's the problem weighted and cost-based routing exist to solve.

Weighted load balancing splits traffic across providers by a set percentage, say 70% to one and 30% to another. It's useful for A/B testing two providers, hedging capacity across vendors, or migrating off one provider gradually without touching application code.

Cost-based routing works differently. It matches each request to the cheapest model that can still hit the quality bar for that task. Short classification jobs go to small, cheap models. Complex reasoning tasks go to frontier models that cost more per token but are worth it. The decision has to use request metadata or content signals, not just a static price list, because a cheap model buried under heavy load can end up costing more per useful response than a pricier model that has spare capacity sitting idle.

Done well, this kind of routing, sending easy prompts to cheap models and saving the expensive endpoints for hard queries, can cut spend by 70% or more without any drop in output quality. A cascading or waterfall pattern pushes this further: run the request on a cheap model first, and only promote it to a stronger model if the confidence score is low or the output looks off. Easy queries stay cheap. Hard queries still get the horsepower they need.

Misrouting has a quiet cost too. Running a premium model for routine, low-stakes tasks racks up spend without adding value anyone notices. Without cost attribution broken out by team, product, or workflow, there's no way to tell which workload is actually driving the overage. This is where a gateway earns its keep: model allowlists tied to virtual keys mean a request asking for a model outside policy gets rejected with an HTTP 403 before a single token gets processed. Policy stops being a suggestion and becomes something enforced automatically.

Rate limit handling and why distributing across keys is as important as distributing across providers

LLM rate limits work per key and per minute, not per IP address. That's a structural quirk with no real parallel in normal API infrastructure, and most retry logic written at the application level doesn't account for it at all.

Hit the limit and the provider sends back an HTTP 429. Without a gateway sitting in front, the app just errors out, or some engineer somewhere writes one-off retry logic for their one service. At scale, 429 errors from a single overloaded key start looking like a reliability problem even though the provider itself is perfectly healthy.

Key-level load balancing fixes this by spreading requests across multiple API keys for the same provider, pooling their combined throughput without changing anything on the application side. The app holds one virtual key. The gateway holds the actual pool of keys behind it, invisible to the calling code.

This matters even more for agentic workloads. A single agent session might fire off dozens of small model calls in sequence, or a handful of calls with huge context windows attached. Rate limits can get hit on request count or on raw token throughput, and those are two separate dimensions that both need tracking independently. A gateway that only counts requests will miss the case where one giant prompt blows through the token budget on its own.

Enforcement usually needs to work in tiers: per-key limits for an individual application or developer, per-team limits aggregated across a department, per-project limits, and an organization-wide ceiling above all of it. Each tier gets checked independently on every request, and when any one of them runs dry, the gateway returns an HTTP 402 before the request ever reaches a provider. The payoff is straightforward: predictable throughput and predictable spend, without every team rebuilding rate-limit logic from scratch.

Automatic failover and the mechanics of zero-downtime provider switching

Failover handles a different failure than rate limits do. HTTP 500 errors, timeouts, and full provider outages signal that something is broken. A 429 means slow down. A 500 means switch.

When a provider starts returning server errors or timing out, the gateway routes to a pre-configured backup automatically, with exponential backoff and retry sequencing built in. No application code changes. No engineer paged at 2am to flip a switch. Retrying the same failed provider over and over wastes time it doesn't have; crossing over to a different provider immediately keeps response times steady even while the primary is fully down.

OpenRouter's documented default routing behavior is a useful reference for how this can work in practice: it prioritizes providers that haven't had a significant outage in the last 30 seconds, and among stable providers, it selects using the inverse square of price, with the remaining providers held as fallback. That's a real example of health signals and cost signals getting combined into one working algorithm rather than treated as separate concerns.

For any of this to hold up, the gateway needs to track provider health continuously, not just check it at the moment a request comes in. Health state has to update fast enough that a provider coming back online re-enters rotation on its own, without someone manually re-enabling it.

Fallback ordering isn't arbitrary either. The backup provider needs to clear the same compliance bar as the primary. A failover that routes a regulated workload, something needing a signed data agreement, to a provider that was never configured for it trades one problem for a worse one. Composed correctly, though, individual providers with less-than-perfect uptime can combine into a system with meaningfully higher effective availability. Diversity across providers is essential here. It's the reliability strategy itself.

Semantic routing and hybrid strategies for workloads where request content determines the right model

Semantic routing looks at what a request is actually asking before deciding which model handles it. A long legal reasoning task gets routed to a model known for strong long-context performance. A short classification task gets routed to something small and fast. The application never has to specify which model to use; the content of the request decides that.

This costs more overhead than simpler routing. Rule-based and cost-based routing run on static signals and barely add latency. Semantic routing means actually analyzing content before the request goes anywhere, which takes real compute. That overhead is worth paying when model-task fit changes the output meaningfully. It's not worth paying on every request a system handles.

Hybrid routing is the pattern that shows up in real production systems: cost-saving rules that cap usage of premium models, layered with live performance signals like response time and error rate, layered again with weighted parameters that get tuned toward cost or toward performance depending on what the workload looks like that week.

Research on routing systems (the PROTEUS paper) found something worth taking seriously here: existing routing approaches try to control the quality-cost tradeoff indirectly, through knobs like confidence thresholds or routing percentages, but the relationship between those knobs and actual output quality is not straightforward and can vary significantly across workloads. Operators can't just dial in a target quality level the way they'd set a latency or throughput target in a traditional serving system.

In practice, that means confidence thresholds and routing splits have to be checked against real output quality on the actual traffic a team runs, not set once from a benchmark and forgotten. Hybrid routing is a system that needs re-measuring, not a configuration file that gets written once.

There's a throughput case for tying routing and load balancing together rather than treating them as separate steps. Research on this pattern (RouteBalance) found a combined routing-and-balancing approach reaching 27.6 requests per second, against 21.8 for a baseline that only optimized routing on its own. Semantic routing earns its cost on high-volume pipelines where request types genuinely differ, like a support system that handles quick triage questions and complex case analysis side by side. If every request looks roughly the same, a simpler weighted split will do the job for less overhead.

Observability as a load balancing requirement, not a monitoring afterthought

Bolt observability on after the fact and the whole routing stack becomes harder to trust. Request data has to get shipped somewhere else, traces splinter across separate tools, and debugging a bad fallback decision or a cost spike turns into a scavenger hunt across systems that don't talk to each other. Sometimes the root cause is just gone by the time anyone looks.

A gateway needs to log, on every single request: which provider got picked, which model ran, the routing path taken, any fallbacks that triggered, input and output token counts separately, latency at each stage, cost in actual dollars, which virtual key made the call, and any policy decision that fired along the way.

Token-level cost tracking matters more than it sounds like it should. AI token spend, per Ramp data through mid-2026, grew 572% year over year. Without cost data attached to team, project, and model at the request level, there's no way to figure out which workload is driving that growth, and no way to retune routing policy against it.

The metrics that actually feed the routing decisions: provider error rates broken out by time window, which feeds failover thresholds; latency percentile distributions by provider and model, which feeds latency-routing weights; cache hit ratios, which show whether semantic caching is doing anything; and cost per successful response by route, which feeds straight back into cost-based policy.

Prometheus and OpenTelemetry are the natural integration points here. Production gateways that expose telemetry through these standards let a team plug straight into whatever Datadog, Grafana, or Splunk setup already exists, rather than standing up a parallel monitoring stack nobody asked for. For multi-step agents, distributed tracing matters even more: a single user request might trigger several model calls across different providers and tools in sequence, and trace context needs to follow that whole chain so an engineer can see exactly which hop added the delay, which one triggered a fallback, and which model actually produced what the user saw.

None of this is passive reporting. It's the feedback loop the whole routing system runs on. Teams that treat load balancing as something configured once and left alone will find their weights and thresholds drifting further from what providers are actually doing in production, month over month.

Putting the layers together: designing a multi-provider routing stack that holds under production pressure

The layers stack in a specific order, and skipping one doesn't save time, it just moves the failure downstream.

Start with capability and compliance constraints. Decide which models are allowlisted and which providers actually meet data residency or contractual requirements for a given workload. This is the filter that narrows the candidate set before any routing logic even runs.

From there, build failover chains within that narrowed set, making sure every backup provider meets the same compliance bar as the primary it's standing in for. Then distribute across API keys within each provider to pool throughput, treating rate-limit headroom as an input to routing decisions rather than just an error to catch after the fact. Only after those three layers are settled should cost or latency weighting get applied, since weighted splits and cost dispatch are supposed to operate on options that have already cleared the reliability and compliance bar, not options picked purely because they're cheap. Semantic or hybrid routing comes next, layered on top for workloads complex enough to earn the extra overhead, not swapped in as a replacement for the structural layers underneath. Instrument every one of these layers as they go in, because a routing stack no one can observe is a routing stack no one can actually tune.

A few failure patterns show up often enough to name specifically: a failover chain that routes a regulated workload to a provider without the right data agreement, cost thresholds calibrated against benchmark data instead of the team's real traffic, latency weights built on historical averages instead of live rolling windows, failover logic that keeps retrying a dead provider instead of crossing over, and no cost attribution by team or project, which makes finding the source of a spend spike close to impossible.

Somewhere in this build, the question of building the gateway in-house versus adopting existing gateway software comes up. Either path works. What matters is that the operational weight, routing logic, credential management, observability wiring, failover configuration, gets owned somewhere deliberate. Left unowned, it ends up scattered across a dozen services, each with its own half-finished retry logic and its own blind spot.