Intelligent LLM Routing Based on Cost, Speed, and Quality

Enterprise spending on LLM APIs jumped from $3.5 billion in late 2024 to $8.4 billion by mid-2025, and Menlo Ventures puts enterprise LLM API spend at $12.5 billion in 2025. That's not a rounding error. It's a sign that generative AI moved from pilot project to core infrastructure faster than most finance teams could build a budget line for it.

Most of that spend still runs on habit, and the habit is the actual problem, not the price tag. Teams pick whichever model they already have an API key for, OpenAI or Anthropic or whoever signed the first contract, and every request goes there regardless of whether the task needs it. Sending everything to one provider instead of matching the task to the model is the entire reason AI budgets blow past forecast as usage scales. Not a side effect. The cause.

The provider landscape makes the habit worse. OpenAI, Anthropic, Mistral, Google, Groq, Cohere, Perplexity, and new entrants showing up on a near-monthly cycle: no single one of these wins on cost, speed, and quality at once. That's not a market inefficiency waiting to resolve itself. It's what happens when different labs optimize for different tradeoffs on purpose. Stay locked into one provider and there's overpayment somewhere, latency risk somewhere else, and no way to tell whether the expensive requests actually needed to be expensive.

Routing fixes this, but only if it's built around the right signals, and picking whichever model launched most recently doesn't count as a strategy. It's the same habit with better marketing behind it.

What intelligent routing actually means and how a gateway enables it

Model routing is the layer that decides, per request, which model or provider actually handles it. Not a config file set once and forgotten. A live decision, made fresh every time a request comes in, based on what that specific request needs.

An AI gateway is broadly described as the technology sitting between applications and AI services, acting as a central point for security, governance, and observability across AI workloads. Architecturally it works like a reverse proxy: applications send requests to one stable endpoint, and the gateway handles provider selection, authentication, and policy enforcement behind the scenes. The application never knows or cares whether the request landed on GPT-4o, Claude, or Mistral. A team can swap providers entirely without touching a line of application code, and that's the entire reason to build it this way rather than wiring provider calls straight into the app.

Why put this logic in a gateway instead of scattering it across application code? Because centralizing it closes off the ways teams route around a policy by accident, whether on purpose or just by forgetting to update three different services at once:

• One enforcement point every request has to pass through, so no team can quietly skip it.
• API keys live in one place, never exposed in application repos.
• Every provider, every team, feeds into one audit trail instead of a dozen separate logs.
• Fallback chains, rate limits, and budget caps get handled once, centrally, instead of rebuilt per project.

Analysts covering Gartner's research expect gateway adoption among teams building multimodel applications to climb sharply by 2028. Whatever the exact number lands on, the direction isn't in question.

None of this matters if the gateway routes on bad information, though. So what should it actually be watching?

Cost as a routing signal: which tasks actually need a frontier model

Frontier pricing makes sense for frontier-difficulty problems. Paying frontier rates for a routine classification task is wasted spend, whatever it's dressed up as. It's just waste.

An ICLR 2025 paper makes the mechanism concrete: a matrix-factorization router tested on MT Bench cut cost by 85% while keeping 95% of GPT-4 Turbo's quality, and it pulled that off by sending only 14% of queries to the expensive model. Fourteen percent. That means 86% of traffic in that benchmark never needed the strong model in the first place. On MMLU, a separate BERT-based classifier hit 45% cost savings at comparable quality, using a different mechanism but proving the same point: most requests are easier than the model they get sent to.

Worth saying plainly: these are benchmark-specific numbers, not a guarantee that transfers to any workload. Actual savings depend on how a team's own traffic breaks down by task type, and that breakdown needs measuring before anyone promises a savings figure to finance.

A rough way to sort requests by cost tier:

• Extraction, classification, short summarization: small, cheap models handle these fine, full stop.
• Multi-step reasoning, code generation, complex synthesis: this is where frontier models earn their price.
• High-volume, repetitive requests: candidates for caching before they ever reach a routing decision.

The tooling has caught up. Vercel's AI Gateway, generally available since August 2025, added a per-request sort: 'cost' strategy that routes across 40+ providers with automatic failover. OpenRouter takes a different default approach, load-balancing with inverse-square price weighting, so a provider charging $1 per million tokens gets tried nine times more often than one charging $3.

There's a spend-stability angle too, and it's the one finance teams actually feel in their gut. Ramp's data shows the median business's AI spend swinging by around 58% month to month, with 61% of businesses seeing swings of 40% or more. Cost-aware routing doesn't just shave the bill down. It makes the bill predictable enough to plan against, which is a different and arguably bigger win. Ramp also found AI token spend grew 572% year-over-year from June 2025 to June 2026, and at that growth rate, unmanaged cost routing doesn't stay a small problem for long. It compounds fast, and it compounds quietly until someone finally opens the invoice.

Latency as a routing signal: matching response speed to what the user actually experiences

Latency is a variable number. It's a distribution that shifts with load, time of day, and whatever the provider's own infrastructure is dealing with that hour. Whichever provider is fastest at 9am might be visibly slower by 3pm, so routing on latency really means routing on current state, not last week's average. Treating it as static is the mistake most routing setups make first.

Two dimensions matter here, and conflating them is a second, separate mistake. Time to first token (TTFT) is what a user actually perceives as responsiveness in a streaming chat interface. Tokens per second (TPS) is throughput, and it matters more for long-form output where the user waits on the whole response rather than the first word of it. Vercel's gateway splits these directly into separate sort: 'ttft' and sort: 'tps' strategies, a fair sign that the industry has stopped pretending "latency" is one number.

The gateway itself has to stay out of the way, too. If the routing layer adds real delay, it defeats the whole point of routing for speed. Some gateways run at single-digit millisecond latency and hundreds of requests per second on modest hardware. Python-based options built for prototyping tend to add hundreds of microseconds to multiple milliseconds once concurrency climbs: fine for a demo, a real liability once production traffic hits it.

This shows up hardest in agentic workflows. A single user action can trigger five or six sequential LLM calls chained together, and per-call overhead from the gateway stacks up across that whole chain. At that point gateway efficiency stops being an infrastructure detail. It becomes part of the product experience.

Fallback chains double as latency resilience, not just uptime insurance. When a provider throws a 429 or a 500, automatic rerouting to a backup keeps the request moving without anyone paging an on-call engineer. That's latency routing under failure conditions, and it's probably the most practical version of latency routing a team runs into day to day, more useful than any dashboard sorted by TTFT.

Routing purely on speed, with no cost awareness built in, can quietly send traffic to a faster but pricier provider when a slightly slower, cheaper one would've done the job just as well. The three signals have to work together. Isolate any one of them and blind spots open up in the other two.

Quality as a routing signal: why benchmark rank is not a routing table

A model that tops a general leaderboard can still be mediocre at the narrow extraction task a product actually runs. Quality is a property of the model in combination with other factors. It's a property of the model doing this specific task with this specific prompt, and a single benchmark ranking is a poor stand-in for a routing rule no matter how impressive the leaderboard position looks.

RouterBench, published by Hu et al. in 2024, built a reference dataset of over 405,000 precomputed inference outputs across 11 different LLMs, tested against multiple tasks including MMLU, MT-Bench, MBPP, HellaSwag, WinoGrande, GSM8K, and ARC. What it actually gives a team is a structured picture of how quality shifts by task across models, which matters more than any single aggregate score. A separate framework, LLMRouterBench, runs its performance-cost evaluation across a pool of 13 flagship models from eight providers, and the spread within that pool, in both capability and price, is itself the argument for why no single routing rule serves every task.

RouteLLM tested four router architectures trained on human preference data and found that matrix factorization routers produced the best tradeoffs in practice. Learned routing beat static rules once the task distribution got complex enough, which means quality routing gets better with data, not with cleverer prompts.

Three approaches show up repeatedly in practice, and they trade accuracy for latency in a fairly predictable order:

• Pre-request rules classify the request type (code, summary, reasoning) up front and route accordingly. Cheapest to run, least precise.
• At-inference cascades send the request to a small model first, check the output against a quality threshold, and escalate to a stronger model only if it falls short. Most accurate, but it costs a round trip.
• Post-response retry re-routes after the fact, on failure or a low-confidence signal. It's the safety net underneath the other two.

OpenRouter's Auto Router has exposed cost-quality tradeoff controls that let teams dial between the most capable and the cheapest model, turning a philosophical argument into one configurable setting instead of a debate at every planning meeting.

Braintrust connects gateway traffic directly into its tracing and evaluation pipeline, so teams run evaluations against actual production traffic instead of synthetic test sets. That's the version of quality measurement worth trusting, and it points at something teams tend to forget: quality isn't a one-time model selection decision. It drifts. Models get updated upstream, prompts evolve, request distributions shift. Measure it once and move on, and the routing policy goes stale without anyone noticing until output quality has already slipped.

How the three signals interact and where single-signal routing fails

Optimize any one of these three signals in isolation and the other two pay for it. The cheapest model is often slower and less accurate. The fastest model often costs more and cuts corners on reasoning. The highest-quality model is usually the slowest and priciest option on the table. There's no way around that tension, only ways to manage it, and pretending one signal can carry the other two is where most routing setups quietly fail.

Timing is where the framework turns usable instead of theoretical:

• Pre-request rules: cheapest to run, best for high-volume, predictable task types where the category is obvious before the request even goes out.
• At-inference cascades: the most accurate option, worth the added latency when quality varies a lot and there's room in the latency budget to absorb a second call.
• Post-response retry: the safety net for compliance-sensitive or high-stakes outputs, where getting it wrong once costs more than the extra round trip.

Building an actual routing policy means answering three questions per workflow, not once for the whole system. What's the latency budget: real-time chat, or an async batch job running overnight? What's the acceptable quality floor: a rough classification task, or a customer-facing response carrying the brand's name on it? And what's the cost ceiling, per request, per team, per month?

A rough taxonomy turns those questions into policy:

• High-volume, low-complexity, async work: optimize for cost, accept a slower model.
• Customer-facing, latency-sensitive, low-stakes: optimize for TTFT, accept paying a premium for it.
• Low-volume, high-stakes, complex reasoning: optimize for quality, accept the cost and the wait.
• Agentic, multi-step workflows: different steps in the same workflow need entirely different priorities. Step one might be a cheap classification call. Step four might need the frontier model.

Static rules age badly here. Pricing shifts, new providers show up, and quality characteristics change every time a provider pushes a model update. A routing policy written six months ago is making decisions based on a market that no longer exists, which is exactly why observability has to sit inside routing, not bolted alongside it. It's the feedback loop that keeps the whole thing honest. Without per-request data on what a decision cost, how fast it came back, and whether the output held up, a team is routing on a guess it made once and never revisited.

What a production routing implementation requires beyond routing logic

Routing logic is the part everyone designs first, and the part that matters least without the infrastructure underneath it.

Virtual keys come first. Teams and applications get scoped keys that never touch a real provider credential, since the gateway holds all of that centrally, which means revoking access is instant and doesn't need a deployment. Budget enforcement has to happen at the token level, not the request level, because a single long prompt can cost a hundred times more than a short one. Counting requests instead of tokens is exactly how teams end up surprised by their own invoice.

Semantic caching sits in front of routing entirely. If two prompts ask the same question in different words, the cached answer should come back before any routing decision gets made at all, cutting cost and latency in the same move. Cache hit rate belongs in the same observability dashboard as routing outcomes, not buried in a separate report nobody opens.

Provider outages aren't rare events. They're a Tuesday. Fallback chains that automatically reroute on a 429 or 500, with zero manual intervention, are a baseline requirement, not a nice-to-have reserved for teams with a bigger budget.

Observability has to be built in, not stitched together after the fact from provider dashboards and a separate monitoring tool. Fragmented logging like that creates real debugging gaps, the kind where a request fails and nobody can tell which layer it failed in. What this actually takes: structured metadata per request (provider, model, tokens in and out, latency, cost, which fallback path fired), metrics in a format that plugs into existing monitoring, and distributed traces that follow a request through the entire routing path, not just the final model call.

Finance needs visibility into this too, and right now most don't have it. SpendHound's 2026 AI Spend Report found 46% of finance and procurement leaders exceeded their AI budget in 2025, compared to a lower share who overspent on traditional finance and accounting software. That nine-point gap says something specific: AI spend is harder to predict and harder to attribute than the software budgets finance teams already know how to manage. Routing decisions carry real financial consequences, and someone outside engineering needs to see spend broken down by team, by project, by model, not one aggregated invoice at the end of the month.

Security can't be an afterthought at the gateway layer either. PII redaction before a request ever reaches a provider, audit logs that hold up in a compliance review, role-based access control over who's allowed to route to which models: none of that is an enterprise upsell. It's a requirement for any team touching customer data at all.

The gateway options on the market span a real range, and the right pick depends on infrastructure a team already runs, how deep its observability needs go, and how much governance the use case demands. Open-source options built around a large model catalog and an OpenAI-compatible API work fine for prototypes and lighter workloads, though some hit real latency and scaling ceilings once concurrency climbs in production. Managed services like OpenRouter offer broad provider coverage across many providers, plus an automatic router that exposes the cost-quality tradeoff as a simple dial, though teams give up some visibility into the internal routing logic in exchange for not running the infrastructure themselves. Vercel's AI Gateway, generally available since August 2025, offers per-request sort strategies for cost, TTFT, and TPS, and automatic failover across 40+ providers.

None of these tools replaces the decision a team has to make first: which signal matters most, for which task, and how that answer changes as the workload does.