Eliminating Provider Credential Sprawl in AI Teams
AI-written code leaks secrets twice as often, and most teams never revoke them.
GitGuardian found 28.65 million new hardcoded secrets in public GitHub commits over 2025, up 34% from the year before. That's the biggest single-year jump on record, and the AI-specific slice of that number is worse than the headline suggests.
Commits written with AI help leaked secrets at a 3.2% rate, against a 1.5% baseline for public commits generally. That's roughly double. Credentials tied to AI services specifically, meaning LLM provider keys, embedding services, AI platform tokens, rose 81% year over year to 1,275,105 detected leaks in 2025. Meanwhile, 113,000 DeepSeek API keys turned up sitting in public repositories that same year. People commit these things without thinking twice.
Leaking a key does damage once. Never revoking it does damage for years, and that's the part most teams get wrong: they treat a leak as a one-time event instead of an open door. Of the secrets leaked back in 2022, 70% were still valid in 2025. A key someone committed to a repo three years ago is probably still live today, sitting there for anyone who stumbles across it.
Here's what doesn't show up in the leak-count headlines: code review pressure doesn't disappear just because an AI wrote the integration. Code you didn't write yourself, even code you technically approved, can move through review faster than code you authored line by line. As AI assistants write more of the integration boilerplate, more credentials get baked in without anyone actually looking.
Why non-human identities multiply faster than security teams can track them
Non-human identities, meaning service accounts, API keys, bot credentials, already outnumber human ones by a wide margin. 1Password puts the ratio at 82 to 144 non-human identities for every single human identity in the average enterprise. That gap is widening, not closing.
Most security teams make the same mistake here: they treat agent credentials like developer credentials, and the two don't behave anything alike. Agents provision and abandon credentials at a pace and volume that human workflows never produced, generating far more credential churn than traditional developer access patterns. Agents also start out overprivileged almost by default, because scoping permissions individually for every agent is real work, and most teams skip it and hope nothing goes wrong. That's the wrong bet to make.
The tooling doesn't help either. Identity governance was built around human accounts and standard IAM reviews. Non-human identities mostly don't appear in any of those places. IBM's 2025 reporting found that 97% of AI-related breaches involved AI systems that lacked proper access controls. Overprivilege isn't a theoretical risk sitting in a whitepaper somewhere; it's the dominant pattern in actual incidents. Shadow AI, meaning models or providers running without security team knowledge, adds another layer on top, and IBM attributes 20% of breaches to shadow AI specifically. The tooling built to manage credentials assumes human-scale key creation. It was never built for the pace at which agents provision and abandon credentials on their own.
What an exploited AI credential actually costs
Credential abuse shows up in nearly 40% of 2025 breaches, per the 2026 Verizon Data Breach Investigations Report. That's the dominant attack vector now, not a rare edge case buried in an appendix.
Once an attacker has a live LLM provider key, the clock starts immediately. They pull data flowing through the API, run inference on the victim's account and dime, or pivot through that same key to reach other connected systems. Breaches involving shadow AI have been found to cost up to $670,000 more than comparable breaches that didn't involve it. The reason is simple: prompts often carry customer data, personal information, or proprietary business logic sent straight to the provider. If the key has no spend cap attached, an attacker can run inference on someone else's account for as long as it stays unnoticed.
What actually hurts teams isn't the breach itself. It's the audit failure that follows. Without central logging, there's no way to answer the basic incident-response question: which requests went out under this key, who sent them, what data did they carry? That turns response into guesswork. Remediation drags too, because rotating a key that's hardcoded across a dozen services means hunting down every reference by hand, and that search gets brutal fast when no registry exists to begin with.
Why the problem compounds as AI usage scales inside an organization
At small scale, sprawl is manageable through sheer memory. One or two engineers know where every key lives and what it's scoped to. It holds together, barely, the way a lot of things held together back when the team was smaller.
Growth breaks that arrangement fast. New teams show up, new projects, new providers, sometimes new contractors, and none of them inherit the tribal knowledge of where the old keys sit or what they were ever meant to do. Agentic pipelines make the math worse: a single user request can now trigger model calls across several providers, each needing its own credential, and the resulting credential surface grows in ways a human developer workflow never did. Hand an agent an unscoped key with no budget ceiling, and it runs up cost and access at the same time, with nobody watching.
The market has already priced this in. Industry analysts project that 70% of software engineering teams building multimodel applications will use AI gateways by 2028, up from roughly 25% in 2025. Teams that skip centralization now aren't avoiding the work. They're deferring it, and deferred sprawl gets paid back with interest. Separately, 53% of AI teams report costs blowing past forecasts by 40% or more while scaling. Spend control and credential control are the same problem, viewed from two angles. Skip one, and it's fair to assume the other is already broken too.
How centralizing credentials through a gateway layer eliminates the root cause of sprawl
The fix has to live in the architecture, not in a memo. Telling engineers to "track your keys better" will not survive contact with a Friday deadline. Instead of every application or agent holding its own provider key, a gateway holds all the provider credentials and hands out internal virtual keys to teams and services. Applications authenticate to the gateway; the gateway authenticates to the providers. The real provider keys never leave that layer, so they're not sitting in application code, not in CI/CD pipelines, not in some developer's dotfiles.
Virtual keys become the thing teams actually manage day to day: scoped, revocable, capped by budget, fully auditable, and none of that touches the underlying provider credential. Cutting off a contractor means revoking their virtual key. The provider key underneath stays untouched and needs no rotation. A virtual key can be locked to a single model, a spending ceiling, a time window, or a specific team. A raw provider key can't do any of that on its own.
There's a second benefit that has nothing to do with security. A team running GPT for summarization and Claude for code generation would otherwise maintain two SDKs, two auth setups, two billing systems in parallel. A gateway collapses that into one integration, with no separate auth flows and no separate billing statements to reconcile at the end of the month. Provider keys rotate at the gateway level too, so teams never touch their code when a credential changes upstream. None of this asks developers to follow a new convention or remember a new rule, which is exactly why it works. The fix sits at the infrastructure layer, and that's the only place it actually holds.
What the gateway layer needs to answer in production: observability, routing, and spend
Running a gateway means answering operational questions in real time, not after the fact. Which provider just returned a bad response? Was that a latency spike or an actual quality regression? Which virtual key just blew through its budget? Did the fallback route kick in the way it was supposed to?
Smart routing pays off on cost and quality both, and the savings come from a real spread in model pricing. Claude Haiku 4.5 runs around $1 per million input tokens; frontier models cost several times that. Routing simpler or batchable tasks to cheaper models is a structural lever, not a marginal tweak. Analysis covering 2.4 billion enterprise API calls found blended AI usage cost fell 67% year over year, from $18.40 to $6.07 per million tokens between Q1 2025 and Q1 2026. Even with that drop, token volume climbs faster than budgets planned for it. The FinOps Foundation's 2026 State of FinOps report found 73% of enterprises saw AI costs exceed original projections, and routing decisions made centrally at the gateway are one of the few real levers available to close that gap.
Spend attribution only works when every request passes through one point. Provider dashboards show account-level totals; they don't show which team, feature, or agent drove the spend. The gateway is the only layer positioned to tag consumption by team, project, virtual key, model, and provider as it happens. By 2026, 98% of FinOps practitioners carry responsibility for managing AI spend, up from 31% in 2025. Without per-request attribution flowing from the gateway, most of them are stuck reading last month's invoice instead of watching a live number.
Architecture matters here too. Gateways that bolt observability on as a separate tool produce fragmented traces, forcing engineers to debug across systems that don't talk to each other. A gateway built with observability native to the request path keeps routing decisions, cost data, and request logs in one place. Worth naming as well: a large share of enterprise AI workload has nobody sitting there waiting on the answer, think nightly batch jobs, bulk classification, report generation. Major providers offer asynchronous batch processing at meaningfully lower cost than real-time calls, and a gateway can route that batchable work to batch endpoints automatically, by default, without anyone remembering to ask.
The security controls a gateway enables beyond credential centralization
Centralizing credentials solves one problem. A stolen key is only half the risk; the other half is what data leaves the building and under what conditions, and that needs its own layer of control sitting at the same choke point.
Without a gateway, nothing inspects outbound requests before they hit a provider. Redaction has to happen inside application code, and it happens inconsistently, integration by integration, because every team implements it slightly differently or skips it under deadline pressure. A gateway applies redaction the same way for every request, catching PII, payment data, or health information before any of it leaves the building, with every redaction reviewable after the fact.
Zero data retention agreements with providers mean little if the gateway itself logs raw prompts on the way through. The audit trail needs to capture metadata, not the sensitive payload, wherever retention rules demand it. Role-based access control at the gateway settles a related question: which teams can reach which models, which providers, which capabilities. It enforces that as a structural rule rather than something everyone's just supposed to remember, and that distinction matters most in regulated industries, where certain categories of data must never reach certain providers, full stop, no exceptions.
Audit logs sitting at the gateway answer exactly what a regulator or auditor will ask: who sent what, to which provider, when, under which key. That's the evidence a SOC 2 audit needs, the same evidence an incident investigation needs, the same evidence a compliance review needs. Rules enforced at the gateway apply to every request from every team, automatically. Rules written into application code are only as strong as the one integration that forgot to implement them, and there's always one.
How teams typically move from direct integrations to a centralized gateway
Most teams don't tear down their stack to adopt a gateway. They insert it as a middle layer, swapping base URLs in existing code rather than rewriting anything from scratch. Gateway providers generally offer endpoints compatible with existing SDKs, so the actual code changes stay small, often a config file and nothing more.
The sequence that avoids breaking things looks about the same across teams. Provider keys move into the gateway first, and new direct integrations stop getting created while the old ones are still running. Virtual keys go out to teams as replacements, scoped by team and by use case rather than handed out generically. Once all traffic is confirmed flowing through the gateway, the old direct provider keys get retired for good. Routing rules, budget caps, and data controls get layered in gradually, as confidence in the setup builds.
There's a build-versus-buy decision buried in here, and one side wins for most teams. Self-hosting an open-source gateway keeps infrastructure costs down, but it puts upgrades, uptime, and security patching on the same engineering team that would rather be shipping product. For a team without spare headcount to babysit an internal tool, that trade rarely pays off; a managed gateway shifts the operational weight onto the provider instead. Evaluating a managed option comes down to three questions worth asking directly: does it cover the full breadth of providers the team uses now and might need later? How granular are its virtual key controls around scoping, budgets, and revocation? And does it carry observability natively, with request logs, cost attribution, and routing traces in one place instead of scattered across a bolted-on tool?
