Token Budgeting and Request Throttling at the Gateway Layer

Enterprise LLM spend hit $12.5 billion in 2025, per Menlo Ventures. At that size, you can't control cost the way most teams still try to: watching request counts and hoping for the best. Most AI teams blow past their forecast by 40% or more once they scale, and the overrun almost always comes back to one thing. Nobody set spending limits at the single point in the stack that can actually see everything happening at once, which is the gateway.

Why request count is the wrong unit for LLM cost control

Old-school rate limiting counted requests. Fine for a normal API, where each call costs roughly the same to serve, but LLMs blow that idea up, since one request might be ten tokens and the next might be ten thousand. A request counter genuinely cannot tell those two calls apart.

I've watched two teams send the same number of requests and land on wildly different bills. One's running a GPT-4-class model with a long system prompt and a fat context window; the other's calling a smaller model with short, tight completions. Count requests and they look like twins. Count tokens and one team is spending ten times the other.

Agentic workflows make it worse. An agent finishing one task might fire off a dozen calls behind the scenes: a retrieval lookup, a tool call, a reasoning step, another tool call, a final answer. Each one lands as a single request, and none of them looks alarming by itself. Add them up, though, and you've got a loop sitting comfortably under any request ceiling while it quietly eats through the token budget.

So you have to start by tracking the right thing. Three numbers matter: prompt tokens (what you send, system prompt included), completion tokens (what comes back), and total tokens (the sum). Most providers already hand these back in the response metadata. A gateway sitting between your app and the provider reads that off every response on its own; you don't touch your application code at all.

Once tokens are the unit you're actually counting, the next question is where you enforce limits on them, and how tightly.

How token budgets and spend limits are structured at the gateway layer

Two control types matter here, and they're not doing the same job. Token limits cap usage in the model's own currency: tokens per minute, per day, per month. Spend limits cap usage in dollars, scoped to any mix of model, provider, team, or app, and they run independent of the token counters.

You need a short fuse and a long one. Short-term limits, per minute or even per second, stop bursts, so one runaway agent loop can't saturate your capacity in sixty seconds. Long-term limits, daily, weekly, monthly, guard the budget itself, so spend doesn't drift past what finance signed off on. A common setup pairs a token-bucket algorithm for the burst side with a fixed window for the cumulative side.

Envoy's AI Gateway documents the loop plainly: a request hits the gateway, which checks whether serving it would push the relevant counter, team, app, model, whatever the scope is, past its configured limit. Breach the limit and the request gets rejected with a 429 before it ever reaches the model provider. Stay within bounds and it gets forwarded, with the gateway reading the real token counts back off the provider's response and updating its counters.

This is also where model-specific policy lives. Set tighter limits on your expensive reasoning models, looser ones on cheap fast models, configure it once, and no service calling that model has to rebuild any of it.

Hitting a limit doesn't have to mean a hard stop, either. Cloudflare's AI Gateway lets you scope dollar limits to a model, a provider, or a custom tag like team or app, on fixed or rolling windows. Hit the ceiling and you can block the request, or reroute to a cheaper fallback model instead of killing the whole workflow.

Here's one thing worth checking before you trust any of this: if the gateway runs across multiple nodes, the counters need to live somewhere shared, usually Redis. Your "global" limit is actually five separate limits otherwise, one per node, none of them talking to each other.

Scoping controls to the right organizational unit

A single global token cap is almost never right. Set it high enough to cover your heaviest workload and everyone else has the door wide open, and set it low enough to rein in casual use and you've choked the team doing legitimate heavy lifting.

Big AI deployments end up multi-tenant whether anyone planned it that way or not. The tenants might be departments, product teams, staging versus production, individual agents, or external customers if you're reselling AI features. Cloudflare has talked about this after conversations with hundreds of companies, and it comes down to the shared API key problem: everyone's hitting the same key, the invoice lands at month's end, and finance can't tell which team, which feature, or which runaway agent produced the number sitting in front of them.

Scoping fixes that along a few lines:

• By team or department, which gives you chargeback reporting and someone to hold accountable
• By application, splitting what customers trigger from what internal tools burn
• By environment, so staging can't eat into a production budget
• By model, tightening the leash on expensive ones and loosening it on cheap ones

Cloudflare's docs describe exactly this setup: limits scoped to any combination of model, provider, or a custom attribute like user, team, or app, on daily, weekly, or monthly windows, fixed or rolling.

This is the point where rate limiting stops being a technical safeguard and starts acting as governance. The same mechanism that stops a runaway job is producing the attribution data finance needs to do chargeback right.

Alert granularity is its own trap, by the way. A monthly budget alert smooths out day-to-day noise, which sounds fine until you realize it means you under-alert for the first three weeks of every billing cycle. A feature that doubles its daily spend on day three won't trip a monthly threshold until real damage is already done. Daily budgets on rolling windows catch that early; monthly views catch it late, after the fact.

Where application-level controls consistently fall short

I get why teams start here. The code's already open, adding a counter feels small, and it doesn't mean standing up new infrastructure. But this approach has a ceiling, and it's a low one.

Limits written into application code only see what that application sends. A limit inside Service A knows nothing about what Service B or Service C is doing against the same provider key. Add a new agent or team six months from now, and someone has to remember to wire up a limit for it, and often, nobody does.

There's a subtler issue too. Application code tracks what it thinks it sent, not necessarily what the provider actually billed. The gap between the app's token estimate and the provider's real count barely shows on any single request, but it compounds over thousands of calls into a bill nobody's mental model matches anymore.

Agentic tool-call chains are the hardest case, because the total cost of a task isn't knowable until the whole chain finishes. Application code often doesn't notice the chain got expensive until it's already over, while a gateway sees every call in that chain as it happens, so it can cut it off mid-stream if it has to.

Provider-level controls have their own limit, and it's a structural one: they exist to protect the provider's infrastructure, not your budget. They apply at the account level, with no concept of "team" or "app" inside your organization.

As of 2025, only about a third of companies had mature AI cost management in place, per CloudZero. Most of the market runs without steady enforcement anywhere, and Deloitte found enterprises using generative AI saw a 35% average jump in cloud spend tied directly to unwatched token use.

The deeper issue is consistency. Policy written into application code only applies where a developer remembered to put it, and it decays as the codebase changes hands. Policy enforced at the gateway applies to every request, full stop, because there's no other path in.

Agentic workflows and why static thresholds need a rethink

Agents don't behave like a person typing into a chat window. A person sends a message, waits, reads the answer. An agent finishing one task might chain tool lookups, retrieval queries, several rounds of reasoning, and a final completion, back to back, with no human pause anywhere in the middle.

Gartner's 2026 research on agentic workflows found they burn far more tokens per task than standard chat use. It calls this one of the biggest hidden cost multipliers showing up in enterprise AI right now, and I'd bet most finance teams haven't priced it in yet.

Two problems stack here. A fast sequence of calls from one agent can slam into a short-term rate limit and fail the task partway through, which is its own expensive mess to clean up afterward. Separately, if any step in the chain hits a retry, token use can spike before application code even notices something's wrong.

Retry budgets handle the second problem directly. Configure the gateway to cap how many times it retries a failed call before it gives up and returns a clear error. Skip that cap and a temporary provider hiccup turns into an unbounded retry storm; a provider incident quietly becomes a cost incident.

Fallback routing handles the first. When a spend limit gets hit, routing the rest of the workflow to a cheaper model, instead of throwing a hard error, keeps the task moving without blowing the budget. That's a decision the gateway makes in one place. Push that logic into application code instead and you're duplicating routing rules across every service that calls a model, the same fragmentation problem all over again.

Throttling agentic work well means the gateway has to understand context: which model this is, what the session has already spent, where it should fall back to. A static threshold with no memory of any of that just doesn't cut it.

What centralized enforcement makes possible for finance and operations teams

Enterprise AI spend averaged $85,521 a month in 2025, according to CloudZero. At that size, waiting for the invoice to land at month's end isn't monitoring, it's delayed discovery. The real question is whether anyone can see spend as it happens, not after the fact.

A gateway gives you visibility a provider invoice never will: which team, app, or agent generated a given cost, whether the model picked was actually the right price for that workload, and whether current usage is on pace to blow through the monthly ceiling before the cycle even closes.

Chargeback reporting falls out of this almost for free. Once limits are scoped by team or department, the same data enforcing those limits is what finance needs to split costs fairly across the org. About 42% of enterprises already run some kind of middleware layer for their AI infrastructure, and most landed on this cost-attribution benefit as a side effect of solving a plain engineering problem. Instrument every request the same way, and attribution comes along for free.

There's the emergency case too. A gateway can cut off every request from a specific team, app, or key right away: a config change that takes seconds, not a code deploy that takes a sprint.

What actually changes is the posture. AI spend moves from something reconstructed after the bill shows up to something watched live, with roughly the same granularity cloud teams have had over compute and storage costs for a decade now.

Where Concentrate and other gateways sit in this picture

There's a real build-versus-buy decision here. Self-hosted gateways like LiteLLM give you full control, but that control has a price tag: somebody owns the deployment, the Redis cluster behind your shared counters, the upgrade cycle, and the on-call shift when the gateway itself falls over. Managed gateways take that weight off your plate. You write the policy; someone else keeps it running underneath.

LiteLLM supports over 100 LLMs and has real traction as a self-hosted option.

Concentrate connects engineering teams to more than 130 LLM providers through one API, no separate provider keys, no custom integration code per provider. Token budgets, spend limits, model-specific rate limiting, fallback routing, all of it applies across that whole set of providers from a single configuration layer, with real-time spend visibility broken out by team, project, key, model, and provider.

Here's what matters most for a team growing fast: once usage outpaces your ability to keep governance current by hand, a managed gateway means the policy stays enforced without a dedicated platform team babysitting the gateway's own infrastructure. Check the fee structure closely, though, since per-token platform fees stack on top of whatever the underlying provider already charges, and that overhead adds up fast once you're at real scale.

Whichever gateway you land on, the test doesn't change. Controls only work if they apply everywhere, automatically, without someone remembering to wire them in. A gateway that needs per-service setup or manual instrumentation leaves the fragmentation problem this piece started with unsolved; the problem just moves somewhere else.