LLM Spend Anomaly Detection in Production Systems

LLM spend behaves nothing like compute or storage spend, and that difference, not its size or its growth rate, is why so many teams get blindsided by it. Google's SRE framework gave the industry four golden signals for production systems: latency, traffic, errors, saturation. Those four signals say nothing about whether a model's output was any good or what it cost to produce, and that gap is exactly where AI spend hides.

Token consumption doesn't move in clean, predictable steps. It scales with prompt length, with how many inference calls a feature makes, with how deep an agent's reasoning loop runs, with which model version is serving a request, and with how much of the context window gets filled on any given call. Changing a prompt template, shipping a new feature, or letting a provider quietly update a model checkpoint can shift every one of those variables at once. Cost degradation in these systems creeps rather than jumps, so a team can be sliding down a bad slope for days before the number gets big enough to notice.

Agentic systems make the problem worse by removing the natural checkpoints that used to exist. A single agent can generate tokens on its own, continuously, without a human in the loop approving each step, and production systems now run dozens of these agents at once. Watching any one call in isolation tells a team almost nothing, because the spend that matters is the sum of thousands of autonomous decisions happening in parallel.

Standard cloud monitoring wasn't built for any of this. It can report, down to the cent, what an organization spent on EC2 or on S3 last month, but it has no concept of a "feature" or a "customer" sitting inside that number. It cannot tell a team that the spike came from one workflow, one account, or one bad prompt. That blind spot is structural, a matter not of turning on a better report but of the monitoring having no concept of features or customers, and it is why a feature that costs a fraction of a cent per request at low volume can turn into one of the largest line items on the monthly bill once it scales. The math compounds faster than most teams budget for, and by the time finance notices, the pattern has often been running for weeks.

How aggregate dashboards hide the anomalies that matter most

Averaging is the enemy here. A dashboard built to show the health of the whole system will, by design, smooth over the handful of traces or tenants that are quietly running up the bill. A platform engineer watching the top-line number might see token costs creep up by a modest percentage over the course of a week and treat it as noise, when in reality a single tenant is generating a runaway-cost trajectory that the gateway never flagged on its own. The global view looked fine. The individual account did not, and nobody was looking at the individual account.

This is the entire reason anomaly detection exists as a discipline separate from ordinary monitoring: most traces in an LLM or agent system are completely normal, and a small number are quietly catastrophic. A response can hit every infrastructure target, fast, cheap, technically successful, and still represent a serious cost anomaly once it's judged against what that workload normally looks like. Monitoring tells a team that a number changed. Explaining why it changed is a different job, and it's the job aggregate dashboards were never built to do.

One tenant lands on an unusual prompt pattern that burns far more tokens than its peers. An agent loop fails to terminate cleanly and keeps calling itself. A provider pushes a model update that silently makes outputs longer. A prompt injection attack tricks a model into generating far more text than the task requires. Each of these produces a real cost event, and each one gets buried the moment it's folded into a system-wide total.

The organizational cost of this blind spot is large: most enterprises miss their AI cost forecasts by more than a quarter, and the visibility gap described here is the direct cause. That's the predictable outcome of asking an averaged, aggregate view to do a job that requires looking at individual traces.

What a production LLM gateway must instrument

None of this can be fixed by better dashboards alone. It requires capturing the right data at the right point in the system, and that point is the gateway. When applications send every model request through a single endpoint rather than calling providers directly, the gateway becomes the one place in the architecture where every call, from every team, through every provider, passes through a shared layer of auth, routing, and policy. That position is what makes it the natural home for metering.

The pipeline built on top of that position has to behave like a stream rather than a batch job. Token counts need to arrive per call and get rolled up into per-team, per-tenant, and per-model series within seconds of the request completing. Aggregate the data on a daily or weekly batch cycle instead, and the resolution needed to catch a spike while it's still small gets lost before anyone sees it.

What gets captured at that layer matters as much as how fast it's captured. Prompt tokens and completion tokens need to be counted separately rather than collapsed into one total, because bloat on the input side and bloat on the output side point to different causes. Every trace needs a model identifier and a provider tag, since pricing differs enough across models that cost-per-trace can't be computed accurately without knowing exactly which one handled the request. Every trace also needs caller identity: which team, which workload, which API key, which tenant generated it, because those are the exact dimensions anomaly detection has to slice along later. Latency needs to be broken into its components, time to first token, time per output token, and total end-to-end latency, because latency spikes and cost spikes often share a root cause. Error type and provider response codes need their own place in the stream, since an elevated failure rate is its own anomaly signal distinct from cost. And for agentic workloads specifically, the gateway needs to track tool calls, sub-agent handoffs, retry counts, and loop depth on every trace, because those are exactly the mechanics behind runaway agent cost.

All of this belongs in the same observability stack that already tracks latency and error rates, where finance can see it in real time rather than in a separate monthly cloud bill reviewed after the fact. Keeping cost data next to performance data is what lets a team correlate the two instead of discovering the connection weeks later. The same logic applies to anomaly scores themselves: a score needs to live on the same span as the LLM call it's judging, because a score sitting in a parallel system is a score nobody doing incident forensics will ever find.

Building baselines that reflect how LLM workloads vary

Once the signals are flowing, the next decision shapes everything downstream: what counts as normal. The single most common mistake at this stage is setting one global baseline for the whole system. Workloads, models, teams, and customer tiers each carry a different normal range, and a threshold built for the average of all of them will be wrong for almost every individual cohort.

A meaningful cohort has to be defined along several dimensions at once. Model and provider come first, since a GPT-4-class call and a call to a smaller, cheaper model produce entirely different cost distributions, and comparing them directly will flag the bigger model constantly for doing what it's supposed to do. Workload type matters just as much: a batch summarization job has a completely different token-per-call profile than a conversational assistant, which in turn looks nothing like an agentic research workflow running multi-step tool calls. Team and feature matter too, because the same underlying model serving customer support and serving internal search will carry different prompt structures and different volume patterns even though it's technically "the same model." And time window has to be part of the baseline, since daily and weekly seasonality produce cost swings that are completely legitimate but will trip a static threshold every single cycle.

One useful approach to building this kind of baseline is to cluster prompt embeddings from a defined window of normal traffic and score every incoming prompt against that cluster's center. Combining that semantic-novelty score with token-cost-per-trace data catches anomalies that a cost threshold alone would miss entirely, since a prompt can be cheap and still be completely unlike anything the system has seen before.

Drift and anomalies are not the same problem, and treating them as one leads to a baseline that quietly stops working. Drift moves the entire distribution over time. An anomaly is a single outlier sitting apart from a distribution that hasn't moved. A baseline that recalibrates too aggressively in response to either will start absorbing real cost growth as the new "normal," at which point it stops catching anything. The only way to know if that's happening is to track how the anomaly score distribution trends over time: a distribution that keeps flattening out is a sign the detector itself is going numb rather than a sign the system got healthier. Calibrating any of this requires a labeled set of historical anomalies to test against, because without one there's no way to measure whether the detector is missing real incidents or just generating alert fatigue with false positives.

The three classes of spend anomaly signal

Production systems generally need three alert types working together: budget alerts for when daily or monthly spend approaches a set threshold, anomaly alerts for unusual patterns like sudden spikes or latency degradation, and error alerts for elevated failure rates. Together, these turn cost tracking from something a team checks on passively into something that actively governs spend.

The first class is the sudden spike: a small number of traces generating unusually high token counts per call, usually traced back to a single tenant, a prompt injection attack, a runaway agent loop, or a provider update that quietly made outputs longer. Left undetected, a prompt injection or a looping agent can generate a very large amount of unplanned spend in a single overnight run. The second class is the slow creep: prompt bloat building up over repeated iterations, a long conversation's context window growing past what anyone budgeted for, or a new feature shipping with a prompt nobody optimized. This class rarely appears as a dramatic number on the aggregate dashboard. It appears as a modest-looking percentage increase system-wide, and only a per-cohort baseline reveals which specific workload is driving it.

The third class runs through errors rather than usage. Elevated provider error rates trigger retry loops, and every retry burns tokens without producing anything useful. Refusal rate, timeout rate, and provider error rate are the equivalent of 4xx/5xx signals, and they belong in the same anomaly detection stack as the cost signals. Agent workloads add a fourth pattern to track on its own: tool-call failures, failed sub-agent handoffs, and unusually deep loops, since a single bad argument passed to one tool can cascade through every step that follows and turn a cheap trace into an expensive one.

Some of these signals turn out to be security events wearing a cost anomaly's clothes. In one documented fintech case, a coordinated probing campaign produced 84 anomalous prompts inside twelve minutes.

Slicing anomalies by model, provider, team, and workload to find the source

An alert that fires without context tells a team that something went wrong, not where to go look. Turning that alert into an actual fix means slicing the same signal along the dimensions that map to real root causes.

Slicing by model usually answers the question first. If a spike is isolated to a single model while every other model's cost stays flat, the likely cause is a provider-side update that changed output length or pricing rather than any change in how people are using the system. Slicing by provider answers a different question: a cost increase spread across several models that all happen to come from the same provider points to a pricing change on that provider's end, a quota problem, or an outage that's pushing traffic over to a more expensive fallback model. Slicing by team or API key is usually the fastest path to a fix, because a spike isolated to one team's key almost always traces back to a specific feature or prompt change that team shipped recently. Giving every team visibility into its own cost, broken down by model, by use case, and by trend over time, turns out to be one of the most effective ways to cut waste on its own, since no team wants to be the one running noticeably higher spend than its peers. Slicing by workload type closes the loop: a spike concentrated in agentic workloads while synchronous completions stay stable points straight at loop behavior or tool-call volume, not at anything related to how the prompts themselves were written.

A gateway at the request boundary makes this slicing possible in practice, because it's the one point in the system that already has prompt and completion tokens counted separately, model and provider identity attached, caller context by team and workload recorded, and latency and error codes logged, all in a single metering stream. Concentrate AI is one tool built around that position, collecting these signals per trace and rolling them up into per-model, per-team, and per-provider series within seconds, which keeps the dimensional slicing described above from being scattered across separate provider dashboards and application logs. Whatever the tooling, the underlying requirement stays the same: find the dimension where the anomaly concentrates, and the root cause is usually sitting right behind it.