Benchmarking LLM Providers on Real Production Workloads

Enterprise LLM spending crossed $12.5 billion in 2025, and a good chunk of that money got allocated on the strength of a leaderboard that has nothing to do with production. Teams check MMLU, glance at a reasoning benchmark, maybe run a coding test, and pick a provider based on numbers measured on a curated test set sitting still. Then real traffic hits the endpoint, and the performance profile looks nothing like what the leaderboard promised. Picking a provider off a leaderboard is close to picking blind, and the teams that keep doing it are the ones rewriting their provider config three months into production.

The five dimensions that production performance actually depends on

Start with the one nobody checks until users start complaining: time-to-first-token, or TTFT. That's the gap between sending a request and seeing the first character stream back. A provider can post great throughput numbers and still feel broken in a chat interface if TTFT drags past a second or two. Users don't experience tokens-per-second. They experience the wait before anything shows up on screen.

Latency under load is the second dimension, and the median, p50, is close to useless here. What matters is p95 and p99, the tail end of the distribution. Those percentiles show what happens once requests stack up during a traffic spike, which is exactly the moment a provider's real behavior stops matching its marketing page.

Third: error rate, broken down by type, not lumped into one scary percentage. Rate limits, 5xx server errors, timeouts, each behaves differently under pressure. Some providers degrade gracefully. Others fail hard and take the whole request chain down with them, and that's the kind of thing you want to know about before your on-call engineer finds out at 2 a.m.

Fourth, cost per completed task, which is a different number entirely from cost per million tokens. A cheap provider that forces three retries per task because the output won't parse isn't cheap at all. Cost per task is tokens spent multiplied by whether the output worked on the first try, and that multiplication is where the real number lives.

Fifth: output consistency. Run the same prompt ten times. Does the format hold steady? Does the length wander? Does instruction-following stay intact? Any pipeline that parses or routes based on a model's output lives or dies by this number, and no leaderboard measures it.

These five dimensions multiply against each other instead of sitting in isolation. A provider that's cheap per token but prone to retries can end up costing more per task than a pricier one that just gets the answer right the first time. That math gets uglier the more complex the workflow gets. Agentic pipelines, the multi-step chains where a model calls tools and revises its own output, can burn 5 to 30 times more tokens per task than a single chat exchange. Any inefficiency baked into a provider's behavior gets amplified by that multiplier, fast.

How to build a benchmark corpus from your own production traffic

Synthetic prompts don't look like real traffic. They miss the length distribution, the ambiguity, the domain-specific vocabulary users actually type, and the weird edge cases that show up 3% of the time but cause 30% of the pain. A benchmark built on made-up prompts measures a workload that doesn't exist, full stop.

Building a real corpus means sampling from production logs on purpose, not at random.

Stratify by request type first. Classification, summarization, multi-hop reasoning, and code generation each stress a provider differently, so lumping them together just hides which one is actually failing. Capture the full prompt-context pair exactly as it arrives: system prompt, conversation history, whatever retrieval context gets injected at runtime. A simplified stand-in prompt tests something different from what actually ships. Sample across time of day and day of week too, since provider latency shifts with infrastructure load, and that load is never flat across a 24-hour cycle. Flag outlier-length requests separately: long-tail prompts can run many times the median length, and those long prompts hit TTFT disproportionately hard.

Before any of this runs, decide what a successful completion looks like for each request type. Set the quality bar first. Skip that step and there's no way to tell a provider that failed from a provider that succeeded on its second try, and retry behavior is exactly the thing this whole exercise exists to catch.

Salesforce Engineering's documented approach is a useful template: separate live LLM dependencies from a mock service used for development and benchmarking. That separation let their validation process handle 16,000 requests per minute with burst capacity above 24,000. Keeping benchmark traffic apart from production traffic wasn't a nice-to-have there. It was the precondition for measuring anything correctly at all.

One more thing before any of this gets replayed against a provider endpoint: production prompts often carry PII or confidential customer data. Scrub it, or swap in synthetic substitutes, before it goes anywhere near a third-party API. A benchmarking exercise that leaks customer data isn't a benchmarking exercise. It's an incident report waiting to be written.

Running the benchmark: load profiles, concurrency, and measurement discipline

Volume isn't the goal here. Shape is. Run the benchmark at the concurrency level production actually sits at, not some stress-test ceiling built to find a breaking point. The question is fitness for the workload in front of you, not the provider's absolute limit, and mixing those two up wastes a whole benchmarking cycle chasing the wrong number.

Ramp concurrency in steps and watch for the point where p95 latency starts climbing sharply. That inflection point is the number that matters: the provider's real operating ceiling for this traffic pattern, not whatever a spec sheet claims.

A few measurement habits are easy to skip and expensive to skip. Record TTFT separately from total response time on every streaming request, since blending the two erases the exact signal being isolated. Classify every non-200 response by type: rate-limit, server error, timeout, malformed output. An aggregated "2% error rate" can hide five very different failure modes behind one number. Use token counts as reported by the provider, not an internal estimate, because input, output, and cache reads price differently and providers don't always count a prompt the same way. For consistency testing, run identical prompts enough times to build a real distribution, then measure format adherence and length variance directly rather than settling for semantic similarity between outputs.

Pricing discipline matters just as much as measurement discipline. Output tokens typically price several times higher than input tokens, and cached reads price below both. Blend those three rates into one number before analyzing anything, or the exact source of the cost gets buried right along with it.

Run every provider against the same corpus in the same time window if at all possible. Providers change infrastructure constantly, so a benchmark run today and one run three weeks later aren't measuring the same thing even with identical prompts. And allow each provider run a brief warm-up period before recording measurements, so early transient behavior doesn't skew the steady-state results.

Reading the results: what each metric pattern actually implies

A low p50 TTFT paired with a high p95 TTFT tells a specific story: the provider runs fine under light load and queues aggressively the moment concurrency rises. That's a risky profile for anything user-facing with unpredictable traffic, and it's the pattern most likely to get missed by a benchmark run at low, steady concurrency.

Stable TTFT with total latency that climbs as output grows longer points to throughput-limited infrastructure on the provider's end. That mostly bites long-generation tasks: document drafting, long-form code, anything where the output itself is large.

High variance in total latency at a fixed prompt length usually means batching behavior on the provider's side. The result is an unpredictable experience even when nothing about the request itself changed.

On the error side, rate-limit errors that only show up above a certain concurrency threshold mean the provider has a hard cap, and retry logic or fallback routing needs to get built around that cap before real traffic hits it. Low error rates during the benchmark that spike once real traffic hits usually mean the provider enforces tighter limits on sustained load than on a short test run, because providers reserve capacity differently for a five-minute benchmark than for a workload running around the clock.

Cost-per-task divergence from cost-per-token is the pattern that catches teams off guard most often, and it's the one worth taking seriously above almost everything else on this list. A provider that looks cheap on the per-token rate but forces more retries, or produces outputs that fail downstream parsing, ends up costing more per completed task than a pricier, steadier alternative. Cost-per-token was never the right number to optimize for in the first place. It just happens to be the number every pricing page puts front and center, so teams keep chasing it anyway.

On consistency: high format variance means downstream parsing needs to get more defensive, or the provider just isn't right for structured-output work. Instruction-following also tends to degrade at longer context lengths, a pattern that's common and almost never visible on a standard benchmark. The only way to catch it is testing with the actual long prompts already sitting in the production logs pulled earlier.

Composite academic scores can point in a completely different direction from what a workload needs, and this is where most teams get it backwards: DeepSeek V3.2 posts a strong performance-per-dollar figure, while premium models like Claude Opus and GPT-5.3 Codex sit at the high-quality, low-performance-per-dollar end of the spectrum. Reaching for the premium model by default, on the assumption that quality always wins, is the exact mistake production benchmarking exists to catch. Which position is right depends entirely on the workload sitting in front of it.

Matching providers to workload types rather than ranking them globally

No provider wins across every workload, and treating this exercise like it should crown a single winner misses the point of running it at all. A good benchmark produces a routing map, not a leaderboard, and teams that go looking for a leaderboard anyway usually end up overpaying for a frontier model on tasks that never needed one.

High-frequency, low-complexity tasks (classification, extraction, basic summarization) get dominated by TTFT and cost-per-task. Quality ceiling rarely constrains work like this, so smaller and faster models are usually the right call. Paying for a frontier model here is close to pure waste, and it's the single most common mistake teams make once a benchmark tells them a bigger model scores higher on paper.

Complex reasoning and multi-hop analysis work the opposite way. Quality and consistency dominate, latency tolerance goes up, and cost-per-task becomes secondary. Streaming, user-facing products care about TTFT above nearly everything else regardless of task complexity, because a slow start feels broken even when the eventual answer turns out great.

Agentic pipelines are their own category, and probably the least forgiving one. Error rate and fallback behavior matter most, since a provider that degrades under load can disrupt an entire multi-step chain that depends on it. Output consistency matters just as much, because each step's output becomes the next step's input, and garbage compounds fast once it's a few steps deep into the chain.

The price gap across tiers makes routing decisions genuinely consequential, not academic. Route a meaningful share of a workload to the wrong tier at that kind of spread, and the cost consequence at scale stops being a rounding error and starts showing up on the invoice. Prompt caching and model routing are among the highest-leverage cost levers available right now, and both only pay off if there's an infrastructure layer built to act on what the benchmark found. A routing decision hardcoded into application code can't benefit from any of this without an engineer going back in to change it by hand, which is exactly the kind of manual patchwork this whole exercise is supposed to replace.

Where a gateway fits into ongoing provider benchmarking

A benchmark run once goes stale fast. Provider infrastructure changes, pricing shifts with every new model generation, and new models show up on a near-constant basis. A benchmark from a year ago describes a cost landscape and a provider lineup that no longer exist.

This is where a gateway, a routing layer sitting between an application and its various model providers, earns its keep. Every production request passing through it becomes a data point automatically: TTFT, total latency, token counts by type, error classification, which provider handled the request. No extra instrumentation needed. Provider and model metrics get tracked natively, so the next benchmarking cycle starts from real production baselines instead of from zero. Fallback trigger rates, how often the system had to reroute away from a failing provider, surface degraded performance before it ever crosses an alerting threshold. No provider dashboard hands that over on its own.

The routing capabilities are what turn benchmark findings into something operational instead of a slide deck nobody acts on. Latency-based routing sends each request to whichever provider is fastest at that moment, acting directly on the TTFT findings. Cost-based routing sends requests to cheaper models when the task doesn't call for a premium one, acting on the per-task cost findings. Fallback chains, triggered by the specific error type observed, act on the error-taxonomy work from the benchmark. Budget enforcement at the team, project, and API-key level closes the gap between what the benchmark projected and what actually lands on the invoice.

Self-hosted, open-source gateway software and fully managed gateway services solve the same problem with a different tradeoff, and self-hosting is the wrong default for most teams. Self-hosting shifts the operational weight, uptime, scaling, security patching, onto the engineering team running it, and that weight rarely gets budgeted for up front. A managed gateway removes it and hands over a single API, observability, and routing controls ready to act on benchmark findings the moment they come in. The real question is blunt: is a team's edge in operating gateway infrastructure, or in building the product sitting on top of it? For most teams, that answer isn't close, and pretending otherwise usually costs an engineer a few months better spent elsewhere.