Intent-Based Routing for Multi-Step AI Workflows

Step intent is the job a single model call has to do inside a bigger pipeline. Not the topic of the call, the function. Most teams route by model preference or by habit, and that's backwards: intent should decide the model, not the other way around, and getting that order wrong is where token budgets quietly bleed out.

The meaning of intent in the context of a workflow step

Two steps can run against the exact same document and still carry totally different intents. "What does this document say?" is a summarization task. "Is this document compliant?" leans toward classification, maybe reasoning, depending on how many rules apply. Same input, different job, different model needs. Treating them the same causes the budget leak to start right there.

Four categories cover most of what shows up in production pipelines:

Classification and extraction labels something, sorts it, or pulls structured data out of unstructured text. Correctness is binary here. Either the label is right or it isn't, which makes it cheap and easy to test.

Summarization and compression shrinks text down while keeping the signal intact. Quality gets measured against the source, so there's a clear target to hit.

Generation and drafting produces new content that meets a spec. Quality gets fuzzier here, closer to a judgment call than a checkbox.

Reasoning and planning covers multi-step inference, chain-of-thought, decision trees. Correctness depends on context, and out of the four, it's the hardest to pin down.

This does not appear on its own in the prompt text; the pattern occurs when intent has to be declared or inferred, either through a developer tagging the call with an intent label in the application code before it reaches the router, or through the routing layer inferring it. Intent has to be declared or inferred. Declared means a developer tags the call with an intent label in the application code before it ever reaches the router. Inferred means the routing layer runs a lightweight classifier over the prompt structure and metadata to guess at intent.

Declared wins, and it isn't close. Inferred routing adds a second point of failure: get the classifier wrong on a reasoning step, route it to a cheap model, and the mistake compounds silently downstream, long after anyone can trace it back to the router. Declared tagging is auditable. A developer wrote down what the step does, and that decision sits in version control where it belongs, not buried inside a classifier's guesswork.

The mapping of the four intent categories to routing decisions in practice

Every routing decision balances cost, speed, and quality, and which one dominates changes by intent. Treating all three as equally important on every single step is the mistake that keeps token budgets bleeding, and it's the default most teams fall into simply because nobody assigned weights on purpose.

Classification and extraction has a high quality bar but a well-defined one. A correct label is just a correct label, full stop. Run offline eval sets first to confirm a smaller model clears that bar on the specific task, then send every classification step to the cheapest model that passes. The RouteLLM paper, presented at ICLR 2025, found that a matrix-factorization router hit 85% cost savings while holding 95% of GPT-4's quality, needing the strong model on only 14% of queries. Most classification work doesn't need a frontier model. It needs a cheap one that's been checked.

Summarization gets scored against the source, either with LLM-as-judge grading or reference summaries calibrated to what counts as acceptable compression. Semantic caching pays off heavily here, since the same document gets queried by different users constantly, and a cached response beats re-running the model every time it comes up. Default to a mid-tier model. Escalate only when the source material is dense and technical, or the summary has to hit a strict length limit.

Generation and drafting has quality that's partly subjective, so a rubric plus an LLM-as-judge step (or a human reviewer) earns its setup cost by showing which model tier actually clears the bar. A 2026 practitioner benchmark run across 38 real tasks and 15 models found Gemini 2.5 Flash hit 97.1% quality at $0.003 for the full 38-test run, with a median response time of 1.1 seconds. That's a non-frontier model landing near-frontier results at a fraction of the price. The model that wins overall doesn't matter here. The model that wins this specific task at this specific quality bar does.

Reasoning and planning is the one category where frontier pricing actually earns its keep, and it's the only one where reaching for the expensive model by default is defensible. Save the top-tier model for steps that genuinely need multi-step inference or tangled decision trees, and run a cascade everywhere else in this category: try the cheaper model first, escalate only when the output falls short. RouterBench offers over 405,000 precomputed inference outputs across 11 LLMs and 7 tasks, which lets teams calibrate escalation thresholds without burning live inference budget just to find the right cutoff.

Routing well, done this way, doesn't just save money. It can beat running everything through the single best model, because it plays to each model's strengths instead of forcing one model to be good at four different jobs at once.

Where routing decisions should be made

Two places tend to host routing logic today. Only one of them holds up past a handful of services, and the other one is where most teams start, which is the problem.

Application code is the first, and it's the wrong one at scale. If/else branches, hardcoded model names, provider SDKs scattered wherever a team happened to build a feature at the time. It works for a while. Then every service ends up re-implementing the same intent detection and fallback logic from scratch, with no single record of which model handled which step. Cost attribution and quality debugging become close to impossible, and provider credentials end up scattered across a dozen codebases for no good reason. When a provider changes pricing or has an outage, every one of those codebases needs its own fix, on its own timeline.

A central gateway layer is the second, and it's the one that scales. The gateway sits between the client and the providers: it terminates the request, applies policy, makes the routing call, normalizes the differences between providers, and logs telemetry before the response goes back out. That makes it the system of record for which model handled which call, at what cost, under which intent tag. Provider complexity belongs at the infrastructure layer, not scattered through application code, and the industry has largely settled this argument already: the LLM middleware gateway market is projected to grow at a 49.6% compound annual rate through 2034, and a large share of enterprises already run some kind of middleware layer.

An API gateway and an LLM gateway are not the same tool wearing a different label, and treating them as interchangeable is a common, expensive mistake. An API gateway routes HTTP traffic and enforces auth and rate limits by counting requests. An LLM gateway handles token-based budgets, semantic caching keyed to what a prompt actually means, failover across model vendors, cost attribution per model, and routing based on intent. A conventional API gateway has no concept of a token or an intent category. It can't do this job, no matter how well it does its own.

Automatic routing that hides its own logic isn't a solution, it's a liability with a friendlier interface. Teams need to see, and control, exactly where their requests go. If the routing layer can't be inspected and audited, it shouldn't be trusted with production traffic, full stop.

The gateways production teams are using in 2026 to implement this routing

For intent-based routing specifically, the factors that matter most are per-request logic granularity, fallback chain configuration, built-in semantic caching, depth of governance controls, support for agentic and MCP traffic, and the deployment model.

Open-source proxy libraries, most built in Python, offer a unified interface across a wide range of providers through a single API. They're popular for prototyping, since they get a team moving fast with almost no setup. Once a workload hits production scale in a regulated or multi-team environment, though, these tools usually need significant extra work layered on top, and their performance under load looks nothing like a gateway built for that scale from the ground up. Fine for teams still deciding whether to build out intent-based routing. Not the endpoint.

Edge-based managed gateways let some providers run LLM traffic through an existing global edge network as a managed service, handling caching, rate limiting, request logging, and basic analytics with no infrastructure setup for teams already on that provider's stack. Edge routing cuts latency for applications with users spread across regions. It fits teams with fairly simple routing needs who are already committed to that provider's broader platform. Past that, the fit gets thin fast.

Platform-extension gateways extend an existing API gateway platform to cover LLM traffic, carrying over the plugin ecosystem, mTLS, rate limiting, and audit features a team already relies on for regular API management. Multi-provider routing, token analytics, and fallback logic get layered on through the existing policy engine. This works best for teams that already run that platform for API traffic and would rather extend it than stand up something new from scratch.

What separates these three in practice comes down to governance depth, how well they support agentic and tool-calling-protocol traffic, and how much control a team keeps over deployment. Those are exactly the axes that matter for multi-step workflows, where every routing decision at every step needs to be both auditable and enforceable, not just fast.

The observability and cost attribution setup intent-based routing requires

Without attribution at the step level, a team knows what the whole pipeline costs and basically nothing else. Which step is bleeding money? Which model is underperforming? Is the escalation threshold even set right? No way to tell, and no way to fix what can't be named.

Real observability for intent-based routing means every model call carries a full set of tags: step name, intent category, model used, provider, token count, latency, cost. Those tags need to roll up to team, project, environment, and cost center; otherwise they just sit logged at the individual request level where nobody will ever look at them again, defeating the entire point of logging them. Distributed tracing has to tie the gateway's routing decisions back to the model calls and the application logic downstream, so a team can follow a request's full path, not just its final cost line.

The gap here isn't theoretical, and the numbers back it up. SpendHound's AI Spend Report, based on 172 finance and procurement leaders, found that 46% of them went over their AI budget in 2025, compared with 37% who overspent on traditional finance and accounting software, according to SpendHound's 2026 AI Spend Report. Run-rate spend across major AI providers and coding tools grew nearly fourfold between December 2024 and December 2025. Budgets keep getting blown for the same reason every time: nobody built the visibility to catch the leak while it was still small enough to patch.