Pub Sub vs Message Queue for LLM Request Pipelines

Message queues and pub/sub solve different problems. Pick the wrong one for a given stage of an LLM pipeline and you pay for it twice, once in duplicate inference calls, once in requests that quietly vanish during a provider outage. Standard async thinking assumes fast execution, cheap retries, and failures that look the same every time. LLM work breaks every one of those assumptions, and that's the whole reason this decision deserves more than five minutes.

Inference latency runs 500ms to 2,000ms per call, well past the low milliseconds you'd get hitting a cache or a database, so every retry costs real wall-clock time. Worse, every retry resends the full prompt, so every redundant call burns tokens on top of the delay. And the failures aren't uniform: transient hiccups, rate limits, outages that stretch for hours, and a fourth kind nobody talks about enough, a provider handing back a 200 status code with a garbage or truncated answer riding inside it.

The messaging layer itself is cheap, by the way, so don't spend your worry budget there. Rule-based routing adds under 1 millisecond. Embedding-based routing adds around 5. Even a full ML classifier tops out around 50 to 100 milliseconds, which is nothing next to an 800ms inference call or a $0.03 token bill. The real question is which pattern keeps you from wasting spend on retries, dropping requests during an outage, or letting one slow subscriber choke the whole pipeline.

What message queues actually guarantee and where that guarantee breaks down

A queue is point-to-point: one message, one destination, one consumer picks it up and does the work. That's the entire model. It sounds almost too simple to matter, but it's the right model more often than people assume once you look at what LLM pipelines actually need.

Durability is the guarantee that carries the weight here. Consumer crashes, worker pool scales to zero, whatever, the message just sits there until something comes along and picks it up. I've watched a provider outage run three or four hours longer than anyone expected, and the difference between a queue and no queue in that moment is the difference between "we lost a batch of user requests" and "we processed a backlog once Anthropic came back online."

Queues also buffer load, which matters more than people give it credit for. Hammer a rate-limited provider without one and every request fires at once, and now you're eating a wall of 429s and paying for retries you didn't need to make. A queue smooths that spike out instead of letting it turn into a mess.

Dead Letter Queues handle cleanup. A message that fails retries, or fails a format check, or keeps tripping the same provider error, lands in the DLQ instead of disappearing. For an LLM pipeline that means a request aimed at a provider that's been down for two hours sits somewhere visible and inspectable, ready to replay the second the provider recovers. No mystery gaps in your logs that some engineer finds three days later and has to explain in a postmortem.

Queues fall apart in two spots: latency-sensitive work, and true fan-out. Enqueue and dequeue overhead adds up fast if you're chasing anything close to real-time. And a queue hands each message to exactly one consumer by design, so if a single LLM response needs to reach three or four downstream systems at once, you're fighting the tool. You'd end up writing the same message to three separate queues, which works, technically, but gets messy in a hurry.

On tooling: Kafka, RabbitMQ, Redis Streams, AWS SQS, and Azure Service Bus all do this job with different tradeoffs on throughput, ordering, and how much operational babysitting you're signing up for. And Postgres as a queue is still a perfectly credible choice for a lot of simpler pipelines. Kafka earns its keep once 200 requests a minute becomes 200,000; below that, simpler tools carry the load fine.

What pub/sub actually guarantees and where that guarantee breaks down

Pub/sub flips the model on its head. A message goes to a topic, and any number of subscribers pick it up on their own, independent of each other. The publisher fires the message and moves on. It doesn't wait for a "got it" from anyone.

That's the guarantee that matters for LLM pipelines: fan-out without coupling. An LLM finishes generating a response, and that single event can hit an audit log writer, a cost attribution service, and a UI notification handler all at once, with none of the three needing to know the other two exist. Add a fourth subscriber next quarter and nothing upstream has to change.

Isolation matters just as much. Say your cost attribution pipeline is doing some heavy aggregation and running slow this week. That slowness doesn't hold up the UI notification even a beat, because each subscriber runs on its own clock.

What pub/sub doesn't give you is exactly-once processing. Some systems redeliver, some subscribers double-process on reconnect, and there's no guarantee baked in that a message gets handled exactly once. For most event types that's a shrug. For LLM inference it's a real risk, because processing the same prompt twice means paying for it twice, and if your downstream services aren't idempotent, you get duplicate writes stacked on top of duplicate spend.

Durability varies a lot by implementation too. Some pub/sub systems are ephemeral, so a subscriber that's offline when the message goes out just misses it, no replay, gone. Kafka and Google Cloud Pub/Sub do better here, with durable storage and replay built in. GCP Pub/Sub in particular gives you push and pull delivery plus monitoring, alerting, and IAM baked in, which matters a lot if your team doesn't have a dedicated messaging engineer on staff.

The structural difference that determines which pattern fits which pipeline stage

One question, asked at every stage: does this step need exactly one consumer per message, or does it need one event to land on several independent consumers? Single delivery, use a queue. Multi-consumer fan-out, use pub/sub. It's a clean split in theory and it gets concrete fast once you map it onto a real pipeline.

Ingestion and inference dispatch is a queue, full stop. One worker grabs one request, sends it to one provider, gets one result. Two workers grabbing the same prompt and calling the API twice isn't redundancy or resilience, it's just a wasted call and a wasted charge on the bill.

Provider fallback and retry is also a queue. A failed message requeues with backoff, or it routes to a DLQ if it keeps failing. A provider going down should never mean silent data loss; it should mean the request waits somewhere safe until it can run again.

Response fan-out is pub/sub. Inference finishes, and the result needs to reach audit logging, cost tracking, a cache layer, and a UI update. Those four don't need to know about each other, and coupling them together defeats the entire point of splitting the work up.

Streaming, token-level output going out to several connected clients, is pub/sub too. You want low-latency broadcast there, not durable replay of every token that streamed sixty seconds ago.

The mistake I see teams make over and over is treating this as a binary choice: pick one pattern, force the whole pipeline through it, done. Most production LLM systems need both, at different stages, and there's a build order that actually works. Get the dispatch queue solid first, since that's where money and correctness live, and layer pub/sub on top for fan-out once dispatch stops wobbling. Flip that order, or lean on pub/sub somewhere a queue belongs, and duplicate inference calls creep in. At low volume that's a rounding error nobody notices. At scale, a fraction of a percent of duplicate calls turns into a bill someone in finance is going to ask about.

How failure modes specific to LLM providers reshape retry and DLQ design

Not every provider failure deserves the same response, and treating them all the same is exactly how teams burn through budget for nothing. Transient errors, rate limits and timeouts, are safe to retry with backoff. A persistent outage running for hours calls for something else entirely: retrying immediately just wastes tokens and piles load onto a system that's already on its knees. Then there's the failure that doesn't even announce itself as one, the provider returns 200 OK and the response is degraded, truncated, or flatly wrong. Standard retry logic never catches that, because there's no error code to hang a retry on.

DLQs exist for the persistent case. Route it, look at it, replay once the provider's back on its feet. Beats a polling loop that keeps hammering a dead endpoint and paying tokens for every failed attempt.

Good retry design for LLM calls comes down to a short list of non-negotiables. Exponential backoff with jitter, that one's expected at this point. A hard cap on retry count before something routes to the DLQ, because uncapped retries against a downed provider just multiply token spend with zero chance of success. And the retry logic has to actually distinguish error types, because a 429 rate limit wants a different backoff curve than a 503 outage does. Treat them the same and you waste time on one while giving up too fast on the other.

Here's a cost wrinkle a lot of retry logic misses entirely: context length. A request carrying a long conversation history resends that entire history on every single retry, full token price, every time. Retry logic that counts attempts but ignores context size is letting your bill creep up quietly, one failed call at a time.

Agentic pipelines make this worse, and fast. One agent spawning sub-agents with no cap, or stuck retrying in a loop, can rack up token volume that dwarfs what your whole team burns in a normal week. Queue-enforced concurrency limits and DLQ routing aren't nice-to-haves in an agentic setup. They're the only thing standing between a bug and a five-figure invoice.

And here's a tell worth watching for: the moment you notice the same retry-and-DLQ logic getting copy-pasted into a third service, that's your signal a gateway belongs in the stack.

Where an LLM gateway sits relative to the messaging layer and what it handles instead

The messaging layer and the gateway split the work cleanly. The queue or topic handles delivery and durability, getting a message from A to B without losing it. The gateway handles the moment that message actually needs to talk to a provider. A queue hands the request off; the gateway decides which provider gets it, checks it against spend limits, logs it, and handles fallback if the first choice fails.

That's a lot of ground a queue was never built to cover. A gateway gives you one API surface across providers, so swapping OpenAI for Anthropic, or adding a fallback path, doesn't touch your application code. It gives you real-time cost attribution, per request, per team, per model, instead of reconstructing spend from an invoice at month's end when it's too late to change anything. It handles routing, sending the easy queries to a cheap model and saving the expensive one for work that earns it, a decision that has no business living inside queue configuration. It reroutes on provider failure without the queue ever needing to know something broke. And it handles PII redaction and compliance logging before a request leaves your perimeter at all.

Watch for this signal: retry-with-backoff code, provider API keys, and spend tracking showing up independently in three or more services means you're already paying the cost of skipping a gateway, just in engineering hours instead of a line item. General guidance puts the tipping point around your second LLM-touching service, or whenever monthly spend turns into a real budget conversation. One analyst projection has roughly 70% of engineering teams building multimodel applications running an AI gateway by 2028, up from about a quarter in 2025. That's a real shift, not a rounding error.

Concentrate is built for exactly this layer: one API across more than 130 providers, real-time spend visibility down to the individual request, fallback routing, and PII redaction, without standing up self-hosted infrastructure or paying a per-token markup on top of what the provider already bills you. With that piece handled, the messaging layer gets to stay boring, which is the goal. The queue does delivery and durability. The gateway does everything that touches how a provider actually behaves.

Practical patterns for combining queues and pub/sub in a single LLM pipeline

Most production pipelines end up hybrid without anyone sitting down and deciding it on purpose: queue at ingestion and dispatch, pub/sub at the response layer. Three patterns cover most of what I've seen in the wild.

Async inference with durable dispatch works like this: a client writes a request to a queue, a worker picks it up and calls the gateway, the gateway routes to a provider. Failure backs off and retries, then DLQs if it keeps failing. Success publishes the result to a pub/sub topic. This fits high-volume batch work, document processing, anything where the caller isn't sitting there waiting on the same breath they asked the question in.

Fan-out from a single result looks different. The finished response lands on a topic, and independent subscribers pick it up on their own schedule, an audit log, cost attribution, a cache writer, a UI push. Each can fail without taking the others down. A slow analytics job doesn't hold the UI update up by even a millisecond, which is the entire point of decoupling them in the first place.

Agentic loops need queue-enforced concurrency limits. Each step an agent takes gets enqueued as its own message, and concurrency gets capped at the consumer level. That cap is what stops a sub-agent spawning problem from saturating a provider's rate limit or racking up token spend nobody approved. A step that keeps failing goes to the DLQ instead of looping forever on the hope that attempt ten works when the first nine didn't.

On tooling: Kafka handles high-throughput pipelines needing durable replay on both dispatch and fan-out sides. Redis Streams is a solid lighter option if you're already running Redis and want queue behavior plus pub/sub without standing up a whole new system. GCP Pub/Sub covers both patterns under one managed roof with monitoring built in, which matters if you already live in GCP. Postgres as a queue still works fine for lower-volume pipelines where operational simplicity beats squeezing out more throughput.

One thing worth building in from day one, capture cost data right at the handoff between queue and gateway: model used, token count, provider, team or project tag. That's the moment the data exists in its cleanest form. Wait until month-end and you're reconstructing it from invoices instead of just reading it off the request as it happens.

Decision checklist for choosing a pattern given a pipeline's actual requirements

Start with delivery. Does exactly one consumer need to handle each inference request? Queue, with a single consumer group. Does one event need to reach several independent consumers? Pub/sub topic. Does the pipeline need both, at different stages? That's the normal case, and the answer is queue for dispatch, pub/sub for fan-out.

Then look at failure tolerance. If losing a message during a subscriber outage is fine, a lightweight pub/sub setup covers you. If losing a request during a provider outage is not acceptable, and for most production LLM work it isn't, you need a durable queue with a DLQ sitting behind it.

Check your retry economics next. Long context windows or expensive models in the path mean capping retries hard and routing to the DLQ early instead of late. Agentic steps that can spawn sub-agents mean concurrency limits at the consumer level aren't optional; they're the guardrail keeping a bug from becoming an incident.

Look at whether provider concerns already have a home somewhere. Retry logic, provider keys, fallback routing, and spend tracking duplicated across two or three services means the gateway needs to go in before the messaging layer gets any more tangled. Once the gateway's in place, the queue's job actually gets simpler too, durable delivery to one well-defined endpoint instead of a pile of conditional multi-provider logic bolted onto the consumer.

Last thing, and it matters more than people admit: be honest about how much infrastructure your team wants to run day to day. A managed pub/sub service paired with a managed gateway gets most teams further, faster, than standing up Kafka clusters and hand-rolled provider routing before the pipeline has even proven it needs that much horsepower.