When You Shouldn’t Use an AI Router — Five Cases Where It’s the Wrong Answer

Most writing about model routing is written by people selling it, which makes the limits hard to find. So here they are. We run OrcaRouter, an AI router covering 40-plus providers, and there are workloads where putting a routing layer in front of your models is machinery you don’t need — or worse, machinery that makes something harder.

Five of them.

1. One narrow task, one model, and it works

If your product does a single well-defined thing, one model handles it correctly, and your volume is modest, routing adds a component to reason about in exchange for savings measured in tens of dollars.

The honest test is the escalation rate: take 200 representative requests, run them through a cheaper model, and count how many fail your quality bar. If almost none fail, you were overpaying and should switch models outright — no router required. If almost all fail, there is nothing to route down to. Routing only pays in the middle, where a meaningful fraction of traffic is routine, and a meaningful fraction isn’t.

A single-model setup also has a virtue worth naming: it is trivially easy to reason about. Every request went to the same place. When something regresses, there is one variable.

2. Every request genuinely needs the best model

There are some workloads that are all equally difficult. Partner review of legal drafting. Security-critical code. Medical documentation. Anywhere with a cost of a wrong answer that is significant enough to outweigh the cost.

In this case, tiering is optimizing what’s wrong. The difference in value of the cheap as compared to the costly saving to a human who believes it is relatively small. Instead, use the best model you can afford; on one model we serve, $0.15 per call for an input that is cached vs. $1.25 per call for the same input, that’s 88% cheaper, with no quality loss whatsoever.

3. Determinism is a hard requirement

If you must be able to say “exactly this model version served this request” — for reproducibility, for a regulated audit, for a customer contract — then a layer that chooses models per request is working against you.

The nuance: this does not say that the router shouldn’t route this traffic; it says that this traffic shouldn’t be adaptively routed. A viable solution is to make the models that need to be pinned adaptive — a routing rule written as code, YAML + condition language, maintained in version control. Determinism when it’s the contract and flexibility when it isn’t.

The good thing about this is that the choice is documented: the grade and selected model are documented on each receipt, and even adaptive traffic can be reconstructed after the fact. If it won’t tell you what it’d picked, then the concerns of determinism are the decisive ones, and you should stop there.

4. You haven’t measured where the money goes

This is the most frequently made mistake, and it isn’t really a routing mistake.

Teams use a router to save money, but don’t understand what’s being billed. Then they send the wrong traffic, save 4%, and say routing is not working. Typically, a few request shapes control most of the bill. Sometimes, the most important cost shouldn’t be a model choice — it’s sending a big stable context on every request that would have saved ~88% by sending a smaller version only in a cache, or a reasoning model that outputs many more tokens than its listed price.

Order of operations: measure cost by model and request type, fix caching if you have a stable prefix, forecast in cost per completed task rather than per token, and only then route. Routing applied to an unmeasured bill is a guess with extra infrastructure.

5. Hard real-time paths

If you’re working with an end-to-end budget of a few milliseconds, ask the question: “What? If your end-to-end budget is a few milliseconds, ask the question: “What? You are not likely to be calling a language model synchronously in that regime anyway — but if you are, you’ll note that routing overhead (under 1ms here by default) is not necessarily the issue. The models are: Our seven-day production telemetry has a range of 444 ms to 7.73 seconds for first-token p50. There’s no layer of routing to save a budget that is blown by a 444ms floor.

Two objections that are *not* good reasons to skip it

“It adds latency.” Do not assume; it’s worth checking. Ask for the number. If you have chosen the slow model, then it’s not the fault of the router, but of your choice of model: that’s what the router is for.

“I don’t want a middleman on my pricing.” Rightly, question, and get in reply: Does the layer add on to the provider’s rate? Pass-through at 0% markup will involve paying the provider’s list price, and the routing needs to be supported by merit. If there’s a per-token surcharge, it’s applied to all traffic; the savings are applied only to traffic that you moved down a tier — and that can be negative.

What to do instead, if you’re in one of the five

• Case 1 — switch models outright if the cheap one passes; otherwise keep it simple.
• Case 2 — buy the best model and optimize caching and prompt structure.
• Case 3 — pin the regulated traffic with explicit rules; route the rest.
• Case 4 — instrument first. Per-call records of model, latency, cost and grade, then decide.
• Case 5 — reconsider whether a synchronous model call belongs in that path at all.

Of those five, note that three of them still have a router in the picture, but doing something less than the pitch suggests. The straight answer: Routing is handy, but not always the correct choice.

And the escalation-rate test that decides which paragraph you’re in costs nothing to run: a rotating set of free LLM API models is callable at $0 per token, which is more than enough for 200 requests and a straight answer.

The takeaway

If your traffic is not so consistent that you have only one model that works for everything, then it’s definitely worth having an AI router. It’s not worth having for a single narrow task on a single model; it’s not worth having for uniformly hard work where the cheap tier can’t be used; and it’s not worth having for replacing a measure and finding out where your bill actually comes from. If determinism is contractual, the solution is not to avoid routing; it’s to point the traffic to be routed. It’s a good idea to check the escalation rate for 200 real requests first; that number tells you which of these paragraphs you are in.

Sourcing note: routing overhead, routing-rule syntax, per-receipt grade reporting, and 0% markup pass-through are OrcaRouter’s own published product descriptions. Latency percentiles and the cached-input rate are our own production figures and the providers’ list prices, respectively. Checked August 9, 2026.