LLM model routing: when a small model is enough and when to use a frontier one

A product that calls a language model usually starts with one model for everything. It is the model the demo was built on, and often the most capable one available. Then the bill arrives, or the replies are slow, or the provider has a bad afternoon and every feature fails at once. The opposite fails too: a team moves everything to a small model to save money, and the hard requests quietly get worse. Model routing is the code that decides, per request, which model answers. This is how we design it, from the routing we wrote by hand for the EroPlay.ai MVP as technology partner to the agents we build now.

Route by task before anything clever

The cheapest router is one you already have: the call site. Your code knows what it is asking for when it asks. A reply the user will read, a title for a conversation, a summary of older turns, a label for an incoming message, an extraction into a schema. These are different jobs with different tolerances for error, delay and cost, and the code that makes each call can say which job it is.

So each call names a route, not a model: reply, summarise, classify, extract. A small table maps each route to an ordered list of models and the settings that go with them, such as the prompt version, the token limit and the timeout. Moving summaries to another model becomes a configuration change that runs through the route’s evaluation set, not a search through the codebase for model strings.

In a roleplay product such as EroPlay.ai, the character’s next line is the product, and people judge the service by it. Summarising the earlier part of a long conversation, so the character stays consistent beyond what a model can read at once, is background work nobody sees directly. Splitting those two jobs is where routing starts.

Background calls such as titles, tags, labels and summaries can outnumber the calls a user waits on, and they are where a small model is most often good enough. Your logs will tell you whether that holds for your product. Count calls per route before deciding anything else.

Routing by difficulty

Within one route, some requests are harder than others. A support question that needs one fact from the documentation and a dispute that needs three policies and the account history arrive through the same endpoint. Routing by difficulty tries to tell them apart before any model has answered.

The simple versions are rules on what you can see: the length of the input, whether there are attachments, the language, how many turns the conversation has, whether the account is on a plan that pays for the better model. Rules are crude, but they are free, instant and easy to explain when someone asks why a request went where it did.

The next step is a classifier, often a small model, that reads the request and predicts whether the cheap model will handle it. RouteLLM trained routers on human preference data to choose between a strong and a weak model, GPT-4 and Mixtral 8x7B in the main experiments, and reports cutting cost by more than half in some cases without losing response quality. The mechanism carries over to current models. The numbers do not carry over to your traffic, because a router trained on public chat preferences has not seen your users.

Two problems are particular to difficulty routing. The router is itself a call, with its own latency and cost, on every request. And its mistakes are invisible in production: when it sends a hard request to the small model, nobody sees the better answer the frontier model would have given. The only way to find those mistakes is to run both models on a sample and compare, which is the evaluation work further down.

Cascades: try small, escalate on a failed check

A cascade does not predict. It sends the request to the small model, checks the answer, and calls the larger model only when the check fails. FrugalGPT describes this approach and reports matching the best single model’s performance at up to 98% lower cost on the datasets its authors tested.

The check is the whole design. The good checks are objective. Does the output parse against the schema? Does every citation point at text that exists in the source? Is the tool the model wants to call one that exists, with arguments of the right type? Is the reply in the user’s language, and not a refusal or an empty string? These checks are cheap and they rarely lie. For structured output they are the same validation you need anyway, which reliable structured output from LLMs covers.

The weak checks ask a model how sure it is. Self-reported confidence is poorly calibrated, and asking a second model to judge the first adds a third call. Token log-probabilities, where a provider exposes them, are a better signal for a short answer such as a label and a poor one for a paragraph.

The arithmetic decides whether a cascade pays. As an example with made-up numbers, say the small model costs 1 unit per request and the frontier model costs 20.

the check accepts	average cost per request
80% of small-model answers	1 + 0.2 × 20 = 5 units
40%	1 + 0.6 × 20 = 13 units
about 5%	1 + 0.95 × 20 = 20 units, no saving
no cascade, everything to the frontier model	20 units

Every escalated request pays for both calls, so with these prices the cascade stops saving money only when about 95% of requests escalate. Latency is less forgiving. An escalated request waits for the small model, the check and then the whole frontier call, so the slowest requests get slower.

That is why cascades suit background and batch work, such as extraction and classification, where an answer that arrives a second later goes unnoticed. They suit a streamed chat reply badly. Once the small model’s tokens are on the screen, escalating means taking them back. For replies the user reads as they arrive, we route up front and keep cascades for the work behind them. In classifying messages at line rate the same idea appears with the escalation moved off the critical path: the small model decides inline, and the uncertain few go to a large model afterwards.

Fallbacks on errors and rate limits

Quality routing decides which model should answer. Fallback decides what happens when that model cannot. They are separate pieces of code, and mixing them is how a product ends up answering half its traffic from the backup model for a week without anyone knowing.

Each failure wants its own response.

failure	what to do
rate limit that says how long to wait	retry after that wait, or send it elsewhere if a person is waiting
429 at a spend cap	not every 429 means wait: Anthropic’s API, for example, returns a 429 with no retry-after header when an organisation reaches its monthly spend cap, and retrying fails until the next month
overloaded or server error	one retry with backoff, then a fallback
context length	not transient at all: use a model with a longer window or a shorter input; retrying the same model is waste
refusal	may or may not be a reason to ask another model; a product decision, not a reliability one

Gateways do part of this for you. On OpenRouter, the models parameter takes an ordered list, and by default any error, including context-length errors, moderation flags, rate limiting and downtime, moves the request to the next model. The request is billed at the rate of the model that actually answered, and the response says which one that was. Below that sits provider routing: the same open model is often served by several hosting providers, and by default the gateway spreads requests across them, preferring providers without recent outages and weighting towards the cheapest, with options to sort by price, throughput or latency instead.

Three things go wrong with fallbacks.

what goes wrong	what we do
the fallback model was never tested on the route, so the day it serves real traffic is the day you learn the prompt reads differently to it	put every fallback model through the route’s evaluation set
the fallback hides the outage, so nobody notices that a feature has run on the second choice for days	log the model that actually answered on every request, and alert when a route’s fallback share rises above its usual level
a streamed reply fails after it has started, once the API has already returned a 200	decide what a half-finished answer looks like in the interface before it happens

What a gateway such as OpenRouter does and does not do

A gateway gives you one API, one key and one bill across many models and providers, plus the provider failover above. For an MVP that is a real saving. Trying a new model on a route is a change to a model string, and nobody has to open an account with another provider. We used OpenRouter for the EroPlay.ai MVP, with the routing between models written by hand on top of it.

OpenRouter also has an auto router that classifies each prompt and picks a model for it, charged at the rate of the model it chose. That is difficulty routing done by someone else, against their idea of quality. It is a reasonable default for a prototype. For a product, the route names, the model lists and the checks belong in your code, because they encode what your product needs, and only your evaluation set can say whether a choice was right.

A gateway costs something too. It adds a network hop and another party in the path of your users’ data, whose data policies deserve the same reading as the model provider’s. Features specific to one provider, such as prompt caching or a strict structured output mode, reach the model through someone else’s interface, so check that each one you rely on actually arrives.

Measure each route on its own evaluation set

A product-level quality score hides routing decisions. The question is never whether the small model is good. It is whether it is good enough at summaries, at titles, at your extraction schema, and those answers differ. So each route gets its own evaluation set, drawn from real requests on that route, with what correct means for that job written down. Building an evaluation set from past cases describes how we make and keep those sets.

For each route we run every candidate model through the set and keep three numbers next to the score: cost per request, median latency and the slow tail, usually the 95th percentile. For a reply the user waits on, time to the first token matters more than total time. Output is generated one token at a time, so a route that writes long answers is where the choice of model moves latency most. A table of routes against models, with those four numbers in each cell, is the document that settles arguments about which model to use.

In production, every call logs its route, the model that answered, whether it escalated or fell back, and its tokens, cost and latency. Two numbers per route deserve an alert.

alert on	why
a cascade’s escalation rate	a sudden rise means either the traffic got harder or the check broke, and a broken check sends everything to the expensive model without raising a single error
the fallback share	a rise means the route is running on its second choice, which the fallback otherwise hides

LLM observability in production covers the logging side in more depth.

Re-run the table when a provider ships a new model, when a model you rely on is deprecated, and when you change a route’s prompt. A route that rightly needed the frontier model in spring may be fine on a small one by autumn, and the table is how you find out.

Where to start

Name your routes by task, give each one an ordered list of two models from different providers, and build a small evaluation set for each route that matters. Put the cheapest model that passes first on each route and a tested fallback behind it. Add difficulty routing or a cascade only on a route where the table shows the small model passing most cases and failing a kind of hard case you can detect. On most products that is one or two routes, and the rest never need more than a task name and a fallback.

If you are building a product on language models and want routing, evaluation and fallbacks designed in from the start, the custom AI apps page describes how we work, and you can write to us.