LLM routing
LLM routing: pick the right model for every request
Short answer
LLM routing means deciding, for each request, which model should answer it. A fast classifier looks at the incoming prompt and sends lookups and simple questions to a small, cheap model, while complex reasoning goes to a large model. Done well, it cuts cost and latency sharply while keeping quality where it matters.
Most AI products send every request to one model. That model is usually chosen for the hardest 10% of requests, so the other 90% pay for capability they don't use. Routing fixes the mismatch: a quick decision in front of your models chooses the cheapest option that will still produce a good answer.
The router itself has to be much cheaper and faster than the models it routes to, or it eats the savings. That makes it a natural job for a decision model rather than another LLM call.
Get Early Access to ClassifierHub: 2× credits in your first paid month.
Common routing strategies
Routers differ in what they decide on:
- Difficulty routing: classify how much reasoning a request needs (lookup, simple, complex) and map each tier to a model.
- Domain routing: send code questions to a coding model, legal questions to a model with the right context, and so on.
- Cascades: try a cheap model first and escalate when a check says the answer is weak. This is more precise but adds latency on escalations.
- Non-LLM routing: many requests don't need a model at all. Route FAQs to templated answers and status questions to an API lookup.
Building a difficulty router
Define tiers with descriptions that match your own traffic, classify each incoming message, and map the label to a model. The important detail is the fallback: when the router isn't confident, choose the stronger model. An unnecessary expensive call costs a fraction of a cent; a bad answer can cost a customer.
// Route each request to the cheapest model that can handle it.
const res = await fetch("https://classifierhub.com/v1/classify", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.CLASSIFIERHUB_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: userMessage,
instructions: "How much reasoning does answering this message need?",
options: {
lookup: "A fact, status or FAQ answer that can be looked up or templated",
simple: "A short, single-step answer a small model can write",
complex: "Multi-step reasoning, code, analysis or a long document",
},
}),
signal: AbortSignal.timeout(3000),
});
const { label, confidence } = await res.json();
// Uncertain? Fall back to the strong model: a wrong cheap answer costs more than tokens.
const tier = confidence >= 0.7 ? label : "complex";
const model = { lookup: "faq-handler", simple: "small-model", complex: "large-model" }[tier];The ClassifierHub API opens to Early Access members first.
Measuring whether routing works
Log the tier, the model used and a quality signal (thumbs up, resolution, escalation) for every request. Compare quality per tier against a sample that always goes to the strong model. If the cheap tier's quality drops, tighten the descriptions or raise the threshold; if it holds, try moving more traffic down.
Track the share of traffic in each tier over time. Shifts usually mean your users or product changed, and your tier descriptions should change with them.
Routing inside AI agents
Agents make many small decisions per task: which tool to call, whether a result is good enough, whether to ask the user. Each of these can be routed the same way. Pairing a decision layer with a large planner model keeps the expensive model focused on planning while cheap decisions handle the branching.
Frequently asked questions
Related guides
Last updated . ClassifierHub is an independent product built on top of the Jev decision model, accessed through OpenRouter. It is not affiliated with or endorsed by TypeSafe or OpenRouter.