What is a model router?

A model router is the component that inspects an incoming request and selects which model handles it, instead of sending everything to one default. It decides using signals from the request itself, such as task type, input length, caller, or an estimated difficulty score, then forwards the request and returns the chosen model's response.

Does model routing hurt response quality?

It can. Routing a hard request to a small model produces a plausible wrong answer rather than an error, so degradation is silent. Two mitigations work. Use cascading, where the cheap model's output is verified and escalated on failure, and replay a fixed held-out set of requests with known-good outputs after every routing change.

What is the difference between model routing and Mixture-of-Experts?

Model routing selects among separately deployed models at the request level, and the choice is observable, since you can log which model answered. Mixture-of-Experts selects among expert subnetworks inside a single model during inference. Routing is an application decision you configure. Mixture-of-Experts is an architectural property of the model you called.

What should you measure to know if routing is working?

Start with cost per completed task rather than cost per token, because a router that halves token price while doubling retries looks like a win on a token dashboard. Then track escalation rate, quality regression against a held-out evaluation set, latency added at the routing hop, and cache hit rate before and after the change.

When should you not build a model router?

Skip it when traffic is low or uniform, when one task type dominates, or when prompt caching is already doing more work than routing would. Switching models can invalidate a cache, and a cold read on a large fixed prefix can cost more than the cheaper model saves. Move stable steps into code instead.

Model Routing: How to Send Each Request to the Right Model

Routing sends each request to the model that fits the job instead of billing every task at frontier rates. Here are the strategies that work, how to tell whether yours is helping, and the one route most teams never configure: the step that no longer needs a model at all.

Rahul RamakrishnanSeptember 7, 2026Updated October 22, 2026

Airport departure board listing flights, a picture of routing each traveler to the right gate

What is model routing?

Model routing is the layer that inspects each request and sends it to the model best suited to it, instead of sending everything to one default. Most teams should route by task type with plain rules, and should not train a router. The cheapest route of all is no model call, because a step that produces the same shape of output every run belongs in code.

Major is built on that last point. When an agent works out a repeatable step, it builds an app for that step and runs the app from then on, so the step leaves the routing table entirely. The model keeps the judgment calls and handles fewer requests.

This has nothing to do with network routing. Packets and BGP share the word and nothing else. Routing also differs from Mixture-of-Experts, which picks among expert subnetworks inside one model. Routing picks among separately deployed models, and you can log which one answered.

Microsoft's documentation for its model router describes the mechanism in three steps: analyze the full request, estimate which model in the pool handles it best, forward it. It says the router is "a lightweight ML model designed to predict which model performs best for a given prompt at minimal latency" (Microsoft Learn).

Why teams route at all

Two conditions make routing worth doing. A single vendor's lineup now spans orders of magnitude in output price between its smallest and largest models. And production traffic is not uniformly hard. Microsoft's docs make the premise explicit: "not all coding questions are equally hard." If a meaningful share of your requests are classification, extraction, or short reformatting, you are paying a reasoning-model rate for work a small model finishes correctly.

Both conditions concern price and difficulty distribution. Routing does not raise the quality ceiling of your system.

Which routing strategy should you use?

Five strategies exist. Here they are, one per line, with what each costs you:

Rule-based: routes on what your code already knows, such as task type, caller, or input length. You maintain a rules table whenever the model lineup changes.
Classifier: a small model scores prompt difficulty. You now operate a second model that nobody has an evaluation set for.
Cascading: try the cheap model, verify, escalate on failure. Escalated requests are paid for twice and answered late.
Learned or preference-based: a trained policy over model-quality features. It needs training data, evaluation infrastructure, and retraining when the pool changes.
No route: the step runs as deterministic code. It costs engineering time once, then nothing per run.

Rule-based routing wins for most pipelines. A support pipeline already knows whether the current step is intent classification or a customer-facing reply. Send the first to a small model and the second to a large one. No inference is needed to decide, so the routing hop adds no latency and no tokens. The standard objection is brittleness, and it is fair: someone revisits the table when a model is deprecated. That is a calendar reminder. A trained classifier is a project.

Classifier routing is reasonable to buy and expensive to build. Microsoft is candid about the follow-on work: "Managed routing doesn't remove the need for evaluation."

Cascading preserves quality best, because the expensive model still sees every request the cheap one got wrong. It degrades into pure overhead when the verifier is itself a frontier call.

Learned routing is where the research is. In Universal Model Routing for Efficient LLM Inference, Jitkrittum and colleagues propose UniRoute, which represents each model as a feature vector so the router can handle models it never saw in training. That matters for large, shifting model pools. It stays rare in production, and for most teams it should.

The route most teams never configure

Every strategy above answers "which model?" with a model. There is a fifth answer. Take a support-ticket pipeline with four step types:

Classify intent: a short enum, so a small model, or a trained classifier once you have labelled volume.
Extract order ID, account, and dates: no model. Schema validation and a lookup against the order table do it, because the ticket format is known.
Draft the customer reply: a large model, because tone and judgment are the point. Cascade to a larger one only on verification failure.
Escalate and write to the system of record: no model. It is a state transition, and sending an irreversible write through a probabilistic step is a governance problem before it is a cost problem.

Two of four steps stop being routing decisions. The remaining two get the model attention they deserve. This is also where determinism pays: code returns the same output every run, and a re-routed model call can drift between runs.

Build the routing-light pipeline on Major

Here is how the ticket pipeline above runs on Major. The model decides far less, and each piece is inspectable.

Connectors: Zendesk for tickets, Postgres or NetSuite for the order table, Slack for escalations. Credentials are scoped per connector, so the agent acts only inside the grants you set.
The app that holds state: a ticket-triage app with tables for tickets, extracted fields, routing decisions, and outcomes. Extraction and validation run in deterministic code inside the app.
The agent and its judgment: the agent classifies ambiguous intent, drafts replies, and decides when to escalate. It built the extraction and write steps once, then stopped re-reasoning them.
The trigger: a workflow fires on new Zendesk tickets, calls the app, and calls the agent only for the judgment steps.
Governance: role-based access on the app, scoped credentials on each connector, and an audit log of every action and which tool made it.
Cost shape: front-loaded while the agent builds the app, then flat for repeated steps, because the model is no longer in the loop for them.

Start building free and describe the first step you want out of the model.

How do you tell whether routing is working?

Instrument five things, in this order.

Cost per completed task comes first. A router that halves token price while doubling retries looks like a win on a token dashboard and is a loss.

Escalation rate is next: the share of requests that started cheap and ended expensive. A rising rate is the earliest sign your thresholds no longer match your traffic.

Then replay a fixed held-out set of requests with known-good outputs after every routing change. Without it you cannot tell savings from silent degradation.

Added latency at the routing hop matters too. Rules add roughly nothing, a classifier adds a full inference, and cascading adds one per escalation.

Last, watch cache hit rate before and after. Teams skip this one, and it is the one most likely to invalidate the exercise.

Microsoft's guidance is worth borrowing for any router: change one lever at a time, keep the workload and baseline fixed, and rerun the evaluation so the change in cost and quality is attributable.

Where routing goes wrong

The router becomes a failure mode. It sits in front of every request and can be down, slow, or stale relative to a lineup that shifted last week.

Routing also fights prompt caching. Anthropic's documentation says cache hits require 100% identical prompt segments, that the default cache lifetime is five minutes, and that cache reads are priced at 0.1 times the base input rate (prompt caching docs). Treat a model switch as a cold cache. On a long, stable system prompt, a warm read on the expensive model can cost less than a cold read on the cheap one. Measure the cache before you route.

Quality drifts quietly. Routing failures do not throw exceptions. The cheap model returns a plausible answer, the pipeline continues, and you find out when a customer replies to a wrong one. The held-out set is the only real answer to that.

Low volume, one dominant task type, or a caching strategy already doing the work all mean you can skip the router. Spend the sprint moving the stable steps into code instead.

What this does not cover

Agent evaluation is a separate and harder problem. And steps whose output varies with the input in ways code cannot express still need a model, so route those carefully using the strategies above. Major does not remove the model. It reduces how often you pay for it on work that no longer needs it.

Start building on Major

Describe the repeatable steps in your pipeline and Major's agents build them as governed apps, with scoped connectors, RBAC, and an audit trail. The routing decisions you keep are the ones that need judgment.

Frequently asked questions

What is a model router?: A model router is the component that inspects an incoming request and selects which model handles it, instead of sending everything to one default. It decides using signals from the request itself, such as task type, input length, caller, or an estimated difficulty score, then forwards the request and returns the chosen model's response.
Does model routing hurt response quality?: It can. Routing a hard request to a small model produces a plausible wrong answer rather than an error, so degradation is silent. Two mitigations work. Use cascading, where the cheap model's output is verified and escalated on failure, and replay a fixed held-out set of requests with known-good outputs after every routing change.
What is the difference between model routing and Mixture-of-Experts?: Model routing selects among separately deployed models at the request level, and the choice is observable, since you can log which model answered. Mixture-of-Experts selects among expert subnetworks inside a single model during inference. Routing is an application decision you configure. Mixture-of-Experts is an architectural property of the model you called.
What should you measure to know if routing is working?: Start with cost per completed task rather than cost per token, because a router that halves token price while doubling retries looks like a win on a token dashboard. Then track escalation rate, quality regression against a held-out evaluation set, latency added at the routing hop, and cache hit rate before and after the change.
When should you not build a model router?: Skip it when traffic is low or uniform, when one task type dominates, or when prompt caching is already doing more work than routing would. Switching models can invalidate a cache, and a cold read on a large fixed prefix can cost more than the cheaper model saves. Move stable steps into code instead.