AI Model Orchestration: Routing Models vs Orchestrating Work

Key takeaways

• Model orchestration coordinates requests across several models: which model handles a task, how inputs and outputs are structured, what falls back on failure, what gets cached.
• The phrase gets used for two other problems. Agent orchestration coordinates multi-step work. Pipeline orchestration manages MLOps training and deployment.
• No NIST, IEEE, or ISO standard defines the term, so vendor definitions expand to fit whatever the vendor sells.
• Routing across models saves real money. RouteLLM reports over 85% cost reduction on MT Bench while retaining 95% of GPT-4's performance, under benchmark conditions.
• Routing sets the price of a model call. It does not ask how many of those calls your workflow needs, and that is the larger lever.

The actual definition

Model orchestration coordinates requests across a pool of models. It decides which model handles which request, structures the inputs and outputs, defines what happens when a call fails, and caches what can be reused. That is the entire scope. It operates on one request and its response.

That definition is hard to find on the pages competing for this phrase. IBM defines AI orchestration as "the coordination and management of artificial intelligence (AI) models, systems and integrations", then files models, agents, compute capacity, databases, data pipelines, external APIs, monitoring, governance, and compliance controls under the same word (IBM, January 2025). GitHub's version is sharper and still spans "models, agents, tools, and data" together (GitHub, April 2026). A word covering the whole stack cannot tell you which layer your problem lives at. We wrote the broad version too, as AI orchestration across models, agents and the app layer. This page is the narrow one.

One thing the competitive set never says: no NIST, IEEE, or ISO standard defines model orchestration. The closest formal terminology is the machine learning function orchestrator in ITU-T Recommendation Y.3179, a different construct in a telecom context. Absent a standards body, vendor definitions expand to match what the vendor sells.

Three things people mean by this phrase

| | What it coordinates | The decision it makes | Typical tools | What it does not solve | |---|---|---|---|---| | Model orchestration | Requests across a pool of deployed models | Which model answers this request, and what happens if it fails | Managed routers such as Azure Foundry model router, gateway and router services | Whether the request should have been a model call at all | | Agent orchestration | Steps, tool calls, and state across a multi-step task | What happens next, with which tool, using what context | Agent frameworks such as LangGraph, orchestrator-worker patterns | Whether each step needs fresh reasoning on every run | | Pipeline orchestration | Training, evaluation, and deployment jobs | When a model gets retrained, validated, and shipped | Airflow, Kubeflow, ModelOps tooling | Anything about runtime request handling |

Model orchestration: routing across models

The unit of work is a request. A router looks at what arrived and picks an endpoint. Everything downstream of that choice is unchanged: same prompt, same tools, same expectations about the response. Swap the model and the workflow does not notice.

Agent orchestration: coordinating multi-step work, tools and state

The unit of work is a task that takes several steps. Anthropic's reference taxonomy separates workflows, where developers specify the sequence in advance, from agents, where the model chooses actions dynamically (Anthropic, Building Effective Agents). It names routing as its own pattern, distinct from orchestrator-workers, where "a central LLM dynamically breaks down tasks, delegates them to worker LLMs, and synthesizes their results." Two patterns in one taxonomy, collapsed by the market into a single word. The longer treatment is how agentic workflows are structured.

Naming drives buying decisions here. LangChain does not use the phrase model orchestration in its positioning, and LangGraph's own title calls it an agent orchestration framework. Search for model orchestration, land on an agent framework, and you have found a product for a different layer.

Pipeline orchestration: MLOps training and deployment

This one is genuinely separate and the leaders agree. GitHub's table places MLOps at model training, deployment, and monitoring across the lifecycle, with model and agent orchestration above it. Pipeline orchestration runs on a schedule or a data trigger and produces an artifact. It has no opinion about which model handles a request at 2pm on a Tuesday.

Why this matters now

Most teams that had one provider wired up in 2024 now have two or three. Capability spreads unevenly across them, price spreads by more than an order of magnitude between small and frontier models, and provider concentration is a live availability concern. Model selection is now a decision someone makes on every request, deliberately or not.

The second pressure is arithmetic. Model cost scales with calls, so a workflow that reasons through the same sequence every run pays for it every run, and the bill grows with adoption. Routing attacks the unit price of the call. Fastest available lever, and the smaller one.

How model routing actually works

What signals a router reads

Microsoft's model router is the clearest first-party description of the mechanism. It is a purpose-built ML model rather than an LLM, and it analyzes the full request including "system message, user message, tool definitions, and conversation history" to judge what the prompt asks for and how hard it is, then estimates which model in the pool delivers the best result (Microsoft Learn, updated August 2026).

That signal list tells you what routing can see. Tool definitions and conversation history are in scope. Business context, the cost of getting this request wrong, and whether the same request was answered last week are outside it. A router reasons about prompt difficulty, and your workflow sits beyond its view. Microsoft also makes a point most vendors skip: the selected model is disclosed in the response, and their guidance is to log that field. A routing layer you cannot attribute after the fact is one you cannot audit.

Routing strategies and when each misfires

| Strategy | What it optimizes | When it misfires | |---|---|---| | Cost-first | The cheapest model that clears the bar, escalating only when the prompt demands it | Underestimates difficulty on prompts that look simple and are not, so quality drops where it is least expected | | Quality-first | Highest-capability model per request regardless of price | Pays frontier prices for classification and formatting work, which is most of the volume in many workloads | | Balanced | Best combination of quality and cost across a mixed traffic profile | Traffic profiles shift faster than the configuration does, so the balance point drifts unnoticed | | Rule-based fallback | Availability, by failing over on error or timeout | Hides degradation. The fallback model answers, the request succeeds, and quality falls with no error to alert on |

Microsoft exposes the first three as Balanced, Cost, and Quality modes, with Balanced as the default. Their operational advice transfers to any router: change one lever at a time, hold the workload dataset fixed, and re-run the evaluation so you can attribute the result. We go deeper on routing strategies and their limits, and compare AI gateways and routers.

What the research measured

The strongest public evidence is RouteLLM, from Berkeley's Sky Computing Lab with LMSYS (arXiv 2406.18665, first posted June 2024, revised February 2025). It trains routers on human preference data to pick between a stronger and a weaker model at inference time, reporting cost reductions of "over 2 times in certain cases" without compromising response quality. The project page gives the specifics: over 85% cost reduction on MT Bench, 45% on MMLU, and 35% on GSM8K, while "achieving 95% of GPT-4's performance" (project page).

Those are benchmark results measured against GPT-4 alone, on three academic benchmarks, with one strong-weak model pair. The mechanism generalizes. The magnitude does not, and a vendor quoting 85% at you without the benchmark attached is quoting it wrong. The more durable finding is transfer: the routers held performance when the strong and weak models were swapped at test time, so difficulty looks like a property of prompts rather than of a model pair.

The hybrid pattern

Production architectures rarely route everything. Microsoft's documented recommendation is a router as the default path for general traffic, with direct deployments kept for specialized, compliance-mandated, or parameter-sensitive workloads. Their compliance mechanism is the model subset: security approves a set of models, you encode it, and new models cannot appear without an explicit opt-in. Some requests need a pinned model because an auditor or a validated evaluation says so. Route by default, pin by exception.

Common misconceptions

Mixture-of-experts is not model routing. MoE gating is conditional computation inside a single model, where a learned gate scores expert subnetworks and activates only the top few per token, so "only parts of the model are used" (Mixture of experts). Both get called routing. One picks subnetworks inside one set of weights. The other picks between deployed endpoints with different prices and providers.

Model orchestration is not the same as an AI gateway. A gateway centralizes auth, rate limits, spend caps, and logging across providers. Some route and some do not. A gateway with a static default model is doing governance rather than model selection, which is worth owning on its own terms. Just do not buy it expecting the other.

More models does not mean better results. Microsoft warns against single-model subsets, which turn a router into an expensive passthrough. The opposite failure is quieter: a bigger pool means more untested paths, and the path a request takes on a bad day is the one nobody evaluated.

The router is a component, and components fail. It adds a hop, a dependency, and a class of error that did not exist when calls were hardcoded. Microsoft characterizes its routing overhead as a negligible fraction of inference time, credible for a purpose-built classifier and still not zero. The worse failure is silent misrouting: the request goes somewhere cheap, gets a weaker answer, and returns HTTP 200.

The orchestration that compounds

Here is the position, and it is why this article exists. Most teams shopping for a model orchestration platform have an agent architecture problem rather than a routing problem.

The test is short. Take a workflow that runs many times a day and ask how much of it is genuinely different on each run. If the answer is most of it, route, because capability and price vary across those calls and picking well matters. If the same seven steps happen in the same order against the same systems, with judgment needed at one of them, model selection is the wrong lever. A language model is re-deriving a known workflow on every execution, and routing makes each redundant call cheaper without making any of them unnecessary. If that describes your system, an honest map of agent frameworks is the better starting point.

Anthropic points the same direction: start with the least complicated approach that works, and add agentic complexity only when testing shows it helps, because agentic designs increase expense, delay, and accumulated error. The industry read that as advice about frameworks. It is also advice about how much of a workflow should touch a model at all.

Routing optimizes the unit economics of a call. Architecture determines how many calls exist. The first is bounded by the price spread between models, large today and shrinking as capable small models get cheaper. The second is bounded by how much of your work is repeatable, which in enterprise operations is most of it. The fuller version is what actually reduces LLM cost.

What this argument does not cover: evaluation. Judging whether a cheaper model's answer was good enough is harder than choosing which model to call, and routing quality depends on it. That deserves its own piece.

What we're building at Major in response

The industry has spent two years improving the unit economics of a model call. Routing it, caching it, compressing the context, quantizing the weights. Comparatively little attention has gone to how many of those calls need to happen, which is a local optimization over a surface most teams have never questioned.

So we built for the other layer. On Major, when an agent works out how to handle a repeatable part of a task, it builds an app for that part and runs the app instead of reasoning through the step again. The app is deterministic code with its own managed database, storage, and logs, so the work runs the same way every time, keeps state between runs, and can be inspected afterwards. The repeatable stretch stops being a model call. The model keeps the judgment, and routing matters more on what remains, because those are the requests where model choice changes the outcome.

We are not arguing against routing. Teams on Major route, and they should. The claim is narrower: routing is a price optimization, architecture is a volume decision, and volume decisions compound. Reason once. Run forever.

If a language model re-derives the same steps for you on every run, model choice is the smaller question. See how Major turns repeatable agent work into deterministic apps with their own database and audit trail.

Related articles

• AI Agent Cost: What It Takes to Build One and What It Costs to Run
• How to Build AI Agents That Survive Run 1,000
• AI Agent Architecture: Components, Patterns, and the App Layer