What is LLM orchestration?

LLM orchestration is the code that coordinates language model calls into a working system: it holds state between steps, decides which model handles which step, calls tools, retries and handles failures, and records what happened. The model itself is one component inside it, since a model turns one input into one output and cannot sequence its own work. Most orchestration designs assume every step keeps running through a model. Major starts from the opposite assumption, having the agent reason once and write a deterministic app so the settled steps stop calling a model at all.

What is the difference between an LLM and an orchestrator?

An LLM turns one input into one output, holds no memory between calls, and cannot act outside its own response. An orchestrator is the code around it: it holds state, chooses which model handles which step, calls tools, handles failures, and records what happened. The model is one component inside the orchestrator. That split is also the lever, because anything the orchestrator can do in code does not need a model call. Major pushes that line as far as it goes, keeping the model for judgment and the app for everything settled.

What is an example of an LLM orchestrator?

LangGraph, CrewAI, AutoGen, LlamaIndex, and Haystack are the frameworks most teams reach for. A typical orchestrated workflow looks like this: a document arrives, a model classifies it, a parser extracts fields, code validates totals and matches a purchase order, the result writes to the system of record, and only genuine ambiguity goes back to the model. Notice how few of those steps actually need reasoning. On Major, an agent builds that pipeline as a deployed app with its own database, and the model handles only the ambiguous cases.

Do I need an orchestration framework?

Not if your workflow is a single prompt with one or two tools, because a framework adds a dependency and a mental model you will maintain. Reach for one when you have branching control flow, state that outlives a single call, and failure paths that need explicit policy. Even then, the framework choice matters far less than deciding which steps need a model. Major is the wrong call for genuinely exploratory work whose shape changes every run, and the right one once a workflow has stabilized enough to be written down as code.

How does LLM orchestration reduce cost?

Two levers do the work, with different math behind them. Routing sends cheap steps to smaller models, prompt caching charges cache hits at a tenth of base input price, and the Batch API halves asynchronous work. Those lower the slope of a bill that still rises with every run. The second lever moves a stabilized step into deterministic code, so that step costs the same at a million runs as at one. Major is built around the second lever, front-loading cost into the reasoning that produces the app and flattening it after.

Is LLM orchestration the same as AI orchestration?

LLM orchestration is the narrower term, covering control flow, routing, state, and tool calls around language models specifically. AI orchestration is the wider coordination of AI systems across an organization, including non-language models, data pipelines, and human approval steps. Our guide to AI orchestration covers that broader layer. Both terms describe coordination that runs through a model on every pass, which is the assumption Major changes by turning the repeatable middle of the pipeline into software the agent deploys and then runs.

LLM Orchestration: The Layer That Should Use Fewer Model Calls

Most orchestration guides teach you to chain more model calls together. That is the expensive path. Here's what LLM orchestration actually is, the patterns worth knowing, and why the best orchestration layer keeps shrinking the work the model does.

Rahul RamakrishnanAugust 26, 2026Updated August 28, 2026

Layered architecture facade separating an upper band from fixed structured blocks below, standing in for the reasoning layer that decides and the deterministic layer that executes

Key takeaways

LLM orchestration is the control layer that decides what runs, in what order, with what context, and on which model.
Orchestration has two jobs the category collapses into one. A layer that decides, and a layer that executes.
Judge an orchestration design by how few model calls it needs per completed unit of work.
Caching and batch discounts lower the slope of your cost curve. Moving a step into code removes the slope for that step.
Move a step out of the model only after its output has stabilized across the input distribution you actually see in production.

What LLM orchestration actually is

LLM orchestration is the control layer that coordinates how one or more language models interact with your data, your tools, and each other to complete multi-step work. It owns control flow, routing, memory, tool invocation, retries, and monitoring. IBM calls the orchestration layer "the backbone of the LLM app stack" (IBM). That is accurate, and it is where most of the category stops.

The part the consensus definition misses is that an orchestration layer does two separable jobs with different economics.

One job is deciding. Which step comes next, what this document appears to be, whether the discrepancy in front of it is a rounding error or a dispute. This work needs a model because the answer depends on judgment applied to an input nobody enumerated in advance.

The other job is executing. Parse the fields, run the arithmetic, join against the purchase order, write the row, emit the log line. This work has a correct answer that does not change between runs.

A well-designed orchestration layer keeps these two jobs visibly distinct and keeps moving work from the first into the second. A poorly designed one runs both through the model and calls the result a chain.

Three terms get used interchangeably and should not be:

Orchestration decides what runs, in what order, with what state.
An agent framework (LangGraph, CrewAI) is a library giving you primitives for writing that orchestration. It is a dependency, and adopting one does not constitute an architecture.
A gateway or router (a proxy in front of several providers) picks which model serves a given call and handles keys, fallback, and spend limits. It is one input to orchestration, not a substitute for it.

All three sit inside AI orchestration more broadly, which also covers non-language models, data pipelines, and human approval steps.

LLM vs orchestrator: the distinction people get wrong

An LLM produces one output from one input. It has no memory of the last call, no view of the steps around it, and no ability to act on anything outside its own response.

An orchestrator is the code around the model. It holds the state, decides which model gets which step, calls the tools, catches the failures, and writes the record of what happened. The model is one component inside the orchestrator, and on a well-built system it is not the busiest one.

The common error is treating the orchestrator as a thin adapter whose purpose is to feed the model more often. Once you accept that framing, every new capability becomes another model call, and your cost per unit of work never improves.

Why this matters now

A prompt chain that costs cents in a notebook costs real money at adoption, because the cost is per run rather than per capability.

Run the arithmetic with published list prices. Take an inbound document workflow at 10,000 documents a month, structured as five model calls per document, roughly 4,000 input and 600 output tokens in total, on Claude Sonnet 5 at $2 per million input tokens and $10 per million output (Anthropic pricing). That is $80 of input and $60 of output, so about $140 a month. At 100,000 documents it is about $1,400. The line is straight, and it stays straight forever.

Now move extraction, validation, matching, and the write into code, leaving one classification call and an escalation path on Claude Haiku 4.5 at $1 and $5 per million. At roughly 800 input and 100 output tokens per document, 10,000 documents cost about $13. At 100,000 it is about $130. (Illustrative volumes and token counts, real published prices.)

The interesting part is the shape rather than the ratio. Prompt caching charges cache hits at 0.1x the base input rate, and the Batch API takes 50% off input and output for asynchronous work (Anthropic pricing). Both are worth doing, and both are slope adjustments. Your bill still rises with every run. A step that has moved into code has no slope at all, because running it a millionth time costs what running it the first time cost. That is an architecture property, and no amount of vendor negotiation produces it.

The six things an orchestration layer actually does

Prompt chaining and control flow

Control flow is the logic that determines which step executes next and what each step receives as input. Frameworks differ mainly in how explicit they make this. Most production chains are a directed graph with a handful of branches, and the branch conditions are usually deterministic checks that people route through a model out of habit. See agentic workflow patterns for the shapes worth knowing before you pick a library.

Since framework choice is over-weighted, here is the whole subject as a table rather than a section. For a fuller treatment, we published an honest map of the framework layer.

LangGraph · What it's for: Stateful, branching agent applications · Control model: Explicit graph of nodes and edges · Watch out for: You own the state schema, and graph complexity grows quickly · Docs: docs
CrewAI · What it's for: Role-based multi-agent collaboration · Control model: Agents with roles delegating tasks · Watch out for: Delegation between agents multiplies model calls per task · Docs: docs
AutoGen · What it's for: Multi-agent conversation patterns · Control model: Agents exchanging messages in a group · Watch out for: Conversation length is the dominant cost driver · Docs: docs
LlamaIndex · What it's for: Retrieval over your own data, plus workflows · Control model: Indexes, query engines, event-driven steps · Watch out for: Retrieval quality dominates output quality · Docs: docs
Haystack · What it's for: Search and RAG pipelines · Control model: Typed component pipelines · Watch out for: Explicit pipelines fit poorly with open-ended agent loops · Docs: docs

Routing

Routing is the decision about which model, or which non-model path, handles a given step. Model routing is the cost lever the category talks about, and it is real. TensorWave names cost inefficiency as a core problem of unmanaged LLM systems and proposes routing to smaller models as the fix (TensorWave). It stops one step short. The cheapest model for a step is sometimes no model.

Memory and state

State is what the system knows between steps and between runs. Context windows are working memory, and they are the most expensive storage you will ever pay for, since every resumed conversation reprocesses its own history.

The research framing agrees. Sumedh Rasal's "A Multi-LLM Orchestration Engine for Personalized, Context-Rich Assistance" (arXiv, October 2024) puts evolving user context into temporal-graph and vector databases rather than the prompt, precisely so it survives and can be retrieved (arXiv:2410.10039). Durable state belongs in a database. The model reads from it.

Tool calls and external actions

A tool call is the model emitting a structured request that your code executes. Tools carry a token tax people forget to budget: on Claude Sonnet 5, enabling tool use adds 354 system prompt tokens with auto, before your own schemas (Anthropic pricing). Every tool definition you attach is paid for on every call, whether or not it is used. Attaching twelve tools to a step that only ever calls two is a recurring charge for nothing.

Retries, guardrails and failure handling

Failure handling is the policy for what happens when a step returns something wrong, malformed, or nothing at all. Naive retry logic is where token bills quietly double, because a retried model call costs full price and a chain that retries at three points can triple its worst-case spend on the runs that were already going badly. Decide up front which failures retry, which fall back to a cheaper deterministic path, and which stop and escalate to a person.

Observability and evaluation

Observability here means being able to reconstruct, after the fact, what the system had available and what it decided. Token dashboards are not this. You need the tool inventory at decision time, the step's inputs, the side effects that landed, and under whose credentials. We wrote separately about observability for agents and why input/output logging falls short.

The question the frameworks don't ask: does this step need a model at all?

Take the inbound document workflow above and label every step honestly.

Classify document type · Judgment or repeatable: Judgment at first, repeatable once the type set closes · Where it should run: Model, then a classifier · Why: Sender and layout patterns stabilize quickly
Extract fields from a known vendor layout · Judgment or repeatable: Repeatable · Where it should run: Deterministic parser · Why: The template is the same every time
Extract fields from an unseen layout · Judgment or repeatable: Judgment · Where it should run: Model · Why: No template exists yet
Validate totals and required fields · Judgment or repeatable: Repeatable · Where it should run: Code · Why: Arithmetic and schema checks
Match to a purchase order · Judgment or repeatable: Repeatable · Where it should run: Database query · Why: A deterministic join
Write to the system of record · Judgment or repeatable: Repeatable · Where it should run: Code, scoped credential · Why: A side effect that must be logged and attributable
Resolve an invoice-to-contract mismatch · Judgment or repeatable: Judgment · Where it should run: Model, escalating to a person · Why: Genuine ambiguity with money attached

Five of seven steps do not need a model once the workflow has run for a while. They needed one to be figured out. That is a different claim.

Here is the sequence we use to decide whether a step is ready to move:

Log the step's real inputs and outputs across a production window, not a curated test set.
Ask whether two identical inputs must produce identical outputs. If yes, the step is a candidate. If no, stop here.
Measure what share of the observed inputs fall into shapes you can enumerate. Below roughly the ninetieth percentile, the deterministic version will spend its life throwing exceptions.
Write the code version and shadow it against the model on live traffic, comparing outputs without acting on the code path.
Promote it when the disagreement rate stops moving, and keep the model as the fallback for inputs the code rejects.

Step five is the one people skip, and skipping it is how this goes wrong. Deterministic code fails on inputs it was never designed for, and it fails silently or loudly rather than gracefully. Moving a step out of the model too early produces a system that is cheap, fast, and wrong on the eleven percent of documents that do not match the template. The fallback path is what makes the move safe.

This argument does not address evaluation, which is a harder and separate problem. Knowing that a model step should stay a model step tells you nothing about whether its output is good. That deserves its own piece.

Common misconceptions

Picking a framework is an architecture decision. It is a dependency decision. LangGraph and CrewAI will both faithfully execute an expensive design. The architecture decision is the one in the table above.

Orchestration means multi-agent. Multi-agent is one topology. Most production workloads are a single reasoning path with tools attached, and adding agents that converse with each other adds model calls faster than it adds capability.

Caching and cheaper models fix the cost curve. They change its slope. Cache hits at 0.1x input and batch at half price are real savings on work you are still paying to redo. The curve only flattens when the work leaves the model.

Deterministic is always safer. Code is predictable within the input distribution it was written for and brittle outside it. A step that has not stabilized belongs in the model, and saying so is part of the discipline.

What we're building at Major in response

The industry settled on a definition of orchestration that means coordinating model calls, and that definition quietly guarantees two things. Cost scales with usage, and behavior stays probabilistic. We think the orchestration layer's real job is to keep making itself smaller, moving each step out of the model as soon as that step has stabilized.

So we built the platform around that move. On Major, when an agent works out how to handle a repeatable part of a task, it builds an app for that part. The app is deterministic code with a managed database, its own file storage, and its own logs. From then on the agent runs the app instead of reasoning through the step again. The agent layer decides. The app layer executes and remembers. State lives in the app's database rather than in a context window that has to be refilled and paid for on every run, and because the work is code with scoped credentials and audit logging, an action can be inspected and attributed months later. Reason once, run forever.

The model does not go away, and we would not claim otherwise. It still handles classification on inputs it has not seen, the ambiguity in a contract, and the judgment call about when to escalate. It just stops re-deriving the arithmetic. If your workflow is short enough to live in one prompt, none of this applies. The argument earns its keep when the workflow is long, branches, touches three systems of record, and someone will eventually have to explain to an auditor what it did in March.

If you are drawing that line between the steps that need judgment and the steps that need code, that boundary is what our platform is organized around: see how Major's agents turn a stabilized step into a deterministic app. Worth walking through from prototype to production first if you are earlier than that.

Frequently asked questions

What is LLM orchestration?: LLM orchestration is the code that coordinates language model calls into a working system: it holds state between steps, decides which model handles which step, calls tools, retries and handles failures, and records what happened. The model itself is one component inside it, since a model turns one input into one output and cannot sequence its own work. Most orchestration designs assume every step keeps running through a model. Major starts from the opposite assumption, having the agent reason once and write a deterministic app so the settled steps stop calling a model at all.
What is the difference between an LLM and an orchestrator?: An LLM turns one input into one output, holds no memory between calls, and cannot act outside its own response. An orchestrator is the code around it: it holds state, chooses which model handles which step, calls tools, handles failures, and records what happened. The model is one component inside the orchestrator. That split is also the lever, because anything the orchestrator can do in code does not need a model call. Major pushes that line as far as it goes, keeping the model for judgment and the app for everything settled.
What is an example of an LLM orchestrator?: LangGraph, CrewAI, AutoGen, LlamaIndex, and Haystack are the frameworks most teams reach for. A typical orchestrated workflow looks like this: a document arrives, a model classifies it, a parser extracts fields, code validates totals and matches a purchase order, the result writes to the system of record, and only genuine ambiguity goes back to the model. Notice how few of those steps actually need reasoning. On Major, an agent builds that pipeline as a deployed app with its own database, and the model handles only the ambiguous cases.
Do I need an orchestration framework?: Not if your workflow is a single prompt with one or two tools, because a framework adds a dependency and a mental model you will maintain. Reach for one when you have branching control flow, state that outlives a single call, and failure paths that need explicit policy. Even then, the framework choice matters far less than deciding which steps need a model. Major is the wrong call for genuinely exploratory work whose shape changes every run, and the right one once a workflow has stabilized enough to be written down as code.
How does LLM orchestration reduce cost?: Two levers do the work, with different math behind them. Routing sends cheap steps to smaller models, prompt caching charges cache hits at a tenth of base input price, and the Batch API halves asynchronous work. Those lower the slope of a bill that still rises with every run. The second lever moves a stabilized step into deterministic code, so that step costs the same at a million runs as at one. Major is built around the second lever, front-loading cost into the reasoning that produces the app and flattening it after.
Is LLM orchestration the same as AI orchestration?: LLM orchestration is the narrower term, covering control flow, routing, state, and tool calls around language models specifically. AI orchestration is the wider coordination of AI systems across an organization, including non-language models, data pipelines, and human approval steps. Our guide to AI orchestration covers that broader layer. Both terms describe coordination that runs through a model on every pass, which is the assumption Major changes by turning the repeatable middle of the pipeline into software the agent deploys and then runs.