LLM Cost Optimization: What Works and What It Misses

Key takeaways

• LLM cost is three factors multiplied together: price per token, tokens per call, and calls per unit of work. Most guides attack only the first.
• Prompt caching is the highest-return first move for agent workloads, because agents resend the same system prompt and tool schemas on every turn.
• The batch discount is documented provider pricing at Anthropic, OpenAI, and Google, not a vendor estimate. Half price, asynchronous.
• Output tokens bill at several times input on every major provider, so constraining response format moves the expensive side of the ledger.
• Every lever here lowers the price of a call the system still makes. Removing the repeatable call is a different move at a different layer.

What LLM cost optimization actually means

LLM cost optimization is the practice of lowering what an application spends on model inference without giving up the quality of what it produces. The definition is easy. The arithmetic underneath it is where teams lose money.

Spend on any model-backed feature resolves to this:

price per token × tokens per call × calls per unit of work

Almost everything published under this keyword works on the first factor and some of the second. Swap to a cheaper model, trim the prompt, cap the response. Those are real reductions worth having. The third factor is where agentic systems actually spend, and it barely appears in the literature. A support agent that answers one ticket may issue a dozen model calls: a classification pass, two tool calls whose results come back into context, a retry after a malformed function call, a summarization step, a final draft. One unit of work, twelve billable events, each carrying the full system prompt again.

Cost work that never touches the third factor buys a discount on a quantity that keeps rising.

Why the bill keeps growing even as token prices fall

The bill grows because calls per task are rising faster than the price per token is falling. Both halves of that sentence are measurable.

Per-token prices have collapsed. Stanford HAI's 2025 AI Index tracked the cost of querying a model at roughly GPT-3.5 quality, measured at 64.8% on MMLU, falling from $20 per million tokens in November 2022 to $0.07 per million tokens in October 2024, more than a 280-fold decline in about eighteen months.

Enterprise spend went the other way. Menlo Ventures put total enterprise generative AI spending at $37 billion in 2025, up from $11.5 billion in 2024, with foundation model APIs accounting for $12.5 billion of the infrastructure layer.

Both numbers are correct at the same time because the unit of consumption changed. A chatbot turn was one call. An agent handling the same request runs a loop: plan, call a tool, read the result, decide again. The agentic workflows that made these systems useful in production are the same patterns that multiply calls per task. Cheaper tokens got spent on more tokens, which is the normal response to a falling input price.

How do you reduce LLM costs? Six levers and the symptom each one treats

You reduce LLM costs by matching a lever to the cost symptom you can see in your usage data, rather than working down a list. Throughout this section, hold one workload in mind: a ticket-triage agent that reads an inbound support message, classifies it, pulls the customer's plan and recent orders, and routes it to a queue. All provider prices below were checked against the vendors' own documentation on 21 August 2026. Re-check them before you plan against them, because they change.

• Same system prompt and tool schemas resent on every call · Lever: Prompt caching · Mechanism: Provider stores the processed prefix and reads it back · Documented pricing effect: Anthropic charges cache reads at 0.1x base input, with writes at 1.25x for five minutes and 2x for one hour · Main tradeoff: Prefix must be byte-stable and placed first
• Work that nobody is waiting on · Lever: Batch processing · Mechanism: Requests queue for asynchronous execution · Documented pricing effect: 50% off input and output at Anthropic, 50% off with a 24-hour window at OpenAI, 50% off eligible Gemini rates · Main tradeoff: Unusable for interactive paths
• One expensive model handling easy and hard alike · Lever: Routing and cascades · Mechanism: Cheap model answers first, escalate on low confidence · Documented pricing effect: Provider price gaps are large: Claude Haiku 4.5 lists at $1 per million input against $5 for Opus 5 (pricing) · Main tradeoff: You need a quality gate you trust
• Long, chatty responses · Lever: Output-token control · Mechanism: Constrained formats and max-token limits · Documented pricing effect: Output bills at five times input across the Claude line, for example $2 in and $10 out for Sonnet 5 (pricing) · Main tradeoff: Truncated reasoning, brittle parsing
• Whole documents pasted into context · Lever: Retrieval instead of stuffing · Mechanism: Fetch only the passages the task needs · Documented pricing effect: Removes input tokens at their listed rate · Main tradeoff: Adds embedding and vector-store cost, and retrieval quality becomes a correctness risk
• Many near-identical requests · Lever: Semantic caching · Mechanism: Serve a stored answer when a new query is similar enough · Documented pricing effect: Avoids the call entirely on a hit · Main tradeoff: A loose threshold returns a confidently wrong answer

Repeated context: prompt caching

Every turn of an agent loop resends the system prompt, the tool definitions, and the conversation so far. Caching stops you paying full input price to reprocess text the provider already read. Anthropic prices a cache read at 0.1x the base input rate, with a 1.25x write for the five-minute TTL and 2x for the hour, which means a five-minute cache pays for itself after a single hit. OpenAI's cached input rate varies by model. Google charges a discounted cached rate plus a storage fee per million tokens per hour. The tradeoff is structural. The cached prefix has to be identical and it has to come first, so a timestamp near the top of your system prompt quietly disables the whole thing.

Work that can wait: batch processing

If a human is not watching a spinner, you are overpaying by half. Anthropic's Message Batches API runs at a 50% discount on input and output, with most batches finishing inside an hour. OpenAI's Batch API applies the same 50% discount against a 24-hour completion window, and Google halves eligible Gemini rates for batch mode. Nightly classification, backfills, evaluation runs, and document enrichment all qualify. Interactive triage does not. The common mistake is assuming a workload is interactive because it currently runs inline, when the only thing making it synchronous is how someone wired it. Sorting your call volume by whether a person is waiting takes an afternoon, and half the answer carries a documented discount.

Mixed task difficulty: routing and cascades

Most request streams are bimodal. A large share is trivial, a small share is genuinely hard, and a single model priced for the hard share serves both. A cascade sends the cheap model first and escalates when confidence is low. Chen, Zaharia, and Zou described this in FrugalGPT in 2023; their reported reductions belong to that paper's benchmarks and to model prices that no longer exist. The mechanism survived the numbers. Major builds with Claude, Kimi, Gemini, Muse Spark, ChatGPT, and Grok, so cheaper-model substitution is available inside Major too. What you need first is a quality gate, because a cascade without one is a way to ship worse answers cheaply.

Verbose responses: output-token control

Output is the expensive side. Across the Claude line, output bills at five times input, so Sonnet 5 lists at $2 per million input and $10 per million output. A model that writes three paragraphs of preamble before a JSON object is charging you a premium rate for the preamble. Structured output formats, explicit max-token limits, and instructions that forbid restating the question move real money and usually improve downstream parsing too. The failure mode is capping a reasoning model mid-thought, which produces a truncated answer you paid for and then have to retry.

Bloated context: retrieval instead of stuffing

Pasting an entire knowledge base into the prompt is the most expensive way to answer a narrow question. Retrieval replaces the paste with a targeted fetch, and the input reduction is arithmetic against your model's listed input rate. The cost moves rather than vanishing. You now run an embedding pipeline, a vector store, and a retrieval step, each with its own bill. The larger risk is not financial. Retrieval quality becomes a correctness property of your system, so a bad chunking strategy shows up as wrong answers rather than as a line item.

Near-identical requests: semantic caching

Semantic caching stores past answers and serves one when a new query is close enough by embedding distance. On a hit, you skip the call. It is the only lever on this list that can silently change what your users see, because the threshold is a guess about equivalence. At 0.95 similarity, "cancel my subscription" and "cancel my last order" can collide, and the user gets a fluent, confident, wrong answer with no error anywhere in your logs. Worth naming the distinction plainly: semantic caching guesses that two requests are the same, while an application encodes that they are.

The one thing all six have in common

Five of the six lower the price of a call your system still makes. Caching lowered the price of re-reading context. Batching lowered the price of waiting. Routing lowered the price of the reasoning. Output limits lowered the price of the answer. Retrieval lowered the price of the input. Only semantic caching skips a call, and it does so by guessing the call was a duplicate.

For a large slice of production work, the call being discounted is the model re-deriving a procedure whose shape stopped changing weeks ago. Your triage agent has classified forty thousand tickets. The routing rules settled in the second week. Every ticket since has paid a model, at a discounted rate, to work out a path it has already worked out.

Another move is available. The agent works out the procedure once, writes it as a deterministic application, and runs that instead. The work leaves the model. Cost front-loads into the reasoning that produced the app, then flattens as the app runs. Determinism, statefulness, and governability arrive together from that single move, because state lives in the app's database rather than a context window, and the action is code someone can read, permission, and audit. That is what an agent platform has to provide before cost stops tracking usage.

When each approach is the right one

Keep work in the model when its rules are still moving, and turn it into an app when the rules have settled. This does not apply everywhere. An application built for work whose rules change weekly costs more to maintain than it saves, and reaching for agentic automation before the pattern is stable is how teams end up maintaining software nobody wanted.

• Keep this in the model: Novel judgment with no precedent · Turn this into an app: Stable inputs and a stable output contract
• Keep this in the model: Tasks that run a handful of times a month · Turn this into an app: Work that runs hundreds of times a week
• Keep this in the model: Rules still being discovered · Turn this into an app: Rules that have not changed in a month
• Keep this in the model: Open-ended drafting and tone work · Turn this into an app: Lookups, joins, and record updates
• Keep this in the model: Edge cases a person will review anyway · Turn this into an app: Steps another team depends on being identical
• Keep this in the model: Exploratory analysis · Turn this into an app: Anything an auditor will ask about later

Common misconceptions

A cheaper model is the biggest lever. Model choice changes the price per token and nothing else. If your agent resends 8,000 tokens of system prompt and tool schemas on every turn, it resends them to the cheap model too. Fix the resend first.

Caching works automatically. Defaults differ by provider, and minimum cacheable prefixes are real: Anthropic requires 512 to 4,096 tokens depending on model, and Gemini sets its own per-model thresholds. Below the floor, your cache_control field does nothing and you pay full price.

A gateway solves cost. A gateway gives you genuine control points for routing, caching, and budget limits, applied once instead of in every service. It does not reduce how much reasoning your application performs. Attributing spend to the decision that caused it is observability that tracks what the agent actually did, which is a different capability from the proxy that routes the call.

Falling token prices will fix this. They have been falling for three years. Enterprise bills went up anyway, because calls per task rose faster than price per token fell.

What we're building at Major in response

The six levers work, and prompt caching alone justifies an afternoon on most agent workloads. The limit is what they leave in place. Each one changes the price of reasoning the system is going to perform again anyway, so the bill still tracks usage, and usage is the thing that grows.

Major is the enterprise platform where AI agents build the software they run on. When an agent works out how to handle a repeatable part of a task, it builds an app for that part and runs the app instead of reasoning through the work again. Reason once, run forever. The cost shape that follows is front-loaded into the reasoning that produces the app, then flat as the app runs, which is a different curve from a discounted version of the same climb. We are not claiming the model goes away. It handles less.

Back to ticket triage. The classification rules, the routing table, the account lookup, and the audit log become application code with a managed database behind it and permissions at the platform layer. The model keeps the ambiguous ticket, the angry customer, the case that does not match anything the app has seen. Because context and memory live in the app rather than in one vendor's context window, swapping in a cheaper model is a change of input rather than a migration, and the choice stays reversible. That is also what makes building an agent that runs in production survive contact with an auditor, since the repeatable path is inspectable code rather than a transcript.

New accounts get $100 in free credits to build your first AI agent or app on Major, which is enough to build and run the ticket-triage agent described in this article. If the third factor in your cost identity is the one that is growing, see how Major moves repeatable work out of the model and into an app.

Related articles

• LLM Router: Routing Strategies and Their Real Limits
• AI Orchestration: Coordinating Models, Agents, and the App Layer
• Claude Agent: What It Is and How to Build One That Lasts