Semantic caching vs model routing: which saves more money on LLM costs?

The Direct Answer: Routing Usually Wins, But the Two Are Not Competitors

If you are deciding where to spend your first engineering dollar on LLM cost reduction, model routing generally delivers the larger and more durable savings. Routing works by matching each request to the cheapest model that can handle it — sending a routine summarization task to a small open-weights model instead of a frontier model can cut per-request cost by 80–95%, because frontier models often cost 10–30x more per token than capable smaller alternatives. Semantic caching, by contrast, only saves money when users ask the same or nearly the same question repeatedly. In real production traffic, cache hit rates typically land between 15% and 45%; enterprise support bots with repetitive queries can exceed 60%, while a coding assistant or an agent pipeline doing novel work may see hit rates under 10%. A 40% cache hit rate against a frontier model saves roughly 40% of inference spend; routing the remaining 60% of requests down-tier can save another 50–70% of what remains.

Also worth reading: What is semantic caching for AI agents and how do you implement it in 2026? · What are the best AI agent model routing strategies in 2026, and how should teams choose between them? · What is the definitive agentic AI ontology modeling guide for building semantic layers in 2026?

The honest framing is that these are stacked layers, not rivals. Caching intercepts duplicate work before it reaches any model. Routing decides which model handles whatever survives the cache. Teams that deploy both commonly report total inference cost reductions of 60–85%, but the split matters for planning: routing savings scale with your traffic volume regardless of query diversity, while caching savings plateau once your query distribution saturates. If you run a task-graph orchestration system — as many product and ops teams now do — routing applies at every node of the graph, whereas caching mainly pays off at nodes with high input similarity, such as document classification or template-based generation steps.

There is also a quality dimension people underestimate. Bad caching silently degrades answers when near-matches get served stale or contextually wrong responses. Bad routing visibly degrades answers when a task exceeds the capability of the cheap model it was assigned to. Both failure modes erode trust faster than they save money, so thresholds and evaluation matter more than headline percentage claims.

How Model Routing Actually Works and Where the Savings Come From

Model routing sits between your application and a pool of models. An incoming request is classified — by rules, by a lightweight classifier model, or by heuristics embedded in the orchestration layer — and dispatched to the cheapest model whose predicted capability meets the task's difficulty threshold. Simple tasks like formatting, extraction from short documents, or yes/no classification go to small models costing fractions of a cent per thousand tokens. Ambiguous or high-stakes requests escalate to mid-tier or frontier models. Cursor's router introduction in 2025 made this pattern mainstream for developer tools, and platforms like OpenRouter, Martian, Not Diamond, and Unify have built businesses around workload-to-model matching.

The economics come from price dispersion across the model market. As of mid-2026, frontier-class models typically price in the range of $3–$15 per million output tokens, while strong small models (7B–14B class) deliver acceptable quality on structured tasks at $0.05–$0.50 per million tokens. That 20x+ spread means every request you successfully downgrade is worth far more than a cached request you avoid entirely — assuming equal volumes. Kearney's analysis of enterprise AI cost curves emphasizes that most enterprise workloads are not actually frontier-difficulty; a large share of tokens spent on expensive models goes to tasks a cheaper model completes within one or two percentage points of quality on evals.

Routing also compounds with token-per-dollar optimization techniques: trimming context windows before dispatch, compressing retrieval results, and batching compatible requests. A well-tuned routing layer typically reports 40–70% cost reduction versus always-frontier baselines, with quality regression held under 2–3% on benchmark suites. The catch is that those numbers depend entirely on how well your router classifies difficulty. Misrouting a hard legal-analysis prompt to a 7B model produces confidently wrong output, which in regulated workflows costs more than any token savings.

How Semantic Caching Works and Its Realistic Hit Rates

Semantic caching stores previous question-answer pairs and, on new requests, embeds the incoming query and retrieves the closest stored entry. If cosine similarity exceeds a configured threshold — commonly 0.90–0.97 depending on tolerance for paraphrase — the stored answer is returned without calling any model. The saving per hit is 100% of that request's inference cost, which sounds superior to routing's partial discount, but the denominator is the problem: you only save on hits, and hits require repeated intent.

Hit rates vary enormously by use case. Internal FAQ assistants and IT helpdesk bots routinely achieve 35–55% hit rates because employees ask the same questions about PTO policies and password resets. Customer support deflection systems can reach 50–70%. RAG pipelines over a stable corpus show moderate rates, though Towards Data Science's cost-control analysis notes that RAG applications burn disproportionate money because every query re-retrieves and re-embeds similar context — a hybrid approach caching both retrieved chunks and final answers attacks both costs. Conversely, agentic workflows, code generation, and personalized analytics rarely exceed 15% because inputs are genuinely novel each time.

Oracle's 2026 benchmarks of semantic caching with Oracle AI Database 26ai and True Cache highlighted another practical factor: latency. Cache hits return in single-digit milliseconds versus hundreds of milliseconds to seconds for inference, so caching improves user experience even beyond its cost contribution. However, embedding the incoming query itself costs money and adds latency — usually negligible (embedding models cost well under $0.02 per million tokens), but at very low hit rates the overhead plus the risk of wrong matches can make caching net-negative. Below roughly a 12–15% expected hit rate, most teams should skip semantic caching entirely and invest in routing instead.

Head-to-Head Comparison

Read the table honestly: routing wins on magnitude and universality, caching wins on latency and per-hit economics. The two risks differ in character too. Routing failures are loud — a bad answer from a weak model gets flagged quickly. Cache failures are quiet — a slightly-off answer that mostly looks right can circulate for weeks, which is why F5 and other infrastructure vendors pushing AI cost-and-safety tooling bundle cache governance with their offerings rather than treating caches as set-and-forget.

Practical Implementation Steps, In Order

Start by measuring before building anything. Instrument your current stack to log per-request token counts, model choice, and cost for two weeks. Segment traffic by task type — extraction, generation, classification, conversation, agentic loops. Most teams discover a Pareto distribution: 20% of task types consume 80% of spend, and much of that spend is on over-provisioned models.

Second, build the routing tier before the cache. Define three tiers (small, mid, frontier), write 50–200 evaluation prompts per task type with graded quality rubrics, and set escalation rules: if the small model's confidence or a cheap verifier flags uncertainty, retry on the next tier up. This cascading approach caps quality loss while capturing most of the discount. Expect one to two engineer-months of work and payback within the first billing cycle if monthly inference spend exceeds roughly $5,000.

Third, add semantic caching selectively, not globally. Apply it only to the traffic segments where measured query similarity is high — support intents, internal FAQs, template-driven generation. Use conservative similarity thresholds (0.95+) initially, attach TTLs to cached entries so pricing changes, policy updates, or data refreshes invalidate stale answers, and log every cache hit with its match score so you can audit near-miss quality later.

Fourth, wire both layers into your orchestration graph rather than bolting them on. In a task-graph system, each node declares its model policy and cache eligibility explicitly. This makes costs attributable per workflow, lets ops teams see which branch of a pipeline burns budget, and prevents the common failure where a cache tuned for one node poisons another node's context.

Fifth, keep evaluating monthly. Model prices drop continuously — prices for equivalent capability fell roughly 10x between early 2023 and late 2025 — so a routing rule that made sense last quarter may be leaving money on the table today, and a cached answer generated by a since-deprecated model may no longer reflect your product's behavior.

Common Mistakes That Erase the Savings

The most expensive mistake is applying one global cache threshold across all traffic. A 0.92 similarity threshold that works fine for FAQ lookups will serve dangerously wrong answers for medical, legal, or financial queries where paraphrase changes meaning. Segment your thresholds by task criticality, and disable caching outright for anything feeding compliance-sensitive decisions.

The second mistake is routing on prompt length instead of task difficulty. Long prompts feel expensive, but length correlates poorly with required capability; a 10,000-token extraction job is trivially easy for a small model, while a two-sentence strategic question may need a frontier model. Route on task type and evaluated difficulty, not token count.

Third, teams forget that cache entries themselves carry hidden costs: vector storage, embedding compute, invalidation logic, and the operational tax of debugging 'why did the bot say that' incidents traced to stale hits. Budget for this — realistically 0.5 to 1 engineer maintaining cache hygiene per 10 million monthly requests.

Fourth, organizations chase vendor routers blindly without measuring their own traffic. Published savings figures (often 50–80%) come from vendors' best-case customers. Run a shadow deployment: route 10% of live traffic through the candidate configuration, compare cost and quality against your baseline for two weeks, then decide. Fifth, many teams skip the boring win of prompt and context compression, which stacks multiplicatively with both routing and caching and requires almost no infrastructure.

When to Act, and What It Costs

Act when monthly inference spend crosses roughly $2,000–$5,000, or earlier if you are on usage-based frontier-model APIs with growing traffic. Below that threshold, engineering time usually outweighs savings, and simple moves — switching default models, trimming context, capping max_tokens — capture 20–30% for a day of work.

Cost-wise, the build-versus-buy math has shifted through 2025–2026. Managed routing platforms charge either a margin on tokens (typically 5–15%), a flat platform fee ($500–$5,000/month at mid-market scale), or free passthrough with paid premium features. Semantic caching is increasingly bundled into inference gateways and databases — Oracle's True Cache integration being one example — rather than sold standalone. Building in-house with open-source components (a gateway, an embedding model, pgvector or a managed vector store) costs one to three engineer-months upfront plus ongoing maintenance, and pays off fastest for teams above roughly $20,000/month in inference spend who want full control over thresholds and data residency.

Timing also interacts with your roadmap. If you plan to move toward multi-step agents in the next two quarters, build routing into the orchestration layer now — retrofitting cost controls onto an existing agent mesh is significantly harder than designing them in, because agent loops multiply token consumption 5–50x versus single-shot calls and amplify every inefficiency.

The Verdict for Product and Ops Teams

For teams running AI inside structured workflows — the typical dotinc-style audience orchestrating task graphs across product and operations — the priority order is clear. First, right-size models per task via routing, targeting 40–60% reduction. Second, compress prompts and retrieval context, adding another 15–30%. Third, add semantic caching only where measured repetition justifies it, expecting 15–40% on those segments and near-zero elsewhere. Combined, mature implementations report 60–85% total inference cost reduction, but treat vendor headlines skeptically until your own shadow tests confirm them on your traffic. The teams that win on AI cost are not the ones with the cleverest cache; they are the ones that measure per-task economics relentlessly and let the data assign each request to its cheapest adequate path.