Semantic Caching for Agent Crews: Cutting Duplicate Token Spend
By wGrow Project Team ·
A team that audits its OpenAI bill finds an embarrassing number: a large sum spent generating answers to questions it has already answered — not the same questions exactly, but the same questions with a swapped preposition, a different article, a rephrased subordinate clause. The LLM does not care. It bills full generation cost every time.
That is the exact-match fallacy. And it is expensive.
The Exact-Match Fallacy in Agent Workflows
Traditional caching operates on cryptographic hashes. SHA-256 the input string, check the cache, return on hit, compute on miss. This works for deterministic function calls where the same bytes produce the same result. LLM prompts are another matter entirely.
A changed comma is a cache miss. A synonym is a cache miss. “What is the penalty for late submission under Section 14?” and “Under Section 14, what are the penalties if a submission is late?” hash to completely different keys. The LLM returns functionally identical answers. You paid twice.
In agentic workflows, this compounds fast. Intermediate agents dynamically assemble prompts from upstream context injection, so the same user intent arrives as a slightly different string every time. Exact-match hashing on dynamically assembled prompts produces hit rates that are functionally zero.
A document processing crew handling a high volume of daily policy queries has different people asking about the same policies in different phrasing. Because each paraphrase hashes to a different key, a cache keyed on SHA-256 hashes against a Redis store would rarely hit, and the misses would be paraphrases of queries the crew had already answered. The team would be paying full LLM generation rates on traffic that is semantically redundant.
Architecture of a Semantic Interceptor
The fix is to cache on meaning, not bytes.
A semantic cache layer can sit in front of the agent crew and intercept duplicate intent before it reaches the model. The stack can be deliberately boring: if you already run Redis Stack, its RediSearch query engine provides HNSW vector indexing. Use that rather than standing up a dedicated vector database — fewer moving parts, one less managed service, one less failure domain. When you’re debugging a production incident at 2 a.m., you want boring.
The embedding model is OpenAI’s text-embedding-3-small. Cache key generation needs to be fast and cheap, not semantically rich — just stable and consistent. text-embedding-3-small fits that requirement well.
Incoming queries are embedded, then run against the Redis index via KNN vector search. If the cosine similarity of the nearest neighbor exceeds the threshold, return the cached answer directly — the crew never runs. Below threshold, the crew executes, and the answer is written along with its embedded query vector and source document metadata back to Redis.
Routing is binary: a hit is a hit. A team can add a lightweight verification pass for high-stakes domains — routing cache hits through a smaller model to confirm relevance before serving. With a well-calibrated threshold that may be unnecessary, but it is worth considering for any use case where a false positive carries real cost.
The Unit Economics of Interception
- Gen Compute (gpt-4o)
- $10.00
- Embed Compute
- $0.02
per 1M output tokens
per 1M tokens
Caching is not free. You trade generation compute for embedding compute plus vector search. The math has to close.
OpenAI’s pricing page listed gpt-4o output at 0.005 per generation.
At the same pricing date, OpenAI listed text-embedding-3-small at 0.0000006 per embedding.
The ratio is approximately 8,300 to 1. Avoiding a single LLM generation pays for the embedding compute on roughly 8,300 incoming queries. The break-even math is so lopsided it barely warrants a spreadsheet.
Latency is a secondary benefit that becomes significant at scale. A Redis HNSW lookup adds only a small delay on the hot path, while a cache miss generating roughly 500 output tokens takes seconds end to end depending on load. A cache hit skips generation entirely, so it returns far faster than the miss path.
Calibrating the Similarity Threshold
Vector caching introduces a failure mode that text caching does not have: fuzzy matching can return wrong answers.
Set the threshold too low, and you start serving cached answers to questions that are superficially similar but semantically distinct. “What is the penalty for late submission under Section 14?” might retrieve an answer generated for a question about Section 14 registration fees. Both questions are about Section 14. They are not the same question. In a regulatory context, that is not a UX problem — it is a liability.
Set it too high, and you approach near-exact-match behaviour. Hit rate collapses, and you have added embedding latency to every miss without gaining anything on the hits.
Do not guess this number. Run historical query logs through a calibration script: embed every logged query pair that produced the same crew output, plot the distribution of pairwise cosine similarities, find the threshold that maximises true positives while holding false positives below an acceptable rate. If the distribution is steep in the relevant range, there is a band where the threshold does useful work, and outside that band it is either useless or dangerous. Plotting the data shows where that band sits.
For a regulatory domain, err on the strict side. At a strict threshold, minor phrasing variation is absorbed while genuinely distinct questions are not collapsed. Track false-positive cache hits, for example through support review of a sampled ticket set, as a monitoring signal rather than a guarantee of zero errors.
The right threshold is specific to the domain. Regulatory queries have tight semantic boundaries; a customer support workflow handling conversational, loosely-worded queries will calibrate lower. The methodology — empirical calibration from production logs — holds regardless of where you land.
Cache Invalidation and State Management
Agent crews operating on live documents face a cache invalidation problem more serious than typical web caching. A stale cached answer about a regulatory penalty that has since been amended is not just wrong — in certain contexts it is actively harmful.
TTL expiration does not solve this. A 24-hour TTL means potentially serving stale regulatory guidance for a full day after a policy update. Shrink the TTL to an hour and you preserve freshness but gut your hit rate — you have traded one problem for another.
Use event-driven invalidation instead. When a source document is updated, the ingestion pipeline fires a webhook to a cache management service. Every cache entry in Redis carries metadata identifying which source document IDs were consulted to generate that answer. The webhook triggers a targeted purge of every entry referencing the updated document ID.
This requires discipline in the generation path: when the crew produces an answer, record which document chunks were retrieved, and that provenance travels with the cache entry. Invalidation is precise. Unaffected entries survive. Hit rate recovers within a query cycle rather than waiting out a TTL window.
The implementation can be a small webhook receiver, for example in FastAPI. The complexity is not in the code — it is in deciding to do it correctly from the start rather than retrofitting after the first production incident.
Compute is for Reasoning, Not Regurgitating
Generating tokens is the most expensive phase of an AI pipeline. Context windows are growing, which means per-query generation costs can rise even as per-token prices fall. Hardware improvements help at the margins. The underlying cost structure is compute-bound, and that is not changing soon.
A semantic routing layer can deliver a meaningful cache hit rate, lower generation cost, a low false-positive rate when the threshold is calibrated, and deterministic invalidation — with no exotic tooling.
The architecture generalises directly to any agent workflow with high query repetition and retrievable answers: customer support crews, internal knowledge-base agents, document processing pipelines. The embedding step is cheap enough that break-even analysis is almost beside the point.
Save the token budget for problems the model actually has to reason through. Let $0.02-per-million embeddings handle the regurgitation.