Every few months a chart goes around claiming Cache-Augmented Generation is “40x faster than RAG,” and another round of “RAG is dead” posts follows. I finally sat down with the actual CAG paper, the long-context research it depends on, and the current provider pricing — and the honest picture is far more interesting than the headline. CAG is a real, measurable win — but only inside a narrow box, and its famous speedup was never measured against RAG at all. If you’re deciding how to ground an AI system this quarter, this is the technical brief I wish someone had handed me.

What I’ll Cover in This Blog

✔️ What RAG and CAG actually are, mechanically — including what a KV cache really caches
✔️ The CAG paper’s own numbers: accuracy, latency, and the misquoted 40x
✔️ The long-context reality: advertised windows vs effective windows
✔️ The cost math with current prices, verified
✔️ What this means for the “RAG is dead” argument

Now, let’s dive in! 🔥

The Two Mechanisms, Honestly Described

RAG (Retrieval-Augmented Generation) — introduced by Lewis et al. at Meta FAIR in 2020 — is the pipeline you know: chunk documents, embed them, index them (vector, BM25 lexical, or hybrid). At query time, retrieve the top-k relevant chunks, optionally rerank, inject them into the prompt, generate. The model only ever sees a small, query-specific slice of the corpus — which means quality lives or dies on retrieval. If the right chunk isn’t fetched, the model can’t use it.

CAG (Cache-Augmented Generation) — from Chan et al., “Don’t Do RAG” — takes the opposite bet: skip retrieval entirely. Preload the whole curated knowledge base into the context once, run one forward pass over it, and save the transformer’s KV cache — the key/value tensors every layer computed for those tokens. Because attention is causal, a fixed document prefix always produces identical KV tensors, so you compute them once and reuse them for every query. Each new question only pays for its own tokens.

🔹 What the cache saves: the expensive “prefill” pass over the corpus, on every query. That’s real and large.

🔹 What the cache does NOT save: the model still attends over the full cached context for every token it generates. The long-context compute cost per generated token — and every long-context quality risk — stays.

🔹 The commercial version: provider “prompt caching” is the managed-API form of the same idea, with two catches — caches are ephemeral (TTL-bound: minutes to an hour, depending on provider and tier) and a cache hit requires a byte-identical prefix. One timestamp at the top of your prompt breaks every hit.

What the Paper Actually Measured

The setup: Llama 3.1 8B, corpora between 21K and 85K tokens from HotPotQA and SQuAD, BERTScore for accuracy, RAG baselines at various top-k. Three findings deserve to be quoted accurately, because almost nobody does:

🔹 Accuracy: CAG won most configurations — by 1 to 3 BERTScore points, on easy datasets. The authors themselves note that sparse BM25 beating dense retrieval “suggests that the datasets may not be sufficiently challenging.” And on the largest corpus (85K tokens), RAG won — the authors concede the gap “narrows” as data grows, and link it to long-context degradation.

🔹 **Latency: the famous 40x is CAG vs uncached long context — not vs RAG. On the large set: uncached in-context loading took 92.08s, CAG took 2.26s. That’s the headline number. But sparse RAG top-3 took ~0.67s — faster than CAG** — because it generates over a few chunks instead of attending over 85K tokens. The misquote (“40x faster than RAG”) reverses the paper’s own table.

🔹 The authors’ own boundary: the approach “becomes impractical for significantly larger datasets,” and they explicitly suggest hybrids — “preload a foundation context and use retrieval only to augment edge cases.”

One more accuracy note for anyone citing this in a deck: the results are one 8B model, one metric (BERTScore), corpora capped at 85K tokens. It’s a solid proof of concept. It is not an enterprise benchmark.

The Long-Context Reality: Effective vs Advertised

CAG’s whole premise is “the corpus fits in context.” The research record says: check which context — the one on the pricing page, or the one the model actually handles well?

🔹 Lost in the Middle (Liu et al., TACL 2024): accuracy follows a U-shaped curve — models use information at the start and end of context far better than the middle. Multi-document QA dropped by more than 20% in the worst cases, sometimes below closed-book performance.

🔹 RULER (NVIDIA): despite perfect needle-in-a-haystack demos, “almost all models exhibit large performance drops as the context length increases.” The CAG paper itself cites RULER to size Llama 3.1’s effective length at 32K (8B) and 64K (70B) — against a 128K advertised window.

🔹 Databricks Mosaic (Nov 2024, 20 LLMs): “only a handful of the most recent state of the art LLMs can maintain consistent accuracy at long context above 64k tokens.” Chroma’s 2025 “Context Rot” study found the same shape across 18 frontier models (worth noting: Chroma sells retrieval infrastructure — but the neutral academic results agree).

🔹 The frontier has largely converged on ~1M-token advertised windows as of this year — and every one of these studies says to plan capacity well below the sticker number.

The practical translation: treat roughly 100–200K tokens as the honest ceiling for “just put it all in context” — which happens to match Anthropic’s own guidance in its Contextual Retrieval post: a knowledge base under ~200K tokens (about 500 pages) can simply ride in the prompt, “with no need for RAG.” That same post, by the way, is retrieval’s best recent result: contextual embeddings plus reranking cut top-20 retrieval failures by 67%. Both sides of this debate are improving.

The Cost Math, Verified

Provider prompt caching changed the economics this year — cached reads now cost roughly a tenth of normal input at the major providers. Here’s the worked example on current, verified Claude pricing (Sonnet 5 at $2/MTok input, cache reads at 0.1x, 5-minute cache writes at 1.25x — as of September 2026):

🔹 Caching pays for itself after about two warm reads — a genuine 90% saving over naive long context.
🔹 But cached CAG still costs ~10x more input per query than RAG, because you pay for the whole corpus on every call, discounted or not. At enterprise query volumes that gap compounds.
🔹 RAG isn’t free either — embeddings, a vector store, a reranker — so compare total cost of ownership, not just tokens. But the per-query asymmetry is structural.

So… Is RAG Dead?

Here’s my read after all of it, and it’s the same conclusion I reached about model size in You Don’t Need a Bigger Model: the loudest framing is the wrong question.

Naive top-k vector RAG is fading. Retrieval is not. Agentic search — a model running grep, calling tools, querying systems at answer time — is retrieval; what changed is who decides what to fetch. Long context raised the threshold at which a retrieval pipeline becomes necessary; it didn’t remove the need. And the strongest voices on each side of the “RAG is dead” fight are, respectively, long-context enthusiasts and retrieval vendors — read the incentives before you read the benchmarks.

What the evidence supports is a workload decision, not a funeral: cache the stable, retrieve the volatile. I’ve turned that into a full decision framework — thresholds, the five CAG conditions, the hybrid prompt anatomy — in part two, and into a real company story in part three.

Conclusion

✔️ CAG = long context + KV-cache reuse. It skips prefill, not attention — the costs and risks of long context remain.
✔️ The 40x is real but misquoted: it’s CAG vs uncached prompting. In the paper’s own table, sparse RAG was faster than CAG.
✔️ Accuracy favored CAG by 1–3 points on easy sets — and flipped to RAG at 85K tokens, exactly where long-context degradation predicts.
✔️ Effective windows are the binding constraint: ~32–64K for the paper’s own model, well under advertised limits across the industry.
✔️ Cached CAG ≈ 10x RAG’s input cost per query on current verified pricing, even with 90% cache discounts.
✔️ The right question was never “is RAG dead” — it’s “which parts of my knowledge are stable enough to cache, and which are too big, too fresh, or too permissioned to be anything but retrieved?”

Read the paper, not the screenshots of the paper.

Connect with me on LinkedIn →

Deciding between long context and retrieval for a real system right now? Tell me your corpus size, churn rate, and permission model — those three answers usually decide it.

What’s the most confidently misquoted benchmark you’ve seen in an AI architecture debate? Tell me on LinkedIn.

About the Author — Abubakar Asif