Some projects stall on technology. This one stalled on a blog war. A 300-person insurance broker came to me with an internal AI assistant that had been “two months from launch” for eight months — not because anything was broken, but because the team had split into two camps that each read a different article. One side had seen “RAG is dead” and wanted the entire knowledge base stuffed into a million-token context. The other had built a vector pipeline and wanted everything indexed, including an 80-page underwriting manual that changes twice a year. Both sides were right about half the corpus — and that sentence, once everyone accepted it, shipped the project in five weeks. Here’s the whole story, numbers included.

What I’ll Cover in This Blog

✔️ The scenario: a stalled assistant and two teams armed with opposite blog posts
✔️ The corpus audit that dissolved the argument
✔️ The architecture: a cached stable pack plus permission-aware retrieval from Salesforce
✔️ The results: latency, accuracy, cost per query, and the update workflow
✔️ What generalizes to your knowledge base

Now, let’s dive in! 🔥

The Scenario

Varena Insurance Brokers (anonymized, details adjusted) runs its business on Salesforce — Sales Cloud and Service Cloud as the system of record for clients, policies, and claims tickets. The goal: an internal assistant for brokers and service staff that could answer two very different kinds of questions:

🔹 “What’s our underwriting position on coastal commercial property?” — answered from the underwriting manual, product guides, and compliance policies.

🔹 “Where does the Hartley renewal stand, and what did we quote them last year?” — answered from live CRM records, claims history, and tickets.

Camp Long-Context had a fair point: the manuals fit easily in a modern context window, and their prototype answered synthesis questions beautifully. Camp RAG had a fair point too: client records change hourly, and brokers must never see accounts outside their book of business. Each camp kept citing benchmarks at the other — mostly the misquoted ones I unpacked in part one. The project needed an audit, not another article.

The Corpus Audit: One Knowledge Base, Two Species

We spent three days measuring what the knowledge actually was, instead of arguing about what architecture deserved it. The split was almost comically clean:

Run the stable half through the five CAG conditions from part two: 140K tokens fits the effective window with headroom ✓, versioned twice a year ✓, one permission scope ✓, queried constantly by 300 people ✓, synthesis-heavy questions ✓. Five for five. Now run the volatile half: it fails four of the five instantly — unbounded scale, hourly churn, per-broker ACLs, record-level citations. No single architecture was ever going to serve both. The knowledge base wasn’t one thing. It was two things wearing one name.

The Architecture: One Assistant, Two Memories

What we built in the five weeks that followed:

🔹 The stable pack, cached. The underwriting manual, product guides, and compliance policies — curated, deduplicated, and versioned as one 140K-token “knowledge pack” with a version header. It rides in the assistant’s prompt as a byte-identical cached prefix: written once per release, read at ~0.1x input cost on every query. No chunking, no embeddings, no retrieval misses on the questions where a miss hurts most — “what’s our position when X and Y apply” questions that need six sections of the manual at once.

🔹 The live half, retrieved from Salesforce. Client, policy, claim, and ticket data stays where it lives — in the CRM — and the assistant fetches it per query through permission-aware access, so Salesforce sharing rules decide what each broker’s assistant can see. Nothing is copied into an index that would need its own security model; the enforcement stays in the platform, the same principle as securing Salesforce-hosted MCP access. Every answer about a client cites the records it read.

🔹 The seam. The prompt is exactly the hybrid anatomy from part two: cached prefix (instructions + tool schemas + knowledge pack, byte-identical), then the uncached suffix (retrieved records + the question). The one incident worth confessing: an engineer added a “current date” line at the top of the prompt in week four, silently killing every cache hit for two days. The bill noticed before anyone else did. Dynamic content goes after the cache boundary — now it’s a lint rule.

The Results

Measured against a 40-question eval set (built before the build, per the deploy-what-you-understand rule) and a month of production traffic:

🔹 Underwriting questions: the cached pack answered synthesis questions the old vector prototype had botched — the eval’s cross-referencing questions (“coastal commercial and prior claims and the compliance rider”) went from the retrieval prototype’s frequent partial answers to consistently complete ones, because nothing relevant could fail to be retrieved. This is exactly the corpus profile where the CAG paper’s results genuinely apply.

🔹 Client questions: answered from live records with record-level citations, and — the part the compliance officer cared about — a broker cannot get an answer about an account they can’t open in Salesforce. The prompt never held data the user wasn’t entitled to, because filtering happened at fetch, not in instructions.

🔹 Cost: the stable pack costs ~$0.03 of cached input per query (140K tokens at 0.1x on verified Sonnet 5 pricing) instead of ~$0.28 uncached — and the alternative nobody now argues for, caching everything clients-included, was impossible anyway at millions of records with per-broker visibility. The retrieved suffix averages a few thousand tokens. Total input cost per query landed around a nickel.

🔹 The update workflow is the quiet win. When the underwriting manual’s new edition ships, one script rebuilds the pack, bumps the version header, and re-warms the cache — a fifteen-minute release. Client data needs no pipeline maintenance at all, because it was never copied out of Salesforce in the first place. Two memories, each maintained the way its nature demands.

What Generalizes

Varena’s pattern transfers to almost any business with a system of record:

✔️ Audit the corpus before choosing the architecture — measure size, churn, permissions, and question types per content set. The argument usually dissolves in the data.
✔️ Most knowledge bases are two species wearing one name: a small stable canon and a large volatile stream. Architect for both, separately.
✔️ Cache the canon: curated, versioned, one permission scope, byte-identical prefix, rebuilt on release.
✔️ Retrieve the stream: live, permission-filtered at the source of record, cited at record level.
✔️ Keep enforcement in the platform — sharing rules and ACLs at fetch time, never “please don’t show broker A broker B’s clients” in a prompt.
✔️ Put an eval set on the seam before launch — synthesis questions for the cached half, freshness-and-permissions questions for the retrieved half.

The teams that stay stuck in the RAG-vs-CAG war are arguing about the whole knowledge base. The teams that ship stopped treating it as one thing.

Connect with me on LinkedIn →

Is your assistant project stuck in the same argument? Send me a one-paragraph description of your knowledge base — I’ll tell you where the split line runs.

What’s in your “stable canon” — the documents you’d cache tomorrow if the war ended today? Tell me on LinkedIn.

About the Author — Abubakar Asif