You Don’t Need a Bigger Model. You Need Better Engineering.

Someone pushed back on my last post with a fair question: “aren’t models still part of the problem?” Here’s the honest answer, including the part most vendors skip.

For roughly 80% of real business work, a lighter, cheaper model will not give you the identical answer a frontier model would. There is a difference. I’m not going to pretend otherwise. But it usually doesn’t matter — because both answers are accurate and acceptable. The cheap one meets the bar. You can’t call it wrong, and you can’t call it below expectations. And it costs a fraction.

The thing doing the real work was never the model. It’s the engineering wrapped around it.

What I’ll Cover in This Blog

✔️ What “meets the bar” means — and how to define it before you argue about models
✔️ The four engineering layers that lift any model
✔️ The research: what the cost and quality numbers actually say
✔️ Routing and cascades — the patterns that let you use the cheap model safely
✔️ When a frontier model genuinely earns its price

Now, let’s dive in. 🔥

The Scenario

Merrow Field Services runs an internal assistant that answers policy and procedure questions for 600 field technicians — about 4,000 questions a week. It was built on the largest available model because that felt like the safe choice, and the monthly bill had become a standing agenda item.

We ran an experiment, not an argument. Fifty real questions, known-correct answers, two configurations: the frontier model as-is, and a much smaller model behind identical grounding, context assembly, prompting and state handling. Scored blind by two technicians who didn’t know which was which.

The frontier model’s answers were, on the whole, a bit better written. More nuance, better structure. On correctness — which was the actual requirement — the smaller model scored within a point of it, and both cleared the acceptance bar on 47 of 50. The three failures were the same three questions for both models, and every one traced back to a document that hadn’t been updated. Not a model problem.

The difference was style. The requirement was accuracy. They were paying premium prices for style.

First, Define “The Bar” — Or You’ll Argue About Models Forever

Almost every “which model?” debate is really a debate about an unwritten quality standard. Write it down and the argument resolves itself:

🔹 Correct — the facts match the source of truth. Binary, not vibes.
🔹 Grounded — every claim traces to a retrievable document or record.
🔹 Complete enough — answers the question asked, including the caveat that matters.
🔹 Safe — refuses what it should refuse, escalates what it should escalate.
🔹 Fast enough — within the latency the workflow tolerates.

That’s the bar. Note what isn’t on it: elegance. If your acceptance criteria don’t require prose that reads like a McKinsey deck, don’t pay for it. And once the bar is written, model selection becomes an empirical question you settle with an eval set instead of a preference you defend in meetings.

The Four Layers That Do the Work

🔹 Grounding (RAG). The answer comes from your data, not the model’s parameters. This is the single biggest lever on correctness, and it’s entirely independent of model size. A small model reading the right paragraph beats a huge model guessing from memory, every time.

🔹 Context. Give it exactly what this task needs, this turn — no more. (Why “no more” matters is the whole argument of Prompt Engineering Didn’t Die: stuffing the window degrades recall.)

🔹 Prompt. Precise instructions pull noticeably more out of a smaller model than a vague prompt pulls out of a big one. Smaller models are less tolerant of ambiguity, which makes prompt discipline pay double here.

🔹 State. Carry the thread across steps so nothing gets dropped. Most “the AI forgot” complaints are missing state management, not missing intelligence.

That stack is model-agnostic. It lifts whatever you put behind it — which is exactly why you should pick the model last.

What the Research Actually Says

I don’t ask anyone to take the 80% figure on faith. Four findings, each traceable:

🔹 Small models are 10–30× cheaper to serve. NVIDIA Research’s position paper Small Language Models are the Future of Agentic AI argues SLMs (under ~10B parameters) are sufficiently powerful and far more economical for most agentic invocations — roughly 10–30× cheaper than 70–175B models, with fine-tuning in hours rather than weeks. (arXiv 2506.02153)

🔹 Routing preserves ~95% of quality at ~85% lower cost. RouteLLM (UC Berkeley, Anyscale and Canva; published at ICLR 2025) trained routers on human preference data from Chatbot Arena and reported over 85% cost reduction on MT-Bench while retaining 95% of GPT-4 performance. (paper · LMSYS)

🔹 Cascades have matched top-model quality at up to 98% lower cost. FrugalGPT (Chen, Zaharia and Zou, Stanford) showed a cascade of cheaper models can match GPT-4-level performance with up to 98% cost reduction. (arXiv 2305.05176)

🔹 The price of “good enough” collapsed. Stanford HAI’s AI Index 2025 found inference cost for GPT-3.5-level performance on MMLU fell from $20.00 to $0.07 per million tokens between November 2022 and October 2024 — a more than 280-fold drop in about 18 months. (AI Index 2025)

Read those together and the conclusion is uncomfortable for anyone whose architecture assumes the biggest model: the capability you needed a frontier model for two years ago is now available near the bottom of the price list. Any design that hard-codes one expensive model is a design that will be overpriced within a year, by default.

The Patterns: Routing and Cascades

You don’t have to choose one model forever. Two patterns let you use the cheap one safely.

Routing classifies the request before it hits a model and sends it to the right tier: policy lookups and summaries to the small model, genuinely novel multi-step reasoning to the frontier one. The classifier itself can be small and cheap.

Cascading tries the cheap model first and escalates only when a confidence or validation check fails. You pay frontier prices exclusively on the requests that actually needed it.

So When Is a Frontier Model Worth It?

Reserve it deliberately, for the genuinely hard minority:

🔹 Novel, multi-step, long-horizon reasoning — problems with no template, where the chain of steps has to be invented.
🔹 Open-ended synthesis — pulling a coherent position out of many conflicting sources.
🔹 Difficult multimodal work — documents, images and layout together.
🔹 High-stakes, low-volume decisions — where the cost of one wrong answer dwarfs the entire model bill. If a mistake costs six figures, don’t optimise pennies.
🔹 Building the eval set itself — I’ll happily use the best available model to generate hard test cases for the cheap one.

That’s a real list. It’s just not 80% of the work.

The Cost Math, Concretely

Merrow’s 4,000 questions a week, at a plausible ~3,000 tokens of context per answer, is roughly 12 million tokens a week. At frontier pricing that’s a line item somebody defends quarterly. Routed — say 80% to a small model, 20% escalated — the same workload lands at a fraction of it, with the identical acceptance rate on the eval set.

The lesson isn’t “small models are better.” It’s that the ~20% that needs a frontier model is a minority you can identify, and paying premium prices on the other 80% is a decision you made by default rather than on evidence.

Common Pitfalls

✔️ Choosing the model before writing the bar. Without acceptance criteria, “better” means “sounds nicer,” and that’s an expensive way to shop.
✔️ Testing on happy-path questions. Any model looks fine on easy questions. Build the eval set from the hard ones.
✔️ Switching models without an eval set. You’ll have no idea whether you saved money or quietly lost accuracy.
✔️ Forgetting the router costs something too. Keep the classifier small; measure the whole path, not just the answer call.
✔️ Blaming the model for a data problem. If your three failures are all stale documents, a bigger model fixes nothing — see AI Won’t Fix Your Mess.
✔️ Hard-coding one model. Prices and capabilities move monthly. Make the model a configuration value, not an architectural assumption.

Conclusion

Anyone can buy a bigger model. Few engineer the context so a smaller one doesn’t need to be.

✔️ Define the bar first — correct, grounded, complete, safe, fast. Elegance isn’t a requirement.
✔️ Four model-agnostic layers do the work — grounding, context, prompt, state.
✔️ The research backs it — 10–30× cheaper small models, ~95% quality at ~85% less via routing, up to 98% savings via cascades, and a 280× collapse in the price of “good enough.”
✔️ Route and cascade — cheap by default, escalate on failure.
✔️ Reserve frontier models for the hard minority — novel reasoning, synthesis, multimodal, high-stakes.
✔️ Pick the model last — and make it swappable when the prices move again.

The model is a commodity. The engineering is the moat.

Connect with me on LinkedIn →

Paying frontier prices for work a routed stack would handle? Send me your use case and we’ll define the bar, build the eval set, and find out — with evidence, not opinions.

What’s your acceptance bar — and have you ever actually written it down? Tell me in the comments.

About the Author — Abubakar Asif