When a 3B model beats the frontier on private-domain RAG (and when it doesn't)

Created on August 02, 2026

2026   ·   rag   retrieval   slm   evaluation   agents   ·   engineering

TL;DR

On a private domain, a frontier model — a top-end general-purpose model — has no memorized advantage. It has to read the retrieved documents, same as a small model. That’s the RAG setup: retrieve relevant documents, then ask a model to answer from them. So I asked a narrow question: can I make a local 3B model competitive with the frontier on private-domain RAG? The honest answer:

  • With clean retrieval — context without irrelevant distractors — the adapted 3B beats the honestly-measured frontier, 82.3% vs 77.0% — at ~1/5 the serving cost, ~94 ms locally, fully on-prem.
  • With noisy retrieval — context that includes those distractors — it’s a near-tie (75.2% vs 76.4%), frontier marginally ahead — the small model wins on cost/latency/privacy, not accuracy. I’m not going to pretend otherwise.
  • The lever wasn’t clever routing (that failed) and it wasn’t model size (a tuned 3B matched a 7B). It was domain adaptation — fine-tuning on examples from the target domain — plus the discipline of measuring the frontier honestly. Its headline RAG number turned out to be inflated by public-benchmark leakage: answers from memorized public facts, not just the retrieved context.

The headline clean/noisy experiment below uses PopQA-440 as a public stand-in for a private corpus. The routing and multi-hop sections use their own named benchmarks.

The question, and why the obvious framings mislead

Enterprise RAG over a private corpus wants high accuracy at low cost and latency, on-prem. The instinct is “use the frontier model, it’s smarter.” But on data it has never seen, smarter mostly means has more world knowledge — and world knowledge is exactly what a private corpus doesn’t reward. Both models live or die by retrieval.

So the real question is narrow and testable: for single-hop extraction — answering from one retrieved source — can adaptation make a local small language model (SLM) competitive with a frontier model restricted to that same context? I answered it by following the evidence, and two of the most useful findings were things that didn’t work.

Detour 1: adaptive routing doesn’t deliver (a useful negative)

The trendy answer is routing: send easy queries to a cheap model, hard ones to the frontier, spend compute by difficulty. The oracle looks amazing. Here, the oracle is a hindsight-only router: it already knows which models answered each query correctly, then picks the cheapest correct one. That foreknowledge makes it a useful but cheating upper bound, not a deployable system. On RouterBench, this oracle chooses across 11 models and hits 95.4% at ~¼ the cost of the best single model. The oracle advantage is that ideal cost-versus-quality gain over sticking with one fixed model.

Then I actually built the router and audited the decision in two ways. Calibration asks whether its confidence numbers are on the right scale. Discrimination, or predictability, asks whether cheap features can tell which tier will be right. Here, capture means the share of the oracle advantage a real router recovers without hindsight. A realistic calibrated router captures only 5.6% of that oracle advantage with a weak cheap tier, and 35.8% even with a strong one. Most of the apparent upside vanishes once the router has to choose without seeing the answer.

The bottleneck isn’t calibration. Putting confidence on the right scale cannot create a missing signal. The hard part is discrimination: can the cheap features tell which tier will be right? And the savings you can reach are unsafe. Cutting serving cost 5× raises the served confidently-wrong rate from 0.1% to 33.5%.

I even tried the appealing fix — generate with the cheap model, then use a groundedness verifier, which checks whether the retrieved context supports the answer, to escalate the unsupported ones. It failed: its AUROC, a threshold-independent measure of how well the score separates correct answers from incorrect ones, was 44.8. Groundedness is not correctness: a cheap answer can be perfectly faithful to a stale or wrong retrieved page. Reporting that refutation of my own idea is the point — it’s what told me routing was a dead end and the lever had to be elsewhere.

The unlock: measure the frontier honestly

Here’s the finding that reframed everything. Frontier-RAG is usually reported with a prompt that lets the model use its own knowledge in addition to the retrieved context. On PopQA that scores 90.5%. But restrict it to reading only the retrieved page — its true behavior on data it never memorized — and it drops to 77.0%.

13.5 of those 90.5 points were public-benchmark leakage, not reading. On a genuinely private corpus, the 90.5 number simply doesn’t exist.

So 77.0% is the honest baseline — and, tellingly, it’s the same score an off-the-shelf 3B gets reading the same page (77.1%). Once you remove world knowledge, the frontier has no comprehension edge on clean single-hop extraction.

If you take one thing from this post: measure your baseline on the data it will actually face, not the public knowledge it happens to have memorized.

The method: adapt the small model

Nothing exotic — the contribution is the honest, measured integration, not a new algorithm:

  1. Domain retrieval over the corpus.
  2. Retrieval-Augmented Fine-Tuning (RAFT) — train the SLM on (retrieved context, question) → answer, loss on the answer tokens only, so it learns to extract from your documents, not recite them.
  3. Distractor augmentation — mix irrelevant retrieved docs into the training examples. That way, robustness to noisy retrieval is learned, not assumed.
  4. A groundedness abstention guard at serving time — flag answers the context doesn’t support so the system abstains instead of asserting (used for abstention, not as the correctness router that failed above).

Results: clean beats, noisy ties

configuration clean noisy latency
3B off-the-shelf + retrieval 77.1 67.3 65 ms
3B + RAFT 81.8 73.9 80 ms
3B + distractor-RAFT 82.3 75.2 94 ms
7B off-the-shelf + retrieval 82.3 71.8 139 ms
7B + RAFT 81.8 75.5 143 ms
frontier-RAG strict (honest) 77.0 76.4 API
frontier-RAG hybrid (public leak) 90.5 — API

Three things I didn’t expect to be so clean:

  • Clean retrieval: the adapted 3B beats the honest frontier, 82.3 vs 77.0.
  • Domain adaptation substitutes for scale here — the tuned 3B (82.3) matches an off-the-shelf 7B (82.3), at 94 ms vs 139 ms. Adapting the small model beat using a bigger one.
  • Robustness to noise is a training property, not a size one. Under noisy retrieval, an off-the-shelf 3B falls to 67 and an off-the-shelf 7B reaches 72. Distractor training lifts that same small model to 75, past the larger off-the-shelf model. Training beat scale in that comparison. That’s what buys the noisy near-tie.

And it’s not a lucky seed: retrained on three seeds, the RAFT 3B is 82.5 ± 0.67, every seed above 77.0.

I also worried it had just memorized PopQA’s ~10 question templates, so I ran a paraphrase check. Most questions were rewritten while keeping the entity, context, and answer fixed. Across original, natural, and telegraphic wording, accuracy stayed flat (82.3 / 81.4 / 83.0). It learned to read, not to pattern-match the template.

Where’s the ceiling, really? (the part I’m most glad I checked)

It would’ve been easy to stop at “3B beats frontier, ship it.” Instead I asked where the system actually breaks. The answer split cleanly in two — and neither is the size of the reader model, meaning the model that reads the retrieved documents and produces the answer, not the person reading this post.

Reasoning. Holding the context fixed and only varying the reader model, single-hop is near its measured ceiling on this proxy (a tuned 3B = a 7B). But multi-hop composition — combining multiple pieces of evidence — is a cliff: the 3B gets 12–17% on MuSiQue, and scaling to 7B only reaches 20–30%. Scale helps, but it does not close the gap. These results support escalating multi-hop cases; they do not establish a decomposition recipe.

Retrieval. My earlier ablation removed retrieval errors: I handed the reader model its gold page, the source page intended for the question. With a real retriever, answer-recall means the share of queries whose retrieved text contains the answer. Full-system accuracy cannot exceed answer-recall, which looked stuck at ~90%. If retrieval never supplies the answer, even a perfect reader model has nothing to extract.

So I dug in. Doc-recall, by contrast, means the share of queries for which the intended source document is retrieved. It was ~100%, meaning the right page was retrieved but its text didn’t contain the answer. The cause was mundane and fixable: my corpus builder used intro-only extracts, dropping Wikipedia infoboxes and the rest of each article. I ingested infoboxes plus full text, and answer-recall jumped 90.9 → 95.4. The “ceiling” was a corpus-construction artifact, not a retrieval limit — only ~1.8% of queries were genuine retrieval failures. Debug the metric before you accept it.

What I’d tell a practitioner

  • On this proxy’s single-hop task, a well-adapted 3B is near the measured reader ceiling. Spend first on retrieval quality and fine-tuning on your own Q&A, before buying a bigger reader model.
  • Measure the frontier honestly (context-only), or you’ll compare against a number that won’t survive contact with your private data.
  • Escalate known reasoning-hard cases, don’t upgrade everything. Multi-hop queries go up a tier; do not use the cheap confidence signals that failed above as the trigger.
  • Report what didn’t work. The routing refutation and the corpus-artifact debugging were more useful than any single accuracy number.

Honest limitations

PopQA is a public proxy for a private domain (its 90.5 hybrid number wouldn’t exist on a real internal corpus). Under noisy retrieval the frontier is marginally ahead on accuracy; the SLM’s win there is cost/latency/privacy. Costs are nominal relative units; latency compares local-GPU-batched vs an API round-trip. Only the headline 3B-RAFT cell is multi-seed. Multi-hop reasoning remains the SLM’s real weakness.

Code + paper

Everything here is reproducible — every number in this post regenerates from the committed scripts and results.

  • Repo: github.com/cuibuaa/private-domain-rag — the scripts, the training/eval harness, and the committed results_*.json behind every figure.
  • Paper — the rigorous write-up, with each claim traced in the verification report.
  • Sibling study — the distilled RAG-groundedness guardrail (ModernBERT student, 58.89 min-AUROC vs LettuceDetect’s 57.3) that provides the abstention guard.