Skip to content
TRUELAB.AI
[Lab note]2026-06-022 min read#rag#retrieval#evaluation

RAG beyond the demo

Every RAG demo works. Most RAG systems don't. The difference is retrieval engineering — and it is measurable.

Retrieval-augmented generation has the best demo-to-production gap in the industry. Twenty minutes with a vector database and a tutorial gets you a demo that answers questions about your documents. Six months later, the same system confidently cites a policy that was replaced in 2023.

The model was never the problem. Retrieval was.

Where RAG systems actually fail

Across the systems we have audited, failures cluster in four places, none of them glamorous:

  • Chunking that ignores document structure. Splitting a contract every 512 tokens produces chunks that begin mid-clause. The retriever finds them; the model misreads them.
  • No recency or authority signal. Two documents disagree, and the cosine similarity doesn't care which one is current. Your ranking function has to.
  • Queries that don't look like documents. Users ask "can I expense a hotel?"; the policy says "accommodation reimbursement". Without query rewriting or hybrid search, embeddings alone miss it.
  • Nobody measures retrieval. Teams eyeball final answers and tweak prompts, while the actual defect — the right passage was never retrieved — stays invisible.

Measure retrieval, not vibes

The fix is unglamorous too: build a golden set and score retrieval separately from generation. Fifty real questions with hand-verified source passages beat any synthetic benchmark.

// The metric that finds most RAG bugs: recall@k on a golden set
const recallAtK = (golden: GoldenItem[], k = 8) =>
  golden.filter((item) =>
    retrieve(item.question, k).some((chunk) =>
      item.sourcePassages.includes(chunk.id),
    ),
  ).length / golden.length;

If recall@8 is 0.6, no prompt engineering will save you — four out of ten answers are generated from the wrong context. We have seen teams spend a quarter on prompt tuning for a problem that was a two-day chunking fix.

The production checklist

What separates the systems that survive:

  1. Structure-aware chunking (headings, clauses, tables stay intact)
  2. Hybrid retrieval: BM25 + embeddings, fused
  3. Metadata filters for recency and authority before ranking
  4. A golden set in CI — retrieval regressions fail the build
  5. Citations the user can click, so trust is verifiable

None of this requires a bigger model. All of it requires treating retrieval as the engineering discipline it is. Your knowledge, grounded answers — but only if the grounding actually holds.