RAG in Production: What Breaks and How to Prevent It

A RAG demo working on ten curated documents proves almost nothing about how it'll behave on ten thousand real ones with real users. Here's what actually breaks, in the order teams usually hit it.

Retrieval quality quietly degrades as the corpus grows

A demo indexes a handful of clean documents and retrieval looks perfect. Production indexes years of inconsistent docs, duplicate content, and outdated pages — and the same top-k retrieval starts returning near-duplicates or stale versions of the same answer. Fix: dedupe at ingestion, tag documents with freshness metadata, and re-rank rather than trusting raw vector similarity alone.

The model answers confidently from nothing retrieved

When retrieval returns weak or irrelevant chunks, most models don't say "I don't know" — they generate a plausible-sounding answer anyway. This is the single most common trust-destroying failure in production RAG. Fix: set a similarity-score floor below which the system refuses to answer, and test that refusal path explicitly — it's the path nobody exercises until a user hits it.

Access control gets bolted on after the fact

Teams build the retrieval pipeline first and add permissions later, which means the vector index has to be re-architected to filter by user, team, or tenant. Retrofitting this is expensive and easy to get wrong in a way that leaks data. Fix: design retrieval filtering by permission scope from the first index you build, not after the first customer asks about it.

Chunking strategy that worked for one document type breaks for another

Fixed-size chunking works fine for prose and falls apart on tables, code, and structured data — you get half a table row as a "relevant chunk" with no context. Fix: chunk by document type, not a single global strategy, and keep enough surrounding context that a retrieved chunk is still meaningful on its own.

Cost and latency scale worse than expected

Re-ranking, multi-query retrieval, and larger context windows all improve answer quality — and all add latency and per-query cost that a demo never has to account for at real volume. Fix: measure cost and latency per query from day one, and treat "good enough" retrieval that's fast and cheap as a real tradeoff, not a compromise.

There's no way to tell if a change made retrieval better or worse

The most common gap: teams ship retrieval changes based on a few manual spot-checks, with no regression suite. Then a prompt tweak or reranker swap silently degrades answers for a query pattern nobody tested. Fix: build a small eval set of real queries with known-good answers before you need one, not after a regression ships.

The pattern behind all of it

Every one of these failures is invisible in a demo and obvious within a week of production traffic, because a demo has curated data, no adversarial users, and nobody measuring quality over time. The fix in every case is the same instinct: build the failure path (refusal, permission filtering, eval regression) before you need it, not after a user finds it for you.

If you're past the demo and hitting these in production — or trying to avoid hitting them at all — tell us what you're building. We run RAG pipeline development as a core practice, not a one-off project.