RAG Works in Demo, Dies in Prod: 6 Failure Modes
Most RAG pilots answer correctly on ten test docs and fail on the ten thousand real ones. The fix is loop engineering: retrieval evals, chunk regression sets, and a weekly failure review that feeds back into parsing and ranking.
The problem in prod
- Chunking destroys tables, scanned PDFs and multi-column layouts common in bank statements and SOP docs.
- Top-k retrieval returns plausible but stale versions: policy v3 instead of v9.
- No ground truth, so nobody can tell whether a wrong answer is a retrieval miss, a ranking miss, or a generation hallucination.
- Latency budget blows up once rerankers and cross-encoders stack on every query.
Loop engineering
Collect 200 real questions with known-good answers from support tickets and auditors. Run them nightly. Track grounded-answer rate, not vibes.
Every wrong answer gets labeled: parse, retrieve, rank, or generate. The label decides the fix. Parsing issues go to the ingestion team, ranking issues to the reranker config.
Freeze 50 hard documents (tables, scans, mixed Malay and English). Any chunking change must keep table-cell recall above 95% on this set before shipping.
Version every source doc. Retrieval filters to current versions by default and cites the version in the answer so auditors can verify.
Production hardening checklist
- Hybrid retrieval (dense plus BM25) so exact account numbers and policy codes still match.
- Rerank only top-30, cache embeddings, stream first token before rerank completes.
- PII redaction at ingestion plus role-based retrieval filters for PDPA scope.
- Canary new indexes on 5% of traffic with automatic rollback on grounded-rate drop.
Deploying this in MY
Malaysian banks hold Malay plus English policy docs, Jawi-adjacent scanned forms, and stamped PDFs. English benchmarks never cover this mix, so build the golden set from real Maybank and CIMB style support queries and test both languages on every run.
Malaysia FDE guide →