Most RAG systems do not break inside the model. They break at the seams between ingestion, retrieval, answer orchestration, and safety checks.
The chapter breaks the live loop into parts and shows how knowledge ingestion, ranking, guardrails, evaluation, and cost control together determine whether an answer is truly useful.
For interviews and architecture discussions, it is useful because it frames RAG not as a quick prototype, but as a system with SLOs, failure modes, and operational trade-offs.
Practical value of this chapter
Design in practice
Translate guidance on production RAG architecture, retrieval quality, and knowledge control into architecture decisions for data flow, model serving, and quality control points.
Decision quality
Evaluate system quality through both model and platform metrics: precision/recall, latency, drift, cost, and operational risk.
Interview articulation
Frame answers as data -> model -> serving -> monitoring, showing where constraints appear and how you manage them.
Trade-off framing
Make trade-offs explicit for production RAG architecture, retrieval quality, and knowledge control: experiment speed, quality, explainability, resource budget, and maintenance complexity.
Primary source
Foundational RAG paper (2020)
The paper that formalized RAG as generation grounded in retrieved context.
GenAI/RAG System Architecture is not a single service around an LLM, but a set of connected planes: data ingestion, retrieval, generation orchestration, guardrails, and operational quality evaluation. The system becomes fit for live use only when these planes are designed as one contract for latency, quality, and cost.
What follows is a practical blueprint you can use to start the architecture of an enterprise AI assistant, internal knowledge assistant, or customer support bot.
Reference GenAI/RAG architecture
The diagram shows the RAG path by layers: from knowledge ingestion and indexing to answer generation, guardrails, and fallback behavior.
What to keep under control
It helps to view the RAG path not only as a chain of services, but as a balance of retrieval quality, response latency, cost, and rollout safety.
Retrieval quality
Live constraints
Safe rollout
Request path: from question to grounded answer
The diagram below shows the synchronous RAG path: from early request checks through retrieval and context assembly to a cited answer with final validation and response shaping.
How a question flows through the RAG path
The synchronous path from user request to cited answer
Active step
Step budget: ~30-80 ms1. Question intake and pre-checks
The system normalizes the request, identifies the scenario, filters out out-of-domain questions, and runs early policy checks before retrieval.
Grounded online answer path
- This path is tightly constrained by latency.
- Retrieval quality influences the outcome as much as the model itself.
- Access checks and guardrails work both before and after generation.
Research
Chroma: Evaluating Chunking Strategies (2024)
A comparison of chunking strategies on shared metrics: the gap between naive and deliberate splitting reaches 9% recall on the same corpus.
Chunking strategies: segmentation sets the quality ceiling
Chunking is the first architectural decision of the knowledge platform: whatever is segmented badly cannot be rescued by embeddings or prompts. Each strategy below follows one frame: how it works, what trade-off it brings, and when to choose it.
Fixed-size windows
How it works: Text is cut into windows of 200-800 tokens with 10-20% overlap, ignoring document structure. It is a one-function implementation, and indexing cost is predictable and easy to estimate up front.
Trade-off: Window boundaries land in the middle of sentences, tables, or code samples, and a fact split in half is found by neither half. Overlap partially insures the boundaries but inflates the index and produces duplicates in the top-k.
When to choose: The baseline for a first launch and for homogeneous texts without strong structure. Start here to get a measurable reference point for your metrics, not as the final answer.
Structure-aware splitting
How it works: Splitting follows natural boundaries: headings, paragraphs, lists, and tables. Recursive splitting walks down the separator hierarchy - sections first, then paragraphs, then sentences - until the chunk fits the size limit.
Trade-off: Chunks align with semantic boundaries, and the heading path lands in metadata for free. The cost: chunk sizes swing from a couple of lines to a whole section, and every format (Markdown, HTML, PDF, wiki) needs its own parser.
When to choose: Documentation, knowledge bases, policies, and contracts where the author has already marked up the structure. On corpora with strong inherent boundaries the gain over naive splitting is largest.
Semantic chunking
How it works: Embeddings of adjacent sentences are compared, and a chunk boundary is placed where similarity drops sharply - that is, where the topic changes. An LLM-driven boundary variant is even more precise, but noticeably more expensive.
Trade-off: In Chroma's 2024 report, semantic and LLM strategies reached up to ~92% recall versus ~88% for recursive splitting - a few percentage points of gain for substantially more expensive indexing. Boundaries are also tied to the embedding model: swapping it changes the segmentation.
When to choose: Long continuous texts without explicit structure: call transcripts, research papers, longreads. Choose it when first-stage recall is limited by unlucky boundaries rather than by embedding quality.
Chunk size: precision versus context
Small chunks (~128-256 tokens) yield pinpoint hits and high precision but tear a fact away from its surroundings: a condition from the neighboring paragraph gets lost. Large chunks (~512-1024) capture the full answer more often but drag noise into the context and burn the model window faster. Tune the size against metrics on your own golden set, not someone else's defaults.
Overlap
A 10-20% overlap insures facts that land on a window boundary: a phrase cut by one chunk lives intact in its neighbor. But every percent of overlap means index growth and near-duplicate neighbors in the top-k, so deduplication of nearly identical fragments is needed before context assembly.
Chunk metadata
Every chunk should carry the source and document version, the heading path, the update date, the ACL or tenant, and the content type. Metadata is what turns an index into a manageable system: it powers pre-search filters, citations in the answer, and targeted invalidation when a changed document is reindexed.
Hybrid search: lexical matching, vectors, and list fusion
No single retrieval mode covers all queries: BM25 over an inverted index finds exact terms literally, dense retrieval captures meaning, and hybrid search merges both lists so that the strengths do not average each other out.
Lexical search: BM25
How it works: The classic of full-text search on top of an inverted index: a term's weight grows with in-document frequency with saturation and is normalized by document length (typical parameters k1 = 1.2, b = 0.75). BM25 is the default in Lucene, Elasticsearch, and OpenSearch.
Trade-off: Exact matches work perfectly: error codes, part numbers, API names, and abbreviations are found literally. But synonyms and paraphrased questions are invisible to BM25: "how to remove my account" and "account deactivation" are different queries.
When to choose: Queries with exact identifiers and rare terms. In enterprise search this is the everyday case: ticket and contract numbers, error codes, names of internal services.
Dense retrieval over embeddings
How it works: A bi-encoder turns every chunk into a vector ahead of time; the query is encoded by the same model, and an ANN index (usually HNSW) finds the nearest vectors in milliseconds even across millions of chunks.
Trade-off: It captures meaning, synonyms, and paraphrases, and works across languages. But rare exact tokens get blurred: a query with the code "ERR-1042" returns chunks about similar errors rather than that one. On internal jargon the embedder never saw in training, quality drops.
When to choose: Natural-language questions, synonymy, multilingual corpora. The primary mode for knowledge-base assistants - but on its own it loses to the hybrid on exact terms.
List fusion: Reciprocal Rank Fusion
How it works: Each source returns its own ranked list, and a document's final score is the sum of 1/(k + rank) across the lists (k = 60 in the original Cormack et al., 2009 paper). Engine scores are ignored - only positions count, so the incomparable scales of BM25 and cosine similarity need no normalization.
Trade-off: RRF rewards consensus: a document ranked tenth in both lists beats the leader of only one of them. The cost is losing the information about match strength; with sources of very unequal quality, the plain sum will need weights.
When to choose: The default fusion method for hybrid search: a single formula with no training and no normalization, built into Elasticsearch, OpenSearch, Qdrant, Weaviate, and most vector engines.
Metadata filters: before or after the search
Filtering by ACL, tenant, and freshness is not an implementation detail but a security boundary: applying the filter before or after the index traversal is a decision that determines both result quality and whether forbidden documents ever participate in ranking at all.
Pre-filter: a mask before index traversal
How it works: Metadata filters (ACL, tenant, freshness, document type) build a mask of admissible chunks before the vector search, and the ANN traversal only visits permitted nodes.
Trade-off: It guarantees that only admissible results reach the top-k. But a highly selective filter on classic HNSW breaks graph connectivity and degrades traversal quality - which is why engines build filterable indexes that embed the filter directly into graph navigation (Qdrant does this, for example).
When to choose: Access rights and tenant boundaries are always pre-filter: a document the user must not see should not even participate in the search. This is a security requirement, not a quality optimization.
Post-filter: pruning after the search
How it works: A regular vector search runs first; then results that fail the metadata filter are dropped from the top-k.
Trade-off: It leaves the index untouched and is trivial to implement, but with a selective filter the result set collapses: you asked for the 10 nearest, 1-2 survive the filter, and you have to re-query with a larger k. For ACL it is also a risk: a forbidden document has already participated in ranking.
When to choose: Weakly selective filters with no security requirements: content type, language, a coarse date cutoff - cases where only a small share of candidates gets filtered out.
Reranking: expensive precision on top of cheap coverage
Reranking separates two incompatible goals into different stages: first-stage retrieval is responsible for coverage and is cheap, while the cross-encoder is responsible for precision and is expensive - so it only runs over a short candidate list.
1. Wide retrieval: top-100 candidates
~30-80 msHybrid search assembles a wide candidate list. Recall is what matters at this stage - the right document must make it into the pool, and ordering precision is secondary: widening k here is almost free.
2. Cross-encoder: reranking the pairs
~50-300 msA cross-encoder reads the "query + chunk" pair in a single pass and sees token-level interaction, so it ranks more accurately than a bi-encoder - but nothing can be precomputed: k full model passes per query. Lightweight MiniLM-class models handle ~100 pairs in tens of milliseconds on a GPU; API rerankers cost roughly 100-400 ms including the network.
3. Selecting the top-5 for the context
~0 msThe LLM context receives a short precise list instead of a long noisy one. Fewer tokens means cheaper generation, a weaker lost-in-the-middle effect, and a lower chance the model leans on an irrelevant chunk.
Cost and payoff
The "retrieve 100 → rerank → top 5" budget adds roughly 100-300 ms to the answer, but in typical measurements yields a double-digit lift in the share of queries where the right document lands in the final context. Economically, a reranker almost always beats the alternative - feeding the LLM 30-50 chunks "just in case": a long context costs more in tokens and is used worse by the model.
What a reranker does not fix
A cross-encoder only reorders what was already found. If the right document is not in the first-stage top-100, reranking has nothing to rescue - fix chunking, the embedder, or source fusion first. That is why the metrics come as a pair: recall@100 for first-stage retrieval and recall@5 after the reranker.
Evaluating retrieval quality: metrics, benchmarks, and drift
Retrieval quality is the only part of RAG that can be measured deterministically and cheaply, without an LLM judge. That is why the evaluation loop starts from retrieval metrics: recall@k, MRR, and nDCG on a golden set catch regressions before rollout, and online signals confirm them on live traffic.
| Metric | What it measures | When to use it |
|---|---|---|
| recall@k | Whether a relevant document made it into the top-k at all, regardless of order. | The key ceiling metric of the system: whatever is missing from the result set, the LLM will never see under any prompt. Track it at the k of the final context and at the k of first-stage retrieval. |
| MRR | How high the first relevant result ranks: the mean of 1/rank across queries. | Scenarios with one correct answer: factual questions, finding a specific policy. The metric is sensitive precisely to the top of the list. |
| nDCG@k | The quality of the entire ranking, with graded relevance and a logarithmic position discount. | Result sets where many documents are relevant to varying degrees; the standard way to compare rankers against each other. |
Golden set: a labeled benchmark
The foundation of offline evaluation: real queries from the logs, labeled with which documents and facts are relevant. Synthetic data (an LLM generating questions from chunks) cheaply expands coverage but needs a human-reviewed sample - otherwise you are measuring the model's ability to answer its own questions.
Trap: Label relevance at the level of documents and facts, not chunk IDs: a chunking strategy change or a reindex renumbers the chunks and silently voids the labels.
Offline loop: a regression gate
recall@k, MRR, and nDCG on the golden set run on every pipeline change: a new chunking strategy, a different embedding model, fusion parameters, a reranker version. This turns retrieval tuning from "feels better" into a controlled experiment.
Trap: Offline metrics are only comparable on a frozen corpus: if the index changed between runs, the recall delta is a mix of the change's effect and data drift.
Online signals
There are no direct labels in production, so you watch proxies: the share of answers with citations, the share of honest "not found" refusals, source clicks, repeated rephrasings of the same question, explicit feedback. Segment by query type: averages hide the degradation of individual scenarios.
Trap: Online signals lag and are noisy: a drop in retrieval quality takes weeks to reach product metrics. Without an offline loop, you will be the last to learn about the degradation.
Corpus drift
The corpus is alive: documents get added, renamed, and outdated; new products and terms appear. A golden set collected a year ago keeps showing green metrics - on year-old questions, not on today's traffic.
Trap: Keep a fresh slice in the golden set (documents and queries from recent weeks), resample production queries regularly, and monitor the share of queries with low retrieval scores: it is the first to grow when the corpus drifts away from the index.
SLO and capacity baseline
Latency
P95 < 2.0s
Break budget into retrieval, model inference, and post-processing components.
Quality
Grounded-answer rate > 90%
Measure whether the answer is actually grounded in retrieved context, not only whether it sounds fluent.
Economics
Cost/task within target range
Control costs via model routing, caching, and context size limits.
Research
Lost in the Middle (Liu et al., 2023)
The paper that showed the U-shaped curve: models use facts from the middle of a long context noticeably worse than from its beginning and end.
RAG failure modes: symptom, cause, architectural response
Most RAG problems look the same - "the model answers poorly" - but are fixed in different parts of the pipeline. It pays to keep a failure catalog in the "symptom → cause → architectural response" format: it moves the conversation from "improve the prompt" to the specific layer where a contract is broken. A hallucination on empty retrieval, for instance, is not a model problem but a missing refusal branch in the orchestrator.
Confident hallucinations on empty retrieval
Symptom: The system answers smoothly and confidently, but without citations or with irrelevant sources; on questions outside the corpus it produces invented details.
Cause: The pipeline cannot tell "found something relevant" from "found nothing": even with weak scores the top-k still goes to the LLM, and the model completes the answer from parametric memory.
Architectural response: A retrieval score threshold with an explicit honest-refusal branch, mandatory citations in the answer contract, and a groundedness check on the output - as a separate pipeline step, not a wish in the prompt.
Lost in the middle
Symptom: The needed fact was definitely in the assembled context, yet the answer ignores it; quality fluctuates depending on the chunk's position in the prompt.
Cause: The U-shaped attention curve: models consistently use the beginning and end of a long context better than the middle (Liu et al., 2023). The more "just in case" chunks, the deeper that middle gets.
Architectural response: Fewer but more precise: rerank down to top-5 instead of top-30, put the most relevant chunks at the start and the end of the context, and treat context size as an explicit budget rather than "whatever fits".
Stale index: answers from the past
Symptom: The assistant cites last year's version of a policy or a deleted page; users complain that "the docs no longer say that".
Cause: Batch indexing once a day or week with no invalidation on source change. The index is a cache of knowledge, and without an invalidation strategy it goes stale as quietly as any cache.
Architectural response: Event-driven reindexing of changed documents instead of full rebuilds, source version and date in chunk metadata with a "current version only" filter, and the document date next to the citation in the answer.
Prompt overflow
Symptom: Quality drops sharply in long conversations, answers lose the system instructions, citations get cut off mid-sentence.
Cause: The context is assembled without a budget: conversation history, retrieval chunks, and instructions pile up until they hit the model window, and on overflow something is silently truncated.
Architectural response: An explicit token budget per block (system, history, retrieval, answer), priority-based eviction - summarize old history first, then drop the worst-scoring chunks - and an alert on the share of requests near the window ceiling.
Vocabulary gap between queries and the corpus
Symptom: Users ask in their own words, and retrieval consistently misses whole classes of queries even though the answers exist in the corpus.
Cause: User jargon and the corpus's official terminology do not overlap lexically, and the embedding model was not trained on domain synonyms.
Architectural response: Query rewriting and expansion before the search, hybrid retrieval instead of dense-only, and a regular review of retrieval misses as the source of a synonym dictionary.
Recommendations
- Design RAG as two connected systems: a knowledge platform (ingestion and index) and a live answer path (retrieval and generation).
- Treat retrieval observability as first-class: hit rate, miss reasons, latency, and segmented quality.
- Stabilize contracts: chunk structure, query filters, context blocks, and the client-facing response format.
- Before rolling out a new model, run both shadow traffic and historical replay on a benchmark task set.
Common pitfalls
- Evaluating only BLEU/ROUGE without product metrics and grounding checks.
- Indexing raw data without cleanup, deduplication, and source version control.
- Trying to fix quality only through prompt wording while ignoring retrieval quality and data freshness.
- Applying access control after generation instead of before context retrieval.
Launch mini-checklist
- A source catalog exists and each knowledge domain has data owners.
- Each use case has defined quality metrics, a latency SLO, and cost guardrails.
- Rule checks are enabled on both input and output, with auditable decision logs.
- Canary and shadow rollout paths are configured together with replay-based regression tests.
- Fallback paths exist for retrieval, reranker, and LLM provider failures.
References
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
- OpenAI cookbook: File Search and RAG patterns
- LlamaIndex: Production RAG guide
- LangChain docs: Retrieval and evaluation
- NVIDIA: RAG 101 and practical trade-offs
- Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)
- Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods (Cormack et al., SIGIR 2009)
- Chroma Research: Evaluating Chunking Strategies for Retrieval (2024)
- Qdrant: A Complete Guide to Filtering in Vector Search
- Weaviate: Evaluation Metrics for Search and Recommendation Systems
Related chapters
- AI Engineering (short summary) - An engineering frame for AI applications: evaluation, release management, and day-two operations.
- Hands-On Large Language Models (short summary) - A foundation for embeddings, retrieval, and the internals of LLM-based systems.
- Prompt Engineering for LLMs (short summary) - Prompt contracts and context engineering practices for RAG.
- Enterprise AI Copilot - An applied GenAI case with ACL-aware retrieval, citations, guardrails, and tenant isolation.
- Evaluation and Observability for AI Systems - How to measure groundedness, retrieval quality, and system behavior after rollout.
- Generative AI System Design Interview (short summary) - Adds the interview frame for RAG: requirements, data, answer quality, safety, and post-launch monitoring.
- An Illustrated Guide to AI Agents (short summary) - Next step after RAG: tool use, planning, and orchestration.
- Data Governance & Compliance - PII control, lineage, and regulatory requirements for knowledge bases.