RAG in Production: Architecture, Evaluation, and Failure Modes

Retrieval-augmented generation (RAG) has become the default pattern for grounding large language models in private, fresh, or domain-specific knowledge. The promise is simple: instead of relying only on parametric memory, retrieve relevant passages at query time and place them in the prompt. In practice, production RAG is a distributed data system with an LLM at the end. It involves ingestion pipelines, chunking, embeddings, vector search, reranking, prompt assembly, evaluation, access control, and observability. This article walks through the architecture and the hard parts that determine whether a RAG application feels reliable or brittle.

What RAG Is and What It Is Not

RAG is not a model architecture. It is an application pattern. A retriever fetches candidate evidence from an external index, and a generator conditions its answer on that evidence. This separates knowledge from reasoning. You can update the index without retraining the model, enforce permissions at retrieval time, and attach citations to answers.

RAG is also not a replacement for fine-tuning. Fine-tuning changes style, format, and latent skills. RAG supplies facts and context. Many production systems use both: a fine-tuned model for domain tone and tool use, plus RAG for current facts and proprietary documents.

Reference Architecture for Production RAG

A production RAG system usually has two pipelines: offline ingestion and online query.

Offline ingestion pipeline

• Source connectors: pull documents from wikis, object stores, databases, ticketing systems, and APIs. Track source IDs, versions, timestamps, and ACLs.
• Parsing and normalization: convert PDFs, HTML, slides, and transcripts into clean text. Preserve headings, tables, and lists because structure carries meaning.
• Chunking: split documents into retrievable units. Chunk size, overlap, and boundaries strongly affect recall.
• Metadata enrichment: attach title, section, author, date, product, tenant, permission labels, and language.
• Embedding: compute dense vectors with an embedding model. Consider dimensionality, latency, cost, and multilingual support.
• Indexing: write vectors to a vector database and optionally write text to a lexical index for hybrid search.
• Versioning: store embedding model version, chunker version, and index schema version. Re-embedding is expensive; you need reproducibility.

Online query pipeline

• Query understanding: normalize, rewrite, expand, or decompose the user query. Apply filters based on user identity and tenant.
• Retrieval: run dense, sparse, or hybrid search. Retrieve a candidate set, not just top-k.
• Reranking: use a cross-encoder or LLM reranker to reorder candidates by relevance to the query.
• Context assembly: select passages that fit the context window. Deduplicate, order, and label chunks with source IDs.
• Generation: prompt the LLM to answer only from context, cite sources, and say when evidence is missing.
• Post-processing: validate citations, redact sensitive data, and format the answer for the UI.
• Observability: log query, retrieved IDs, scores, prompt, response, latency, and user feedback.

Chunking: The Highest-Leverage Decision

Chunking determines what can be retrieved. If a chunk is too small, it lacks context. If it is too large, it dilutes relevance and wastes tokens. Fixed-size chunking is easy but often cuts sentences and separates tables from captions. Semantic chunking uses embeddings or model-based segmentation to keep related ideas together. Hierarchical chunking stores parent documents and child chunks so retrieval can find a precise child and generation can use a broader parent.

Good chunking strategies include:

• Structure-aware splitting: split by headings, sections, and list boundaries.
• Overlap: include 10 to 20 percent overlap to preserve continuity across boundaries.
• Metadata prefixes: prepend title and section to each chunk before embedding. This improves retrieval for ambiguous text.
• Small-to-big retrieval: embed small chunks for search, but return the surrounding window or parent section to the LLM.
• Table handling: keep tables with their headers. Convert complex tables to Markdown or text summaries.

Embeddings and Vector Search

Embedding models map text to vectors. The best model depends on domain, language, latency, and cost. General-purpose models work well for many English tasks, but domain-specific models can improve recall in legal, medical, or code search. Always evaluate on your own retrieval set rather than relying on public benchmarks.

Vector databases provide approximate nearest neighbor search. Key choices include:

• Index type: HNSW for low latency, IVF for large scale, or DiskANN for memory-constrained workloads.
• Distance metric: cosine similarity, dot product, or Euclidean distance. Match the metric to how the embedding model was trained.
• Filtering: metadata filters for tenant, date, document type, and permissions. Pre-filtering is faster but can reduce recall if not supported well.
• Hybrid search: combine dense vectors with BM25 or SPLADE. Dense search captures semantics; lexical search captures exact terms, IDs, and names.

Query Understanding and Routing

Users rarely write perfect search queries. Query understanding improves retrieval before the vector search runs. Common techniques include:

• Rewriting: turn a conversational question into a standalone query using chat history.
• Expansion: add synonyms, acronyms, and domain terms.
• Decomposition: split multi-part questions into sub-queries, retrieve for each, then merge.
• Routing: decide whether to use RAG, a database query, a calculator, or a direct answer.
• Filter extraction: infer date ranges, product names, and categories from natural language.

Reranking and Context Selection

Initial retrieval optimizes for speed and recall. Reranking optimizes for precision. A cross-encoder reads the query and each candidate together and outputs a relevance score. This is slower than vector search but much more accurate. In production, retrieve 50 to 200 candidates, rerank them, then keep the top 5 to 15 chunks.

Context selection must also manage the context window. More context is not always better. Irrelevant passages can distract the model, increase cost, and push important evidence into the middle where models often lose attention. Use these tactics:

• Deduplicate near-identical chunks.
• Keep the highest-scoring evidence near the beginning and end of the context.
• Label each chunk with source, date, and section.
• Limit total context tokens based on the model and task.

Generation: Grounding, Citations, and Refusal

The generation prompt is a contract. It should tell the model what to do when evidence is missing, how to cite sources, and what format to use. A robust prompt includes:

• Role and task: answer the user question using only the provided context.
• Evidence rules: if the context does not contain the answer, say you do not know.
• Citation format: cite source IDs inline, for example [doc:123#section-2].
• Style constraints: concise, no speculation, no external knowledge unless allowed.
• Output schema: JSON if the downstream system needs structured data.

Citations should be validated after generation. Map cited IDs back to retrieved chunks. If a citation does not exist, either regenerate or flag the answer. For high-stakes domains, use extractive answers or require human review.

Evaluation: You Cannot Improve What You Do Not Measure

RAG evaluation has two layers: retrieval and generation. Build a golden dataset of questions with known relevant documents and reference answers. Then track metrics over time.

Retrieval metrics

• Recall at k: what fraction of relevant documents appear in the top k results.
• Precision at k: what fraction of top k results are relevant.
• MRR: mean reciprocal rank of the first relevant result.
• nDCG: normalized discounted cumulative gain, which rewards relevant results ranked higher.

Generation metrics

• Faithfulness: is every claim supported by the retrieved context.
• Answer relevance: does the answer address the user question.
• Context precision: are the retrieved passages relevant and necessary.
• Context recall: did retrieval include all information needed to answer.
• Citation accuracy: do citations point to the correct source and span.

Use LLM-as-judge for fast iteration, but calibrate it against human labels. LLM judges can be biased toward long answers or their own style. For critical systems, combine automated metrics with human review and online feedback such as thumbs up and down.

Common Failure Modes and Fixes

• Hallucination despite context: The model ignores evidence or blends prior knowledge. Fix with stricter prompts, refusal training, and citation validation.
• Lost in the middle: Key evidence is buried in a long context. Fix with reranking, context compression, and placing top chunks at the edges.
• Chunk boundary loss: The answer spans two chunks and neither is retrieved. Fix with overlap, parent-child retrieval, or sentence-window retrieval.
• Stale index: Documents changed but embeddings did not. Fix with incremental ingestion, versioning, and freshness filters.
• Embedding drift: A new embedding model changes the vector space. Fix by versioning indexes and re-embedding in a shadow pipeline before cutover.
• Query mismatch: Users use different vocabulary than documents. Fix with query expansion, hybrid search, and domain synonym dictionaries.
• Permission leaks: Retrieval returns documents the user cannot see. Fix with ACL-aware filtering at query time, not post-filtering.
• Context poisoning: Malicious or incorrect documents enter the index. Fix with source allowlists, content sanitization, and provenance tracking.

Production Concerns: Latency, Cost, Security, and Operations

RAG systems must meet service-level objectives. Latency comes from query rewriting, embedding, vector search, reranking, and generation. Cache frequent queries and embeddings. Use streaming for generation so users see tokens early. Parallelize retrieval and reranking where possible. For cost, track tokens per query, embedding costs, and vector database read units. Compress context and use smaller models for routing and reranking.

Security and privacy are non-negotiable. Retrieval must respect tenant isolation and document-level permissions. Redact PII before indexing when possible. Encrypt data in transit and at rest. Log prompts and responses carefully because they may contain sensitive data. Provide audit trails for which documents were retrieved and which were cited.

Operational maturity includes:

• Index versioning: know which embedding model and chunker produced the index.
• Canary deployments: test new retrievers or prompts on a small traffic slice.
• Regression tests: run golden questions on every change.
• Observability: trace each request from query to retrieved chunks to final answer.
• Feedback loops: use user feedback to mine hard negatives and improve retrieval.

A Minimal Implementation Sketch

The following pseudocode shows the core online flow. It is intentionally simple, but it captures the separation between retrieval, reranking, and generation.

def answer(query, user):
    rewritten = rewrite_query(query, history)
    filters = build_acl_filters(user)
    candidates = hybrid_search(rewritten, filters, top_k=100)
    reranked = rerank(rewritten, candidates, top_n=10)
    context = assemble_context(reranked, max_tokens=3000)
    prompt = build_prompt(question=query, context=context)
    response = llm.generate(prompt, temperature=0)
    validate_citations(response, reranked)
    return response

In a real system, each step would have retries, timeouts, fallbacks, and metrics. You might cache embeddings for repeated queries, fall back to lexical search if the vector database is slow, or return a refusal if retrieval confidence is low.

Conclusion

Production RAG is an information retrieval problem wrapped in an LLM application. The model matters, but retrieval quality, chunking, evaluation, and operations usually determine success. Start with a clear golden dataset, measure retrieval and generation separately, and iterate on the highest-leverage bottleneck. Treat the index as a product, the prompt as an interface, and every answer as a traceable claim. Done well, RAG lets teams ship AI features that are fresh, permission-aware, and grounded in evidence rather than guesswork.