Retrieval-augmented generation is the most practical pattern for building AI applications that need access to private, domain-specific knowledge. In theory, it's straightforward: embed your documents, store them in a vector database, retrieve relevant chunks at query time, and pass them to an LLM for generation.
In practice, every step of that pipeline involves decisions that dramatically affect quality. After deploying RAG systems in healthcare, financial services, and enterprise support, here are five lessons we learned the hard way.
Lesson 1: Chunking Strategy Is Everything
The default chunking approach — split documents into fixed-size chunks of 500-1000 tokens — is a reasonable starting point and a terrible ending point. How you chunk your documents is the single biggest lever for retrieval quality.
We've tested four chunking strategies across production deployments:
Fixed-size chunking works for homogeneous documents where context doesn't span sections. For FAQs, product descriptions, and short articles, fixed chunks of 200-400 tokens with 10-20% overlap perform well.
Semantic chunking splits documents at natural boundaries — paragraph breaks, section headers, topic shifts. This preserves the coherence of ideas and dramatically improves retrieval quality for long-form content like policy documents, technical manuals, and legal contracts.
Hierarchical chunking stores both parent chunks (full sections) and child chunks (paragraphs within sections). At retrieval time, you search against the child chunks for precision, then return the parent chunk for context. This is our default approach for complex documents.
Document-level metadata enrichment adds a summary, key topics, and document type to each chunk as metadata. This enables pre-filtering (only search within 'policy documents' or 'product manuals') that dramatically reduces noise in the retrieval results.
For one healthcare deployment, switching from fixed-size to hierarchical chunking with metadata improved answer accuracy from 71% to 89% — without changing the model, the prompt, or anything else in the pipeline.
Lesson 2: Embedding Model Selection Matters More Than LLM Selection
Teams agonize over which frontier LLM to use for generation. In our experience, the embedding model has a larger impact on end-to-end quality than the generation model.
General-purpose embeddings (like OpenAI's text-embedding-3-large or Google's gemini-embedding-001) work well for general queries. But for domain-specific applications — medical terminology, legal language, financial jargon — fine-tuned or domain-specific embedding models like Voyage-4 consistently outperform general-purpose alternatives on retrieval benchmarks.
We've seen 15-20% improvement in retrieval precision by switching from a general-purpose embedding model to one fine-tuned on domain data. That improvement propagates through the entire pipeline — better retrieval means better context, which means better generation.
Our recommendation: start with a strong general-purpose model, measure retrieval quality with a domain-specific evaluation set, and only invest in fine-tuning if retrieval precision falls below your threshold.
Lesson 3: Re-Ranking Is a Force Multiplier
Vector similarity search returns the chunks that are closest in embedding space to the query. But 'closest in embedding space' doesn't always mean 'most useful for answering the question.'
Adding a re-ranking step — using a cross-encoder model to re-score the top-K results from vector search — consistently improves answer quality by 10-15%. Cross-encoders process the query and document together, which captures relevance signals that embedding similarity misses.
The architecture: vector search retrieves top-20 candidates (fast, cheap), then a cross-encoder re-ranks those 20 and returns the top-5 (slower, more expensive, but only on 20 documents). This gives you the speed of vector search with the precision of cross-encoding.
We now include re-ranking in every production RAG deployment. The compute cost is marginal (you're re-ranking 20 documents, not thousands), and the quality improvement is significant.
Lesson 4: Evaluation Is the Foundation — Build It First
The biggest mistake we made early on was building the RAG pipeline first and evaluation second. Without a systematic way to measure retrieval quality and generation quality, every architectural decision becomes a guess.
Before building the pipeline, we now create: a set of 50-100 representative questions with known correct answers, a retrieval evaluation that measures whether the correct document chunks are in the top-K results, a generation evaluation that measures whether the final answer is correct and grounded in the retrieved context, and an automated pipeline that runs these evaluations on every change.
This evaluation suite is the single most valuable artifact in the project. It turns every decision — chunk size, embedding model, prompt template, re-ranking threshold — from a debate into a measurement.
Lesson 5: Production RAG Needs More Than Retrieval
The RAG pattern as typically described — retrieve, then generate — is a starting point. Production systems need additional components:
Query preprocessing: Rephrase ambiguous queries, expand acronyms, detect the user's actual intent. A query classification step before retrieval significantly improves results for complex or multi-part questions.
Citation and grounding: Every claim in the generated response should be traceable to a specific source document. This isn't just good practice — in regulated industries, it's a requirement.
Confidence estimation: The system should know when it doesn't know. If retrieval returns low-relevance results, the system should say 'I'm not sure about this' rather than hallucinate a confident-sounding answer.
Feedback loops: Users thumbs-up or thumbs-down responses. That signal feeds back into retrieval tuning, prompt optimization, and identifies gaps in the knowledge base.
Architecture Recommendations
For teams starting a new RAG project, our recommended starting architecture: hierarchical chunking with metadata enrichment, a strong general-purpose embedding model (evaluate domain-specific models if precision is insufficient), a vector database with metadata filtering support (we use Pinecone or Weaviate), cross-encoder re-ranking on top-20 results, query preprocessing with intent classification, and citation-grounded generation with confidence scoring.
Start with this architecture, build your evaluation suite, and iterate based on measurements. The teams that ship high-quality RAG systems are the ones that measure relentlessly and optimize based on data.