Retrieval-Augmented Generation (RAG): A Practical Implementation Guide

Ask an LLM a question about your internal documentation, your product’s changelog, or last quarter’s support tickets, and it will confidently make something up — because none of that was in its training data. Retrieval-augmented generation is the standard fix: instead of relying on what the model memorized during training, you retrieve relevant, current information at query time and hand it to the model as context.

It’s the architecture behind most production “chat with your docs” and internal-knowledge-base features shipping today, and it’s usually the right first move before anyone should be talking about fine-tuning.

What RAG Actually Solves

An LLM’s knowledge is frozen at training time and generic to whatever it was trained on. RAG addresses three problems that causes:

• Staleness — the model doesn’t know about anything that happened after its training cutoff, or anything private to your organization
• Hallucination — without grounding, a model will generate plausible-sounding but fabricated specifics
• Traceability — a RAG system can cite which document an answer came from; a fine-tuned model’s knowledge is baked into its weights with no audit trail

RAG vs. Fine-Tuning vs. Prompt Engineering

These get conflated constantly. They solve different problems and are often used together, not as alternatives:

Approach	Good for	Not good for
Prompt engineering	Adjusting tone, format, few-shot examples	Injecting large or frequently changing knowledge
RAG	Grounding answers in current, private, or large document sets	Teaching a model a new skill or style
Fine-tuning	Teaching consistent behavior, format, or domain-specific reasoning	Keeping knowledge current — you’d have to retrain on every update

The practical default: reach for RAG first. It’s cheaper to stand up, cheaper to update (edit a document, not retrain a model), and — critically — you can swap the underlying LLM without losing anything, since the knowledge lives in your retrieval layer, not the model.

The RAG Architecture

At its core, RAG has two pipelines: one that runs ahead of time to prepare your data (ingestion), and one that runs on every user query (retrieval + generation).

Ingestion (offline, runs on a schedule or on document change):

1. Load documents (PDFs, Markdown, Confluence pages, database rows)
2. Split them into chunks
3. Generate an embedding vector for each chunk
4. Store the chunks and vectors in a vector database

Query time (runs on every user request):

1. Embed the user’s query with the same embedding model
2. Search the vector database for the most similar chunks
3. Insert those chunks into the prompt as context
4. Send the augmented prompt to the LLM and return its answer

Choosing a Vector Database

The right choice depends more on your existing stack than on raw benchmark numbers — most of the popular options are within a reasonable range of each other on latency and recall at normal scale.

• pgvector — a Postgres extension. The right default if you already run Postgres and don’t want a new piece of infrastructure to operate. Scales further than people assume.
• Pinecone — fully managed, minimal ops overhead, a reasonable default if you don’t want to run infrastructure at all.
• Weaviate and Qdrant — open-source, self-hostable, strong hybrid-search (keyword + vector) support out of the box.
• Chroma — the easiest option for prototyping and small-to-medium production workloads; embeds directly in your application process for local dev.

If you’re not sure, start with pgvector or Chroma. Migrating vector stores later is far less painful than migrating your ingestion pipeline, so don’t over-index on this decision early.

Chunking Strategy

Chunking is where most RAG quality problems actually live, and it gets the least attention.

• Fixed-size chunking (e.g., 500 tokens with 50-token overlap) is simple and a reasonable starting point, but it cuts through sentences and sections arbitrarily.
• Semantic / structure-aware chunking — split on headings, paragraphs, or sections instead of a raw token count — produces noticeably better retrieval for structured content like documentation.
• Overlap matters. Some overlap between consecutive chunks (10-20%) prevents an answer from being split exactly across a chunk boundary and becoming unretrievable in either half.

A good rule of thumb: chunk size should roughly match the size of a single coherent idea in your source material. Legal contracts and API documentation want very different chunk sizes.

Retrieval Beyond Naive Similarity Search

Pure vector similarity search is the starting point, not the ceiling:

• Hybrid search — combine vector similarity with traditional keyword search (BM25). Pure embedding search is bad at exact matches (product SKUs, error codes, proper nouns); keyword search fixes that gap.
• Reranking — retrieve a larger candidate set (say, top 20) with a fast vector search, then use a more expensive but more accurate cross-encoder reranker (Cohere Rerank, or an open-source model) to pick the true top 5. This consistently improves answer quality for a small latency cost.
• Metadata filtering — attach metadata (document date, source, department) to each chunk and filter before or during vector search. Essential once you have more than one type of document in the same index.

A Working Example

A minimal but real ingestion-and-query pipeline using LangChain and Chroma:

from langchain_community.document_loaders import DirectoryLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain.chains import RetrievalQA

# --- Ingestion (run once, or on a schedule) ---
loader = DirectoryLoader("./docs", glob="**/*.md")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=120,
)
chunks = splitter.split_documents(documents)

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db",
)

# --- Query time ---
retriever = vectorstore.as_retriever(search_kwargs={"k": 5})
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True,
)

result = qa_chain.invoke({"query": "How do I rotate the staging database credentials?"})
print(result["result"])
print([doc.metadata for doc in result["source_documents"]])

This is intentionally minimal — no reranking, no hybrid search, no metadata filtering — but it’s a real, runnable pipeline and the right shape to extend once you know where your specific quality gaps are.

Evaluating RAG Quality

“It seems to work” isn’t an evaluation strategy. Two dimensions matter, and they’re independent — a system can be strong on one and weak on the other:

• Retrieval quality — did the system fetch the right chunks at all? Measured with recall/precision against a labeled set of question-to-correct-chunk pairs.
• Answer faithfulness — given the retrieved chunks, did the model’s answer actually stick to them, or did it hallucinate on top of correct context? Tools like RAGAS automate this scoring using an LLM-as-judge approach.

Build a small evaluation set (even 30-50 realistic questions with known-correct answers) before shipping. It’s the only way to know if a change to chunk size, retriever, or prompt made things better or worse, instead of guessing.

Production Considerations

• Caching — cache embeddings for unchanged documents, and consider caching answers for frequently asked questions to cut both latency and API cost.
• Latency budget — retrieval + reranking + generation adds up; know your target response time before choosing a reranker or a larger context window.
• Cost — embedding cost scales with your corpus size at ingestion, and generation cost scales with context length on every query. A bigger k isn’t free.
• Re-ingestion strategy — decide upfront how document updates propagate. A nightly batch job is fine for most internal knowledge bases; anything customer-facing with frequently changing data usually needs event-driven re-indexing.

Where This Fits

RAG is infrastructure, not a one-off script — it needs the same production discipline as any other service: monitoring, versioned embeddings, a rollback path when you change chunking or the embedding model, and a real evaluation set you run before every change ships. That operational layer is what turns a promising notebook demo into something a client can actually rely on.