RAG: Retrieval-Augmented Generation

Retrieval-Augmented Generation demystified: architecture, chunking, embeddings, and the practical gotchas.

RAG (Retrieval-Augmented Generation) pairs a language model with a search step over your own documents. The model still generates the answer, but only after retrieving the most relevant chunks of source material to ground itself.

The five-stage pipeline

1. Ingest — pull documents from sources (docs, tickets, PDFs, wikis).
2. Chunk — split into 300–800 token pieces with 10–20% overlap.
3. Embed — turn each chunk into a vector using an embedding model.
4. Store — write vectors to a vector DB (pgvector, Pinecone, Weaviate, Qdrant).
5. Retrieve + generate — at query time, embed the question, fetch the top-k chunks, and prompt the LLM with them.

Minimal Python example (pgvector + OpenAI)

import openai, psycopg
q_emb = openai.embeddings.create(model="text-embedding-3-small", input=question).data[0].embedding

with psycopg.connect(DB_URL) as db:
    rows = db.execute(
      "SELECT content FROM chunks ORDER BY embedding <=> %s LIMIT 5",
      (q_emb,)
    ).fetchall()

context = "\n---\n".join(r[0] for r in rows)
answer = openai.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role":"system","content":"Answer only from the context. Cite sources."},
        {"role":"user","content":f"CONTEXT:\n{context}\n\nQ: {question}"},
    ],
)

Where RAG goes wrong

• Bad chunking. Splitting mid-sentence kills semantic quality. Prefer semantic or heading-aware chunkers.
• Wrong embedding model. Old (ada-002) is cheap but weaker; newer small models are 3–4× better on retrieval.
• No re-ranking. Top-k similarity is often noisy — a cross-encoder re-rank on the top 20 → keep top 5 gives large quality gains.
• Not deduplicating. Near-duplicate chunks starve the context of variety.
• Ignoring metadata. Attach source URL, timestamp, permissions — you need them for citations and for filtering.

Evaluating a RAG system

Measure two things separately:

• Retrieval quality — recall@k on a labeled question/passage set.
• Answer quality — LLM-as-judge or human review on hallucination, faithfulness, completeness.

A great RAG system is 70% retrieval engineering and 30% prompt engineering. Don't skip the boring half.