Stackvora.tech

RAG: Retrieval-Augmented Generation Done Right

Retrieval-Augmented Generation (RAG) significantly enhances the utility of Large Language Models (LLMs) by grounding their responses in external,…

Retrieval-Augmented Generation (RAG) significantly enhances the utility of Large Language Models (LLMs) by grounding their responses in external, up-to-date, and domain-specific information. This approach mitigates common LLM limitations such as hallucination and knowledge obsolescence. Building an effective RAG system involves careful consideration of data preparation, retrieval mechanisms, and integration with the LLM. This article details best practices for constructing a robust RAG pipeline.

Data Ingestion and Chunking Strategies

The foundation of any RAG system is a well-structured and searchable knowledge base. Raw documents—PDFs, web pages, internal wikis, database records—must be processed into discrete, meaningful units, or "chunks." The effectiveness of subsequent retrieval steps heavily depends on the quality of these chunks.

Optimal Chunk Sizing and Overlap

A common heuristic for chunk size is approximately 250-750 tokens, roughly corresponding to 1-3 paragraphs of text. However, the optimal size is domain-dependent and often requires experimentation. For instance, highly technical documentation might benefit from smaller, more focused chunks (e.g., a single code snippet with its description), while narrative content might allow for larger chunks. A practical starting point is around 500 tokens.

Crucially, chunks should have strategic overlap to preserve context across chunk boundaries. An overlap of 10-20% of the chunk size (e.g., 50-100 tokens for a 500-token chunk) is generally recommended. This ensures that a relevant piece of information isn't split across two chunks, making it harder to retrieve comprehensively.


# Example: Python function for basic text chunking with overlap
import tiktoken

def chunk_text_with_overlap(text, max_tokens=500, overlap_tokens=50):
    tokenizer = tiktoken.get_encoding("cl100k_base") # Common for OpenAI models
    tokens = tokenizer.encode(text)
    chunks = []
    i = 0
    while i < len(tokens):
        chunk_tokens = tokens[i : i + max_tokens]
        chunks.append(tokenizer.decode(chunk_tokens))
        if i + max_tokens >= len(tokens):
            break
        i += (max_tokens - overlap_tokens)
    return chunks

# Example usage:
long_document = "This is a very long document that needs to be broken down into smaller pieces. Each piece, or chunk, should retain enough context to be meaningful on its own, but also overlap slightly with adjacent chunks to avoid losing critical information at the boundaries. The process involves tokenizing the text and then segmenting the token stream."
processed_chunks = chunk_text_with_overlap(long_document, max_tokens=100, overlap_tokens=20)
for j, chunk in enumerate(processed_chunks):
    print(f"Chunk {j+1}: {chunk}\n---")

Semantic Chunking Techniques

Beyond fixed-size chunking, semantic chunking aims to create chunks based on the natural structure and meaning of the document. This can involve:

  • Recursive splitting: Break a document by large delimiters (e.g., chapter, section) then recursively apply smaller delimiters (e.g., paragraph, sentence) until chunks are within target token limits. Libraries like LangChain's RecursiveCharacterTextSplitter implement this.
  • Markdown/HTML awareness: For structured documents, chunking can respect headings, lists, and code blocks, ensuring that logical units are not arbitrarily split.
  • Sentence Transformers: Embed individual sentences, cluster them, and then group adjacent sentences belonging to the same cluster into a chunk. This is more computationally intensive but can yield highly coherent chunks.

Embedding Models and Vector Databases

Once documents are chunked, each chunk is transformed into a high-dimensional vector (an embedding) that captures its semantic meaning. These embeddings are then stored in a vector database for efficient similarity search.

Selecting an Embedding Model

The choice of embedding model is critical. It should:

  • Match your domain: General-purpose models (e.g., text-embedding-ada-002, bge-large-en-v1.5, e5-large-v2) are good starting points. For highly specialized domains (e.g., legal, medical, financial), fine-tuned or domain-specific models often perform better.
  • Be sufficiently powerful: More parameters generally mean better semantic understanding, but also higher computational cost. Balance accuracy with inference speed and resource requirements. Models like bge-large-en-v1.5 (released by BAAI) and e5-large-v2 (from Microsoft) are strong open-source contenders, often outperforming older OpenAI models for general use cases.
  • Support adequate context length: Ensure the model can handle your chunk sizes.

When deploying, consider self-hosting open-source models via platforms like Hugging Face's transformers library or vendor-specific inference APIs (e.g., OpenAI, Cohere). Batching embeddings can significantly improve throughput.

Vector Database Selection and Configuration

Vector databases store and index embeddings, enabling fast similarity searches (e.g., K-Nearest Neighbors, Approximate Nearest Neighbors). Popular choices include:

  • Pinecone: Managed service, good for scalability and ease of use. Offers hybrid search (vector + metadata filtering).
  • Weaviate: Open-source, supports semantic search, GraphQL API, and custom modules. Can be self-hosted or managed.
  • Qdrant: Open-source vector similarity search engine, offering high performance and rich filtering capabilities.
  • Chroma: Lightweight, easy-to-use open-source vector database, suitable for smaller-scale projects or local development.
  • Pgvector: An extension for PostgreSQL, allowing you to store and query embeddings within your existing relational database. Good for simple RAG where you already use PostgreSQL.

When indexing, ensure that alongside the vector, you store essential metadata for each chunk: the original text, source document identifier, page number, section heading, and any timestamps. This metadata is crucial for contextualization, filtering, and citation.

Retrieval Mechanism: Querying and Re-ranking

After a user query is received, it's also embedded into a vector. This query vector is then used to search the vector database for the most semantically similar chunks. However, a simple top-k similarity search isn't always sufficient.

Initial Retrieval (Top-K)

Perform an initial similarity search to retrieve a larger set of potentially relevant chunks, often k=20 to k=50. This cast a wide net, ensuring that relevant information isn't missed due to slight semantic misalignments. The choice of k depends on the complexity of your knowledge base and the expected diversity of answers.

When searching, always consider metadata filters. For example, if a user query specifies a date range or document type, apply these as pre-filters to narrow down the search space in the vector database before performing similarity calculations.

Re-ranking for Precision

The initial top-k retrieval often returns chunks that are semantically similar but not necessarily the most *relevant* to answer the specific user query. A re-ranker model refines this initial set, ordering them by true relevance. Re-rankers take the original query and each retrieved chunk as input and output a relevance score.

  • Cross-encoders: Models like cross-encoder/ms-marco-MiniLM-L-6-v2 are specifically trained for re-ranking. They consider the query and document pair together, allowing for a deeper contextual understanding of relevance than a simple embedding similarity score.
  • Monolingual models: For English, models trained on datasets like MS MARCO are highly effective. For other languages, look for re-rankers fine-tuned on equivalent datasets.
  • Cost and latency: Re-rankers add latency. Typically, you'd apply a re-ranker to the top 20-50 initial results, reducing this to the top 3-5 most relevant chunks to pass to the LLM.

This two-stage retrieval (initial large retrieval + re-ranking) significantly improves the quality of the retrieved context passed to the LLM, leading to more accurate and focused responses.

LLM Integration and Response Generation

With the most relevant chunks identified, the final step is to integrate them into a prompt for the LLM and generate a coherent response.

Prompt Engineering for RAG

Construct a prompt that clearly instructs the LLM on its role, the provided context, and the desired output format. A typical prompt structure includes:

  1. System message: Define the LLM's persona and general guidelines (e.g., "You are an expert technical assistant...").
  2. Context injection: Present the retrieved chunks, clearly labeling them as "Context" or "Retrieved documents." Include source metadata (e.g., document title, page number) alongside each chunk.
  3. User query: The user's original question.
  4. Instructions: Explicitly tell the LLM to answer *only* based on the provided context. Instruct it to cite its sources and to state if it cannot find an answer in the given context.

Example prompt structure:


"You are a helpful and accurate technical assistant.
Answer the user's question ONLY using the provided context.
If the answer is not found in the context, clearly state 'I cannot answer this question based on the provided information.'
Always cite the source for each piece of information by referencing the 'Source' label.

Context:
---
Source: Document A, Page 3
Chunk Text: 'The TCP/IP model has four layers: Application, Transport, Internet, and Network Access. The Application layer combines the functions of the Session, Presentation, and Application layers of the OSI model.'

Source: Document B, Section 2.1
Chunk Text: 'OSI model, developed by ISO, consists of seven layers: Physical, Data Link, Network, Transport, Session, Presentation, and Application.'
---

User Question: What are the layers of the TCP/IP model according to the provided documents, and how does its Application layer relate to the OSI model?
"

Citations and Transparency

Always instruct the LLM to cite its sources directly in the response. This builds user trust and allows for verification. Citations should refer to the metadata you stored with each chunk (e.g., "According to 'Document A, Page 3'...") rather than just "Source 1."

Handling Unanswerable Questions

It's crucial to explicitly guide the LLM on how to handle queries where the provided context does not contain the answer. Without this instruction, LLMs are prone to "hallucinating" plausible-sounding but incorrect information. A clear instruction like "If the answer is not found in the context, state 'I cannot answer this question based on the provided information.'" is essential.

Common Pitfalls and Troubleshooting

  • Suboptimal Chunking: Chunks that are too large dilute relevance; chunks too small break context. Experiment with sizes (e.g., 256, 512, 768 tokens) and overlap.
  • Mismatched Embeddings: Using a general-purpose embedding model for highly specialized content often leads to poor retrieval. Consider fine-tuning or using domain-specific models.
  • Insufficient Retrieval (Low K): If your initial k for retrieval is too small, you might miss relevant documents. Increase k and use a re-ranker to filter.
  • Lack of Re-ranking: Relying solely on vector similarity can return semantically close but contextually irrelevant chunks. A re-ranker is almost always beneficial.
  • Poor Prompt Engineering: Ambiguous or unspecific instructions to the LLM can lead to ungrounded or incomplete answers. Be explicit about using only the provided context and citing sources.
  • Ignoring Metadata: Not storing or utilizing metadata (source, page, date) limits filtering capabilities and the quality of citations.
  • Scaling Challenges: For very large document sets, ensure your vector database and embedding infrastructure can scale with indexing and query load. Batching embedding requests can mitigate API rate limits and improve throughput.

Back to the knowledge base · Ask the AI assistant