Working through this module?0 of 19 done
Module 07 · Page 8 of 20 · 9 sections

RAG System Design

Retrieval Augmented Generation end to end: ingestion, chunking, embeddings, retrieval, prompt construction and generation.

06LLM System Design08Agentic AI System Design14Evaluation and Testing
7.1

What is RAG?

  • Retrieval Augmented Generation is a technique that combines retrieval of relevant information from external knowledge sources with generative LLMs to produce accurate, up-to-date and context-aware responses.
  • It helps reduce hallucination and allows LLMs to use domain-specific knowledge.
7.2

Why RAG is Important?

  1. 1Provides access to latest and domain-specific data.
  2. 2Improves accuracy and reduces hallucination.
  3. 3Useful for enterprise and private data.
  4. 4Can be updated without retraining the model.
  5. 5Cost effective compared to fine-tuning.
  6. 6Works well for long and specific documents.
7.3

High Level Architecture

User Query
Query Processing (optional)Rewriting, intent detection, expansion
RetrieverVector, keyword, hybrid search
Relevant DocumentsTop-k chunks, re-ranking, filtering
LLM (Generator)Answer, use context, show citations
Response to UserAccurate, relevant, cited source
Feedback and follow-up questions
7.4

1. Document Ingestion Pipeline

  1. 1Load documents from different sources (PDF, DOCX, web, database, APIs).
  2. 2Extract text and metadata.
  3. 3Split text into smaller chunks (chunking).
  4. 4Generate embeddings for each chunk.
  5. 5Store embeddings and metadata in a vector database.
Related03Data Pipeline and Data Engineering
7.5

2. Text Chunking

  • Split large documents into smaller meaningful chunks to improve retrieval accuracy.
1
Fixed size chunking
For example 500 tokens per chunk.
2
Recursive chunking
Based on headings and paragraphs.
3
Semantic chunking
Using embeddings to find meaning boundaries.
7.6

3. Embedding and Vector Database

  • Convert text chunks into vector embeddings using embedding models (e.g. OpenAI, SentenceTransformer).
  • Store embeddings in a vector database for similarity search.
  • Popular vector databases: Pinecone, Qdrant, Weaviate, FAISS, Chroma, Milvus.
DocumentsPDF, DOCX, etc.
Text Chunking
Embedding Modeltext-embedding-3-small, all-MiniLM-L6-v2
Vector DatabasePinecone, Qdrant, etc.
7.7

4. Retrieval Techniques

1
Vector Search
Find similar chunks using embeddings.
2
Keyword Search
Traditional BM25 search.
3
Hybrid Search
Combine vector and keyword search.
4
Re-ranking
Re-rank top results using a cross-encoder or LLM.
5
Metadata Filtering
Filter by source, date or category.
7.8

5. Prompt Construction

  • Combine the user query and retrieved context to create a well-structured prompt.
  • Include clear instructions and citation format.
prompt
You are a helpful assistant. Use the following context to
answer the question. If the answer is not in the context,
say you don't know.

Context: {retrieved_chunks}
Question: {user_question}
Answer:
7.9

6. Generation and Response

  1. 1Send the prompt to an LLM (e.g. GPT-4, Claude, Llama).
  2. 2Generate a clear and concise answer.
  3. 3Include citations (source of information).
  4. 4Support follow-up questions using conversation history.
  5. 5Optionally, implement guardrails to prevent unsafe responses.
Related12Security, Privacy and Responsible AI14Evaluation and Testing