Basic RAG Pipeline

RAG (Retrieval-Augmented Generation) is a technique that combines information retrieval with text generation. Instead of relying solely on the knowledge the LLM absorbed during training, you provide relevant context in real time — the model responds based on that context.

A basic RAG pipeline has two phases: indexing and querying.

Phase 1: Indexing

The offline process of preparing documents for search.

1. Document loading

Documents are loaded from some source — PDFs, web pages, text files, databases, etc.

2. Chunking

Documents are split into smaller pieces called chunks. This is necessary because LLMs have a context limit and because smaller chunks increase search precision — a well-defined chunk tends to cover a single topic.

The chunking strategy directly affects response quality. Chunks that are too large introduce noise; chunks that are too small lose context.

3. Embedding

Each chunk is transformed into a numerical vector (embedding) by an embedding model. Semantically similar vectors end up close to each other in the vector space.

text chunk → embedding model → vector [0.12, -0.34, 0.87, ...]

4. Vector database storage

The vectors are stored in a vector database (Pinecone, Qdrant, Weaviate, pgvector, etc.) along with the original chunk text and its metadata.

Phase 2: Querying

The online process that happens with each user question.

1. Query embedding

The user's question goes through the same embedding model used during indexing, generating a vector.

2. Similarity search

The query vector is compared against the stored vectors. The chunks with the highest similarity (cosine similarity) are returned — these are the top-k chunks.

3. Prompt construction

The retrieved chunks are inserted into the prompt along with the user's question:

Context:
[chunk 1]
[chunk 2]
[chunk 3]

Question: {user query}

4. Generation

The LLM receives the prompt and generates a response based on the provided context.

Full flow

[documents] → chunking → embedding → vector db
                                          ↓
[query] → embedding → similarity search → top-k chunks
                                          ↓
                             [chunks + query] → LLM → response

Limitations of basic RAG

• Chunking quality — poorly defined chunks degrade the entire pipeline.
• Purely semantic search — can fail on queries with specific technical terms or proper nouns. That's why advanced pipelines combine dense search (embeddings) with sparse search (BM25).
• Fragmented context — chunks retrieved in isolation can lose the thread of the original document.

These limitations motivate more advanced techniques like hybrid search, reranking, and HyDE, which we'll cover in upcoming articles.

Practical examples are available in this repository.