RAG Tutorial: Build a Retrieval Pipeline With LangChain and LlamaIndex

Build a RAG pipeline in Python twice: once with LangChain and once with LlamaIndex. Load, chunk, embed, index, and query your own documents step by step.

By Galactic News Editorial · Published 2026-11-09T13:28:36+00:00 · Updated 2026-11-09T13:28:36+00:00

RAG Tutorial: Build a Retrieval Pipeline With LangChain and LlamaIndex

Large language models do not know your private documents. They cannot answer questions about your company wiki, your client reports, or a PDF from last week. Retrieval-augmented generation, or RAG, fixes that. Before the model answers, your code retrieves the most relevant passages from your own files and passes them into the prompt. The model then answers from what it was just shown, not from what it half-remembers from training.

This RAG tutorial in Python builds the same pipeline twice: once with LangChain and once with LlamaIndex. You will load documents, split them into chunks, embed them, store them in a vector index, and query them. The code follows the official documentation and current tutorials for LangChain 1.x and LlamaIndex 0.12+. Nothing here claims firsthand testing of every line. Run the code yourself and check the output.

What RAG is, and when to use it

RAG has two steps. Retrieve, then generate. The retriever finds the most relevant chunks for a question. The generator feeds those chunks to the language model as context. The answer is grounded in your documents.

RAG beats fine-tuning for private data. Fine-tuning teaches the model new facts, but it is slow and costly. RAG leaves the model alone and changes the input instead. When your documents change, you re-index. You do not retrain.

Use RAG when the answers live in documents you own. Use it for support bots, internal Q and A, and research tools. Do not use it when the answer needs live data that changes every minute. For that, call an API instead.

The six building blocks

Every RAG system has the same parts. Learn them once, and both frameworks make sense.

• Documents: your text plus metadata, such as the source filename.
• Chunks: documents split into small pieces that retrieve well.
• Embeddings: each chunk converted into a vector of numbers that captures its meaning.
• Vector store: an index that finds the closest chunk vectors to a question vector.
• Retriever: the search call itself. Question in, best chunks out.
• Generator: the language model that answers using the retrieved chunks.

RAG tutorial Python: the LangChain pipeline

LangChain is a general-purpose framework. It gives you document loaders, text splitters, vector stores, and retrievers as composable pieces. The LangChain 1.x stack uses langchain-core, langchain-text-splitters, and partner packages such as langchain-openai and langchain-huggingface.

Step 1: install the packages

pip install -U langchain langchain-text-splitters langchain-openai langchain-community faiss-cpu

FAISS is the local vector store in this example. It runs in memory and needs no server. Chroma is a common alternative when you want persistence.

Step 2: load and split your documents

from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter

docs = [Document(page_content="Your leave policy text here.", metadata={"source": "leave-policy.md"})]

splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)

Chunk size matters. A chunk of 500 characters with 50 characters of overlap is a reasonable start. Overlap stops a key sentence from being cut in half at a chunk boundary. LangChain also ships dozens of ready-made loaders for PDF, Notion, Slack, and web pages in the langchain-community package.

Step 3: embed and store the chunks

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = FAISS.from_documents(chunks, embeddings)
retriever = vector_store.as_retriever(search_kwargs={"k": 3})

The k=3 setting returns the three most similar chunks for each question. For most beginner projects, a k of 2 or 3 is a reasonable starting point. If you prefer local embeddings, swap in HuggingFaceEmbeddings from langchain-huggingface. Nothing leaves your machine then, except the final language model call.

Step 4: retrieve, then generate

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini")

def answer(query: str) -> str:
    retrieved = retriever.invoke(query)
    context = "\n\n".join(d.page_content for d in retrieved)
    prompt = f"Answer using only this context:\n{context}\n\nQuestion: {query}"
    return llm.invoke(prompt).content

That is the whole two-step loop. Retrieve the chunks. Pass them to the model. The quality of retrieval decides the quality of the answer. If the wrong documents come back, the model has no chance. Test retrieval on its own before you judge the answers.

The LlamaIndex pipeline

LlamaIndex is a RAG-first framework. Where LangChain gives you general-purpose pieces, LlamaIndex ships retrieval, chunking, reranking, and generation as first-class primitives. Its core mental model is simple: data, then nodes, then index, then retriever, then query engine, then app.

Step 1: install the packages

pip install llama-index llama-index-embeddings-openai llama-index-llms-openai

LlamaIndex 0.10 and later uses a modular package structure. Install only what you need. The core package is llama-index-core. The OpenAI wrappers live in their own packages.

Step 2: load documents and build the index

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.llm = OpenAI(model="gpt-4o-mini")

docs = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(docs)

SimpleDirectoryReader loads every file in a folder. The global Settings object holds the embedding model, the language model, and the defaults. The index wraps the vector store. It handles chunking into nodes behind the scenes, using a sentence splitter with defaults you can override.

Step 3: query through the query engine

query_engine = index.as_query_engine(similarity_top_k=5)
response = query_engine.query("What is the refund policy for annual plans?")
print(response.response)

The query engine is the end-to-end pipeline. It takes a natural language question and returns an answer, along with the reference context it retrieved. One line of code runs retrieval and generation together. That is the main difference from LangChain. LlamaIndex hides the two steps inside the engine. LangChain shows them to you.

Step 4: improve retrieval

Basic retrieval returns the top chunks by vector similarity. Production systems add more. LlamaIndex supports reranking with a cross-encoder, hybrid search that mixes keyword and vector results, and response modes such as compact synthesis. A common setup retrieves 20 chunks, reranks them, keeps the top 5, and passes those to the model.

from llama_index.core.postprocessor import SimilarityPostprocessor

query_engine = index.as_query_engine(
    similarity_top_k=20,
    node_postprocessors=[SimilarityPostprocessor(similarity_cutoff=0.7)],
    response_mode="compact",
)

Start simple. Add reranking only when you can measure a retrieval problem. Every extra stage adds latency and cost.

LangChain vs LlamaIndex: which to pick

Pick LlamaIndex when the job is document Q and A and you want the shortest path to a working pipeline. Pick LangChain when the pipeline is one part of a larger system with agents, tools, and chains. Many teams use both. The retrieval concepts transfer. The chunking, embedding, and evaluation lessons are the same.

Common mistakes to avoid

• Chunks too large. A 5,000-character chunk dilutes the signal. The model reads mostly irrelevant text. Start at 500 characters and tune.
• No overlap. Without overlap, a key sentence splits across two chunks and neither retrieves well. Keep 10 percent overlap.
• Skipping evaluation. Build a test set of 20 questions with known answers. Check retrieval before you check generation. Bad retrieval is the most common failure.
• Trusting the answer without sources. Return the source chunk with every answer. A RAG answer without a citation is just a guess with extra steps.
• Indexing junk. Old drafts, duplicate files, and boilerplate pollute the index. Clean the corpus before you index it.

Bottom line

A RAG pipeline has six parts: documents, chunks, embeddings, a vector store, a retriever, and a generator. LangChain shows you each part. LlamaIndex wraps them in a query engine. Build the LangChain version first to learn the mechanics. Then build the LlamaIndex version to ship faster. Test retrieval with real questions, return sources with every answer, and keep your chunks small. That is the whole tutorial. The rest is tuning.