AImy.blog Logo
← Back to Latest Intelligence
·AI Tutorials

Build Your Own Local RAG Pipeline: A Practical AI Tutorial with Ollama & LangChain

Unlock the power of private, cost-effective AI by building a Retrieval-Augmented Generation (RAG) pipeline with local LLMs. This tutorial guides you step-by-step using Ollama and LangChain to query your own documents without sending data to the cloud.

Christina
Christina
AImy Editor
Build Your Own Local RAG Pipeline: A Practical AI Tutorial with Ollama & LangChain

Build Your Own Local RAG Pipeline: A Practical AI Tutorial with Ollama & LangChain

In an era dominated by cloud-based AI, the ability to run large language models (LLMs) and sophisticated AI pipelines locally offers significant advantages: enhanced privacy, reduced costs, and greater control over your data. This tutorial will walk you through setting up a Retrieval-Augmented Generation (RAG) pipeline using Ollama for local LLMs and LangChain for orchestrating the data flow, allowing you to query your own documents securely.

Why Local LLMs and RAG?

  • Privacy & Security: Your data never leaves your machine, crucial for sensitive information.
  • Cost-Effectiveness: Eliminate API costs associated with cloud LLMs.
  • Offline Capability: Run AI applications without an internet connection.
  • Customization: Fine-tune models and RAG components to your specific needs.
  • RAG's Power: RAG enhances LLM responses by grounding them in specific, relevant information from your documents, reducing hallucinations and improving factual accuracy.

Prerequisites

Before we begin, ensure you have the following installed:

  • Python 3.8+: Download from python.org.
  • pip: Python's package installer (usually comes with Python).
  • Ollama: Download and install Ollama from ollama.com. Ollama makes it incredibly easy to run open-source LLMs locally.

Step 1: Set Up Ollama and Download a Local LLM

First, install Ollama and download an LLM. We'll use Llama 3 8B Instruct for this tutorial, a powerful and versatile model that runs well on consumer hardware.

  1. Install Ollama: Follow the instructions on ollama.com for your operating system (macOS, Linux, Windows).

  2. Download Llama 3: Open your terminal or command prompt and run:

    ollama pull llama3

    This might take some time as the model files are several gigabytes.

  3. Verify Installation: Once downloaded, you can test it by running:

    ollama run llama3 "Tell me a fun fact about AI."

    You should see a response from the local Llama 3 model.

Step 2: Prepare Your Data for RAG

For this tutorial, let's use a simple text file. Create a file named my_documents.txt with some content. For example:

Artificial intelligence (AI) is intelligence demonstrated by machines, unlike the natural intelligence displayed by humans and animals.
Machine learning (ML) is a subset of AI that focuses on the development of algorithms that allow computers to learn from data without being explicitly programmed.
Deep learning is a subset of machine learning that uses neural networks with multiple layers to learn from vast amounts of data.
Retrieval-Augmented Generation (RAG) is an AI framework that combines the strengths of retrieval-based and generation-based AI models. It improves the factual accuracy and relevance of generated responses by retrieving information from an external knowledge base before generating a response.
Ollama is a platform that allows you to run large language models locally.
LangChain is a framework designed to simplify the creation of applications using large language models.

Step 3: Build the RAG Pipeline with LangChain

We'll use LangChain to orchestrate the RAG process. This involves loading documents, splitting them, creating embeddings, storing them in a vector database, and finally querying with our local LLM.

3.1 Install Python Libraries

First, install the necessary Python packages:

pip install langchain langchain-community langchain-chroma beautifulsoup4 unstructured sentence-transformers
  • langchain: The core LangChain library.
  • langchain-community: Contains integrations for various LLMs, document loaders, etc.
  • langchain-chroma: ChromaDB integration for vector storage.
  • beautifulsoup4, unstructured: For more advanced document loading (not strictly needed for a simple text file, but good for general use).
  • sentence-transformers: For local embedding models.

3.2 Create the RAG Script

Create a Python file named local_rag.py and add the following code:

from langchain_community.document_loaders import TextLoader
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.llms import Ollama
from langchain.chains import RetrievalQA
import os

# 1. Load Documents
print("Loading documents...")
loader = TextLoader("my_documents.txt")
documents = loader.load()
print(f"Loaded {len(documents)} document(s).")

# 2. Split Documents into Chunks
print("Splitting documents into chunks...")
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = text_splitter.split_documents(documents)
print(f"Split into {len(chunks)} chunks.")

# 3. Create Embeddings
# Using a local sentence-transformer model for embeddings
print("Creating embeddings...")
# Ensure you have 'sentence-transformers' installed: pip install sentence-transformers
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")

# 4. Initialize ChromaDB (Vector Store)
# This will create a local directory 'chroma_db' to store embeddings
print("Initializing ChromaDB...")
vector_store = Chroma.from_documents(chunks, embeddings, persist_directory="./chroma_db")

# 5. Initialize Local LLM (Ollama)
print("Initializing Ollama LLM...")
# Ensure 'llama3' model is pulled in Ollama: ollama pull llama3
local_llm = Ollama(model="llama3")

# 6. Create RetrievalQA Chain
print("Creating RetrievalQA chain...")
# 'stuff' chain type combines all retrieved documents into a single prompt
rqa_chain = RetrievalQA.from_chain_type(
    llm=local_llm,
    retriever=vector_store.as_retriever(),
    chain_type="stuff",
    return_source_documents=True
)

# 7. Query the RAG Pipeline
print("\n--- RAG Pipeline Ready. Enter your queries (type 'exit' to quit) ---\n")
while True:
    query = input("Your query: ")
    if query.lower() == 'exit':
        break

    print("Processing query...")
    response = rqa_chain.invoke({"query": query})
    print("\n--- Response ---")
    print(response["result"])
    print("\n--- Source Documents ---")
    for doc in response["source_documents"]:
        print(f"- {doc.metadata['source']} (Page {doc.metadata.get('page', 'N/A')}):")
        print(f"  {doc.page_content[:200]}...") # Show first 200 chars
    print("\n")

print("Exiting RAG pipeline.")

Code Explanation:

  • TextLoader: Reads your my_documents.txt file.
  • RecursiveCharacterTextSplitter: Breaks down your document into smaller, manageable chunks. This is crucial for RAG, as LLMs have context window limits.
  • HuggingFaceEmbeddings: Converts your text chunks into numerical vectors (embeddings) using the all-MiniLM-L6-v2 model. This model runs locally, keeping everything private.
  • Chroma: Our local vector database. It stores the embeddings and allows for efficient semantic search (finding chunks similar to your query).
  • Ollama: Integrates your local Llama 3 model into LangChain.
  • RetrievalQA: This is the core RAG chain. It takes your query, finds relevant chunks from the vector_store (retrieval), and then passes those chunks along with your query to the local_llm to generate a grounded answer (generation).

Step 4: Run Your Local RAG Pipeline

  1. Ensure Ollama is Running: Make sure the Ollama application is active in your system tray or running in the background.

  2. Execute the Script: Open your terminal in the directory where you saved local_rag.py and my_documents.txt, then run:

    python local_rag.py

  3. Interact with Your RAG System: The script will prompt you for queries. Try asking questions related to the content in my_documents.txt:

    • "What is RAG?"
    • "What is Machine Learning?"
    • "How does Ollama help with LLMs?"

    You'll notice the responses are directly informed by your provided documents, and the script will even show you which source documents were used.

Conclusion & Next Steps

Congratulations! You've successfully built a fully functional, local RAG pipeline. This setup provides a powerful foundation for building private, domain-specific AI applications.

Further Enhancements:

  • Different Document Types: Explore DirectoryLoader, PyPDFLoader, CSvLoader from langchain_community.document_loaders to process PDFs, CSVs, and more.
  • Advanced Chunking: Experiment with different TextSplitter configurations or even context-aware chunking.
  • Hybrid Search: Combine semantic search with keyword search for even better retrieval.
  • Chat History: Integrate ConversationalRetrievalChain to maintain chat history and context.
  • UI Development: Build a simple web interface (e.g., with Streamlit or Gradio) to interact with your RAG system more intuitively.
  • Model Experimentation: Try other local models available on Ollama, such as Mistral, Mixtral, or Gemma, to find what works best for your use case.

By taking control of your AI infrastructure, you open up a world of possibilities for secure, efficient, and tailored AI solutions.

Tags & Entities

#Local LLM#RAG Pipeline#Ollama#LangChain#AI Tutorial#Private AI#HuggingFaceEmbeddings#ChromaDB#Llama 3