Ollama-WebUI: 10+ Steps to a Free, Private RAG AI Stack with Docker

Alexandra

Employee Experience (EX) Manager, WPRiders

Published: August 12, 2025

TL;DR

• Ollama-WebUI + Docker Compose = a free, private RAG stack (Ollama for models, Open WebUI for chat, and a Vector DB for embeddings) running locally.
• Setup: Install Docker (add NVIDIA Container Toolkit for GPU), run docker compose up, pull a model, ingest your docs, and start chatting with your data.
• Privacy & Cost: No cloud lock-in, no per-token fees; everything stays on your machine.
• Performance: Works on CPU; add a GPU for major speedups and larger workloads.
• Integration: Exposes APIs for apps, automations, or WordPress (chatbots, docs assistants, internal tools).

Ever wondered what it would feel like to have ChatGPT-level AI running right on your laptop? No monthly subscriptions. No sending your sensitive documents to unknown servers. Just pure AI power that you control completely.

That dream becomes reality with Ollama-WebUI. You can run advanced large language models on your own hardware without paying a cent to cloud providers or worrying about where your data ends up.

Ollama acts like the engine behind this local AI magic—an open-source platform that makes running AI models locally surprisingly straightforward. Pair it with Open WebUI, a clean browser interface that lets you chat with these models just like any AI service, and you’ve got something special. The whole setup comes together with Docker Compose, making what used to be a nightmare installation process into a few simple commands. Your computer becomes your private AI playground, delivering the privacy and cost savings that cloud alternatives simply can’t offer.

But here’s where it gets really interesting. Add a vector database to this mix, and you can build Retrieval-Augmented Generation (RAG) systems that let your AI reference your own documents before answering questions. Think of it as giving your AI photographic memory for your specific information—whether that’s company policies, research papers, or personal notes.

This guide walks you through building your own AI infrastructure from scratch. You’ll set up Ollama with Open WebUI using Docker Compose, configure GPU acceleration to make everything lightning-fast, and create RAG systems that can help with everything from writing code to crafting emails based on your own knowledge base.

Whether you’re a developer tired of API limits, a business owner concerned about data privacy, or just someone who thinks having AI on tap sounds pretty cool—this is your roadmap to AI independence.

Understanding the Components of Your AI Stack

Building your own AI infrastructure isn’t just about installing software—it’s about understanding how three key pieces work together to create something powerful. Think of it like assembling a high-performance car: the engine, the dashboard, and the memory system all need to work in perfect harmony.

Let’s break down what each component actually does and why you need all three.

What is Ollama and how it runs LLMs locally

Ollama acts as the foundation of your local AI setup. Picture it as “Docker for AI models”—it packages everything needed to run complex language models into a single, manageable file called a Modelfile. This approach turns what used to be a nightmare of dependencies and configurations into something surprisingly straightforward.

Here’s what makes Ollama special:

• Runs powerful LLMs directly on your hardware without any cloud connections
• Gives you access to a growing library of models, including Llama, Gemma, and others
• Lets you customize models with specific system prompts and parameters
• Connects to other applications via a REST API for integration

The platform automatically detects your GPU during installation and uses it for acceleration—though it’ll work on CPU-only mode too (just expect slower responses). Want to test a model? Just type ollama run gemma3 and it downloads and starts a conversation immediately.

But here’s where things get interesting: Ollama can handle multi-modal models that work with both text and images. These models can analyze photos, diagrams, or any visual content right within your local environment, keeping everything completely private.

Role of Open WebUI in managing LLMs

Open WebUI solves a critical problem: command-line interfaces are powerful, but they’re not exactly user-friendly for everyday AI interactions. This browser-based interface becomes your mission control center for everything Ollama.

The real magic happens in how it handles conversations. You can chat with multiple AI models simultaneously, compare their responses side-by-side, and switch between different models without touching a terminal. Open WebUI also includes granular permission controls and user groups, making it work well for teams.

What sets it apart is the seamless Ollama integration. Open WebUI automatically discovers and loads your Ollama models, letting you switch between different AI personalities without any technical hassle. This makes advanced AI accessible to users who prefer clicking buttons over typing commands.

The flexibility extends beyond local models too. Open WebUI supports OpenAI-compatible APIs, so you can mix local models with cloud services like LMStudio, GroqCloud, and Mistral. This hybrid approach lets you use local models for sensitive data while accessing cloud models for specific tasks.

Why use a Vector DB for RAG pipelines

Here’s where your AI stack gets really smart. Vector databases solve a fundamental limitation of language models: they only know what they learned during training, with a hard cutoff date for their knowledge.

Think of vector databases as giving your AI photographic memory for external information. Unlike the parametric memory baked into LLMs during training, vector databases provide non-parametric memory—essentially a searchable knowledge vault that models can access on demand.

The technical magic happens through embeddings—numerical representations that capture the semantic meaning of text, images, or other data. Your AI doesn’t just do keyword matching; it understands context and meaning.

A vector database transforms your AI system by enabling it to:

1. Store domain-specific knowledge as searchable vector embeddings
2. Instantly retrieve the most contextually relevant information for any query
3. Access up-to-date information beyond the model’s training cutoff
4. Deliver accurate answers on specialized topics without expensive model retraining

This approach costs significantly less than continuously retraining models while delivering more accurate, relevant answers. Your local AI stack becomes a complete system that understands both general concepts and your specific business or personal information needs.

Open WebUI makes this remarkably accessible with built-in RAG capabilities. You can upload documents directly into conversations or build document libraries accessible with the # command. This tight integration means building knowledge-enhanced AI applications doesn’t require a computer science degree.

Setting Up Ollama-WebUI with Docker Compose

Now comes the fun part—actually building your AI fortress. Docker makes this whole process surprisingly painless. No more wrestling with dependencies or wondering why something works on one machine but not another.

The beauty of using Docker? Your entire AI stack becomes portable and predictable. Set it up once, and it works everywhere.

Install Docker and NVIDIA Container Toolkit

First things first—you need Docker on your machine. Docker Desktop is your best bet for most setups, giving you everything you need in one package. Linux users will need Docker Engine and Docker Compose installed separately, but that’s pretty standard.

Here’s where things get interesting: GPU acceleration. Trust me, you want this. The difference between CPU and GPU performance is like the difference between walking and driving a sports car.

For Ubuntu/Debian systems, here’s how to get the NVIDIA Container Toolkit running:

1. Add the NVIDIA GPG key and repository:
   • curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey |
   • sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -s -L
   • https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list |
   • sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg]
   • https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
2. Update and install:
   • sudo apt-get update sudo apt-get install -y nvidia-container-toolkit
3. Configure Docker for NVIDIA runtime:
   • sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker

Create docker-compose.yml for Ollama and WebUI

Time to write the blueprint for your AI stack. Create a docker-compose.yml file—this single file defines your entire setup:

version: '3'
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    environment:
      - "OLLAMA_API_BASE_URL=http://ollama:11434/api"
    depends_on:
      - ollama
    volumes:
      - open-webui:/app/backend/data
    restart: unless-stopped

volumes:
  ollama:
  open-webui:

GPU configuration for Ollama container

Want to unlock your GPU’s full potential? Modify your Ollama service to include GPU support:

ollama:
  image: ollama/ollama:latest
  container_name: ollama
  ports:
    - "11434:11434"
  volumes:
    - ollama:/root/.ollama
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: all
            capabilities: [gpu]
  restart: unless-stopped

This configuration hands over all your NVIDIA GPUs to Ollama. Your models will thank you for the speed boost.

Port mapping and volume setup for persistence

Let’s talk about the plumbing. Port mapping makes your services accessible:

• Port 11434 exposes Ollama’s API to your machine
• Port 3000 lets you access Open WebUI through your browser

The volume setup is crucial—without it, you’d lose your downloaded models every time you restart. Nobody wants to re-download several gigabytes of AI models because they forgot to persist data. Docker volumes keep everything safe and isolated.

Run the stack using docker compose up

Ready to see the magic happen? Fire up your AI stack with one command:

docker compose up -d

The -d flag runs everything in the background. First run will download about 4GB of container images—grab a coffee while it works.

Once everything’s running, point your browser to http://localhost:3000. You’ll find yourself face-to-face with Open WebUI, ready to download models and start chatting with AI that runs entirely on your own hardware.

No APIs. No subscriptions. Just you and your local AI assistant.

Pulling and Running LLM Models in Ollama

Your AI stack is up and running. Now comes the fun part—actually putting it to work. Ollama gives you access to dozens of cutting-edge models, each with different strengths and hardware appetites.

Choosing a model from Ollama registry

Think of model selection like picking a car. You want the most powerful engine your garage can handle. Visit the Ollama website and click “Models” to browse what’s available. You’ll see models organized by size, capabilities, and resource requirements.

The naming system is straightforward: model_family_name:tag where tags tell you the model size (1b, 3b, 7b, etc.). Bigger numbers mean more knowledge but hungrier hardware. Here’s what you need to know about resource requirements:

• 7B models need at least 8GB RAM
• 13B models require 16GB
• 33B models demand 32GB or more

For vision tasks, check the “Vision” category, where you’ll find models like LLaVA that can analyze images alongside text. Pretty cool stuff for a local setup.

Running ollama run to download and start model

The ollama run command does all the heavy lifting. It downloads the model (if needed) and starts it up in one go:

ollama run llama3.2

First time running a model? Ollama automatically grabs it from their registry. Once it’s loaded, you’ll see this prompt:

>>> send a message (/? for help)

You can even test multimodal models with images:

ollama run llava
>>> what is there in this image? ./logo.jpeg

Testing model with curl API call

Want to get programmatic? Ollama exposes a REST API on port 11434 that you can hit directly:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

The API handles temperature control, context length limits, and even structured JSON output. Perfect for building your own applications on top.

Accessing model via Open WebUI interface

Command line not your thing? Open WebUI makes model management point-and-click simple. Head to Admin Settings, then Connections > Ollama > Manage (look for the wrench icon).

You can also download models on the fly. Just type a model name in the Model Selector—if it’s not installed, Open WebUI offers to grab it from Ollama automatically.

The interface even lets you run two models side-by-side to compare their responses. It’s like having a conversation between different AI personalities right in your browser.

Building a RAG Pipeline with Open WebUI

Your AI setup is running smoothly, but now comes the game-changer: teaching your models about your specific information. RAG (Retrieval Augmented Generation) turns your local AI from a generalist into a specialist that knows your business, your documents, and your unique needs.

Think of it this way—your AI currently knows what the internet taught it up to its training cutoff. RAG gives it access to your private library of knowledge, making every conversation smarter and more relevant.

Create a Knowledge Base in Open WebUI

The Knowledge section works like a digital filing cabinet where your AI can quickly find and reference information during conversations. It’s your personal reference library that makes responses laser-focused on what actually matters to you.

Setting up a knowledge base takes just a few clicks:

1. Navigate to Workspace > Knowledge in the Open WebUI interface
2. Click the + Create a Knowledge Base button
3. Give it a descriptive name (like “Research Papers” or “Product Documentation”)
4. Add a purpose description if you want (such as “Assistance” or “Technical Support”)
5. Click Create Knowledge to finish

Upload Documents to the Knowledge Base

Now for the fun part—feeding your AI the information it needs to become truly useful. Open WebUI makes this process flexible with several upload options:

• Upload File: Drop individual documents straight into the system
• Upload Directory: Import entire folders of files at once
• Sync Directory: Keep a folder automatically synchronized
• Add Text Content: Type information directly into the system

Your best bet is uploading markdown (.md/.mdx) files, PDFs, or plain text documents. Open WebUI automatically processes everything, creating embeddings that capture the meaning behind your content.

Link Knowledge Base to a New Model

Here’s where the magic happens—connecting your knowledge base to a model so it can actually use that information:

1. Go to Workspace > Models > + Add New Model
2. Enter a Name for your custom model
3. Select a Base Model (Llama3 or any other Ollama model)
4. Under Knowledge Source, choose your knowledge base
5. Click Save & Create to seal the deal

This creates a specialized version of your chosen model that can pull from your documents when answering questions.

Test RAG Model with Document-Based Queries

Time to see your enhanced AI in action:

1. Start a New Chat in Open WebUI
2. Select your custom model from the dropdown
3. Ask questions related to your uploaded documents

Here’s a neat trick: you can reference specific knowledge bases in any chat by typing the # symbol before your query. This lets you temporarily give any model access to your documents without permanently linking them.

The difference is immediate. Instead of generic responses based on training data, your AI now grounds its answers in your specific documents. It’s like having a research assistant who’s memorized every file in your system and can instantly find the most relevant information for any question.

Tuning Parameters and Using REST API

Your AI stack is running, but getting the best performance means tweaking the right settings. Think of this as fine-tuning your engine—small adjustments make a big difference in how well your AI handles different tasks.

Adjusting chunk size and overlap for documents

Document processing success comes down to chunk size and overlap settings. Smaller chunks (200-300 tokens) work great when you need precise, factual answers, while larger chunks (800-1000 tokens) keep more context for complex topics. A solid starting point? 512 tokens with 128 token overlap.

Head to Settings > Admin Settings > Documents to adjust these settings. The sweet spot depends on your use case—legal documents might need larger chunks to preserve context, while FAQ databases work better with smaller, focused chunks.

Changing the embedding model in Admin Settings

Your embedding model determines how well your AI understands document relationships. From the Documents section in Admin Settings, you can choose between SentenceTransformers, Ollama, and other engines. For better performance, try BAAI/bge-m3 or configure Azure OpenAI embeddings by providing the Base URL, API Key, and model name.

Different embedding models excel at different tasks. Some handle technical documentation better, others work great with conversational content.

Modifying temperature and max tokens

Temperature controls response randomness—this setting dramatically affects your AI’s personality. Lower values (0-0.2) produce focused, deterministic outputs perfect for code generation, while higher values (0.7-0.8) encourage creativity.

For coding tasks specifically, set mirostat to 0 and top K to 1. Max tokens (n_predict) limits response length, with 4096 recommended for complex code generation. These settings prevent your AI from rambling when you need concise answers.

Calling Open WebUI via REST API endpoints

Want to integrate your local AI into applications? Open WebUI’s REST API makes this straightforward. First, grab your API key from Settings > Account.

The primary endpoint for interactions is:

POST /api/chat/completions

This endpoint supports model selection and RAG integration with uploaded files. You can reference individual files and entire collections in your queries by including them in the request body. This means your applications can tap into both the AI’s training and your specific knowledge base simultaneously.

The API follows OpenAI’s format, making it easy to swap out cloud services with your local setup without changing application code.

Key Takeaways

Building a local AI stack with Ollama, Open WebUI, and vector databases provides a powerful, privacy-focused alternative to expensive cloud AI services while maintaining complete control over your data.

• Set up your free AI infrastructure using Docker Compose – Deploy Ollama and Open WebUI containers with GPU acceleration for local LLM hosting without subscription fees
• Create RAG pipelines by uploading documents to knowledge bases – Enhance AI responses with your specific information using Open WebUI’s built-in vector database capabilities
• Choose models based on your hardware capacity – 7B models need 8GB RAM, 13B models require 16GB, while larger models demand 32GB+ for optimal performance
• Fine-tune parameters for different use cases – Adjust temperature (0-0.2 for coding, 0.7-0.8 for creativity) and chunk sizes (200-300 tokens for facts, 800-1000 for context)
• Access your AI stack programmatically via REST API – Integrate local models into applications using OpenAI-compatible endpoints while maintaining data privacy

This approach eliminates cloud dependencies, reduces costs, and ensures your sensitive data never leaves your local environment while providing enterprise-grade AI capabilities.

FAQs

Q1. What are the main components of a local AI stack using Ollama and Open WebUI? 

The main components are Ollama for running LLMs locally, Open WebUI for managing the models through a browser interface, and a vector database for implementing RAG (Retrieval Augmented Generation) capabilities.

Q2. How much RAM is required to run different sizes of language models? 

Generally, 7B parameter models need at least 8GB of RAM, 13B models require 16GB, and larger 33B models demand 32GB or more for optimal performance.

Q3. Can I use GPU acceleration with this setup? 

Yes, GPU acceleration is supported and recommended for better performance. You’ll need to install the NVIDIA Container Toolkit and configure your docker-compose file to pass GPU capabilities to the Ollama container.

Q4. How do I create a RAG pipeline using Open WebUI? 

To create a RAG pipeline, create a knowledge base in Open WebUI, upload relevant documents, link the knowledge base to a new model, and then test the model with document-based queries.

Q5. Is it possible to access the AI models programmatically? 

Yes, both Ollama and Open WebUI provide REST APIs that allow you to interact with the models programmatically. You can make API calls to generate text, ask questions, or utilize the RAG capabilities in your applications.

In Conclusion…

You’ve just built something most people pay hundreds of dollars a month for. Your own AI infrastructure. No subscriptions. No data sharing. No limits.

The setup you’ve created gives you something cloud providers can’t—complete control. Your documents never leave your machine. Your conversations stay private. Your models run when you need them, not when someone else’s servers decide to cooperate.

But here’s what really matters: this isn’t just about saving money or protecting privacy. You’ve gained independence from the AI gatekeepers who decide what models you can use, when you can use them, and how much you’ll pay for the privilege.

The RAG system you’ve built changes the game entirely. Instead of generic AI responses, you get answers grounded in your specific knowledge. Your business documents. Your research. Your expertise. The AI becomes an extension of your own thinking rather than a replacement for it.

Sure, there’s a learning curve. Docker commands. Configuration files. GPU drivers that occasionally throw tantrums. But once it’s running, you’ll wonder why you ever trusted cloud providers with your most sensitive work.

Your local AI stack grows with you. New models appear in Ollama’s registry regularly. Better embedding models. Faster inference engines. You adopt them on your timeline, not when some company decides to roll out updates.

The real power isn’t in the technology—it’s in the autonomy. When everyone else is rationing their API calls or waiting for rate limits to reset, your AI is ready. When others worry about service outages or policy changes, you keep working.

This is AI on your terms. Start experimenting with your free AI stack and discover what’s possible when you’re not asking permission to innovate.

Do you like this article? Share it and send us your feedback! Check out our articles page, where you might find other interesting posts. Also, if you want to learn more about business, check out the WPRiders Blog!

Don't forget to subscribeReceive WPRiders' newsletter for the freshest job openings, sent directly to your inbox. Stay informed and never miss a chance to join our team!

Email Address*

Name

I accept the terms and conditions