Self-Hosting AI with Ollama and Open WebUI: Run LLMs Locally

Running large language models locally used to require deep knowledge of Python environments, model quantization, and GPU drivers. Today, two tools make it remarkably simple: Ollama handles downloading, running, and serving LLMs through a clean CLI, and Open WebUI provides a polished chat interface on top of it. Together, they give you a private, self-hosted alternative to ChatGPT that runs entirely on your own hardware.

Photo by Ladislav Sh on Unsplash

No API keys, no usage fees, no data leaving your network.

What Ollama Does

Ollama is a lightweight runtime for large language models. Think of it as Docker for LLMs -- you pull a model by name, and Ollama handles downloading the weights, loading them into memory, and serving an OpenAI-compatible API.

# Install and run a model in two commands
ollama pull llama3.1:8b
ollama run llama3.1:8b

Under the hood, Ollama uses llama.cpp for inference, which means it supports GGUF-quantized models and can run on both CPU and GPU. It exposes a REST API on port 11434, making it easy for other tools to interact with it.

What Open WebUI Provides

Open WebUI (formerly Ollama WebUI) is a self-hosted web interface that connects to Ollama's API. It gives you a ChatGPT-like experience in your browser, with features that go well beyond a basic chat box:

• Multi-model switching -- swap between installed models mid-conversation
• Conversation history -- persistent chat logs stored locally
• Document upload -- basic RAG (retrieval-augmented generation) for chatting with files
• User management -- multiple accounts with separate conversation histories
• Model management -- pull, delete, and configure models from the UI
• Prompt templates -- save and reuse system prompts

Docker Compose Setup

The simplest way to run both together is a single Docker Compose file.

CPU-Only Setup

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    volumes:
      - ollama_data:/root/.ollama
    ports:
      - "11434:11434"
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    volumes:
      - open_webui_data:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama_data:
  open_webui_data:

docker compose up -d

Open http://your-server:3000, create an account (the first account becomes admin), and you're ready to go. You still need to pull a model -- either from the Open WebUI interface or via the CLI:

docker exec -it ollama ollama pull llama3.1:8b

NVIDIA GPU Setup

For GPU acceleration with an NVIDIA card, install the NVIDIA Container Toolkit, then modify the Ollama service:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    volumes:
      - ollama_data:/root/.ollama
    ports:
      - "11434:11434"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

Verify GPU access after starting:

docker exec -it ollama ollama run llama3.1:8b "Hello, are you using GPU?"
# Check nvidia-smi to confirm GPU utilization
nvidia-smi

GPU vs CPU: What to Expect

The performance difference between GPU and CPU inference is dramatic. Here are rough expectations for generating tokens with Llama 3.1 8B:

CPU inference is viable for small models and occasional use. If you plan to use LLMs regularly, a GPU with at least 8 GB of VRAM makes a significant quality-of-life difference.

Recommended Models

Not all models are equal, and bigger is not always better. Here's a practical guide to which models work well for self-hosting:

Best Starting Points

Larger Models (If You Have the Hardware)

Pull any of these with:

docker exec -it ollama ollama pull llama3.1:8b
docker exec -it ollama ollama pull mistral:7b
docker exec -it ollama ollama pull gemma2:9b

Hardware Requirements

Minimum Viable Setup

• CPU: Any modern x86_64 with AVX2 support (most CPUs from 2015+)
• RAM: 8 GB (for 7B models with CPU inference)
• Storage: 10 GB per model (varies by quantization level)
• GPU: Not required, but strongly recommended

Comfortable Setup

• CPU: 8+ cores
• RAM: 32 GB
• GPU: NVIDIA card with 12+ GB VRAM (RTX 3060 12 GB is the sweet spot for price/performance)
• Storage: 100 GB SSD for model storage

Memory Rule of Thumb

For GGUF quantized models (which Ollama uses by default), expect roughly:

• Q4_0 quantization: Model parameter count in billions x 0.6 = GB needed
• Example: Llama 3.1 8B Q4_0 needs about 4.7 GB of RAM/VRAM

If a model doesn't fit entirely in VRAM, Ollama will split it between GPU and CPU memory, which works but reduces speed.

Practical Configuration Tips

Customizing Model Behavior

Create a custom model with a system prompt:

docker exec -it ollama bash -c 'cat <<EOF | ollama create my-assistant -f -
FROM llama3.1:8b
SYSTEM "You are a helpful technical assistant. Be concise and accurate."
PARAMETER temperature 0.7
PARAMETER num_ctx 4096
EOF'

Increasing Context Length

By default, most models run with a 2048-token context window. For longer conversations:

docker exec -it ollama ollama run llama3.1:8b --num-ctx 8192

More context uses more memory. A 7B model with 8192 context needs roughly 2 GB more RAM than the default.

Exposing Ollama to Your Network

By default, Ollama only listens on localhost. To allow other devices (or Open WebUI on a different machine) to connect:

environment:
  - OLLAMA_HOST=0.0.0.0

If you do this, make sure Ollama is behind a firewall or VPN. There is no built-in authentication.

Practical Use Cases

Self-hosted LLMs shine in specific scenarios:

• Privacy-sensitive queries -- medical questions, legal research, financial planning, anything you wouldn't type into a cloud service
• Code assistance -- local Copilot-like functionality without sending your codebase to a third party
• Document analysis -- upload contracts, papers, or reports to Open WebUI and ask questions about them
• Offline access -- works without internet once models are downloaded
• Learning and experimentation -- try different models, fine-tune prompts, understand how LLMs work without per-token costs

Self-Hosted LLMs vs Cloud APIs

The honest truth: cloud models like GPT-4o and Claude are still significantly more capable than anything you can run locally on consumer hardware. Self-hosted LLMs are best as a complement to cloud services, not a replacement. Use them for privacy-sensitive tasks and experimentation, and cloud APIs when you need maximum quality.

Keeping Things Updated

Ollama and Open WebUI both move quickly. Update regularly:

docker compose pull
docker compose up -d

To update a model to the latest version:

docker exec -it ollama ollama pull llama3.1:8b

Verdict

Ollama and Open WebUI are the easiest way to run LLMs on your own hardware. The setup takes five minutes with Docker Compose, and the experience is genuinely good -- especially with a decent GPU. You won't match GPT-4 quality with a 7B model, but for many everyday tasks, local models are more than capable. The privacy and zero ongoing cost make it worth running alongside whatever cloud services you already use.