Open Source & Self-hosted RAG LLM Server with…

Updated August 7, 2026

Most developers approaching RAG (Retrieval-Augmented Generation) systems hit the same wall: they want to run private LLMs locally without sending data to OpenAI or Claude, but the setup feels like assembling a spaceship with incomplete instructions. If you’ve tried deploying Ollama and wondered why it felt clunky, or you’re wrestling with GPU access on Windows Subsystem for Linux 2 (WSL2), this guide strips away the guesswork. We’ll walk through self-hosting a production-grade RAG server using open-source tools that actually work together—Ollama for the language model, a vector database for semantic search, and Docker to make it reproducible. By the end, you’ll have a system that runs entirely on your hardware, pulls documents through intelligent retrieval, and costs you nothing beyond electricity. The prerequisite is simple: Docker installed on your machine. Everything else builds from there.

Understanding RAG Before You Deploy It

Retrieval-Augmented Generation sounds academic, but the concept is borrowed from how humans actually think. When you’re asked a specific question about a domain you know well, you don’t generate an answer from thin air—you retrieve relevant knowledge first, then synthesize an answer based on that context. RAG systems work the same way. An LLM alone (whether Ollama’s Llama 2, Mistral, or another model) generates text based on its training data, which has a knowledge cutoff and knows nothing about your proprietary documents. RAG bridges that gap by embedding your documents into a vector database, retrieving the most relevant chunks when a query comes in, and feeding those chunks as context to the LLM before it generates a response.

Here’s the practical difference: asking a standard LLM “What does our Q3 2024 earnings report say about R&D spending?” returns hallucinated numbers. Asking the same question to a RAG system retrieves the actual Q3 report, feeds it to the LLM, and you get an accurate answer grounded in your real data. This matters at scale—enterprises spend 18-24 weeks deploying RAG systems because accuracy directly impacts decision-making. The self-hosted approach cuts that timeline dramatically because you skip the vendor negotiation and compliance overhead. Ollama (which bundles models like Llama 2 7B, Mistral 7B, and Neural Chat 7B) handles the LLM inference. Vector databases like Milvus, Qdrant, or Weaviate handle embeddings and retrieval. Docker containerizes everything so you’re not managing Python versions, library conflicts, or dependency hell across machines.

Your Hardware Prerequisites and What They Actually Mean

Docker is mandatory—that’s non-negotiable. But what’s less obvious is that your GPU situation determines whether this setup runs in minutes or hours. If you’re on Windows with an NVIDIA GPU, you’ll want CUDA toolkit installed (version 11.8 or higher, released June 2023). AMD GPU owners on Linux can use ROCm. Apple Silicon users get Metal acceleration automatically through recent Ollama releases. The reason this matters: running Llama 2 7B (the smallest usable model) on CPU alone consumes 30-40 minutes per inference query on a mid-range processor. The same query on a modest NVIDIA RTX 3060 (12GB VRAM) takes 5-8 seconds. That’s not a marginal difference—it’s the difference between a usable system and a paperweight.

Let’s talk specifics. A Docker host with 16GB RAM and 8 CPU cores is the practical minimum for running Ollama plus a vector database without throttling. The model itself matters: Llama 2 7B requires roughly 4GB VRAM (quantized), while Mistral 7B uses 5GB. Larger models like Llama 2 13B or 70B need 9GB and 40GB respectively. If your GPU sits below 8GB (like an RTX 3050), stick with 7B models or quantize larger ones to 4-bit or 5-bit precision. Quantization trades 5-10% accuracy for 40-60% smaller model size—an acceptable tradeoff for most enterprise use cases where you’re pulling specific facts, not generating poetry.

Docker Setup and Ollama Deployment

Start by verifying Docker runs on your system. Open your terminal and run docker --version—you should see version 20.10 or newer. If Docker isn’t installed, grab it from docker.com; the installation takes 15 minutes on Windows, macOS, or Linux. Next, decide your deployment strategy: Ollama can run as a native WSL2 service with GPU access (faster, simpler), or you can containerize Ollama itself using a Docker image. We’ll cover the WSL2 native approach first because it’s what most Windows developers actually use.

On Windows with WSL2: Download Ollama from ollama.ai—the installer is 200MB and handles WSL2 integration automatically if you have NVIDIA drivers installed. After installation, open WSL2 and run ollama serve. This starts the Ollama server on localhost:11434. To pull a model (let’s say Mistral 7B, which is 4.1GB), run ollama pull mistral in another terminal. The first pull takes 8-12 minutes depending on internet speed. Verify it works by running curl http://localhost:11434/api/generate -d '{"model":"mistral","prompt":"Hello"}'—you should see JSON output with the model’s response. That confirms Ollama is listening and the model is loaded.

For GPU access on WSL2, the secret is that NVIDIA’s CUDA toolkit must be installed on the Windows host, and WSL2 shares that GPU automatically if your drivers are recent (550.54 or newer, released January 2024). Verify this by running nvidia-smi inside WSL2—if you see your GPU listed, GPU acceleration is working. If you see an error, your drivers are outdated. Update them from nvidia.com (the GeForce or Studio drivers depending on your card), reboot, and try again. This is where most people get stuck: they assume WSL2 “just works” with GPU, but older driver versions break that bridge.

Vector Database Configuration and Document Ingestion

Now you need somewhere to store and search your documents. Qdrant is a strong choice for self-hosting—it’s lightweight, Docker-native, and requires minimal tuning. Milvus is more powerful but heavier; Weaviate is modern but adds operational complexity. Let’s use Qdrant for this guide. Pull the Qdrant Docker image: docker run -p 6333:6333 qdrant/qdrant:latest. This starts the vector database on localhost:6333. That single command gives you a persistent database at ./qdrant_storage (check with docker inspect if you need the exact path).

Here’s what happens next: you create a “collection” (think of it as a table) to hold embeddings. Each document gets split into chunks—typically 512 tokens with 100-token overlap—and each chunk gets vectorized (converted to a list of 1024 or 2048 numbers representing semantic meaning). Ollama can generate embeddings using models like nomic-embed-text (274MB), which outputs 768-dimensional vectors. A workflow looks like this: (1) ingest a PDF or text file, (2) split it into overlapping chunks, (3) embed each chunk using nomic-embed-text, (4) store vectors and metadata in Qdrant, (5) when a user queries, embed the query, search Qdrant for the closest vectors (using cosine similarity), (6) feed those top-K results to your LLM for synthesis. If you have 10,000 documents averaging 3,000 tokens each and you split them into 512-token chunks with 50% overlap, you’re storing roughly 60,000 vectors. Qdrant handles that comfortably in under 2GB RAM.

Python is the practical glue here. Use LangChain (v0.1.0+) or LlamaIndex (formerly GPT Index) to automate this pipeline. A minimal LangChain example: load a PDF with PyPDF2, split it with RecursiveCharacterTextSplitter (chunk_size=512, chunk_overlap=100), embed chunks using OllamaEmbeddings (model=”nomic-embed-text”, base_url=”http://localhost:11434″), and store in Qdrant using LangChain’s Qdrant wrapper. The entire pipeline for a 50-page report runs in 3-5 minutes, depending on your embedding model speed. Store embeddings to disk using Qdrant’s snapshot feature (PUT /snapshots endpoint) so you don’t re-embed documents if your server restarts.

Building the Complete Self-Hosted Stack with Docker Compose

Manually spinning up Ollama, Qdrant, and your application separately is tedious. Docker Compose orchestrates all three with a single file. Create a docker-compose.yml that defines three services: ollama, qdrant, and your rag-app (a Python Flask or FastAPI service). Here’s the structure:

1. Ollama service: Uses the official Ollama Docker image, exposes port 11434, mounts a volume for model persistence so re-pulling models isn’t necessary, and sets environment variables for GPU (CUDA_VISIBLE_DEVICES).
2. Qdrant service: Official Qdrant image, port 6333, persistent storage volume, and memory limits to prevent runaway allocation.
3. RAG application: Your custom image (built from a Dockerfile) that depends on both Ollama and Qdrant, exposes port 8000 for the API, and includes LangChain or LlamaIndex for orchestration.

A real example Dockerfile for the RAG app (Python 3.11 base): copy requirements.txt (containing langchain==0.1.4, ollama==0.0.11, qdrant-client==2.7.0, fastapi==0.104.1, uvicorn==0.24.0), run pip install -r requirements.txt, copy your application code, expose port 8000, and set the entrypoint to uvicorn main:app --host 0.0.0.0. Inside your FastAPI app, define endpoints: POST /ingest (accepts file uploads, orchestrates the embedding pipeline, returns success/failure), POST /query (accepts a query string, retrieves context from Qdrant, calls Ollama, returns answer), and GET /health (for Kubernetes or orchestration tools). The /query endpoint is the money shot: it typically runs in 2-8 seconds depending on model and context length.

To bring the stack online, run docker-compose up --build. Docker Compose waits for dependent services (Ollama and Qdrant) to be healthy before starting your app—you define health checks in the compose file using curl commands that ping the service ports. After 30-60 seconds, you have a fully functioning RAG system. Test it: curl -X POST http://localhost:8000/ingest -F "[email protected]", then curl -X POST http://localhost:8000/query -d '{"text":"What is mentioned about budget?"}'. You get back a JSON response with the LLM’s answer grounded in your documents.

GPU Acceleration on WSL2 With Real Configuration

Windows developers face the trickiest setup because GPU passthrough to Docker containers inside WSL2 requires precise configuration. Here’s what actually works (tested on Windows 11 with an RTX 4070): ensure NVIDIA drivers are 550.54 or newer (check nvidia.com/Download/driverDetails.aspx), install WSL2 with a recent Linux kernel (run wsl --update), and verify nvidia-smi works inside WSL2. Then, when running Ollama natively in WSL2 (not containerized), the GPU is automatically available—Ollama detects and uses it.

If you want Ollama containerized for consistency, Docker for Windows (not Docker Desktop with WSL2 integration) has better GPU support. Alternatively, use the docker-nvidia-runtime: install nvidia-docker, then in your docker-compose.yml, add runtime: nvidia to the Ollama service. Pass CUDA environment variables: environment: CUDA_VISIBLE_DEVICES: "0" (assuming single GPU). After restarting Docker, run docker exec CONTAINER_ID nvidia-smi to confirm the container sees the GPU. Common failure: the nvidia-docker runtime isn’t installed. Fix: distribution=$(. /etc/os-release;echo $ID$VERSION_ID), then follow nvidia-docker’s install instructions per your Linux distribution. This takes 5 minutes and is well-documented on nvidia-docker’s GitHub.

For non-NVIDIA hardware: AMD users on Linux use ROCm (install rocm-core, verify with rocm-smi), then pass HSA_OVERRIDE_GFX_VERSION environment variables to Ollama. Apple Silicon runs on Metal acceleration natively as of Ollama v0.1.0 (released March 2024)—no special configuration needed. Benchmark your setup before putting it in production: run a standard query 10 times, measure the median latency, and compare against baseline (CPU-only). A 7B model on RTX 3060 should hit 40-80 tokens/second; on CPU alone, expect 5-15 tokens/second. If you’re below 20 tokens/second on GPU, your GPU isn’t being used—check docker logs and nvidia-smi inside the container.

Production Hardening and Scaling Considerations

A working RAG system and a production RAG system are different animals. Add observability: instrument your FastAPI endpoints with OpenTelemetry or Prometheus to track query latency, embedding success rates, and LLM API errors. Log everything—queries, embeddings, retrieved context, and LLM responses—to a centralized store (ELK stack, Datadog, or simple file rotation). This matters because when an LLM generates a hallucinated answer, you need to audit what context it received and why retrieval failed. Implement rate limiting (slowapi library for FastAPI, 100 requests/minute per IP) and authentication (JWT tokens or OAuth2) to prevent abuse.

Scaling horizontally means running multiple instances of your RAG app behind a load balancer. Ollama and Qdrant benefit from that too: run Ollama with OLLAMA_NUM_PARALLEL=4 to handle 4 concurrent requests, and replicate your Qdrant cluster using Qdrant’s built-in replication (Enterprise feature) or by running multiple independent Qdrant instances and managing consistency yourself. For most mid-market use cases (100-1000 queries/day), a single machine with a decent GPU and 32GB RAM is sufficient. Beyond that, consider managed services like Supabase’s pgvector (PostgreSQL vector extension) for vector storage, or stay self-hosted but upgrade to a larger instance (AWS p3.2xlarge with 1x V100 GPU costs $3.06/hour; that’s $2,200/month for constant availability).

Cost transparency: running a self-hosted RAG system 24/7 on a modest home server (100W average draw) costs roughly $8-12/month in electricity. A cloud VM with equivalent specs costs $30-50/month. The tradeoff is operational burden—you’re responsible for backups, updates, security patches, and monitoring. For small teams or internal tools, self-hosting wins. For customer-facing products requiring 99.9% uptime and compliance auditing, managed services or hybrid approaches (self-hosted compute, managed vector DB) make sense. The boundary is typically around 50,000 documents or 1,000 queries/day—above that, operational overhead grows faster than the cost savings justify.

Common Pitfalls and How to Avoid Them

Disclosure: This article may contain affiliate links. If you make a purchase through these links, we may earn a small commission at no additional cost to you. We only recommend products and services we believe will add value to our readers.