All writeups

The Self-Hosted AI Tier: Private Inference with Ollama, OpenWebUI, and ComfyUI

August 26, 2026

OllamaOpenWebUIComfyUILLMGPU PassthroughNVIDIAEdge TPUFrigateSelf-HostedPrivacyHomelab

I wanted a private ChatGPT and a private image generator, a place to ask questions and generate images where not one token leaves my infrastructure. The same instinct that makes me self-host search makes me self-host inference: if the queries are mine, the model serving them should be mine too. This tier is where the lab spends its heaviest compute, and it is built around one rule: one clean front door, and everything expensive behind it.

Why run models locally at all

Three reasons, in order. Privacy: prompts, documents, and generated output stay inside the lab, full stop, with no third-party retention policy to trust. Control: I pick the models, the versions, and when they change, instead of waking up to a silently "improved" endpoint. Cost: once the hardware exists, inference is just electricity, and I already own the hardware.

The trade is that I am now the capacity planner. That constraint shaped every placement decision below.

Architecture

                       ┌─────────────────────────────┐
  client ──HTTPS──▶ reverse proxy ──▶ │  OpenWebUI  (the front door) │
                       └───────┬──────────────┬──────┘
                               │ OpenAI-compat │ workflows
                               ▼               ▼
                        ┌────────────┐   ┌──────────────┐
                        │  Ollama    │   │  ComfyUI ×2  │
                        │ LLM serving│   │  diffusion   │
                        └────────────┘   └──────────────┘
                               │
                               ▼
                        ┌────────────┐
                        │ Qwen3-TTS  │  speech synthesis
                        └────────────┘

 heavy compute pinned to the two beefiest nodes:
   a Ryzen 9 5950X (32 threads) and a 36-core Xeon E5-2697 v4 (256GB RAM)
 GPU passthrough:
   Tesla M40 -> Ollama (LLM)  ·  GTX 1080 Ti -> ComfyUI (diffusion)  ·  Tesla M60 -> spare
 edge inference:
   Coral Dual Edge TPU (M.2) + Coral SOM -> Frigate NVR object detection

Everything a user touches goes through OpenWebUI. Behind it, Ollama serves language models, two ComfyUI instances handle image generation, and Qwen3-TTS does speech. The expensive workloads are deliberately concentrated on the two most capable nodes in the cluster.

GPU passthrough

The first version of this tier ran inference on CPU. It worked, but it made everything patient. The upgrade was passing real GPUs through to the containers that need them, and three cards now sit behind the tier, each matched to the work it suits:

  • NVIDIA Tesla M40 serves the LLMs in Ollama. It has the largest frame buffer of the three cards, and VRAM is the ceiling on local language-model serving: a model that fits in the card's memory stays resident and answers quickly, while one that spills to system RAM slows to a crawl. The biggest-buffer card draws the memory-bound workload.
  • NVIDIA GeForce GTX 1080 Ti runs the diffusion in ComfyUI. Image generation is where a GPU earns its keep most obviously, turning multi-minute CPU renders into something interactive. The Pascal 1080 Ti and its 11GB are a comfortable fit for SDXL-sized workflows.
  • NVIDIA Tesla M60 is spare capacity. The M60 presents as two GPUs on one board, which makes it useful for a second model or a parallel job without contending with the cards already doing primary work.

Two things the older datacenter cards make you plan for: the Tesla M40 and M60 are passively cooled, with no onboard fan and no display output, so they need forced airflow to stay inside their thermal budget, and they want the matching NVIDIA driver installed inside the guest rather than a desktop one. The passthrough itself is the usual Proxmox routine: enable IOMMU, hand the device to the guest, and make sure the driver lands where the container can see the card.

Ollama: the model server

Ollama is the engine room. It pulls and manages models, keeps them warm, and, most usefully, exposes an OpenAI-compatible API, which means anything that speaks the OpenAI protocol can point at it with a base-URL change and nothing else. It runs as a container on the Ryzen 9 node with the Tesla M40 passed through: the card's large VRAM pool is exactly what LLM serving wants, since the whole point is to keep a model resident in the frame buffer instead of paging it against slower memory.

# ollama environment (excerpt): expose on the internal network, keep models warm
OLLAMA_HOST=0.0.0.0:11434
OLLAMA_KEEP_ALIVE=30m         # don't reload weights between every message
OLLAMA_MAX_LOADED_MODELS=2    # cap concurrent models so RAM doesn't blow up
OLLAMA_NUM_PARALLEL=2

By default Ollama binds 127.0.0.1, so it has to be bound to the internal address (as above) before OpenWebUI on another container can reach it. From there the reverse proxy and firewall, not the bind, are what control who gets to it.

OpenWebUI: the front door

OpenWebUI is the single interface over everything: chat, model switching, multi-user accounts, and document RAG. It is the heaviest-provisioned container in this tier for a reason (it runs on the 36-core Xeon node, which has 256GB of RAM): it is doing embeddings and retrieval on top of proxying chat.

# OpenWebUI environment (excerpt): point the front door at Ollama, keep it internal
OLLAMA_BASE_URL=http://ollama.internal:11434
WEBUI_AUTH=true               # accounts on, even internally
ENABLE_SIGNUP=false           # I add users; the internet does not

Only OpenWebUI is published, behind the reverse proxy and authentication; Ollama and ComfyUI stay reachable only from inside the tier. That is the point of a single front door rather than three: an unauthenticated inference endpoint left on the network is a free GPU for whoever finds it first.

ComfyUI: diffusion on demand

Image generation lives in two separate ComfyUI containers on the Ryzen node, with the GTX 1080 Ti passed through to do the actual rendering. Two instances, not one, so I can pin different model sets and workflows without them fighting over the same memory: one can be mid-render on a big SDXL graph while the other stays clean for quick jobs.

They are the only guests in this tier I run stopped by default. Diffusion is bursty and heavy, and there is no reason to hold that much allocation hostage 24/7 for something I use in sessions. I spin them up when I need them and shut them down after. Left running but unused, two large containers make the cluster look full when it is idle, which is exactly the high RAM allocation with no usage data that the posture scanner flags, and the nudge to keep on-demand workloads actually off when they are off.

Qwen3-TTS: giving it a voice

Qwen3-TTS rounds out the tier with local text-to-speech, so the "assistant" can talk without a cloud voice API in the loop. Same principle as the rest: the audio is generated in the lab and stays there.

Edge TPUs: the other end of the spectrum

The GPUs above are for generative work: big models, bursty load, lots of VRAM and watts. The opposite corner of the lab's AI hardware is a pair of Google Coral Edge TPUs, which do the small, constant, latency-sensitive inference it would be a waste to spin a GPU up for.

An Edge TPU is a tiny ASIC that runs INT8-quantized TensorFlow Lite models at a few TOPS on about two watts. It cannot run a language model or a diffusion graph, and that is exactly the point: it is built to run one small vision model, fast, all day.

In the lab they drive Frigate, the self-hosted NVR. Object detection on a camera feed is the textbook Edge TPU workload: a small detector, every frame, in real time. Offloading it to the TPU keeps it off the CPU entirely, so detection runs in a handful of milliseconds per frame while the host barely notices, and the footage never leaves the lab for a cloud detection service.

The two devices sit at opposite ends of the form-factor range:

  • Coral M.2 Accelerator with Dual Edge TPU provides the detection over PCIe. The one catch with the dual version: the second Edge TPU only comes online if the M.2 slot exposes two PCIe lanes, since each TPU wants its own. A single-lane slot sees just one of the two.
  • Coral SOM (System-on-Module) is the other shape entirely: a complete module with its own SoC, RAM, and eMMC alongside the Edge TPU, so it runs the same class of edge inference as a self-contained device rather than an accelerator bolted into a host.

Same privacy argument as the rest of the tier: the frames and the detections stay inside the lab.

Why it's designed this way

  • One front door. OpenWebUI is the only published surface. Every model backend sits behind it on the internal network, so there is exactly one thing to authenticate and one thing to watch.
  • Compute follows the workload. LLM and diffusion land on the nodes with the GPUs and the RAM, with each model backend pinned to the card that suits it: the big-VRAM Tesla M40 for language models, the GTX 1080 Ti for diffusion. The small low-power nodes run DNS and telemetry, not transformers.
  • On-demand beats always-on for the heavy stuff. Chat is cheap enough to keep warm; diffusion is not, so it stays stopped until asked for.
  • Privacy is the whole point. The tier exists so that prompts, documents, and outputs never become someone else's training data.

What's next

  • Tensor-core accelerators. The current cards are Maxwell (M40, M60) and Pascal (1080 Ti), so none have tensor cores. A newer GPU would speed up FP16 language-model inference specifically, which is where these cards are weakest.
  • Shared model storage. Pull each model once to a shared volume instead of duplicating weights per container.
  • Wire the tier into the lab's own assistant. The inventory platform already runs a cluster-aware assistant, so pointing internal tooling at the self-hosted Ollama endpoint instead of a hosted API is the natural convergence.