Why do it yourself?
The goal was simple: a private ChatGPT. A modern chat interface wired to a real LLM, but with inference running on my own hardware — no data leaving for a cloud provider. All of it on my home Kubernetes cluster (MireCloud), deployed cleanly via GitOps.
Two building blocks are enough for the core:
vLLM — the inference engine. It exposes an OpenAI-compatible API (/v1/chat/completions), it's fast, and it serves quantized models.
Open WebUI — the front end. A chat UI that looks like ChatGPT, with user management, RAG, and — crucially — enterprise SSO.
This is the first article in a series. It covers the two foundations: serving the model (vLLM) and logging in (Open WebUI + Keycloak SSO). The next articles build on top of it — groups, per-group data isolation, and the RAG engine that answers from your own documents. One layer per post.
Open WebUI running Qwen2.5-14B-Instruct-AWQ.
The hardware
The inference node (node-gpu) is a VM with:
NVIDIA RTX 5060 Ti — 16 GB VRAM, Blackwell architecture (compute capability 12.0 / sm_120). Driver 595.71.05, CUDA 13.2. 24 GB host RAM, 96 GB disk.
That "Blackwell" detail looks harmless. It isn't. More on that below.
The model being served: Qwen2.5-14B-Instruct-AWQ — a 14-billion-parameter model quantized to 4-bit (AWQ), about 8.5 GB of VRAM. A good quality/VRAM trade-off for a 16 GB card.
The architecture
Keycloak (SSO / OIDC + groups)
|
▼
Open WebUI ──► vLLM (Qwen2.5-14B-AWQ) ← plain chat
(UI + RBAC) RAG API (pgvector + groups) ← private, per-group isolated dataNote the two backends. Open WebUI doesn't talk to a single engine — it exposes two selectable "models" in the dropdown: the raw vLLM chat model, and a second, OpenAI-compatible API of our own that does RAG. More on that second one below — it's where the real product value lives.
Everything is deployed by Argo CD from a Git repo (apps/vllm, apps/openwebui). The workflow: I edit the values.yaml files on my laptop, git push, and Argo syncs the cluster automatically (prune: true, selfHeal: true). No manual changes on the server — self-heal would revert them anyway. That's the whole point of GitOps: the Git repo is the state of the cluster.
Prerequisites: this stands on infrastructure I already had
An honest disclaimer: this went as smoothly as it did because a big part of the platform was already in place. vLLM and Open WebUI didn't land on a bare cluster — they plugged into a foundation I'd built (and written about) earlier:
HashiCorp Vault — my source of truth for secrets and my internal PKI / Certificate Authority. Every secret in this post (the OIDC client secret, database credentials) is pulled from Vault via External Secrets, never hardcoded. See: HashiCorp Vault Dynamic Secrets — Homelab Part 8.
Keycloak as the OIDC provider — the identity layer, the realm, the clients, and the group model were already stood up. This post consumes that; it doesn't build it. See: Eradicating Static Kubernetes Credentials with Zero-Trust OIDC — Homelab Part 5.
The internal CA certificate, issued by Vault's PKI and distributed across the cluster — that's the mirecloud-ca-cert that Open WebUI mounts so it can trust Keycloak over TLS. And GitOps with Argo CD, already wired to the repo.
So if some steps below look like "just add a values block," it's because the hard, foundational plumbing — secrets, certificates, identity — was solved in the previous articles. If you're starting from scratch, read those first.
Part 1 — vLLM, and the Blackwell trap
Deployment uses the vllm-stack Helm chart. The values.yaml describes the model, resources, and storage. On paper, trivial. In practice, the pod got OOMKilled (137) over and over.
The diagnosis
vLLM 0.11.0 does not ship precompiled CUDA kernels for sm_120 (the card is too new). So on the first forward pass it compiles kernels just-in-time (JIT). With no limits, it spawns ~6 cicc processes (NVIDIA's CUDA compiler) in parallel, at ~1.3 GB each → a 15–20 GB host-RAM spike → Memory cgroup out of memory.
The trap: it's not the vLLM Python process blowing up (it used only ~0.6 GB) — it's the cicc processes. Confirmed in the OOM-killer's dmesg. Raising the pod's memory limit doesn't help — the node only has 24 GB.
The fix
We cap JIT compilation via environment variables, and disable torch.compile:
env:
- name: MAX_JOBS
value: "2" # max 2 parallel compiles (~2.6 GB)
- name: NVCC_THREADS
value: "1"
- name: TORCH_CUDA_ARCH_LIST
value: "12.0" # compile ONLY for sm_120
- name: VLLM_CACHE_ROOT
value: "/data/vllm_cache" # JIT cache on the PVC -> compiled once
- name: TORCH_EXTENSIONS_DIR
value: "/data/torch_ext"
vllmConfig:
maxModelLen: 8192
dtype: "half" # AWQ = float16
enablePrefixCaching: true
extraArgs:
- "--enforce-eager" # no torch.compile/cudagraph -> avoids a 2nd host-RAM OOM
- "--quantization"
- "awq_marlin" # fast kernel
- "--gpu-memory-utilization"
- "0.90"With these settings plus a 12Gi/20Gi memory request/limit, the pod starts cleanly. cgroup peak ~10.8 GiB, well under the limit.
Two things worth remembering
First boot is slow (~4 min): it's compiling kernels. init engine took 241s. That's normal, and it happens once. And the cache that matters is FlashInfer's (/root/.cache/flashinfer, ~416 MB) — it is not covered by VLLM_CACHE_ROOT. To avoid recompiling for 4 minutes on every restart, persist it to the PVC via FLASHINFER_BASE_DIR=/data.
/var/lib/containerd bloated to 61 GB of accumulated images) → DiskPressure → cascading pod evictions. The node runs containerd, not docker: cleanup is ctr -n k8s.io images prune --all then systemctl restart containerd. On a homelab, watch ephemeral-storage as closely as you watch VRAM.Two more traps that hid the real one
The Blackwell OOM was actually the last problem in the chain — two infrastructure traps kept the pod from ever reaching the GPU. Both fail silently or misleadingly:
| Trap | What actually happened | Fix |
|---|---|---|
| Umbrella-chart nesting (silent) | My vllm app is a wrapper chart depending on vllm-stack. In Helm a dependency's values must be nested under its name. servingEngineSpec/routerSpec at the top level were handed to the wrapper (ignored) and never reached the subchart — which ran on example defaults: no engine Deployment, empty storageClass. |
Nest everything under vllm-stack:. (Same lesson as the Open WebUI chart later: misplaced keys don't error, they vanish.) |
| Proxmox CPU type hid AVX (misleading) | node-gpu is a Proxmox VM. Its CPU type x86-64-v2-AES doesn't expose AVX to the guest — though the physical Ryzen 5 7600 supports it. vLLM's UCX dependency is built with AVX → illegal instruction → SIGSEGV on startup. |
Set the VM CPU type to host (safe — PCIe passthrough already rules out live migration). |
If you want the full forensic version — nine stacked problems, from a FailedBinding PVC to reading the kernel OOM-killer log with kubectl debug node — I wrote it up as a runbook and a post-mortem, both linked at the end.
Part 2 — Open WebUI and Keycloak SSO
Inference is up, the API responds. What's left is the UI — and above all enterprise SSO via Keycloak (OIDC). This is where most of the silent traps lived. Five of them, each costing a debugging round. But first, the client itself.
The goal of this section: a “Continue with Keycloak” button that delegates login via OIDC.
Creating the OIDC client in Keycloak
Before Open WebUI can delegate login, Keycloak needs a client representing it. In the mirecloud realm: Clients → Create client.
1. General: Client type OpenID Connect, Client ID openwebui.
2. Capability config — the part that matters:
Client authentication: On — makes it a confidential client, so Keycloak issues a client secret (a public client would have none; Open WebUI needs one). Standard flow: On (browser authorization-code flow). Direct access grants: On — optional, I only enable it to mint test tokens via the password grant (handy for testing the RAG API's group filtering later). Implicit flow, Service accounts, Authorization: Off.
3. Access settings — the redirect URI is where people get stuck. Open WebUI's callback path is /oauth/oidc/callback:
Valid redirect URIs: https://openwebui.mirecloud.com/oauth/oidc/callback
Web origins: https://openwebui.mirecloud.com
Valid post logout redirect URIs: https://openwebui.mirecloud.com/*4. Grab the secret: the Credentials tab now shows the client secret. Don't paste it into values.yaml — store it in Vault and let Open WebUI read it via an External Secret (the openwebui-oidc / client-secret reference below).
The one mapper that makes groups work
By default the token carries the user's identity but not their groups. Since group-based access is the whole point (and the basis for the RAG RBAC coming later), add a Group Membership mapper:
• Client scopes → openwebui-dedicated → Add mapper → By configuration → Group Membership
• Token Claim Name: groups
• Full group path: Off (so the claim reads finance, not /finance)
Then verify before wiring anything else: Client scopes → Evaluate, pick a user, and confirm the token actually contains the groups claim. This single check would have saved me one of the debugging rounds below.
Secrets and certs come from Vault, not values.yaml
Remember the prerequisite: nothing sensitive is hardcoded. The two things Open WebUI needs to reach Keycloak — the OIDC client secret and the internal CA certificate — are both pulled from Vault by External Secrets, which materialize them as native Kubernetes Secrets. That's why values.yaml only ever references a secret name, never a value.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: openwebui-oidc
namespace: openwebui
spec:
refreshInterval: 1m
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: openwebui-oidc # ← the K8s secret the chart reads (clientExistingSecret)
creationPolicy: Owner
data:
- secretKey: client-secret
remoteRef:
key: secret/openwebui/secret
property: client-secret
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: mirecloud-ca-sync
namespace: openwebui
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: mirecloud-ca-cert # ← mounted by Open WebUI as /etc/ssl/certs/vault-ca.crt
creationPolicy: Owner
data:
- secretKey: ca.crt
remoteRef:
key: secret/mirecloud/ca
property: tls.crtTwo things to notice. The client secret refreshes every minute — rotate it in Vault and the cluster follows, no redeploy. And the CA that Open WebUI mounts to trust Keycloak (trap #2 below) is the same CA my Vault PKI issues cluster-wide — one root of trust, synced by a one-hour loop. Exactly the payoff of having built the Vault layer first.
Exposing it: Gateway API + cert-manager
Open WebUI reaches the outside world through the Kubernetes Gateway API (Cilium as the implementation), with cert-manager minting the TLS cert from that same Vault-backed CA. Three objects:
# 1. The certificate - issued by the Vault CA ClusterIssuer
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: openwebui-tls-cert
namespace: openwebui
spec:
secretName: openwebui-tls-secret
issuerRef:
name: mirecloud-ca-issuer # ← the same internal CA, now issuing a server cert
kind: ClusterIssuer
commonName: openwebui.mirecloud.com
dnsNames:
- openwebui.mirecloud.com
---
# 2. The Gateway - terminates TLS at the edge
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: mirecloud-gateway
namespace: openwebui
spec:
gatewayClassName: cilium
listeners:
- name: https
protocol: HTTPS
port: 443
tls:
mode: Terminate # ← decrypts here; the pod sees plain HTTP
certificateRefs:
- kind: Secret
name: openwebui-tls-secret
allowedRoutes:
namespaces: { from: Same }
---
# 3. The route - sends the hostname to the Open WebUI service
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: openwebui-route
namespace: openwebui
spec:
parentRefs:
- name: mirecloud-gateway
hostnames:
- "openwebui.mirecloud.com"
rules:
- matches:
- path: { type: PathPrefix, value: / }
backendRefs:
- name: openwebui-open-webui
port: 80Notice mode: Terminate. The Gateway does the TLS handshake and forwards plain HTTP to the pod. Convenient — but it's exactly what sets the trap in the next section: Keycloak, behind the same pattern, then has no idea it was reached over HTTPS.
The 5 SSO traps
1. The right chart keys — and the "silently ignored" trap. The open-webui chart (v14.8.0) silently ignores keys it doesn't recognize. A misspelled key = no error, just… nothing works. You need sso.oidc.* (not a top-level oidc:) and sso.enabled: true as the global switch.
sso:
enabled: true
enableSignup: true
mergeAccountsByEmail: true # links OAuth to the pre-existing local admin account
enableGroupManagement: true
oidc:
enabled: true
clientId: "openwebui"
clientExistingSecret: "openwebui-oidc"
providerUrl: "https://keycloak.mirecloud.com/auth/realms/mirecloud/.well-known/openid-configuration"
providerName: "Keycloak"
scopes: "openid email profile"2. Mounting the internal CA — same silent-key trap. For Open WebUI to trust Keycloak's internal certificate, you must mount the CA. But the chart expects volumes + volumeMounts.container, not extraVolumes/extraVolumeMounts. Wrong keys → the secret was never mounted → SSL_CERT_FILE pointed at a missing file → FileNotFoundError in httpx → 500 on /oauth/oidc/login.
3. Keycloak behind a TLS-terminating Gateway. The Gateway terminates TLS, so the Keycloak pod only sees plain HTTP. Without --hostname=https://keycloak.mirecloud.com, the OIDC issuer comes back as http:// → mismatching_issuer → 500. (While you're at it: drop the old KC_PROXY=edge, deprecated in Keycloak 26.)
4. Do NOT request a groups scope. Keycloak rejects a groups scope that doesn't exist (Invalid scopes). Keep scopes: "openid email profile" and add the Group Membership mapper (claim groups) on the openwebui-dedicated client scope. Verify under Keycloak → Clients → openwebui → Client scopes → Evaluate.
5. DNS split-horizon. Cluster pods couldn't resolve keycloak.mirecloud.com (Name or service not known). Fix in the CoreDNS ConfigMap: a block that forwards the domain to the LAN DNS (the one that knows the gateway's IP). Trap within the trap: use forward (ask the DNS), not the template plugin hardcoding an A answer — otherwise pods hit the DNS box on :443 → All connection attempts failed.
Group sync (the seam to the next article)
The full RBAC story — mapping groups to models, tools, and data — is the next article's job. But SSO already lays the plumbing: Keycloak groups can flow into Open WebUI as groups. Two subtleties worth locking in now:
• enableGroupManagement (ENABLE_OAUTH_GROUP_MANAGEMENT) only assigns users to groups that already exist. To auto-create them from the token, you also need ENABLE_OAUTH_GROUP_CREATION=true (not exposed by the chart → added via extraEnvVars).
• Groups sync on the member's login, not the admin's. The logged-in user must genuinely be a member of the Keycloak group.
New SSO users land in the pending role (manual approval) — a deliberate choice over an open DEFAULT_USER_ROLE=user.
A sneak peek: the second "model" is really a RAG API
One last thing worth showing now, even though its details belong to the next article. Open WebUI points at two OpenAI-compatible URLs — the vLLM router (raw Qwen chat) and a homemade RAG API of our own:
extraEnvVars:
- name: OPENAI_API_BASE_URLS
value: "http://vllm-router-service.vllm.svc.cluster.local/v1;http://rag-api.rag.svc.cluster.local:8000/v1"
- name: ENABLE_FORWARD_USER_INFO_HEADERS
value: "true" # forwards email/groups to the RAG API -> per-user isolationBecause that second service speaks the OpenAI API, it shows up in Open WebUI as just another model in the dropdown — but it isn't a model at all. It's a FastAPI service that, on every question, retrieves the relevant private documents, injects them into the prompt, and then asks the same Qwen-14B to answer with sources. Retrieval-Augmented Generation, transparent to the user.
The ENABLE_FORWARD_USER_INFO_HEADERS flag is the linchpin: Open WebUI forwards the logged-in user's identity and Keycloak groups (from the OIDC token) to the RAG API on every call. That's what makes per-user, per-group isolation possible — and it's exactly where the next article picks up.
Where the series goes next
That's the foundation done: the model is served, and people log in with SSO. From here the series adds one layer per article:
Part 11 — Groups & RBAC: turning Keycloak groups into real access control (different groups → different models, tools, and data).
Part 12 — Multi-tenant RAG: the FastAPI engine above, so the assistant answers from your documents, with hard per-group isolation.
Here's a taste of what that RAG article will cover. The design goal is hard data isolation between groups, driven entirely by the OIDC identity:
• Vector store: PostgreSQL with the pgvector extension. Documents are chunked, embedded (with a multilingual model — the embedder choice matters a lot for retrieval precision), and stored alongside an allowed_groups column.
• Isolation is a single SQL clause. Each row carries the groups allowed to see it; every query filters with WHERE allowed_groups && <groups-from-the-JWT>. The groups come straight from the Keycloak token — so a user cannot widen their own access from the client side. An admin group bypasses the filter.
• This is where the OIDC groups pay off. The very same groups claim that assigns a user to an Open WebUI group also decides which documents the RAG API will ever show them. One identity, one source of truth, enforced at the database level.
Two people ask the exact same question, and each gets an answer grounded only in the documents their group is allowed to see — the other group's data never even reaches the model.
What I took away
• The LLM is not the hard part. Once the Blackwell trap is understood, vLLM serves Qwen-14B without complaint. The real work is everything around it: SSO, RBAC, DNS, certificates, data isolation.
• Helm charts that silently ignore unknown keys are a productivity killer. The moment a config "has no effect," check the exact expected key names first.
• GitOps pays off. Every fix above is a commit. The cluster's state is reproducible, versioned, and I never touch the server by hand.
• A recent consumer GPU (Blackwell) is production-usable at home — as long as you accept a one-time JIT compile and cache it.
From this foundation — SSO + per-group access + per-user connectors — you can build a real private enterprise AI assistant, whose core pitch fits in one line: your data stays home.
Comments
Post a Comment