Eradicated Static Kubernetes Credentials with Zero Trust OIDC - mirecloud homelab part 5

Mirecloud — Production Engineering

How I Eradicated Static Kubernetes Credentials with Zero Trust OIDC

From the nightmare of admin.conf to Keycloak SSO with centralized RBAC. A complete production architecture explained from A to Z.

By Emmanuel Catin

Kubernetes v1.34 Keycloak (Quarkus) Zero Trust kubelogin v1.35.2

Table of Contents

• 1. The Illusion of Security & The "Ghost" in the Machine
• 2. The Concept: Architecture & Authentication Flow
• 3. Phase 1 — Keycloak Configuration (The Easy Way)
• 4. Phase 2 — Mutating the Kubernetes API Server
• 5. Phase 3 — The RBAC Bridge (Authorization)
• 6. Phase 4 — The Developer Experience (Client Demo)
• 7. Knowledge Base: Troubleshooting
• 8. What's Next: Part 6 — Audit Logging & Observability

1. The Illusion of Security & The "Ghost" in the Machine

Picture the scene: it's late, you're auditing your production Kubernetes cluster logs. You scrutinize the events and notice that every Pod deletion, every ConfigMap modification, and every access to a critical secret is attributed to one single user: kubernetes-admin.

It's a ghost. Is it you? A colleague? The CI/CD pipeline? Or is it a hacker who got their hands on that infamous /etc/kubernetes/admin.conf file someone left in a Slack channel six months ago? It's impossible to tell.

This file is a loaded gun left on the living room table. It grants total cluster-admin access. It never expires. It doesn't require multi-factor authentication (MFA). And crucially, if you need to revoke the access of an ex-employee who has it, you must force the rotation of your cluster's entire PKI (Public Key Infrastructure), potentially causing service interruptions.

The Zero Trust Revolution: The principle is simple: "Never trust, always verify". Instead of handing out a universal master key, every developer must prove who they are via a secure portal. They then receive a very short-lived token, and Kubernetes cryptographically verifies this token on every single request.

The solution? OpenID Connect (OIDC). In this article, I'll show you how I transformed my Mirecloud bare-metal cluster into a Zero Trust fortress by connecting the API Server directly to Keycloak.

2. The Concept: Architecture & Authentication Flow

To understand the magic of this integration, we need to look under the hood. Kubernetes will never handle your passwords. It delegates this task to Keycloak, which returns an unforgeable digital passport: the JWT (JSON Web Token).

How does it work in practice?

flow — authentication sequence

The anatomy of a token: The JWT generated by Keycloak contains "claims". The most important ones for us are the email (which becomes the username in Kubernetes) and the groups (which will dictate permissions).

3. Phase 1 — Keycloak Configuration (The Easy Way)

Prerequisite: I am using Keycloak deployed directly inside my cluster. If you want to see how I built this foundation, check out my previous article: Meet my Homelab Part 3 — Identity & Access.

Kubernetes (via kubelogin) has very strict requirements on how the OIDC client must be configured in Keycloak. Rather than clicking through 20 different menus, I've prepared a complete JSON file that you can import directly.

Why is this JSON file crucial?

• It creates a Public Client ("publicClient": true). This is mandatory, as a command-line interface (CLI) on a dev's machine cannot securely store a "client secret".
• It authorizes redirects to the local loopback (http://localhost:8000/*) required by the kubelogin plugin.
• It configures the Group Mapper with the "full.path": "false" option. This is trap #1! If this isn't done, Keycloak sends the groups formatted as /k8s-admins (with a leading slash) instead of k8s-admins, and Kubernetes RBAC will systematically deny access.

Import the configuration: Grab my keycloak-client-kubernetes.json file, go to the Clients tab in Keycloak, and click Import client.

The Golden Rule: Verified Email

checklist — keycloak user

┌─────────────────────────────────────────────┐
│          Keycloak User Checklist            │
├─────────────────────────────────────────────┤
│  ✅  Email address populated                │
│  ✅  Email verified → ON (MANDATORY)        │
│  ✅  Password configured                    │
└─────────────────────────────────────────────┘

Warning: By default, Keycloak creates users with an unverified email. If you do not toggle the Email Verified switch to ON, the API Server will silently reject the token with a completely mute 401 Unauthorized error.

4. Phase 2 — Mutating the Kubernetes API Server

It's time to operate on the brain of the cluster. We need to tell the API Server (kube-apiserver) to accept tokens signed by Keycloak. On a bare-metal (kubeadm) cluster, this is done in the static manifest.

yaml — /etc/kubernetes/manifests/kube-apiserver.yaml

- command:
  - kube-apiserver
  # ... other parameters ...
  - --oidc-issuer-url=https://keycloak.mirecloud.com/auth/realms/mirecloud
  - --oidc-client-id=kubernetes
  - --oidc-username-claim=email
  - "--oidc-username-prefix=oidc:"
  - --oidc-groups-claim=groups
  - "--oidc-groups-prefix=oidc:"
  - --oidc-ca-file=/etc/kubernetes/pki/keycloak-ca.crt

Explanation of choices: I add the oidc: prefix for security reasons. This ensures that a user named "admin" in Keycloak cannot accidentally (or maliciously) spoof the internal "admin" system account of Kubernetes.

The Deadly YAML Trap: Did you notice the quotes around "--oidc-groups-prefix=oidc:"? In YAML, a word ending with a colon (:) is interpreted as a dictionary key. If you forget the quotes, the YAML parser crashes, and your API Server will never start again.

5. Phase 3 — The RBAC Bridge (Authorization)

Authentication (Keycloak) proves who you are. Authorization (RBAC) defines what you are allowed to do.

Since our API Server now applies the oidc: prefix, we will bind a Keycloak group (e.g., k8s-admins) to a Kubernetes role:

bash — RBAC bindings

# Grants full "cluster-admin" rights to the Keycloak k8s-admins group
kubectl create clusterrolebinding oidc-admins \
  --clusterrole=cluster-admin \
  --group=oidc:k8s-admins

# Grants limited "edit" rights to developers
kubectl create clusterrolebinding oidc-developers \
  --clusterrole=edit \
  --group=oidc:k8s-developers

The Operational Advantage: Now, to grant or revoke access to a developer, you never have to touch Kubernetes again. You simply add or remove them from the group in the Keycloak UI. Does their session expire? Their access is instantly revoked.

6. Phase 4 — The Developer Experience (Client Demo)

This is where the magic happens. On the developer's side (on their Windows or Mac PC), the experience must be seamless. We install kubelogin, an official plugin that acts as the bridge between the command line and the web browser.

1. Installation and Kubeconfig setup

powershell — client workstation

# 1. Installation on Windows
winget install int128.kubelogin

# 2. Configure the user profile to use the OIDC plugin
kubectl config set-credentials oidc-user `
  --exec-api-version=client.authentication.k8s.io/v1beta1 `
  --exec-command=kubectl `
  --exec-arg=oidc-login `
  --exec-arg=get-token `
  --exec-arg=--oidc-issuer-url=https://keycloak.mirecloud.com/auth/realms/mirecloud `
  --exec-arg=--oidc-client-id=kubernetes `
  --exec-arg=--insecure-skip-tls-verify

2. The Real-Time Demo

Here is exactly what happens in the developer's terminal during their first connection of the day:

powershell — Action!

PS C:\WINDOWS\system32> kubectl oidc-login clean
Deleted the token cache from C:\Users\emman\.kube\cache\oidc-login
Deleted the token cache from the keyring

PS C:\WINDOWS\system32> kubectl config set-context oidc-context --cluster=kubernetes --user=oidc-user
Context "oidc-context" modified.

PS C:\WINDOWS\system32> kubectl get node
# =========================================================================
#  At this exact moment, your Web Browser opens automatically!
# It displays the Keycloak login page (https://keycloak.mirecloud.com...)
# The developer enters their Email, Password, and MFA code.
# Once validated, the page displays "Authenticated".
# The terminal instantly unblocks and displays the result below:
# =========================================================================

NAME     STATUS   ROLES           AGE   VERSION
node-2   Ready    <none>          32d   v1.34.2
node-3   Ready    <none>          32d   v1.34.2
node-4   Ready    control-plane   32d   v1.34.2

PS C:\WINDOWS\system32>

The token is now securely cached locally. Subsequent kubectl commands will execute immediately, without reopening the browser, until the session expires.

7. Knowledge Base: Troubleshooting

Because production never goes completely without a hitch, here is my cheat sheet of encountered errors and how to resolve them:

Symptom / Error	Root Cause	Fix
Resource not found on the issuer URL	Incorrect path. Often missing the /auth/ segment.	Verify the discovery URL via /.well-known/openid-configuration. It must match the --oidc-issuer-url flag character for character.
unauthorized_client	The Keycloak client is demanding a secret.	Set Client authentication to Off in Keycloak.
state does not match	Corrupted cache from multiple attempts or leftover open tabs.	Close all browser tabs and run kubectl oidc-login clean.
Silent 401 Unauthorized	YAML comment error in the manifest OR unverified email.	Ensure Email Verified is set to ON. Never place a comment (#) on the same line as a YAML flag.
403 Forbidden (RBAC fails)	Keycloak is sending groups with a leading slash (/k8s-admins).	Make sure the Full group path option is set to OFF in the Keycloak mapper.

8. What's Next: Part 6 — Audit Logging & Observability

We have successfully eradicated the ghost in the machine. Every developer now authenticates with a real identity, backed by MFA, and bounded by strict RBAC rules. But identity without visibility is only half the battle won.

 Coming Soon: Meet my Homelab Part 6

In the next installment of this series, we will transform these raw identities into actionable security intelligence. I will show you how to:

• Configure the Kubernetes Audit Policy to log every single API request.
• Deploy Promtail to parse and ship these JSON logs securely.
• Aggregate the data in Loki and build real-time Grafana Dashboards to monitor exactly who connects, what they execute, and instantly flag 403 Forbidden anomalies.

Stay tuned as we close the loop on our Zero Trust architecture by building a true SecOps observability stack!

---

EC

Emmanuel Catin

Platform Engineer — Mirecloud

Stack: Kubernetes · Keycloak · Vault · Cilium