Skip to main content

Meet my homelab part-2

OIDC the Hard Way: Deploying Keycloak on Bare-Metal Kubernetes

OIDC the Hard Way: Deploying Keycloak on Bare-Metal Kubernetes (Part 2)

Production-grade identity infrastructure: Vault secrets, clustered Keycloak, Gateway API, and zero credentials in Git.

Overview

Part 1 established the foundation: HashiCorp Vault as the single source of truth for credentials, External Secrets Operator bridging Vault into Kubernetes-native Secrets, cert-manager automating TLS certificate lifecycle, and ArgoCD deploying everything declaratively from Git.

Part 2 builds the identity layer on top of that foundation: Keycloak — an open-source identity and access management solution deployed as a production-grade, 2-replica cluster with PostgreSQL persistence, every credential sourced from Vault, and exposed via the Cilium Gateway API with automatic TLS.

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2dba82', 'edgeLabelBackground':'#0d1117', 'tertiaryColor': '#1c2128'}}}%% graph TD %% Styles Définitions classDef git fill:#f34f29,stroke:#fff,stroke-width:2px,color:white classDef k8s fill:#326ce5,stroke:#fff,stroke-width:2px,color:white classDef vault fill:#666,stroke:#fff,stroke-width:2px,color:white classDef app fill:#e0e0e0,stroke:#333,stroke-width:1px,color:#333 classDef user fill:#fff,stroke:#333,stroke-width:2px,color:#333 %% Components User((User)):::user GitRepo[Git Repository
No secrets]:::git subgraph K8S [Kubernetes Cluster] direction TB Gateway[Cilium Gateway
192.168.2.204]:::k8s subgraph MGT [Management] ArgoCD[ArgoCD]:::k8s Vault[("Vault (Secrets)")]:::vault ESO[External Secrets]:::app Secret[K8s Secret]:::app end subgraph APPS [Apps] Keycloak[Keycloak x2]:::app DB[Postgres]:::app end end %% Flux User -->|HTTPS| Gateway Gateway -->|HTTP| Keycloak GitRepo -->|Sync| ArgoCD ArgoCD -->|Deploy| Keycloak ArgoCD -->|Config| ESO ESO -->|Auth & Pull| Vault ESO -->|Create| Secret Secret -.->|Inject| Keycloak Keycloak <--> DB

By the end of this article, you will have:

  • A highly available Keycloak cluster with distributed session state via Infinispan
  • PostgreSQL backend for persistent storage
  • Admin and database credentials managed entirely through Vault
  • TLS certificates issued and renewed automatically by cert-manager
  • External access via Cilium Gateway API with proper proxy header handling
  • Zero secrets visible in Git — complete GitOps compliance

Why Keycloak?

Keycloak provides enterprise-grade identity and access management with support for:

  • OpenID Connect (OIDC) and SAML 2.0 protocols
  • User Federation with LDAP, Active Directory, Kerberos
  • Social Login (Google, GitHub, etc.)
  • Multi-factor Authentication (TOTP, WebAuthn)
  • Fine-grained Authorization with role-based and attribute-based access control
  • Centralized Session Management across multiple applications

In a homelab or small enterprise environment, Keycloak eliminates the need to manage separate user databases in every application. Every service delegates authentication to Keycloak. Add a user once, grant them access to multiple services through realm roles. Revoke access in one place when they leave.

Prerequisites

The following components must be operational before proceeding:

  • Vault initialized, unsealed, and Kubernetes auth configured
  • ClusterSecretStore named vault-backend in Valid state
  • cert-manager operational with ClusterIssuer mirecloud-ca-issuer in Ready state
  • ArgoCD connected to the repository

Verify:

kubectl get clustersecretstore vault-backend
# NAME            AGE   STATUS   CAPABILITIES   READY
# vault-backend   2d    Valid    ReadWrite      True

kubectl get clusterissuer mirecloud-ca-issuer
# NAME                  READY   AGE
# mirecloud-ca-issuer   True    2d

Layer 0 — PostgreSQL

Keycloak requires a relational database for persistent storage of realms, users, sessions, and configuration. The embedded H2 database is explicitly unsupported in clustered deployments and should never be used outside of local development.

PostgreSQL is deployed as a StatefulSet in a dedicated namespace:

helm install postgres oci://registry-1.docker.io/cloudpirates/postgres \
  -n postgres --create-namespace

The chart generates a random password on first install and stores it in a Kubernetes Secret. This is the one time this credential is handled manually:

# Retrieve the generated password
kubectl -n postgres get secret postgres \
  -o jsonpath='{.data.postgres-password}' | base64 -d

# Store it in Vault immediately
kubectl -n vault exec -ti vault-0 -- vault kv put secret/keycloak/db \
    password='<retrieved-password>'

Layer 1 — Secrets in Vault

Every sensitive value is stored in Vault before any Kubernetes manifest is applied. This is the contract the entire pipeline depends on.

# Keycloak admin account
kubectl -n vault exec -ti vault-0 -- vault kv put secret/keycloak/admin \
    password='StrongAdminPassword'

# Keycloak database credentials
kubectl -n vault exec -ti vault-0 -- vault kv put secret/keycloak/db \
    password='DbPassword'

Two paths. Two credentials. Zero Git commits.

Layer 2 — External Secrets

The ClusterSecretStore from Part 1 is already in place. Two ExternalSecret resources declare which Vault paths to sync and what Kubernetes Secret objects to create. 

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: keycloak-admin-es
  namespace: keycloak
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: keycloak-admin-password
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: secret/keycloak/admin
      property: password

Critical: The remoteRef.key path. ESO handles KV v2 path construction internally — it appends /data/ to the path when calling the Vault API. If you include /data/ in the key yourself, the resulting path becomes /data/data/secret/keycloak/admin, which returns a 404.

Layer 3 — Keycloak Helm Chart

Values Configuration 

keycloakx:
  command:
    - "/opt/keycloak/bin/kc.sh"
    - "start"
    - "--http-enabled=true"
    - "--http-port=8080"
    - "--hostname-strict=false"
    - "--proxy-headers=xforwarded"

  replicas: 2

  extraEnv: |
    - name: KEYCLOAK_ADMIN
      value: admin
    - name: KEYCLOAK_ADMIN_PASSWORD
      valueFrom:
        secretKeyRef:
          name: keycloak-admin-password
          key: password
    - name: KC_PROXY
      value: edge

  dbchecker:
    enabled: true

  database:
    vendor: postgres
    hostname: postgres.postgres.svc
    port: 5432
    database: postgres
    username: postgres
    existingSecret: byo-db-creds

Configuration Rationale

--proxy-headers=xforwarded instructs Keycloak to respect X-Forwarded-For and X-Forwarded-Proto headers injected by the upstream Gateway. Without this flag, Keycloak ignores the forwarded headers and constructs redirect URIs based on what it sees directly — which is plain HTTP on port 8080.

KC_PROXY=edge is the complementary setting. It tells Keycloak it operates behind a TLS-terminating reverse proxy and should accept forwarded headers as authoritative. These two flags are paired — neither is sufficient without the other when TLS is terminated at the Gateway.

Layer 4 — Cilium Gateway API and TLS

This deployment uses the Kubernetes Gateway API rather than the legacy Ingress resource. Gateway API provides cleaner separation between infrastructure concerns (Gateway, GatewayClass) and application routing concerns (HTTPRoute).

Object Managed By Responsibility
Certificate cert-manager Issues and renews TLS certificate from internal CA
Gateway Cilium Binds a LoadBalancer IP, terminates TLS, forwards HTTP internally
HTTPRoute Cilium Maps keycloak.mirecloud.com to the Keycloak service

Distributed Sessions: What Infinispan Provides

With replicas: 2, the keycloakx chart automatically configures Keycloak nodes to form a distributed Infinispan cache cluster. Pod discovery uses the headless Kubernetes service, which exposes individual pod addresses directly rather than load-balancing across them.

The clustering lifecycle is visible in the Keycloak logs:

ISPN100002: Starting rebalance with members
             [keycloak-keycloakx-0-35812, keycloak-keycloakx-1-53189],
             phase READ_OLD_WRITE_ALL, topology id 2
ISPN100010: Finished rebalance with members
             [keycloak-keycloakx-0-35812, keycloak-keycloakx-1-53189],
             topology id 5

Once complete, both nodes share distributed session state. A user authenticated through pod-0 can have their request served by pod-1 without being prompted to log in again.

Security Posture

At the completion of this deployment:

  • No credential appears in Git in any form — no base64, no Helm --set flags, no inline stringData
  • Vault is the authoritative source for every sensitive value in the cluster
  • TLS is enforced on all external endpoints, certificates issued and renewed automatically by cert-manager
  • ExternalSecrets refresh every 60 seconds — a rotated Vault secret propagates to Kubernetes within one minute
  • Session state is distributed across Keycloak replicas — no single point of failure

What's Next: Part 3

Keycloak is now operational as a standalone identity server. The next step is integrating it with an actual application to provide SSO.

Part 3 will cover the complete OIDC integration with Grafana, including:

  • The OpenID Connect Authorization Code Flow (with diagram)
  • Front-channel vs. back-channel URL configuration
  • Client secret management via Vault and ESO
  • Role mapping from Keycloak realm roles to Grafana permissions
  • Eliminating the Grafana native login form entirely

The complete repository is available at github.com/mirecloud/home_lab.

Emmanuel Catin — Senior Platform Engineer | Kubernetes, GitOps, Zero Trust
CKA | Montréal, QC

#Kubernetes #Keycloak #GitOps #Vault #ExternalSecrets #CiliumGateway #GatewayAPI #PostgreSQL #Infinispan #DevSecOps #HomeLab #PlatformEngineering #ZeroTrust

Comments

Post a Comment

Popular posts from this blog

FastAPI Instrumentalisation with prometheus and grafana Part1 [Counter]

welcome to this hands-on lab on API instrumentation using Prometheus and FastAPI! In the world of modern software development, real-time API monitoring is essential for understanding usage patterns, debugging issues, and ensuring optimal performance. In this lab, we’ll demonstrate how to enhance a FastAPI-based application with Prometheus metrics to monitor its behavior effectively. We’ve already set up the lab environment for you, complete with Grafana, Prometheus, and a PostgreSQL database. While FastAPI’s integration with databases is outside the scope of this lab, our focus will be entirely on instrumentation and monitoring. For those interested in exploring the database integration or testing , you can review the code in our repository: FastAPI Monitoring Repository . What You’ll Learn In this lab, we’ll walk you through: Setting up Prometheus metrics in a FastAPI application. Instrumenting API endpoints to track: Number of requests HTTP methods Request paths Using Grafana to vi...
Post a Comment
Read more

Join Ubuntu 20.04 to Active Directory with SSSD and SSH Access

Join Ubuntu 20.04 to Active Directory with SSSD and SSH Access  Overview This guide walks you through joining an Ubuntu 20.04 machine to an Active Directory domain using SSSD, configuring PAM for AD user logins over SSH, and enabling automatic creation of home directories upon first login. We’ll also cover troubleshooting steps and verification commands. Environment Used Component Value Ubuntu Client       ubuntu-client.bazboutey.local Active Directory FQDN   bazboutey.local Realm (Kerberos)   BAZBOUTEY.LOCAL AD Admin Account   Administrator Step 1: Prerequisites and Package Installation 1.1 Update system and install required packages bash sudo apt update sudo apt install realmd sssd libnss-sss libpam-sss adcli \ samba-common-bin oddjob oddjob-mkhomedir packagekit \ libpam-modules openssh-server Step 2: Test DNS and Kerberos Configuration Ensure that the client can resolve the AD domain and discover services. 2.1 Test domain name resol...
Post a Comment
Read more

Observability with grafana and prometheus (SSO configutation with active directory)

How to Set Up Grafana Single Sign-On (SSO) with Active Directory (AD) Grafana is a powerful tool for monitoring and visualizing data. Integrating it with Active Directory (AD) for Single Sign-On (SSO) can streamline access and enhance security. This tutorial will guide you through the process of configuring Grafana with AD for SSO. Prerequisites Active Directory Domain : Ensure you have an AD domain set up. Domain: bazboutey.local AD Server IP: 192.168.170.212 Users: grafana (for binding AD) user1 (to demonstrate SSO) we will end up with a pattern like this below Grafana Installed : Install Grafana on your server. Grafana Server IP: 192.168.179.185 Administrator Privileges : Access to modify AD settings and Grafana configurations. Step 1: Configure AD for LDAP Integration Create a Service Account in AD: Open Active Directory Users and Computers. Create a user (e.g., grafana ). Assign this user a strong password (e.g., Grafana 123$ ) and ensure it doesn’t expire. Gather Required AD D...
Post a Comment
Read more