Meet my HomeLab Part-3

PIDC the Hard Way: Integrating Grafana with Keycloak via OIDC (Part 3)

Eliminating password databases: OpenID Connect, front-channel vs. back-channel, role mapping, and the end of local authentication.

Overview

Parts 1 and 2 built the foundation: Vault manages all credentials, External Secrets Operator bridges them into Kubernetes, cert-manager automates TLS, and Keycloak runs as a production-grade identity provider with clustered session state.

Part 3 is where that infrastructure proves its value: integrating Grafana with Keycloak via OpenID Connect to eliminate Grafana's native login form entirely. By the end, there is no Grafana password database. No local admin account. Every login redirects to Keycloak, authenticates against the central identity layer, and maps realm roles to Grafana permissions automatically.

The deliverables:

• Understanding the OIDC Authorization Code Flow
• Configuring Keycloak as an Identity Provider (IdP)
• Configuring Grafana as a Relying Party (RP)
• Managing the client secret through Vault and ESO
• Front-channel vs. back-channel URL configuration (the detail most guides get wrong)
• Role mapping via JMESPath expressions

A Primer on OpenID Connect

Before diving into YAML, it is worth understanding what OpenID Connect actually does — because every configuration decision that follows is a direct consequence of how the protocol works.

The Problem It Solves

Without SSO, every service in your cluster has its own user database, its own password policy, its own session management. Add a user, you add them five times. Rotate a password, you rotate it five times. An employee leaves, you hope you remembered to revoke access in all five places.

OpenID Connect (OIDC) is an identity layer built on top of OAuth 2.0. It defines a standard protocol by which an application (the Relying Party, e.g., Grafana) can delegate authentication to a trusted external service (the Identity Provider, e.g., Keycloak). The application never handles passwords. It only receives a verified identity token.

The Authorization Code Flow

This is the flow used by Grafana when a user attempts to log in:

Step-by-step breakdown:

1. User navigates to Grafana → GET /
2. Grafana redirects to Keycloak → 302 with auth_url
3. Browser follows redirect to Keycloak → GET /auth/realms/mirecloud/protocol/openid-connect/auth
4. Keycloak renders login form → User sees username/password fields
5. User submits credentials → POST to Keycloak (Grafana never sees this)
6. Keycloak redirects back to Grafana → 302 with code=AUTH_CODE
7. Browser follows redirect to Grafana callback → GET /login/generic_oauth?code=...

From here, the flow switches to back-channel (server-to-server, no browser involved):

8. Grafana exchanges code for tokens (back-channel) → POST /token
9. Keycloak returns tokens → { access_token, id_token }
10. Grafana requests user info (back-channel) → GET /userinfo
11. Keycloak returns user claims → { sub, email, realm_access.roles }
12. Grafana creates session → Sets grafana_session cookie

Front-Channel vs. Back-Channel

The diagram reveals a critical distinction that most tutorials ignore:

Front-channel calls travel through the user's browser as HTTP redirects. The auth_url is a front-channel URL — the browser navigates to it directly. It must be publicly reachable: https://keycloak.mirecloud.com/...

Back-channel calls are made directly between Grafana's pod and Keycloak's pod, inside the Kubernetes cluster. The browser is not involved. These are the token exchange (token_url) and user info (api_url) calls.

This is why token_url in the Grafana configuration uses the internal Kubernetes service DNS name (keycloak-keycloakx-http.keycloak.svc.cluster.local) rather than the public hostname.

Key Concepts

Term	Role	In this deployment
Identity Provider (IdP)	Authenticates users, issues tokens	Keycloak
Relying Party (RP)	Delegates auth to the IdP	Grafana
Client ID	Identifies the RP to the IdP	grafana
Client Secret	Shared secret between RP and IdP	Stored in Vault
Realm	Isolated tenant in Keycloak	mirecloud

Step 1 — Configure Keycloak (One-Time Setup)

Create a Realm

Navigate to the Keycloak admin console → Create Realm.

• Realm name: mirecloud
• Enabled: true

Create a Client for Grafana

Inside the mirecloud realm, navigate to Clients → Create Client.

General Settings:

• Client type: OpenID Connect
• Client ID: grafana

Capability config:

• Client authentication: ON
• Authentication flow: Enable "Standard flow"

Login settings:

• Valid redirect URIs: https://grafana.mirecloud.com/login/generic_oauth

Retrieve the Client Secret

Navigate to Clients → grafana → Credentials tab. Copy the Client Secret and store it in Vault:

kubectl -n vault exec -ti vault-0 -- vault kv put secret/grafana/sso \
    client_secret='<client-secret-from-keycloak-ui>'

Step 2 — ExternalSecret for Grafana

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: grafana-keycloak-es
  namespace: monitoring
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: grafana-keycloak-secret
  data:
  - secretKey: client_secret
    remoteRef:
      key: secret/grafana/sso
      property: client_secret

Step 3 — Grafana OIDC Configuration

kube-prometheus-stack:
  grafana:
    enabled: true
    envFromSecret: grafana-keycloak-secret

    grafana.ini:
      auth.generic_oauth:
        enabled: true
        name: "Keycloak"
        client_id: "grafana"
        client_secret: $__env{client_secret}

        # Front-channel URL (browser navigates here)
        auth_url: "https://keycloak.mirecloud.com/auth/realms/mirecloud/protocol/openid-connect/auth"

        # Back-channel URLs (pod-to-pod)
        token_url: "http://keycloak-keycloakx-http.keycloak.svc.cluster.local:80/auth/realms/mirecloud/protocol/openid-connect/token"
        api_url: "http://keycloak-keycloakx-http.keycloak.svc.cluster.local:80/auth/realms/mirecloud/protocol/openid-connect/userinfo"

        scopes: "openid profile email"
        allow_sign_up: true

        # Role mapping
        role_attribute_path: "contains(realm_access.roles[*], 'admin') && 'Admin' || 'Viewer'"

Configuration Breakdown

auth_url uses the public DNS name: https://keycloak.mirecloud.com/...

token_url and api_url use internal cluster DNS to avoid DNS hairpin issues in homelab environments.

Test the OIDC Flow

Navigate to https://grafana.mirecloud.com.

Click Sign in with Keycloak.

The browser redirects to Keycloak. Enter credentials for a user in the mirecloud realm.

Grafana exchanges the authorization code for tokens (back-channel, invisible to you) and creates a session.

You land on the Grafana dashboard. Your role (Admin or Viewer) is determined by the admin realm role assignment.

Security Posture

• No Grafana password database — all authentication delegated to Keycloak
• Client secret managed through Vault and ESO — never visible in Git
• OIDC tokens transmitted securely (TLS on front-channel, internal service mesh for back-channel)
• Role assignment driven by Keycloak realm roles — access control changes do not require Grafana restarts

What's Next: Part 4

Part 4 will cover GitLab OIDC configuration with discovery: false, explicit OAuth endpoint definition, and CA injection.

The complete repository is available at github.com/mirecloud/home_lab.

Emmanuel Catin — Senior Platform Engineer | Kubernetes, GitOps, Zero Trust
CKA (90%) | CKS in preparation | Montréal, QC

#Kubernetes #OIDC #Keycloak #Grafana #SSO #OpenIDConnect #GitOps #Vault #ExternalSecrets #DevSecOps #HomeLab #PlatformEngineering #ZeroTrust