I Was kubectl apply-ing Everything.
Here's How I Stopped.
Building MireCloud — the right way, from the ground up.
I have a confession.
For months, my homelab was held together with notes, memory, and hope.
Keycloak was running. Grafana was up. GitLab was accessible. But if you asked me why something worked, half the time the honest answer was: "because I ran some commands six weeks ago and I haven't touched it since."
Passwords lived in a notes file. Certificates were generated once with OpenSSL and forgotten until they expired. Secrets were committed to Git — sometimes as plaintext, sometimes base64-encoded, which is the same thing with extra steps. Every rebuild started with 30 minutes of archaeology through old terminal sessions asking: "What was that Keycloak admin password again? Which node has the CA key?"
This is the story of how I fixed that. Not by being more careful. Not by taking better notes. By building infrastructure that doesn't need me to remember anything.
This is Part 1 of the MireCloud series — the foundation. By the end of this article, you'll understand the secret and certificate pipeline that runs underneath everything else. In Part 2, I'll show you how Keycloak gets deployed on top of it: SSO, OIDC, database credentials from Vault, TLS from cert-manager — all GitOps-native, zero secrets in Git.
But you can't tell that story without telling this one first.
01 — What I Already Had
Before I started fixing things, I had a working multi-node Kubernetes cluster running at home. Four nodes, bare metal, Ubuntu. No cloud provider. No managed services. Everything from scratch.
The networking layer was already solid:
- Cilium as the CNI — full eBPF mode, kube-proxy completely replaced, no iptables anywhere
- Cilium L2 announcements handling LoadBalancer IPs natively via CiliumLoadBalancerIPPool and CiliumL2AnnouncementPolicy — no MetalLB needed
- ArgoCD managing deployments from Git
That last part is important. I had already committed to GitOps. Every service was supposed to be declarative. Every change was supposed to go through Git.
The problem was: I was cheating on secrets.
Everything structural was in Git. But every credential was applied manually — kubectl create secret, helm install --set password=..., copy-paste from a notes file. The GitOps principle was there in spirit. The execution had a gaping hole.
02 — The Design Decision
Before writing a single line of YAML, I made one rule:
If this cluster burns down tomorrow, git clone + ArgoCD sync should give me everything back. No manual steps. No notes. No memory required.
That rule forced three questions:
- Where do secrets live if not in Git? → Vault
- How do secrets get from Vault into Kubernetes? → External Secrets Operator
- Who signs and renews TLS certificates? → cert-manager
And a fourth question most tutorials skip: how does all of this get deployed itself without manual helm install commands? The answer: ArgoCD. The infrastructure deploys the infrastructure.
03 — The Repo Structure
Everything lives in github.com/mirecloud/home_lab. The structure is deliberate:
home_lab/ ├── clusters/ │ └── home-lab/ ← one ArgoCD Application per service │ ├── vault-app.yaml │ ├── external-secrets-app.yaml │ ├── external-secrets-config-app.yaml │ ├── cert-manager-app.yaml │ └── ... ├── infrastructure/ ← Helm wrapper charts, platform │ ├── vault/ │ ├── external-secrets/ │ ├── external-secrets-config/ │ └── cert-manager/ └── apps/ ← Helm wrapper charts, applications ├── keycloak/ ├── gitlab/ └── ...
The pattern: every deployable unit is a Helm wrapper chart — a thin Chart.yaml declaring an upstream dependency, plus a values.yaml with your overrides. Nothing is deployed with helm install. ArgoCD reads the Application manifest, finds the chart, deploys it.
apiVersion: v2 name: vault version: 1.0.0 dependencies: - name: vault version: 0.31.0 repository: https://helm.releases.hashicorp.com
Same pattern repeats for cert-manager, ESO, Keycloak, GitLab, Grafana — everything. One consistent mental model. No helm upgrade. No state drift between what's running and what Git says.
04 — Layer 1 — Vault: The Single Source of Truth
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: vault namespace: argocd spec: source: repoURL: "git@github.com:mirecloud/home_lab.git" targetRevision: HEAD path: infrastructure/vault destination: server: https://kubernetes.default.svc namespace: vault syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ignoreDifferences: - group: admissionregistration.k8s.io kind: MutatingWebhookConfiguration jsonPointers: - /webhooks/0/clientConfig/caBundle
Why ignoreDifferences? Vault's injector auto-generates a webhook certificate on every pod restart. Without this block, ArgoCD shows Vault as OutOfSync forever and constantly tries to revert it — noise with no effect.
vault: server: dev: enabled: false # never dev mode ha: enabled: true replicas: 1 raft: enabled: true config: | ui = true listener "tcp" { tls_disable = 1 address = "[::]:8200" } storage "raft" { path = "/vault/data" } service_registration "kubernetes" {}
After ArgoCD deploys it, one manual bootstrap — intentionally the only manual step in the entire pipeline:
# Initialize — save unseal keys in a password manager, not Git kubectl -n vault exec -ti vault-0 -- vault operator init vault auth enable kubernetes vault write auth/kubernetes/config \ kubernetes_host="https://kubernetes.default.svc:443" \ disable_iss_validation=true vault policy write vault-backend - <<EOF path "secret/*" { capabilities = ["create", "read", "update", "delete", "list"] } EOF # Bind ESO ServiceAccount to the policy vault write auth/kubernetes/role/vault-backend \ bound_service_account_names=external-secrets \ bound_service_account_namespaces=external-secrets \ policies=vault-backend \ ttl=24h
That last command is the critical link. Vault trusts the ESO ServiceAccount JWT for auth. No static tokens. No passwords. Kubernetes rotates the token automatically.
vault kv put secret/keycloak/admin password='StrongAdminPassword' vault kv put secret/keycloak/db password='DbPassword' vault kv put secret/grafana/sso client_secret='OIDCClientSecret'
05 — Layer 2 — External Secrets Operator: The Bridge
ESO watches for ExternalSecret CRDs and materializes them into native Kubernetes Secret objects. The CRDs are safe to commit — zero sensitive data, only references to paths in Vault.
I split this layer into two separate ArgoCD Applications. That split prevents a race condition that will silently break your cluster.
Part A — The operator
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: external-secrets namespace: argocd spec: source: repoURL: "git@github.com:mirecloud/home_lab.git" targetRevision: HEAD path: infrastructure/external-secrets destination: server: https://kubernetes.default.svc namespace: external-secrets syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true - ServerSideApply=true # required — ESO CRDs are large - Replace=true
external-secrets: fullnameOverride: "external-secrets" # must match Vault role binding installCRDs: true webhook: create: true certManager: enabled: false
fullnameOverride is not optional. It controls the ServiceAccount name ESO creates. If it doesn't exactly match external-secrets — the name bound in the Vault role — every secret sync silently fails with a 403. This one cost me a few hours.
Part B — The configuration
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: external-secrets-config namespace: argocd spec: source: repoURL: "git@github.com:mirecloud/home_lab.git" targetRevision: HEAD path: infrastructure/external-secrets-config destination: server: https://kubernetes.default.svc namespace: external-secrets syncPolicy: automated: prune: true selfHeal: true
apiVersion: external-secrets.io/v1 kind: ClusterSecretStore metadata: name: vault-backend spec: provider: vault: server: http://vault.vault.svc.cluster.local:8200 path: secret version: v2 auth: kubernetes: mountPath: kubernetes role: vault-backend serviceAccountRef: name: external-secrets namespace: external-secrets
Why two Applications? The ClusterSecretStore CRD is installed by Part A. Bundle both together and ArgoCD tries to create the object before the CRD exists — permanent failure. Separate Applications: sync Part A, wait for Healthy, then sync Part B.
Always use vault.vault.svc.cluster.local — never vault-active. During Raft leader elections, the active endpoint briefly disappears. The stable service never does.
06 — Layer 3 — cert-manager: Certificates on Autopilot
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: cert-manager namespace: argocd spec: source: repoURL: "git@github.com:mirecloud/home_lab.git" targetRevision: HEAD path: infrastructure/cert-manager destination: server: https://kubernetes.default.svc namespace: cert-manager syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true - ServerSideApply=true
The CA private key lives in Vault. ESO injects it into cert-manager's namespace via an ExternalSecret:
apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: mirecloud-ca-es namespace: cert-manager annotations: argocd.argoproj.io/sync-wave: "-2" # inject CA before ClusterIssuer spec: refreshInterval: "1m" secretStoreRef: name: vault-backend kind: ClusterSecretStore target: name: mirecloud-ca-key-pair creationPolicy: Owner template: type: kubernetes.io/tls data: - secretKey: tls.crt remoteRef: key: mirecloud/ca property: tls.crt - secretKey: tls.key remoteRef: key: mirecloud/ca property: tls.key
sync-wave: "-2" is essential. ArgoCD applies this ExternalSecret before anything else in the Application — the CA secret must exist before the ClusterIssuer references it. Without it, cert-manager enters a failed state and stays there until you manually resync.
apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: mirecloud-ca-issuer spec: ca: secretName: mirecloud-ca-key-pair
One issuer for the whole cluster. Any namespace can now request a signed, auto-renewing certificate. No OpenSSL. No manual copy-paste. No calendar reminder.
07 — The Deployment Order That Actually Works
selfHeal: true on every Application means ArgoCD continuously reconciles cluster state with Git. Someone manually changes something? Reverted. The cluster always converges back to what Git says.
08 — What This Feels Like Now
# 1. Store the credential in Vault — Git never sees it vault kv put secret/newapp/db password='Value' # 2. Create the wrapper chart # infrastructure/newapp/Chart.yaml ← declare upstream dependency # infrastructure/newapp/values.yaml ← your overrides # 3. Declare what secrets to inject — no sensitive data in Git # apps/newapp/templates/external-secret.yaml # 4. Request a certificate — 3 lines referencing mirecloud-ca-issuer # apps/newapp/templates/certificate.yaml # 5. Register with ArgoCD # clusters/home-lab/newapp-app.yaml # 6. Push. Done. git push
No kubectl create secret. No helm install. No OpenSSL. No notes. No archaeology.
The cluster knows what to do.
09 — The Honest Part
The initial setup takes time. Vault initialization is intentionally manual — you should be present for it, save the unseal keys in a real password manager, and run the Kubernetes auth config yourself. That's appropriate for something that holds every credential in your cluster.
The fullnameOverride quirk, the ServerSideApply requirement, the sync wave on cert-manager, the ignoreDifferences on Vault — none of these are prominently documented. They're the things you find by staring at a 403 or a permanently stuck ArgoCD sync. I'm writing them down so you don't have to find them the hard way.
But they're one-time costs.
Every service I've added since — and there are several — follows the same pattern. The infrastructure layer stops being something I think about. It just works.
The full repo is at github.com/mirecloud/home_lab. Questions, feedback, or war stories about your own secrets setup — drop them in the comments.
Comments
Post a Comment