Where We Left Off
Part 8 closed two concrete threats. First, encryption at rest — an attacker who steals a disk or exfiltrates an etcd backup gets nothing useful. Second, the Vault CSI Driver — Kubernetes Secrets no longer exist at all, so anyone with kubectl get secrets sees an empty list.
Those were relatively clean wins. The threat we deferred at the end of Part 7 and 8 is harder:
A3 — Compromised workload. Pod exec, sidecar abuse, mounted ServiceAccount token.
What makes A3 uncomfortable is that it operates inside the perimeter we just built. The attacker doesn't need to touch etcd. They don't need cluster-admin. They just need a shell — and shells appear through CVEs, deserialization bugs, dependency compromises, things that happen to production workloads regardless of how well the infrastructure is configured.
This article is about what you do after that shell opens.
The State of the Cluster
After Part 8, secrets flow like this:
Vault ──► CSI Driver ──► /mnt/secrets/ (tmpfs) ──► Application Pod
│
└── No Kubernetes Secret created.
No etcd entry. No backup exposure.
A1 ✓ A2 ✓The application reads credentials directly from the tmpfs mount on every request:
# app.py
def get_db_credentials():
with open("/mnt/secrets/db-username", "r") as f:
user = f.read().strip()
with open("/mnt/secrets/db-password", "r") as f:
password = f.read().strip()
return user, passwordReading from disk on every request is intentional — the credential rotates every 5 minutes, and this pattern ensures the app always uses the current one without any restart or signal handling. From the outside, this looks solid. It has a problem.
Part 1 — What Rotation Doesn't Fix
Here's the thing about a 5-minute TTL: it doesn't matter if the attacker reads the file before the rotation happens.
kubectl -n test-dynamic exec -it deployment/secure-test-app -- /bin/bash
appuser@secure-test-app-5dc5d9f97d-qhwh2:/app$ cat /mnt/secrets/db-password
gXH2tx-vLIQ5AcFL7gPDThat's it. One command. The Vault audit log shows a normal credential issuance. The CSI Driver mount is readOnly: true. Nothing in kubectl get events moves. And the attacker has a working PostgreSQL credential with up to 300 seconds on the clock — enough to dump the public schema, exfiltrate over HTTPS, or plant something that outlives the credential.
The rotation limits the blast radius. It doesn't prevent the read.
Where the CSI Driver's responsibility ends
The CSI Driver's job is to answer one question: is this pod allowed to receive this secret? Once the answer is yes, the credential crosses into the pod's mount namespace and the CSI Driver is done. What happens inside the pod is not its concern.
Inside the pod, the filesystem has no idea who's reading it. python app.py opening /mnt/secrets/db-password to serve a request and an attacker opening the same file during an intrusion look identical at the syscall level. Same openat. Same path. Same bytes returned.
This isn't a bug in Vault. It isn't a bug in the CSI Driver. It's just where those tools stop and something else needs to start.
Part 2 — It's Not Just cat
When most people think about detecting a credential read, they think about flagging cat. That's a reasonable starting point — but in practice, cat is probably the last tool a real attacker reaches for.
The image mirecloud/vaultest:3.0 is minimal by design. A lot of the usual suspects aren't installed — less, strings, xxd, curl, pgrep all come back as command not found. But what's left is more than enough. Here's what we tested directly on the running pod:
appuser@secure-test-app:/app$ cat /mnt/secrets/db-password
gXH2tx-vLIQ5AcFL7gPD
appuser@secure-test-app:/app$ head /mnt/secrets/db-password
gXH2tx-vLIQ5AcFL7gPD
appuser@secure-test-app:/app$ dd if=/mnt/secrets/db-password
gXH2tx-vLIQ5AcFL7gPD
0+1 records in / 0+1 records out / 20 bytes copied
appuser@secure-test-app:/app$ awk '{print}' /mnt/secrets/db-password
gXH2tx-vLIQ5AcFL7gPD
appuser@secure-test-app:/app$ sed '' /mnt/secrets/db-password
gXH2tx-vLIQ5AcFL7gPD
appuser@secure-test-app:/app$ sh -c "cat /mnt/secrets/db-password"
t7m8AxyuKS8j-zeG4nYu ← rotated between attempts, pod never restarted
appuser@secure-test-app:/app$ python3 -c "print(open('/mnt/secrets/db-password').read())"
t7m8AxyuKS8j-zeG4nYuSeven methods. The credential rotated between some of these — the pod didn't restart once. Every attempt succeeded.
The python3 entry is the one that should make you nervous. It's the same binary that runs the application. There's no filesystem-level way to distinguish python app.py handling a request from python3 -c "print(open(...).read())" run by someone who just exec'd into the pod. They both open the same file. They both succeed.
There's also the application vector — if the Flask app exposes a debug endpoint, or if an exception handler leaks local variable state into a response, the credential surfaces over HTTP without any shell access at all. It's worth mentioning because it's the kind of thing that gets overlooked when the infrastructure hardening is solid but the application isn't.
Part 3 — Falco: At Least We Can See It
Falco runs as a DaemonSet and instruments the Linux kernel via eBPF. Every syscall that matches a rule produces an alert — timestamped, with full process and container context. It doesn't prevent anything. But knowing something happened, exactly when, exactly how, is the foundation of incident response.
How Falco actually hooks in
It helps to know where in the execution path Falco fires. When a process calls open():
userspace process calls open()
│
▼
glibc / syscall wrapper
│
▼
syscall enters kernel ◄── Falco eBPF probe fires here
│ alert written before return
▼
VFS resolves the path
│
▼
file descriptor returned to callerFalco fires at kernel entry, before the file descriptor comes back. The alert is written before the attacker has the data in hand. The call still completes — Falco can't stop it — but the record is already there.
This is the key difference between Falco and Tetragon, which we'll get to in Part 9. Both hook at the syscall boundary. Only Tetragon can deliver a SIGKILL before the return.
Starting simple
cat <<EOF > mirecloud-falco-rules.yaml
customRules:
mirecloud_rules.yaml: |-
- rule: MireCloud Intruder Activity
desc: Detect any unauthorized read in the test-dynamic namespace
condition: >
open_read
and fd.name startswith "/mnt/secrets/"
and container
and k8s.ns.name = "test-dynamic"
output: "ALERTE MIRECLOUD: Unauthorized command execution (user=%user.name pod=%k8s.pod.name ns=%k8s.ns.name cmd=%proc.cmdline)"
priority: CRITICAL
EOF
helm upgrade falco falcosecurity/falco \
-n falco \
--reuse-values \
-f mirecloud-falco-rules.yaml
kubectl -n falco get all -wWhat the live cluster produced
One cat /mnt/secrets/db-password from inside the pod:
17:57:59.315968470: Critical ALERTE MIRECLOUD: Unauthorized command execution
(user=appuser pod=secure-test-app-5dc5d9f97d-qhwh2 ns=test-dynamic
cmd=cat /mnt/secrets/db-password) container_id=9c45982d256e
container_image_repository=docker.io/mirecloud/vaultest
container_image_tag=3.0
17:57:59.315990847: Critical ALERTE MIRECLOUD: [...]
[...]
17:57:59.316724451: Critical ALERTE MIRECLOUD: [...]16 events in 758 microseconds. That's not 16 separate reads — it's Falco capturing each individual openat syscall that cat emits while reading the file. Sub-millisecond granularity, nanosecond timestamps, container ID, image tag.
Two things stood out. First, user=appuser — not root. Falco doesn't need to see root to fire. An attacker who gets a shell as appuser gets detected as cleanly as one who escalates to root. Second, container_image_tag=3.0 — if this intrusion came through a compromised dependency in a specific image version, that version is stamped on every single alert.
The gap — proved on the live cluster
The rule catches anything that opens /mnt/secrets/. But it has to allow python3, because the application is python3. So we added proc.name = "python3" to the whitelist. Then we ran this:
appuser@secure-test-app:/app$ python3 -c "print(open('/mnt/secrets/db-password').read())"
nZrram3e3xA39rYPQk-ZAnd checked the logs:
kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco \
--tail=20 | grep "ALERTE MIRECLOUD"
17:57:59.316286789: Critical ALERTE MIRECLOUD: [old cat attempt]
17:57:59.316724451: Critical ALERTE MIRECLOUD: [old cat attempt]Nothing new. The credential nZrram3e3xA39rYPQk-Z was read and printed to the terminal. Falco saw nothing. Whitelisting the binary name creates a blind spot the attacker can walk straight through. We need a tighter anchor.
Part 4 — The Production Ruleset
The fix is to whitelist on the command line, not just the binary name. Our application starts as python app.py — that's PID 1 in the container. An attacker who exec's in and runs python3 -c "..." has a completely different command line. Same binary, different invocation. We can tell them apart.
A note on cmdline vs process ancestry
proc.cmdline startswith "python app.py" is effective for the standard kubectl exec intrusion scenario. Its limitation is that cmdline is user-controllable — a determined attacker with enough persistence could launch a process with the right name to match the filter. In practice this requires a level of access that goes well beyond a simple shell.
The stronger approach is process ancestry — checking that the calling process descends from PID 1. Falco's support for this is limited. Tetragon does it natively via matchProcessAncestors, which is part of why Part 9 matters: the Tetragon layer uses the actual kernel process tree rather than a string heuristic. For the Falco layer, cmdline filtering is the right tradeoff.
cat <<EOF > mirecloud-falco-rules.yaml
customRules:
mirecloud_rules.yaml: |-
- macro: mirecloud_secret_path
condition: >
fd.name startswith "/mnt/secrets/"
- macro: mirecloud_authorized_procs
condition: >
proc.name in (python3, python)
and proc.cmdline startswith "python app.py"
- macro: mirecloud_sensitive_procs
condition: >
proc.name in (python3, python, sh, bash, ash, dash)
# Rule 1 — Direct file read
- rule: MireCloud - Vault Secret Direct Read
condition: >
open_read
and mirecloud_secret_path
and not mirecloud_authorized_procs
and container
and k8s.ns.name = "test-dynamic"
output: >
ALERTE MIRECLOUD [READ]: Vault secret file opened
(user=%user.name proc=%proc.name cmd=%proc.cmdline
file=%fd.name pod=%k8s.pod.name ns=%k8s.ns.name
image=%container.image.repository:%container.image.tag
container_id=%container.id)
priority: CRITICAL
# Rule 2 — Interactive shell
- rule: MireCloud - Interactive Shell Spawned
condition: >
spawned_process
and mirecloud_sensitive_procs
and container
and k8s.ns.name = "test-dynamic"
and proc.tty != 0
output: >
ALERTE MIRECLOUD [SHELL]: Interactive shell opened in pod
(user=%user.name proc=%proc.name cmd=%proc.cmdline
pod=%k8s.pod.name ns=%k8s.ns.name
image=%container.image.repository:%container.image.tag
parent=%proc.pname)
priority: WARNING
# Rule 3 — Network exfiltration
- rule: MireCloud - Secret Exfiltration Attempt
condition: >
spawned_process
and proc.name in (curl, wget, nc, ncat, python3, python)
and container
and k8s.ns.name = "test-dynamic"
and not mirecloud_authorized_procs
output: >
ALERTE MIRECLOUD [EXFIL]: Potential secret exfiltration process
(user=%user.name proc=%proc.name cmd=%proc.cmdline
pod=%k8s.pod.name ns=%k8s.ns.name
image=%container.image.repository:%container.image.tag)
priority: CRITICAL
# Rule 4 — In-memory dump
- rule: MireCloud - Process Memory Read
condition: >
open_read
and fd.name glob "/proc/*/mem"
and container
and k8s.ns.name = "test-dynamic"
output: >
ALERTE MIRECLOUD [MEM]: Process memory read attempt
(user=%user.name proc=%proc.name cmd=%proc.cmdline
pod=%k8s.pod.name ns=%k8s.ns.name target_fd=%fd.name)
priority: CRITICAL
# Rule 5 — Tamper
- rule: MireCloud - Vault Secret Tampered
condition: >
open_write
and mirecloud_secret_path
and container
and k8s.ns.name = "test-dynamic"
output: >
ALERTE MIRECLOUD [TAMPER]: Write on read-only secrets mount
(user=%user.name proc=%proc.name cmd=%proc.cmdline
file=%fd.name pod=%k8s.pod.name ns=%k8s.ns.name)
priority: CRITICAL
EOF
helm upgrade falco falcosecurity/falco \
-n falco \
--reuse-values \
-f mirecloud-falco-rules.yamlA word on Rule 4 — the /proc/*/mem vector
This one deserves a nuance. Reading /proc/<pid>/mem gives an attacker direct access to the Python process heap — where the credential already lives in memory after the first HTTP request. No filesystem open required. The openat to /mnt/secrets/ never fires. Rules 1 through 3 see nothing.
In practice though, this attack requires CAP_SYS_PTRACE or a permissive ptrace_scope setting. On this cluster, the container runs as appuser, CAP_SYS_PTRACE is not granted, and the kernel default ptrace_scope = 1 applies.
/proc/*/mem attack fails before Falco even needs to fire. The rule is still worth keeping: if the container ever gets misconfigured with elevated capabilities, or if an attacker escalates to root inside the container, this is the only rule that catches a silent heap dump.
What the production ruleset covers
cat, head, tail, dd, awk, sed ✓ Rule 1 — Direct Read
python3 -c "open(...)" ✓ Rule 1 — cmdline mismatch
sh -c "cat ..." ✓ Rule 1 — sh not in authorized procs
kubectl exec → /bin/bash with TTY ✓ Rule 2 — Shell Spawned
/proc/pid/mem (if CAP_SYS_PTRACE) ✓ Rule 4 — Memory Read
Write on /mnt/secrets/ ✓ Rule 5 — Tamper
python app.py reading secrets ✓ authorized — no alertPart 5 — Being Honest About What This Buys Us
In every scenario above, the credential was already read before Falco fired. The alert came after. The forensic record is excellent — nanosecond precision, image tag, exact command — but it's a record of something that already happened.
That's not a failure of Falco. It's just what detection means. You know what happened, when, who, from which image version. That's genuinely valuable, especially when the alternative is finding out three weeks later during a PostgreSQL audit.
But if the goal is to stop the read before it completes, detection isn't enough. You need something that runs below user space — below the container runtime, below the filesystem, below any binary the attacker can touch.
Where We Stand After Part 8
Vault ──► CSI Driver ──► /mnt/secrets/ (tmpfs) ──► python app.py ✓
│ │
│ Falco 5-rule set
│ ├── [READ] open_read + fd.name + cmdline
│ ├── [SHELL] kubectl exec flagged on TTY
│ ├── [EXFIL] network tools flagged
│ ├── [MEM] /proc/pid/mem (if escalated)
│ └── [TAMPER] write on RO mount
│
└── Rotation every 5 min
A1 ✓ A2 ✓ A3 👁| Threat | Control | Status |
|---|---|---|
| Disk / backup exfiltration | Vault dynamic secrets + rotation | ✓ Closed — Part 7 |
| etcd / kubectl get secrets | CSI Driver — no Secret object | ✓ Closed — Part 7 |
| Compromised workload — visibility | Falco 5-rule production set | 👁 Visible — Part 8 |
| Compromised workload — prevention | Tetragon kernel enforcement | ⏳ Part 9 |
A3 is half-closed. We can see the intrusion the moment it happens, with enough forensic detail to reconstruct the timeline precisely. We proved live that a naive binary whitelist fails silently, and we replaced it with something that holds up against the realistic attack surface of this image. What we can't do yet is stop the read before it lands.
🔮 Coming in Part 9 — Tetragon
Where Falco records the openat after it returns, Tetragon intercepts it before the kernel hands back the file descriptor.
The tricky part is that our app reads /mnt/secrets/ on every HTTP request. A blunt SIGKILL on any openat to that path kills the application. The solution is process ancestry — Tetragon walks the actual kernel process tree and can distinguish python app.py as PID 1 from python3 -c "..." spawned by an exec session, even when both use the same binary. Where Falco filters on cmdline as a heuristic, Tetragon uses the real process lineage.
# After the TracingPolicy is applied:
kubectl -n test-dynamic exec -it deployment/secure-test-app -- \
python3 -c "print(open('/mnt/secrets/db-password').read())"
# Killed
kubectl -n test-dynamic get pods
# RESTARTS: 0Exit code 137. Credential never read. Pod still running. And Falco still fires on the attempt — because Tetragon kills the syscall and Falco captures it. They're not doing the same job. Together, A3 is closed.
Comments
Post a Comment