One password in 13 places: inventory and rotation of home-lab secrets

When I finally counted the credentials in my home lab, one database password turned up in 13 places: five database accounts, the connection strings of a dozen apps, and two manifests committed to Git. Rotating it by hand would have meant editing every one of them, restarting everything that read them, and hoping I had not missed the fourteenth.

So I did not rotate it by hand. I spent two days in October 2026 building an inventory, a small registry that knows where every credential lives but never stores its value, and a guided rotation that changes a password in the vault, the database and every Kubernetes Secret in one go, then proves the old one no longer works. This post covers what the inventory found, how rotation works, and the traps that only showed up during real rotations.

Contents · 10 sections

TL;DR #

• Start with an inventory. Mine found 62 plaintext credentials in workload specs, 15 ConfigMaps with credential fields, 18 host environment files and secrets committed to Git.
• Move every literal value into a Kubernetes Secret reference first, with the same value. That step changes nothing at runtime and makes rotation possible.
• Keep a registry of pointers, never values: which vault item, which Secret keys, which apps, which database account.
• Rotate in a fixed order: vault first (keep the previous value), then the database, then Secrets with read-back, then restart only the consumers, with rollback.
• Verify by failure: the old password must be rejected over the network.
• Watch for literal passwords in health probes, and for checks that “pass” without authenticating.

Step 1: count everything #

The inventory was a script that read the cluster, the Git repositories and the host. On 9 October 2026 it reported:

• 69 credential Secrets (plus 119 TLS Secrets), 24 of them used by no workload at all;
• 62 credential values written directly into workload env blocks;
• 15 ConfigMap files with password or token fields;
• 18 environment files on the host;
• a container registry configuration with credentials that was world-readable;
• a database password pasted into a project’s security notes;
• the SSO portal’s signing and encryption secrets committed in plain text, in two stale copies of its configuration that had also drifted about 360 lines from what was actually running.

None of this was exotic. It is what accumulates when you set things up one evening at a time and copy a working example.

Step 2: move values into Secrets, without changing them #

The first phase changed where values were read from, not what they were. Every literal became a reference:

env:
  - name: DATABASE_PASSWORD
    valueFrom:
      secretKeyRef:
        name: <secret-name>
        key: DATABASE_PASSWORD

Same value, so nothing could break except typos, and each workload was restarted once and checked. The committed Secret objects in Git were stripped of their values, so Git describes the shape and the cluster holds the data. The registry file got mode 0600.

The databases needed a short maintenance window, because several of them shared that one password. A backup CronJob turned out to have its database password inline in its command line. The scanner had only looked at env, so I extended it to look at command and args too, and verified the backup with a manual run afterwards. At the end of the phase there were zero literal credentials left in workload specs.

Values that were ever committed to Git are still in its history. Moving them is not enough; they have to be rotated.

Step 3: a registry of pointers #

Next I built the registry, a small web app in its own namespace with a restricted Pod Security profile, a NetworkPolicy that only admits the reverse proxy, and read-only RBAC. For each credential it records:

• what it is and why it exists;
• the vault item that holds the value (in Vaultwarden, in a dedicated organisation collection);
• every Secret and key that carries it, including connection strings that embed it;
• the workloads that read those keys;
• for database credentials, the database account and engine.

It never stores the value. To detect drift it compares keyed fingerprints, an HMAC of the value with a key only the registry holds, so it can say “the vault and the Secret disagree” without either value leaving its home.

The fingerprints also found the reuse: grouping by fingerprint showed the 13 keys sharing one value across PostgreSQL, MySQL, MongoDB, Redis, a Git server and a video platform.

Talking to the vault #

The registry uses the Bitwarden CLI against Vaultwarden. Two gotchas:

• The CLI refuses plain-HTTP servers. In-cluster clients go through the reverse proxy over HTTPS, with the local CA passed to Node.js via NODE_EXTRA_CA_CERTS.
• The CLI crashed on the first unlock after a fresh API-key login. One retry fixed it.

The vault is unlocked in memory only, and locks again after 10 minutes idle.

Step 4: guided rotation #

A rotation is a plan, shown before it runs: the vault item, every Secret key, every connection string, every workload that will restart, and any literal-value warnings that must be fixed first. Then, after a step-up code from an authenticator app:

1. Vault first. Write the new value, and keep the old one in a “previous” field. If anything later fails, the old value is still recoverable.
2. Database account. For database credentials, change the password on the server (PostgreSQL, MySQL, MongoDB or Redis), without restarting the database.
3. Secrets, with read-back. Patch each Secret key and connection string, then read it back and compare fingerprints.
4. Restart only the consumers, and wait until they are ready.
5. Roll back on any failure, using the previous value.

Write access is narrow by design. The registry’s service account cannot edit Secrets in general. Each credential gets its own Role, limited by resourceNames to exactly the Secrets that credential lives in, and the app checks with a SelfSubjectAccessReview before it starts:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: <credential-role>
  namespace: <namespace>
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["<secret-name>"]
    verbs: ["get", "patch"]

The first end-to-end test, in a throwaway namespace, rotated a credential in 14 seconds. In the following two days 15 rotations ran live. The biggest was the shared PostgreSQL account: 8 Secrets, 11 connection strings and 20 restarts.

Verify by failure #

“The apps still work” is not proof that a rotation happened. Every rotation is verified three ways:

# the old password must be refused, over the network, not via a local socket
PGPASSWORD='<old>' psql "host=<db> user=<account> dbname=postgres" -c 'select 1'
# expected: password authentication failed

# the new one works for every connection string, and consumers log no auth errors
kubectl logs -n <namespace> deploy/<app> --since=10m | grep -i -E 'auth|password'

When the Redis password was rotated, all 13 clients were checked this way, and the old password got WRONGPASS. After the last rotation, a sweep of Secrets, specs, ConfigMaps and host configuration found the old shared value nowhere.

The traps that only real rotations found #

Passwords in health probes #

After rotating the MongoDB root password, the database pods started failing their liveness checks. The probes ran a shell command with the password written into the probe itself. The rotation updated every Secret and missed the probe, so Kubernetes started restarting a healthy database.

Two fixes: probes now use commands that need no login (mongosh --eval 'db.adminCommand("ping")', mysqladmin ping), and the rotation plan runs a pre-rotation guard that scans the involved workloads, including probes, for the literal value and refuses to start until it is gone.

A check that passes without authenticating #

redis-cli ping exits with status 0 even when the server answers NOAUTH. A probe or a script that only checks the exit code will report a healthy, authenticated connection that is neither. Check the reply, not the exit status.

The guard that matched too much #

The literal-value guard initially matched the value anywhere in a spec. One app’s password was a common word that also appeared in container names and labels, so the guard blocked its rotation everywhere. It now only checks positions where a value acts as a credential: env values, arguments, probe commands, connection strings.

A wrong root cause, stated too early #

After two rotations that changed connection strings, the registry showed “out of sync” for credentials that were fine. I told myself, in writing, that a scan had raced with the rotation. It had not. The rotation stored connection-string fingerprints under one key name and the scanner looked them up under another. Verify a root cause in the code before you state it, and correct the record as soon as the guess is disproved.

Browser autofill #

Step-up codes kept failing on one rotation form. The browser was autofilling a saved password into the code field. autocomplete="off" and a clearer error message fixed it.

Two more rules I learned the hard way #

Notes hold pointers, never values. My project notes, which are committed and pushed automatically, had contained a vault admin token since April. A note should say “the admin token is in vault item X”, nothing more.

A healthy vault can be empty. During my October home-folder recovery I learned that Vaultwarden does not fail when its SQLite database disappears: it creates a fresh empty one and keeps answering its health endpoint with 200. Monitor the number of users and items, not /alive.

Checklist #

• Run an inventory: Secrets, env literals, ConfigMaps, command lines, host files, Git.
• Move literals to Secret references with the same value first.
• Keep a registry of pointers and compare fingerprints, never values.
• Rotate one credential at a time, vault first, keeping the previous value.
• Scope write access with per-credential Roles and resourceNames.
• Restart only the consumers, wait for ready, roll back on failure.
• Verify the old password is rejected over the network.
• Probes must not contain credentials.
• Check replies, not exit codes (redis-cli ping).
• Monitor the vault’s content, not just its health endpoint.
• Anything ever committed to Git gets rotated.

Frequently asked questions #

Where should I start with secrets in a home lab? #

Count them. An inventory of Secrets, environment literals, ConfigMaps, command lines, host files and Git history shows what you actually have. Then move literals into Kubernetes Secrets with unchanged values, which is safe and makes every later step possible.

Should my password manager be the source of truth for Kubernetes Secrets? #

For me the vault holds the value and the history, and Kubernetes holds the copies apps read. A registry ties them together with pointers and fingerprints, and rotation always writes the vault first, so the vault is never behind.

How do I prove a password rotation worked? #

Try the old password against the real service, over the network, and expect a refusal. Then confirm every consumer connects with the new one and logs no authentication errors.

Why did my database restart after a password change? #

Check its liveness and readiness probes. If a probe logs in with a literal password, it fails after rotation and Kubernetes restarts the pod. Use probes that do not authenticate.

The lesson #

Rotation is hard when you do not know where a secret lives, and easy when you do. The two days went mostly into the boring part: finding every copy, moving values without changing them, and writing down where each one lives. After that, rotating the password from 13 places was one guided run behind a step-up code, and the proof was the old password being refused.