Moving secrets out of files: the night I built a key registry

The first commit is stamped 19:46 on Friday, 9 October 2026: “Credentials phase 1: secret values out of git (live Secrets unchanged)”. The last one of the stretch is stamped 10:50 the next morning. In between there are 34 commits, and a small application that did not exist at dinner time could, by breakfast, change a password in a vault, in Kubernetes and in a database, restart only the programs that use it, and put everything back if any step failed.

Its first end-to-end rotation took 14 seconds.

This is not a post about how to rotate a password. I wrote that one already, with the traps that only real rotations found, in One password in 13 places. This is about why I stopped keeping secrets in files, what pushed me over the edge, and the one design decision that everything else follows from: the system that knows where every secret lives is never allowed to know what any of them is.

Contents · 10 sections

In one minute #

• For most of the lab’s life, passwords and tokens lived where they were used: in configuration files, in environment variables, in manifests in Git, sometimes in all three.
• In early October I kept losing track of which credential belonged where. A few days earlier an AI assistant’s test had deleted my home folder, files and all.
• On 9 October I decided on a registry: an app that records what each credential is, where it lives, why it exists, who owns it and when it was last rotated, and never the value itself.
• Values moved into an encrypted vault and into Kubernetes Secrets. Rotation became a guided, reversible procedure that needs a fresh one-time code.
• My local AI agent can look a credential up and propose a change. It can never read a value.

The before: secrets where they were needed #

A home lab grows one service at a time, and each service arrives with its own secret. A database password in a manifest. A token in a service unit file. An API key in a config file, because that is what the README said to do. A connection string with the password inside it, copied from one app to the next because it worked.

None of this felt wrong at the time. Every secret was exactly where the program that needed it would look. The problem only shows when you ask the question in the other direction: not “where does this app get its password?”, but “where does this password live?”

In my case the honest answer was: I did not know. The registry’s first scan would later find credentials in Kubernetes Secrets, in plain environment values in workload specs, inside connection strings, in container command lines, in health checks, in host environment files and in the Git history of my manifests repository. How secrets get into a GitOps repository, and why deleting them from the files does not remove them, is in Secrets in GitOps history.

If you have ever searched your own disk for a password you know you set, and found it in four places, two of them out of date, you know this feeling. It is not fear exactly. It is the slow realisation that you are not in control of something you thought you owned.

The trigger: two things in one week #

On Saturday 3 October an AI coding assistant ran a test against old, unfixed code on my real machine, and in about ten seconds my home folder disappeared. That story is in The test that deleted my home folder. One of the changes I made afterwards is listed there in a single line: “secrets are no longer reachable from plain files the assistant can read.”

The second thing was quieter. In the days that followed I kept losing track of credentials, and my classroom app was the latest example: I could not say with confidence which credential it used, or where its copies were.

On 9 October I wrote down the decisions before writing any code:

• a separate application, not a feature inside my SSO manager, so that a bug in one cannot expose the other;
• reachable only from the home network, behind two-factor login, with a second, in-app code for anything that changes a secret;
• it stores metadata and salted fingerprints, never values;
• and an order of work: inventory and design first, then quick fixes, then a read-only app, then the link to the vault, and only then rotation.

That order matters. It is tempting to start with the exciting part, the button that rotates a password. Starting there means rotating things you have not found yet.

The build: one night and a morning #

The Git log of that night reads like a list of promises kept one by one.

19:46. Secret values out of Git. The live Secrets in the cluster were not touched; only their copies in the repository were removed.

20:02. Plain credential values written directly into workload specs moved into Secrets, with the same values. Nothing rotated yet. First, put every secret in a place that can be managed.

20:19. Version 1.0.1 of the registry, read-only. Scanners for Kubernetes, the Git checkout and the host. Every finding lands in an inbox: adopt it into a credential, or dismiss it with a reason.

20:30. The link to the vault. I use Vaultwarden, a self-hosted server compatible with Bitwarden’s clients, so values are encrypted end to end. The registry talks to it through the official command-line client, with a dedicated account that can see one collection only, and whose master password is never stored: I unlock it on a page, the session key stays in memory and locks after 10 minutes idle.

21:36. Guided rotation, with a step-up code from the app’s own authenticator, per-credential rights, the vault updated first and a rollback path.

22:06 to 22:54. A report with default rotation policies and a daily reminder. A runbook for database passwords that also updates connection strings. Rotation of host environment files through a small root agent that only touches targets listed in its own allow-list. Detection of credentials written into container command lines. Grouping of credentials that share the same value, because one value can serve many names.

23:09. The first real fix: breaking up a password that turned out to be shared. The details are in the earlier post; the short version is that the registry found it, and the rotations were then run one at a time, by me, not by an agent.

00:41 to 01:30. A guard that refuses to rotate a value that still appears literally somewhere it would not be updated. Clearer messages when a step-up code fails. Scans and rotations made mutually exclusive.

Then a gap, and at 09:00 the morning starts with backups: the vault and the registry in every backup layer, all Secrets included in the cluster backups, and a freshness alarm for both. A vault you cannot restore is just a new way to lose everything.

09:24. An emergency entrance. If the SSO portal is down, the one tool that knows where every credential lives must still open, with its own login, an authenticator code or one-time recovery codes. The task that asked for it is one line long: “Owner must not lock themselves out during troubleshooting.”

09:39. The agent API.

10:50. Version 1.11.4.

Building a security tool through the night is exactly the kind of habit a security tool should protect you from. What kept it safe was the order: nothing was rotated before it had been found, and nothing was rotated that could not be put back.

The technical heart: who is allowed to know what #

The single idea underneath the whole thing is a split between knowing where and knowing what.

Who	Knows where a secret lives	Knows its value
The registry app	yes: names, locations, key names	no: only keyed fingerprints
The vault	no	yes, encrypted, unlocked by a human
Kubernetes Secrets and host env files	no	yes, for the program that uses them
My local AI agent	yes, through a read and propose API	never
Me	yes	only when I unlock the vault

Three mechanisms make that table true.

Fingerprints instead of values. At every scan the registry computes a keyed hash (an HMAC) of each value it finds and stores only that. If a fingerprint changes, the credential shows as changed. If the vault and the cluster disagree, a check compares them field by field and reports in sync, drift or missing in vault, without ever showing either value. A fingerprint cannot be reversed, and because it is keyed, it cannot be checked against a list of common passwords either.

Rights per credential, granted from outside. The app cannot grant itself anything. Each credential it is allowed to rotate needs a Kubernetes Role in that Secret’s namespace, with get and patch on exactly the named Secrets and consumer workloads:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: keyring-c42
  namespace: <namespace>
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["<app>-credentials"]
    verbs: ["get", "patch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    resourceNames: ["<app>"]
    verbs: ["get", "patch"]

The rotation page shows this YAML when it is missing. I apply it and commit it. Deleting it takes the right away again.

Values only in memory, only in flight. During a rotation the new value exists in the runner’s memory, is written to the vault first with the old one kept as “previous”, then patched into every Secret that holds it and read back, then the consumers are restarted and checked. Values reach the vault client through environment variables or standard input, never as command-line arguments, which other processes can see. If anything fails after the vault step, the old value goes back everywhere.

The wrong turns #

The record of those hours is not a victory lap. A few entries I keep re-reading:

A step-up code that kept failing. For one rotation the in-app authenticator code was rejected again and again. The most likely cause, in the end, was the browser helpfully autofilling the form. Version 1.7.7 turned autofill off on that form and logs why a code fails: already used, phone clock off by a number of seconds, or a code from the wrong authenticator entry.

A race that was not a race. After some rotations the registry showed credentials as “changed” when they were not. The first diagnosis, stated with confidence, was a race between scanning and rotation. The real cause was simpler and in the code: one part stored fingerprints under the key name, the other compared them under a prefixed name. The lesson in the record: verify a root cause in code before stating it.

My own emergency door. The first version of the emergency entrance decided whether a request had come in through the emergency port by reading a value derived from the request itself, which a client can influence. It was found and fixed the same morning: the decision now uses the socket the server actually accepted. The lesson is general: authorisation decisions must never rest on anything the client can write.

A guard that saw secrets everywhere. The safety check that looks for leftover literal values flagged a common word that happened to be a password, in every place that word appeared. It now looks only where a value acts as a credential.

The after: what changed #

The most visible change is the direction of the work. The aim now is that a new service’s credentials are registered on the day it is deployed, not discovered months later; since version 1.10 the registry can generate them for new deployments, and the scan inbox catches the ones I forget.

The second change is in what an AI agent can do. My local agent can ask the registry where a credential lives, when it was last rotated and what the runbook says, and it can file a proposal for me to review. It cannot read a value, and it cannot rotate anything. Rotations need my unlocked vault and a fresh code from my phone.

The third change is relief, and it is measurable. After each rotation the old value is tried over the network and must be refused. It is the rare check where an authentication error is exactly the result you hope for.

What it cost #

It cost one night and a morning, and then a long tail: rotations run one at a time over the following days, health checks that still had old passwords written into them, a notification token that existed in dozens of copies and needed its own small service, described in One notification relay instead of 40 copies of a token.

It cost simplicity. A password in a file is easy to understand. A registry, a vault, Secrets, grants, fingerprints and a break-glass door are not. The registry itself is now one more critical system that must be backed up, monitored and kept reachable when everything else is down.

And it cost a comfortable belief: that because I built every service myself, I knew where its secrets were. I did not.

What changed #

• Before: credentials in files, manifests, environment variables and Git history; nobody could say where a value lived.
• The trigger: losing track of credentials, a few days after a test deleted my home folder.
• After: a registry of where, why, who and when; values only in an encrypted vault and in the Secrets of the programs that use them.
• Rotation: guided, vault-first, read back, reversible, behind a one-time code; 14 seconds for the first full run.
• Agents: may look up and propose; may never read a value.

Questions people ask me #

What is a secrets vault, and do I need one at home? #

A vault is an encrypted store for passwords and keys that programs and people unlock when they need them, instead of keeping copies in files. At home it starts paying off as soon as you have more than a few services, or as soon as an automated tool, including an AI assistant, can read your files.

Why not store the passwords in the registry itself? #

Because the registry is the one place that knows where every credential lives. If it also held the values, one bug or one stolen session would expose everything at once. Splitting “where” from “what” means the registry can be wrong, or compromised, without leaking a single password.

Is it safe to let an AI agent manage secrets? #

Let it manage the knowledge about secrets, not the secrets. Mine can look up locations, ages and runbooks and propose changes. It has no permission to read values, and every rotation needs a human to unlock the vault and enter a fresh code. That keeps the agent useful and the blast radius small.

How often should home-lab passwords be rotated? #

Rotate on a schedule that you will actually keep, and immediately when a value may have leaked. The schedule matters less than the ability to rotate quickly and safely. If changing a password means editing ten files by hand, it will not happen. If it takes one guided procedure and a code, it will.

The closing #

Before October, every secret in my lab was exactly where the program that used it would look. That is the definition of convenient, and it is also why nobody, including me, could answer the simplest security question there is: where is this password?

The turning point was choosing to build something that knows where every secret is and is forbidden to know what any of them says. If you run your own services, try asking the question in that direction for one credential today. Not where does my app get its password, but where does this password live. The answer is usually longer than you expect.