Passwords that only fail after a reboot

I rotated a database password, updated every secret I knew about, and everything kept working. For days. Then I rebooted the machine, and 2 services that had been running happily all week refused to start: password authentication failed.

Nothing had changed in those services. The rotation had happened long before. They were broken from the moment of the rotation. They just hadn’t noticed yet.

This post is about that delay: why a credential change can stay invisible for days, how to find the services that are hiding it, and how to make them fail the moment something is wrong instead of a week later.

Contents · 8 sections

TL;DR #

• Long-running services often authenticate once, when they open a connection, and then keep that connection open for days.
• If you rotate a password, those services keep working on their existing connections. They only fail when they reconnect, usually at a restart or a reboot.
• Worse, some code carries a hard-coded default password used when the environment variable is missing. That default quietly becomes wrong at the next rotation.
• Fixes: fail closed when a credential is missing, restart every consumer after a rotation, and test a fresh connection, not the old one.

What happened #

My home lab runs dozens of services on one machine: a small Kubernetes cluster, some containers, and some plain system services. Several of them share a PostgreSQL database and a Redis instance.

Some weeks earlier I had rotated the database password. I updated the Kubernetes Secrets, restarted the obvious consumers, checked that the main applications could log in, and moved on.

Then the machine was rebooted after a routine system upgrade. My post-reboot check script, which walks through every service I care about, turned up these failures:

• the honeypot log shipper, a small system service that copies honeypot events into the database and into the log store, was crash-looping with “password authentication failed”;
• a book-fetching service in the cluster was crash-looping with “invalid username-password pair” against Redis;
• earlier, a mail server had shown the same pattern: it survived a rotation only on old connections and would have lost its mail store at the next restart.

All three had been “working” since the rotation. All three were already broken.

Why the failure was invisible #

Connections outlive credentials #

A database checks your password when you connect, not on every query. A service that opens a connection pool at start-up and keeps it alive can run for weeks on that single successful login. Rotating the password on the server does not drop existing sessions in PostgreSQL or Redis by default. The service is effectively grandfathered in.

So “the service is still working after the rotation” proves nothing about whether it can log in. It only proves it has not needed to.

A default that quietly went stale #

When I looked at the two services’ code, both had the same pattern:

DB_PASSWORD = os.environ.get("DB_PASSWORD", "<an old literal password>")

and

REDIS_URL = os.environ.get("REDIS_URL", "redis://:<an old literal password>@redis:6379/2")

The intention was convenience: run it on a laptop without setting anything. The effect was a trap:

1. In production the environment variable was never set, because the default “just worked”.
2. The default was the old password.
3. After the rotation, the service kept running on its existing connection, so nobody noticed.
4. At the next restart it tried the default, and failed.

A missing credential should be an error the moment the service starts. Instead it silently fell back to a value that was correct once and wrong forever after.

Restarts are rare, so the delay is long #

On a stable system, services restart rarely: at upgrades, reboots, crashes. The gap between “the credential became wrong” and “the service found out” is the gap between restarts. That can be days or months, and by then nobody connects the failure to the rotation.

How I fixed it #

1. Give each service its credential explicitly #

For the cluster service, the Redis URL now comes from a Kubernetes Secret, wired into the Deployment:

env:
  - name: REDIS_URL
    valueFrom:
      secretKeyRef:
        name: app-secrets
        key: REDIS_URL

For the system service, systemd reads the password from a root-only environment file through a drop-in, so it never sits in the unit file or in the code:

# /etc/systemd/system/<service>.service.d/10-db.conf
[Service]
EnvironmentFile=/etc/<service>.env

with the file mode set to 600. The value itself was copied straight from the existing Secret through a pipe, so it was never printed or pasted.

2. Fail closed: no defaults for secrets #

The most important change is in the code. A secret must have no default:

import os, sys

DB_PASSWORD = os.environ.get("DB_PASSWORD")
if not DB_PASSWORD:
    sys.exit("DB_PASSWORD is not set; refusing to start")

Now a missing or mis-wired credential fails at start-up, loudly, where a deployment check or a crash loop alert sees it immediately, instead of degrading into a stale value. I then searched all my repositories for the same pattern and found several more fallbacks: in tests, in archived code, and in at least one live service. Each one is either removed or scheduled for removal.

A literal password in code has a second problem: it is now in git history forever, so it must be treated as exposed and never reused.

3. A rotation is not done until every consumer has reconnected #

My rotation checklist now ends with a forced reconnection:

1. Find every consumer of the credential, not just the obvious ones: system services, cron jobs, containers outside the cluster, scripts.
2. Update the secret store.
3. Restart every consumer, so each one performs a fresh login with the new value.
4. Watch for authentication errors in the next minutes, while the change is still fresh in your head.
5. Only then, revoke the old credential.

Step 3 is the one people skip because “it’s still working”. That is exactly the case that hides the problem.

4. Make the database tell you who is still using old sessions #

Most databases can show you current sessions. After a rotation, any session that started before the rotation belongs to a client that has not proven it can log in with the new password. In PostgreSQL:

SELECT usename, application_name, client_addr, backend_start
FROM pg_stat_activity
WHERE backend_start < '<time of the rotation>'
ORDER BY backend_start;

Every row is a service to restart before you can call the rotation finished. Optionally, terminate those sessions deliberately during a maintenance window, so the failures happen on your schedule, not at 3 a.m. after an unplanned reboot.

Related trap: addresses that move at a reboot #

The same reboot exposed a cousin of this problem. One of my network policies allowed traffic to a container by its IP address. The container was recreated during the reboot, received a different address on its bridge network, and the policy silently stopped matching. Same pattern: a value captured once, correct at the time, wrong after the next restart, with no error until something needs it.

The cure is the same too: never pin something that can change without telling you, or, when you must, add an automated check that compares the pinned value with reality after every restart. My post-reboot script now does exactly that.

Checklist: hunting credentials that only fail later #

• grep your code for getenv( / environ.get( calls whose second argument looks like a password, a token or a URL with credentials in it.
• Every secret is read from the environment or a secret store, with no default, and the service refuses to start without it.
• Literal credentials found in git history are rotated, not just deleted.
• Each rotation ends with restarting every consumer and checking for authentication errors.
• After a rotation, list database sessions that predate it; each is a client you have not verified.
• After every reboot, run a check script that tests real logins, not just “is the process running”.

Frequently asked questions #

Why do services keep working after a password change? #

Because they authenticated when they opened their connection, and the database does not re-check passwords on open sessions. They fail only when they need a new connection, typically after a restart.

Should I terminate all sessions after rotating a database password? #

If you can schedule it, yes: it turns hidden failures into immediate, visible ones while you are watching. At minimum, restart every known consumer and list the sessions that started before the rotation.

Is a default password in code ever acceptable? #

Not for anything that reaches a real system. Use defaults for harmless settings like ports or log levels. For secrets, a missing value must stop the program. Local development can use a .env file that never leaves the developer’s machine.

How do I find every consumer of a credential? #

Search your manifests, unit files, compose files and code for the secret’s name and for the host it connects to, and check the database’s current sessions for client addresses and application names. Anything you did not expect is the one that will fail later.

The lesson #

“It still works” is not evidence. It is often just the delay before a failure. Credentials should fail early and loudly, rotations should end with every consumer proving a fresh login, and anything captured once, like a password or an address, deserves a check that runs after every restart.