One docker rm -f took down my whole k3s cluster

On 25 August 2026, a single shell pipeline took my entire Kubernetes cluster offline for about 20 minutes. Every website, every API, every background job: down at once. No hardware failed, no update went wrong. Someone needed a free port for a throwaway test database, and freed it the fast way.

The port belonged to the one container on the machine that the whole cluster depends on: the database that holds Kubernetes’ own state.

This post explains how a k3s cluster can end up depending on a plain Docker container, why docker rm -f by filter is so dangerous next to shared infrastructure, and what I changed so the cluster’s brain is never “just another container” again.

Contents · 10 sections

TL;DR #

• My single-node k3s cluster does not use its embedded datastore. It stores all cluster state in an external PostgreSQL that runs as a Docker container on the same host.
• While setting up a disposable test Postgres, the first docker run failed with “port already allocated”. The next command removed every container publishing that port: docker ps -a --filter publish=<port> -q | xargs docker rm -f.
• That removed the k3s datastore. The API server lost its database and crash-looped waiting for it. Every workload behind it went down with it.
• Recovery was quick because the data lived in a bind-mounted directory, not inside the container: recreate the container with the same data directory and k3s comes back.
• 20 minutes of total outage, from one command and one container.
• 1 hour of cluster changes was the worst-case loss, thanks to an hourly dump of the datastore.
• The lessons: inspect before you remove, never free a port by killing unknown containers, give test databases their own port range, and label the datastore so nobody (human or AI assistant) can mistake it for clutter.

How k3s can depend on a Docker container #

Kubernetes keeps all of its state, every Deployment, Secret, ConfigMap and lease, in a key-value store. Upstream Kubernetes uses etcd. k3s ships with an embedded SQLite or etcd by default, but it also supports external SQL databases through kine, a small shim that translates the etcd API into SQL.

You point k3s at an external database with one flag:

k3s server \
  --datastore-endpoint="postgres://<user>:<password>@<host>:<port>/<database>"

kine then creates one table and stores every Kubernetes object in it. From k3s’ point of view, that database is the cluster. If it disappears, the API server cannot read or write anything.

My cluster was set up this way early on: the datastore is a postgres container on the host’s Docker engine, listening on a non-default host port, with its data directory bind-mounted from the host disk. k3s itself runs as a systemd service. It is a perfectly valid design. It also means one of the most important processes on the machine shows up in docker ps right next to scratch containers, test databases and leftovers.

What happened #

The task was harmless: verify a database migration against a disposable copy of a Postgres database before running it for real. The plan was to start a temporary Postgres container, load the data, run the migration, compare, delete.

The first docker run failed:

Bind for 0.0.0.0:<port> failed: port is already allocated

The port that had been chosen for the test database happened to be the port of the k3s datastore. Instead of picking another port, the next command “freed” it:

# DO NOT DO THIS
docker ps -a --filter publish=<port> -q | xargs docker rm -f

This reads as “remove whatever is in the way”. What it actually says is “force-remove every container, running or stopped, that publishes this port, without telling me what they are”. There was exactly one such container. It was the datastore.

Within seconds:

• the k3s API server lost its database connection;
• k3s crash-looped in its start-up phase, waiting for the database port to answer;
• with no API server, nothing could be scheduled, probed or routed, and services across the cluster started returning errors.

The command was run by an AI coding assistant working on a database task. It did not know that the cluster state lived in a Docker container on that port. Honestly, nothing on the machine told it. The container had an ordinary name, an ordinary image and an ordinary port.

Why the recovery was fast #

docker rm -f removes the container. It does not remove a bind-mounted host directory. Because the datastore’s data directory was a bind mount, every byte of cluster state was still on disk.

The recovery was to recreate the container with the same image, the same database name and credentials, the same port and the same data directory, with a restart policy:

docker run -d --name <datastore-name> --restart always \
  -e POSTGRES_USER=<user> -e POSTGRES_PASSWORD=<password> \
  -e POSTGRES_DB=<database> \
  -p <host-port>:5432 \
  -v <host-data-dir>:/var/lib/postgresql/data \
  postgres:15-alpine

Postgres started on its existing data directory, and k3s found its database again. The whole outage lasted about twenty minutes.

If the data had lived in an anonymous Docker volume and the cleanup had included -v or a volume prune, this would have been a cluster rebuild, not a restart.

The second safety net: an hourly dump #

Separately from the bind mount, the datastore has an hourly logical dump taken by cron. It was not needed this time, but it is the reason I was calm while diagnosing: even in the worst case, at most an hour of cluster changes would have been lost.

One general warning if you do the same: a dump of the Kubernetes datastore contains every Secret in the cluster. Secrets in Kubernetes are only base64-encoded at rest unless you enable encryption. Treat the dump with the same care as a password vault: restricted permissions, encrypted storage, never a world-readable share.

Root cause #

The command was the trigger, not the cause. Three things lined up:

1. Invisible criticality. The single most important container on the host looked like every other container. Nothing in its name, labels or documentation near the work said “the cluster dies without me”.
2. Destructive commands by filter. --filter ... | xargs docker rm -f acts on a set you have not looked at. It is the container equivalent of rm -rf $DIR/* with an unchecked variable.
3. Shared port space. Test databases and production databases picked ports from the same small range on the same host. A collision was only a matter of time.

What I changed #

Inspect before you remove #

The rule is now explicit in the notes that every agent and every session reads: before removing a container, run docker inspect and know what it is. If a port is busy, pick another port. Freeing a port is never a reason to delete something.

# What is on that port, and what is it?
docker ps --filter publish=<port> --format '{{.ID}} {{.Names}} {{.Image}} {{.Status}}'
docker inspect <id> --format '{{.Name}} {{json .Mounts}} {{json .Config.Labels}}'

A separate port range for scratch databases #

Disposable test databases use their own range of high ports, well away from anything long-lived. The datastore’s port is documented as “never touch” in the same place.

Mark the datastore as critical #

A pinned note marked critical and do-not-touch now says that the cluster state lives in an external Docker Postgres, which port it uses, where its data directory is, and the exact command to recreate it. It is tagged so that a search for k3s, Postgres or the datastore finds it. Labels on the container itself help too:

docker run ... \
  --label role=k3s-datastore \
  --label do-not-remove=true \
  ...

Labels do not stop docker rm -f, but they make an inspect impossible to misread, and cleanup scripts can exclude anything with do-not-remove=true.

Check it first after any disruption #

When the machine came back after an equipment move in September, the system check confirmed “datastore container up” alongside node and pod status. If the datastore is down, nothing else in the cluster is worth checking yet, so it belongs at the top of any post-reboot list.

Consider running it outside Docker #

A database that the cluster cannot live without is arguably better as a systemd service or a managed package than as a container in the same engine as test workloads. I have not moved it yet, but it is on the list. If you are building fresh, the embedded datastore removes this whole class of problem for a single node.

Checklist #

• Know where your cluster state lives: embedded SQLite, etcd, or an external database. Write it down next to the machine.
• If it is a container, keep its data in a bind mount or named volume and know the exact command to recreate it.
• Take a regular dump of the datastore and protect it like a secrets vault.
• Never run docker rm -f, docker volume prune or xargs cleanups on a filter you have not listed and read first.
• Use a separate port range for disposable databases.
• Label critical containers and make cleanup scripts skip them.
• Give AI assistants and scripts written notes about what must never be touched, and keep them where they will be read.

Frequently asked questions #

Can k3s use PostgreSQL instead of etcd? #

Yes. k3s supports external datastores (PostgreSQL, MySQL, MariaDB, or etcd) through kine, configured with the --datastore-endpoint flag. The database then holds all cluster state, so its availability is the cluster’s availability.

What happens to k3s if the datastore goes down? #

The API server cannot read or write state, so k3s fails to start or crash-loops until the database answers again. Already-running containers may keep running for a while, but nothing new can be scheduled, and anything that depends on the API (controllers, probes, ingress updates) breaks.

Does docker rm -f delete the container’s data? #

It deletes the container and its writable layer. Data in a bind-mounted host directory survives. Data in a named volume survives unless you also remove the volume, and anonymous volumes are removed if you pass -v. That difference decided whether this was a 20-minute outage or a rebuild.

How do I find out what is using a port before freeing it? #

Use docker ps --filter publish=<port> to list containers that publish it, then docker inspect on each result to see its name, image, mounts and labels. On the host, ss -ltnp shows which process holds a listening port. Then choose a different port for your test.

The lesson #

The most dangerous component on a machine is the one that looks ordinary. My cluster’s entire state sat in a container with a plain name on a plain port, and a perfectly reasonable cleanup command treated it like clutter. Bind mounts and an hourly dump made the mistake survivable. Writing down what the container is, labelling it and keeping test work in its own port range make it unlikely to happen again. Before you delete anything to get it out of your way, find out what it is.