-
Moving secrets out of files: the night I built a key registry
For months my lab's passwords lived wherever the programs needed them. A week after a test wiped my home folder, I spent one night building a registry that tracks where every secret lives, never its value, with guided and reversible rotation.
-
One docker rm -f took down my whole k3s cluster
Freeing a busy port with docker rm -f by filter removed the external Postgres that holds my k3s cluster state: about 20 minutes of total outage. How k3s datastores work, why recovery was quick, and the guard rails I added.
-
The cleanup job that deleted 239,000 live search chunks
A nightly orphan-chunk sweep in my Elasticsearch library deleted 239k live chunks from 19.4k documents for four weeks. Why its liveness check was wrong, how I recovered, and the count-first circuit breaker that followed.
-
DNS is not a firewall: splitting Traefik into LAN and public
My internal services had no public DNS, yet anyone could reach them by sending the right Host header to my public IP. How I proved it over Tor, split Traefik into LAN and public entry points, and the two test mistakes I made.
-
A post-reboot check script for a one-machine home lab
A cold boot of my lab produced 37 failed jobs; three were real defects hidden for up to two weeks. Here is the read-only OK/FAIL script I now run after every reboot, and the failures it catches.
-
NetworkPolicy gotchas: hostNetwork Traefik and Docker DNAT
A hostNetwork ingress shares the node IP with everything on the host, and Docker rewrites destinations before kube-router filters egress. What that broke on my k3s cluster, how a forged header created an admin, and how to write exact policies.
-
Velero said Completed. It had backed up a RAM disk.
Velero's Kopia backup skipped my main Postgres because it used hostPath, saving only an in-memory temp volume; 13 of 17 databases had no backup anywhere. PV fixes, two OOM kills, and a byte-exact restore.
-
My dynamic DNS job got my home IP banned by the registrar
Polling GoDaddy's API every 5 minutes to keep an apex A record current got my home IP banned. How to diagnose an API ban, switch to change detection with zero calls on a stable day, and avoid two related DNS traps.
-
Running 414 Kiwix ZIMs as an offline reference shelf
What I learned serving 414 Kiwix ZIM files (973 GB) beside my own library: undated aliases need -z, lazily opened ZIMs break when moved, 240 lack full-text indexes, and encyclopedias should be federated, not embedded.
-
One query wrote 376 GB of temp files: shared Postgres lessons
A six-way LEFT JOIN on my shared PostgreSQL spilled 376 GB to pgsql_tmp in 3.5 hours. The limits every shared instance needs, a RAM-disk temp tablespace and its restart trap, and two more shared-database lessons from k3s.
-
Authelia SSO pitfalls: a policy protects nothing if the route skips it
Three services my Authelia config marked two-factor served pages to anyone: the Traefik routes never called Authelia. Six SSO pitfalls from a home lab, all fixed, with config.
-
Falco on Linux 7.0: 222 restarts and a BPF verifier error
Falco 0.43 restarted 222 times on a Linux 7.0 kernel. The path to the fix: OOM kills, dropped syscalls, a BPF verifier rejection fixed in Falco 0.45.1, and Helm 4 server-side apply conflicts.