-
From watching to answering: when my home lab began to defend itself
In April 2026 my lab blocked a hostile probe by itself in under 60 seconds. Then I learned how little of that protection was mine, and how automated defence can fail silently or hurt its own network.
-
13 hours of question marks: a canary for LLMs that look healthy
After a llama.cpp router restart my main model answered every request with question marks for 13 hours while every health check stayed green. A known-answer canary now catches it within 10 minutes.
-
A post-reboot check script for a one-machine home lab
A cold boot of my lab produced 37 failed jobs; three were real defects hidden for up to two weeks. Here is the read-only OK/FAIL script I now run after every reboot, and the failures it catches.
-
A passive GPU in a tower: VRAM hit 106 °C while the core throttled
My new 32 GB passive GPU had no plateau under LLM load and its memory reached 106 °C. Measurements, the limits that were silently ignored, and the fan, power cap and clock limit that fixed it.
-
Why my long CrowdSec bans vanished after two days
CrowdSec's alert flush deleted my year-long bans after ~48 hours, a systemd timer sat dead for 24 days, and the fix triggered 18 router re-add storms. Mechanisms and fixes.
-
Falco on Linux 7.0: 222 restarts and a BPF verifier error
Falco 0.43 restarted 222 times on a Linux 7.0 kernel. The path to the fix: OOM kills, dropped syscalls, a BPF verifier rejection fixed in Falco 0.45.1, and Helm 4 server-side apply conflicts.
-
A chat-ops bot is a doorway, not a convenience
My infrastructure chat bot went silent for six hours because the router had blocked the messaging API's one IP. That and a self-heal webhook wired to nothing taught me how to build and test a privileged chat-ops bot.
-
Prometheus WAL on a RAM disk: the trade-offs, and my empty tmpfs
My Prometheus config said the WAL lived on a RAM disk. The tmpfs held 0 bytes; the 785 MB WAL was on the SSD all along. Why, whether it was ever a good idea, and where RAM disks do help.
-
Backups that lie, and the freshness alarm that catches them
My backup jobs reported OK while 13 of 17 databases had no backup, dumps were truncated and two crons had been dead for months. One freshness check over the artifacts of every backup set now catches all of it.
-
One notification relay instead of 40 copies of a token
The same chat-bot token lived in about 40 places in my lab, some in git. A small relay now holds the credentials once; senders carry only a revocable per-sender token and address channels, not platforms.