After the wipe: six habits that changed how I run my home lab

One of the first things I did after my home folder disappeared was not to restore anything. It was to stop my disk from forgetting.

My server’s root disk is an encrypted SSD, and a timer regularly tells the drive which blocks are free so it can erase them. On a normal day that keeps the drive healthy. On 3 October 2026 it would have erased the only evidence I had. Deleted files stay on disk until those blocks are reused or trimmed, so I disabled the trim timer before doing anything else. Later that day, scanning the raw disk recovered 351 records of a deleted transcript, enough to find the exact command that had wiped everything.

I have already written about what happened that Saturday in the test that deleted my home folder: an AI coding assistant proving a path bug in my own agent by running the old, buggy code on the real machine. In about ten seconds it removed everything my user account could delete.

This post is not that story again. It is about the month after, and about what I do differently now. Some of it is technical. Most of it is habit.

Contents · 12 sections

In one minute #

• Freeze before you fix. Stop anything that can make the damage worse or erase evidence, before you restore.
• Reboot on purpose. A running program can hide a deletion until it restarts. My first deliberate reboot found three dead databases.
• Back up the boring things, every hour, and alarm on age. My host backup now runs hourly, and a freshness check watches 33 backup sets.
• Commit or lose it. Every script now lives in Git. AI sessions that work on my machine are blocked from ending a turn with uncommitted work.
• Agents get their own user. My local AI agent runs as its own account and reaches my folders only through grants I approve.
• Nobody watching means nobody acting. Unattended runs propose; they do not change things.

The before: a home folder that was also a server #

Before that morning, my home folder was not just a home folder. It was where my lab kept the things I had not got around to putting anywhere else.

The scripts behind four of my backup jobs lived there and nowhere else. The local certificate authority for my internal sites lived there. A small web app that manages my router’s address leases ran from there. So did the lock files of a security job that runs every minute, and the SSH keys that my monitoring tools use to reach the host. The backup of the home folder itself had silently stopped 11 months earlier.

None of this was a decision. Each one was a “temporary” place that became permanent because nothing broke. Looking at the list afterwards, the pattern was obvious: every one of those things existed in exactly one copy, on the machine that was supposed to be protected by them.

Habit one: freeze before you fix #

The instinct after a disaster is to start repairing. The first hours taught me to do the opposite.

Disabling the trim timer was one freeze. The second was the agent. By 11:22 that morning I had stopped and masked all 12 services and timers of my local AI agent, so nothing could start them again by accident, and commented out the scheduled jobs of the older agent version. Later that afternoon, with my approval, the AI monitoring agent inside the cluster was suspended too. Nothing that could act on the machine on its own was allowed to run while I was still finding out what had happened.

Freezing has a cost. For more than a day I had no agent, no automated reports and no self-healing. It was the right trade. You cannot investigate a scene while something is still walking through it.

The freeze also taught me that recovery creates its own incidents. While I was cleaning up the database of my intrusion-prevention engine, I stopped it for a few minutes. The job that copies its block list to my router read zero entries while it was down, concluded that every block had been lifted, and removed them all. Refilling took 25 minutes. That job now refuses to act on a failed read and refuses mass removals, and my recovery notes have a new line: pause the copier before you stop the engine.

Habit two: reboot on purpose #

On 4 October at 11:34 I rebooted the server deliberately, while I was still in recovery mode. I did not want to find the next surprise weeks later, by accident.

Three MySQL databases did not come back. Their data folders had been writable by my user’s group, so the deletion had removed their core files. MySQL had kept running for a full day on files it already had open. On Linux, a deleted file stays alive for any process that holds it open, so everything looked fine until the reboot closed those handles. All three were restored from the nightly dumps taken a few hours earlier.

A post-reboot check script came out of that weekend: one command that checks the kernel, every important service and container, the cluster’s pods, the security pipeline and the age of the latest backup, and prints OK or FAIL for each. After that first reboot, 57 of its checks were green. I described it in detail in a post-reboot check script for a one-machine home lab. It is the first thing I run after every restart now, and it keeps earning its place: a later reboot surfaced two services that had only survived on old database connections with a password that had since been rotated, the kind of problem described in passwords that only fail after a reboot.

The habit is simple to say: after any big change, reboot while you are still paying attention.

Habit three: back up the boring things, every hour #

The cluster’s own data came back almost perfectly, because it had real backups. The things that hurt were boring host files that nobody had thought of as data.

The new host backup runs every hour to my NAS, encrypted and deduplicated, and covers exactly what no other job covers: the parts of my home folder that are not already backed up elsewhere, the system configuration, local tools, and the state folders of services that run outside the cluster. About 42 GiB. It keeps 24 hourly and 30 daily snapshots.

The part that matters more is the alarm. A freshness check now watches every backup set I have, 33 of them at the time, and raises an alert when any of them is older than it should be. A backup that has silently stopped is worse than no backup, because you believe in it. I have written about that before, in backups that lie, and the freshness alarm that catches them and in my first restore test reported 0 tables and 0 errors.

The technical heart: what file permissions really protected #

If you remember one mechanism from this post, make it this one. It is what decided what I lost.

rm -rf running as a user deletes every file that user can unlink. Unlinking a file is a permission on the folder, not on the file. So a root-owned file inside a folder my user owned was deleted. A folder owned by a container’s user, with no write access for my group, survived untouched.

That one rule explains the whole map of the damage:

Situation	Result
My files in my folders	deleted
Root-owned files inside folders I owned	deleted
Container data in folders owned by the container user, mode 750	survived
Container data in folders writable by my group (mode 775)	deleted
System folders owned by root	survived
Network shares my user could not write	survived

It also explains the creepiest detail of the recovery. My password manager’s data folder was one of the casualties. Its database disappeared at 10:12, and a few minutes later the service created a fresh, empty database file and kept running. For about two hours it served an empty vault, politely, with sign-ups disabled. My single sign-on service did something similar: it kept running from memory on a folder that no longer existed.

The restore pattern I used for every damaged volume is worth copying, because it never makes anything worse:

# per damaged volume, one at a time
kubectl -n <namespace> scale deploy/<app> --replicas=0
mv <data-dir> <data-dir>.partial-<date>          # keep the damaged copy for diffing
mkdir <data-dir> && chown <uid>:<gid> <data-dir> && chmod 750 <data-dir>
<restore snapshot into data-dir>                 # read-only connection to the backup repo
# verify: restored bytes == the byte count the backup recorded
kubectl -n <namespace> scale deploy/<app> --replicas=1

The verification line is the one people skip. Every volume restored from the cluster backups that day was checked against the byte count the backup system had recorded when it made the snapshot. When the numbers matched exactly, I moved on. Where they could not be checked, I wrote that down.

And the fix that follows from the table: data folders are now owned by the service that uses them, with no write access for anyone else, including me. Convenience permissions were the holes.

Habit four: commit or lose it #

The most humbling part of the recovery was rebuilding things that only ever existed in my home folder.

Four backup scripts had to be reconstructed from their own output: the file names they produced, their retention, the layout of their archives, the permission rules that had been written for them. Each rewrite was tested with pruning turned off, so a new bug could not delete the old backups I still needed.

The router lease manager was stranger. Its source code was gone, but the program was still running. Its Python code was recovered from the running process’s memory, and the result was bytecode-identical to what had been running. Its configuration values had to be rewritten from memory.

Everything that came back went into Git, pushed to a server that is not my home folder. Then I made the rule stick. On 17 October I told the AI assistant I work with, in so many words: “commit, push, update roadmap (every single time!)”. The roadmap is the shared written record where I keep every task, mistake and runbook. Since 18 October a hook in the assistant’s tooling blocks the end of a turn when the assistant has left edits uncommitted or changed something without writing it down. A background job commits and pushes anything left quiet for too long. My local AI agent gets a nudge, then writes the journal entry itself.

It sounds bureaucratic. It is the opposite. It means I never again have to remember where something lives.

Habit five: agents get their own user #

My local AI agent used to run as me. On 4 October it came back as its own system user, with no sudo, no Docker access and no access to my home folder: it could not even list it. Its code was installed root-owned from Git, so it could not rewrite itself, and its service ran in a systemd sandbox: read-only system folders, no capabilities, a system-call filter, and outbound network denied except to the few places it needed. Today it reaches my folders only through time-limited grants that I approve.

Then it got a gate: every tool call is rated read, write or critical, writes need my approval, and critical actions need a code sent to my phone. That is described in read, write, critical: permission tiers for a local AI agent.

The tests changed too. Any test that deletes, traverses paths or writes outside its own folder runs in a throwaway container built from a clean copy of the code, with no network. If such a test escapes its sandbox now, it destroys an empty container. The agent’s whole test suite runs that way; at the last count it was more than a thousand tests.

Habit six: nobody watching means nobody acting #

After the wipe, anything that runs without a human present only proposes. Scheduled reports, monitors and helper agents can read freely, but every change they want lands in an inbox as a proposal with the exact command and a reason. I read it later and decide.

The AI assistant follows a version of the same rule. It prepares changes, tests them and commits them, but privileged applies on the live system are often mine to run. Its own safety layer stops it when a change would weaken security, and waits for me. It slows us down. Some days I am the bottleneck between a finished change and a deployed one. That is the point.

This rule has since been loosened, carefully and only for a few reversible actions. That is the subject of another post.

What it cost, and what is still not perfect #

About two days of recovery. A morning of vector-database writes that were not in any backup. Three small apps whose folders had no backup at all and came back empty. Some alert silences. An SSH key that could not be recovered, and a few credentials that were printed in a terminal during the recovery and so had to be rotated.

And the open list. The audit I ran that weekend produced a long list of things to fix. Many are closed. Not all of them are. I would rather say that than pretend a disaster ends when the services come back.

Questions people ask me #

What should I do in the first hour after accidental data loss? #

Stop writing to the affected disk, including scheduled trim jobs, and stop any automation that can act on the system. Then find the time window from logs, and only then start restoring. Every hour of writes reduces what raw-disk recovery can find.

Do you still let AI coding assistants work on your machine? #

Yes, every day. They work with narrower permissions, destructive tests run in disposable containers, every change is committed and recorded, and privileged steps go through me. An assistant’s mistake still runs with whatever access you give it.

How often should a home lab be rebooted? #

Often enough that a reboot is not an event. Mine gets a deliberate reboot after any large change, followed by the check script. Problems that only appear on restart are much cheaper to find on purpose than by surprise.

What is the cheapest change with the biggest effect? #

An alert on backup age. It costs almost nothing, and it turns “I think we have backups” into a fact you are told about every day.

The lesson #

The deletion took ten seconds. Nothing I changed afterwards is clever. Freezing, rebooting on purpose, hourly backups with an alarm, Git for everything, a separate user for the agent, proposals instead of actions when nobody is watching: each one is a small, boring habit. Together they mean the next bad morning will be an inconvenience, not a loss.

Disasters rarely come from the thing you were worried about. They come from the folder that was writable “for now”, the script that lived in one place “until later”, the backup that stopped “a while ago”. Make your temporary things permanent, or make them disappear.