From watching to answering: when my home lab began to defend itself

On Friday, 24 April 2026, I ran a small test from inside my lab that reached out to the internet. Within seconds, at least five scanning machines from four countries turned around and probed me back. I did not touch anything. My defences noticed them, decided they were hostile, dropped them at the edge of my network and sent a message to my phone. The whole chain, from the first probe to the block, took under 60 seconds.

I had spent a month watching attacks. That was the first time I watched the lab answer one by itself.

This post is about that shift: from a system that records what happens to it, to a system that acts. It is also about what I learned once it did. That I was not alone. That my answers could hurt my own house. And that a defence you cannot see working is a defence you should assume has stopped.

Contents · 9 sections

In one minute #

• From March 2026 my lab watched: a honeypot recorded attacks, dashboards counted them, and I read the numbers.
• On 24 April a test proved the lab could also answer: detect, decide, block at the edge, and notify me, in under a minute, with no human in the loop.
• On 8 May I learned how little of that answer was mine: every one of 42 addresses in two targeted campaigns had already been flagged by a community of other defenders.
• Automating the answer created new ways to fail. In June the defence sent 2,131 requests to my router in a few minutes and crashed it. In September a scheduled job turned out to have been silent for 24 days.
• The lesson I carry: every automatic answer needs its own proof of life.

The before: a very good audience #

My honeypot went live on 21 March 2026. It is a decoy server that pretends to be an easy target and records everyone who tries to get in. Within days the numbers were large enough to stop feeling personal. By 24 April the public page I built that day quoted 331,000 events from 3,182 attacking addresses.

I had also installed CrowdSec, an open-source engine that reads logs, recognises hostile behaviour and issues “decisions”, which are bans with an expiry. The oldest note about it, from 2 April, describes three layers: detection on the host, a firewall component that enforces the bans locally, and a sync that copies bans to the router at the edge of my network. The same note says there were already more than 21,000 active bans.

So technically the lab was already defending itself. But my relationship with it was that of a spectator. I looked at dashboards. I read session logs. I counted. If you have ever installed a security camera and then spent evenings watching the footage, you know the mood: fascinated, a little uneasy, and entirely passive.

Watching has a ceiling. You learn a lot. You protect nobody, including yourself.

The trigger: the lab hits back #

The test on 24 April was meant to check something else. A test pod in my cluster reached out to a target on the internet. What came back was not the answer I was looking for.

Internet-wide scanning services keep sensors everywhere. When they see a new source of outbound traffic, they look at it. Within seconds, my note says, “5+” of their nodes, from Singapore, Germany, Canada and the United States, were probing my public services at the same time.

And then the part I had built but never seen run end to end. The record lists the chain in one line: my outbound test, their counter-scan, detection by CrowdSec on the host, a block on the MikroTik router at the edge, a notification on Telegram, “all within 60 seconds”.

I did not decide anything. A parser read my web proxy’s access log, a scenario matched the behaviour, a decision was created, a bouncer pushed it to the router, and a bot told my phone. By the time I read the message, the addresses were already being dropped.

If you have ever built something and then watched it work without you, you know that small, quiet click. It is not triumph. It is closer to hearing a lock shut behind you in a house you built yourself.

I wrote the incident onto the new Defense page that same day, under the name “scanner backlash”. The page also carried a figure I had not paid much attention to until then: more than 33,000 signals already pushed from my lab to the community. I was answering, and I was also, without quite noticing, talking.

The humbling: everyone had already seen them #

The moment that changed how I think about all of this came two weeks later.

On 8 May I finished a long investigation into two targeted SSH campaigns against my honeypot. Part of it was a cross-check of every attacking address against my own defences.

The result was not about me at all. The record is short:

All 42 IPs from both targeted SSH campaigns were already banned in CrowdSec via the community threat-intel-feed.

All 42. The community, a network of other servers sharing what they see, had flagged every one of them, either before they reached me or with my own honeypot’s help. My careful investigation had arrived at a conclusion the network already held.

Two months later I measured the scale of that. On 12 July my host was enforcing about 52,870 active decisions. About 32,520 came from the community feed. About 19,750 came from public blocklists I import every day. My own lab’s decisions numbered about 600.

Around 1 per cent.

If you have spent months building something alone at night, that number does two things at once. It deflates you: the wall protecting your house is almost entirely built by strangers. And it lifts you: you are not alone, and the bricks you add go into everyone else’s wall too. My lab’s own shares are small, but they are real, and they protect servers I will never see. That idea later became its own post.

A second discovery from the same summer made the point sharper. In July, surveying 152 web campaigns, I found that 57% of the attackers I had classified were running from the big cloud providers. Their addresses are recycled constantly. Reputation, the idea that an address is “bad”, barely works there. What works is behaviour: what a visitor does, not where it comes from. A community that shares behaviour-based verdicts, quickly, beats any single defender’s memory.

The build: when the answer becomes a hazard #

Once a system answers by itself, it can also answer wrongly, too loudly, or not at all. I found all three.

The defence that slept every night #

On 11 June I noticed scanners that should have been banned, and were not. Every one of the misses fell in the same window: the hours after midnight, when the web proxy’s log was rotated. The rotation method kept the file but emptied it, and the detection engine kept reading from its old position, far past the new end of the file. It saw nothing until the file grew back. My notes put it at 7.5 hours of blindness, every night.

The fix was a different rotation method and a second, independent watchdog that scans the same logs every few minutes as a safety net. Its first run caught 10 addresses the main engine had missed. A later check found it catching about three slow scanners an hour that slipped past the primary detection. I wrote the detailed version of this class of bug in the honeypot pipeline post.

The answer that never left the building #

On 22 June two mistakes surfaced on the same day, and they are mirror images.

The first: the job that copies bans from the host to the router had a comment in the schedule, but not the schedule line itself. For a while, bans were being decided and never reached the router. The lab was answering in a room with no door.

The second came when I fixed that. The sync script had no lock, no rate limit and no tolerance for small timing drift, so on every run it rewrote close to 200 entries that had not really changed. It sent 2,131 add requests and ran 40 sync cycles in minutes. The router, a small embedded device, could not keep up and crashed. My defence had knocked over my own front door.

In September a bigger version of the same storm happened, for a different reason, and in the same investigation I found that a scheduled job meant to lengthen bans for repeat offenders had not fired for 24 days. That whole chain is in the post about vanishing bans. I mention it here only for the shape: every time I made the answer stronger, I found a new way for it to fail silently.

The answer that hit the wrong people #

The honest weakness of an automatic defence is that it cannot be modest. It does not hesitate. In October I found that my own router had been banning the crawlers of the search engines I wanted to be found by; Google’s tools reported only a vague connection error. That story is here. Automation answers at the speed of a rule, and a rule is only as wise as its exceptions.

The gap that was not a gap #

Not every scare was real. On 12 July I saw that my router held far fewer bans than the host, and assumed the sync was broken again. I traced the whole path, end to end, before changing anything. There was no gap. Enforcement happens in more than one place by design, and each place holds what it is meant to hold. I made no code change that day. Sometimes the bravest operational act is reading your own system carefully enough to leave it alone.

The technical heart: an answer is a loop, and every step needs a pulse #

Here is the model I use now, in words, for any automated defence. It is general; the same shape applies to rate limiting, fraud rules or abuse handling.

1. Sense. Something reads raw signals: logs, sessions, requests.
2. Decide. Something turns signals into a verdict with an expiry.
3. Enforce. Something applies the verdict where traffic actually flows.
4. Share. Verdicts go out to others, and theirs come in.
5. Prove. Something independent checks that steps 1 to 4 are still happening.

Most people build the first four. Most of my real failures were a missing fifth step. Sensing went blind after log rotation; enforcement never ran because a schedule line was missing; a renewal job sat dead for 24 days. Each stage looked healthy from inside, because a stage that is not running reports nothing at all.

So each stage now leaves a heartbeat, and something else reads it. A generic version, with placeholders:

#!/usr/bin/env bash
# Proof of life for one stage of an automated defence.
# The stage writes a timestamp each time it completes a run:
#   date +%s > /var/lib/<stage>/last-success
# This check runs from a different scheduler and alerts if it is stale.
set -euo pipefail
stamp=/var/lib/<stage>/last-success
max_age=<seconds>

now=$(date +%s)
last=$(cat "$stamp" 2>/dev/null || echo 0)
age=$(( now - last ))

if (( age > max_age )); then
  notify "<stage> has not completed a run for ${age}s"
  exit 1
fi

Three rules came with it, each paid for by one incident above:

• A heartbeat must be written on success, not on start. A job that starts and dies should look dead.
• The checker must not share a failure mode with the checked. If the same scheduler runs both, one broken timer silences both.
• A failed read is not an empty answer. If a stage cannot read its own state, it must stop, not assume there is nothing there. Treating a timeout as “empty” is what turned a slow router into a storm.

And one rule for the machines on the other end of the cable: rate-limit anything that talks to an embedded device. Your router is not a database.

What changed #

• My role. I went from spectator to operator. I no longer watch attacks; I watch the answers, and especially the silences between them.
• What I trust. Behaviour over reputation, shared verdicts over my own memory, and an independent check over a green status.
• How I measure. Not “how many did I ban” but “how fast, and can I prove the chain still works today”.
• How I feel about the community. About 1 per cent of my protection is my own work. That is a reason to contribute more, not less.

The costs, plainly: an automated defence is a second system to run. It has crashed my router, gone blind at night, gone silent for weeks, and banned visitors I wanted. Each of those cost an evening or several, and each left a new check behind it.

Questions people ask me #

Is a community blocklist worth using on a home server? #

In my data, overwhelmingly yes. On one day in July, roughly 99 per cent of the active bans protecting my lab came from the community feed and public blocklists, not from my own detection. In one investigation, all 42 attacking addresses were already flagged before I finished looking.

How fast can a home lab block an attacker automatically? #

In my first end-to-end test, under 60 seconds from the first hostile probe to a block at the router, with a notification on my phone. The speed is not the hard part. Keeping every step of that chain alive, every day, is.

What is the most common way automated defences fail? #

Silently. A stage stops running and reports nothing, so everything looks green. Mine failed after log rotation, after a missing schedule line and after a timer stopped firing. Give each stage a heartbeat written on success and check it from somewhere else.

Can an automated defence harm your own network? #

Yes. Mine crashed my own router by flooding it with updates, and later banned search engine crawlers I wanted. Rate-limit anything that talks to network devices, abort on failed reads, and review what the automation blocks, not just how much.

The closing #

The first time the lab answered by itself, it took under a minute and I was not needed. That is still the goal. But I have stopped thinking of the defence as a wall. It is closer to a conversation: thousands of servers telling each other, in seconds, what they have just seen. My lab’s voice in that conversation is small, about one word in a hundred.

What I can control is whether that voice is still speaking tomorrow. So I no longer ask the lab what it blocked today. I ask it to prove it is still awake.