Why I threw away my first security lab after one month

At 07:53 on Saturday 23 May 2026 my home security lab was declared clean. 72 minutes later its replacement existed, built from scratch. The 07:53 roadmap entry is proud of it: the flat directory split into components, a 3,964-line web page split into separate CSS and JavaScript files, a 3,206-line script decomposed into 23 modules. “All pods healthy.”

Seventy-two minutes later, at 09:05, the next entry reads: “Created [the v2] microservice platform from scratch: 6 services, frontend SPA, CronJobs, full K8S deployment.”

I had just spent the morning cleaning the house, and then I built a new one next to it. By the evening the new platform was committed to Git, its numbers matched the old one exactly, and two weeks later the old one was switched off for good.

This post is about that day: why a working system was thrown away after about a month, what the rewrite looked like hour by hour, and what it cost.

Contents · 12 sections

In one minute #

• My security lab watches attacks on my own home network: an SSH honeypot that pretends to be a weak server, web-traffic analysis, and the firewall and blocking tools behind them. It is defensive. The attackers come to me.
• Version 1 grew fast in April 2026: one dashboard, one database file, an AI assistant, more than 50 tools, everything in one place.
• By late May it was too big to change safely and too slow to trust. I rewrote it in a day as six small services with clear jobs.
• The rewrite was not done until the new system reproduced the old numbers exactly: 17,347 sessions and 4,288 attacking addresses. The first version of v2 showed about half of that.
• Version 2 became the base for everything that followed, including more than 800 public threat reports.

The before: a lab that grew in a month #

Version 1 was born in April. By 24 April an inventory counted “52 MCP tools, 10 tool groups, 3 scan profiles, 4 model tiers, 9 UI views, 20+ API endpoints, 7 external integrations” across 13 source files. MCP is the protocol that lets an AI model call tools; I will come back to it in another post.

It had everything I wanted from a small security operations centre: a dashboard for the honeypot, runtime security alerts from Falco, the defence pipeline that bans attackers at the router, a weekly report, and an AI agent you could ask questions in plain language.

It also had every shortcut I took to get there in weeks.

The honeypot data lived in one SQLite file. About 88 MB plus a 78 MB write-ahead log. The honeypot wrote sessions and credentials into it while the enrichment job wrote into the same file, so they fought over the lock. One fix in the record raised the wait timeout to 120 seconds and added retry loops. For a while the dashboard’s own API returned empty results while direct SQL queries against the same file “work perfectly”.

The web interface was one file. By May the main page was 3,964 lines. Kubernetes stores configuration objects with a size limit, and the UI files together passed 262,144 bytes, so even deploying them needed a special flag.

The weekly report had been failing silently. On 23 April I found that the reporter job had been broken since 19 April: eight syntax errors in its format strings and wrong parameter names for the tools it called. It ran every Sunday and produced nothing.

The AI agent was slow in a way that broke things. It ran a 26-billion-parameter model on the CPU, so each model call took 30 to 60 seconds. An investigation that used four tools easily passed five minutes, and the reverse proxy’s write timeout cut the live stream in the middle. The record names it precisely: “Traefik writeTimeout is the SSE killer”.

None of these was a disaster. Together they meant that every change touched everything, and I could not tell from the dashboard whether a number was true.

A moment that taught me what this lab is for #

One observation from 24 April is worth telling before any bug.

A test scan from inside the lab went out to the internet. Within seconds, five or more scanning nodes from four countries probed my own infrastructure back, looking for known Jira vulnerabilities, exposed .env files and Docker registries. The defence stack caught all of it: the security engine detected the probes, the router blocked them, and a Telegram notification arrived, “all within 60 seconds”.

If you have never run a server on the open internet, this is the part that is hard to believe. You are not anonymous for long. The moment your address does something interesting, machines you have never heard of come to look.

Read today, that entry shows what the lab is actually good at: not the scanning, but the watching, the recording and the defending. Version 2 was built around those.

The trigger: you cannot refactor trust into a monolith #

The record does not explain the decision in words. It shows it. The 07:53 entry fixed version 1 in place: split the page, split the scripts, move the code into folders per component. It worked. The pods were healthy.

But splitting files does not split responsibilities. The dashboard still talked to one big backend. The honeypot, the tools, the agent and the reports still shared one process and one fate. A slow scan could block a page. A broken query could take the agent with it.

The next entry is the new platform.

The build: one long Saturday #

The work sessions of that day are in the record with their timestamps, and they read like a race.

09:05. The new platform exists: “112 files: 6 Python services (gateway, honeypot, tools, agent, reporter, frontend), 2 CronJobs, Vite SPA with 12 code-split views”. All six pods running, all health checks passing.

09:37 to 10:26. The honeypot views are rebuilt one after another: overview, timeline, intelligence, sessions, credentials, commands, downloads, fingerprints, networks, campaigns, sharing. The reason recorded for several of them is honest: “The v2 honeypot tabs were still thin v1-style tables and lacked intelligence drill-downs.” A rewrite is only an improvement if you rebuild what people actually used.

16:15. The scripts that feed live numbers to my public website are moved from the v1 APIs to v2.

16:49. The whole path from honeypot to database to security engine to router is verified. Along the way I found that the service shipping honeypot logs into PostgreSQL had been stale for two days; 6,124 sessions were backfilled. And I found that automatic banning from the honeypot had never actually been implemented, because the security engine cannot parse the honeypot’s logs on its own. It was built that afternoon. That story, and why a honeypot that accepts every password can end up banning nobody, is in My SSH honeypot accepted every password.

17:51. “Committed entire [v2] platform to Gitea.”

19:55 and 22:05. The AI agent is rewritten twice. The small model ignored stop instructions whenever tool definitions were in its context, and looped forever. The fix was a two-phase design: one call picks a tool, a second call with a clean context writes the answer. The outcome line is my favourite in the whole project: “Agent responds in 48s (greeting) or 108s (tool query). No more looping.”

Forty-eight seconds to say hello. On one machine, with a small local model, it counted as a win.

The technical heart: parity before switch-off #

The most important step of the rewrite was not in the new code. It was a comparison.

When v2 first showed its headline numbers, they were about half of what v1 showed for the same honeypot. Nothing had been lost. The new API was simply asking different questions:

• its default time window was 30 days, while there were 63 days of data;
• it counted unique attackers from the sessions table, which only gets a row when a login succeeds, instead of from the credentials table, which records every attempt (1,107 against 4,288);
• it summed counters stored on sessions instead of counting the rows of the commands and downloads tables.

Every one of those is a reasonable choice in isolation. Together they made the new system look like it had found half as many attackers. If I had switched v1 off that morning, I would have believed it.

The rule I took from it, in generic form:

-- Run the same question against the old and the new system,
-- over the SAME window, with the SAME denominator, before you retire anything.

-- old system
SELECT count(DISTINCT src_ip) FROM credentials;                 -- every attempt

-- new system, first draft (wrong denominator)
SELECT count(DISTINCT src_ip) FROM sessions
 WHERE start_time > now() - interval '30 days';                 -- successes only, last 30 days

-- new system, fixed
SELECT count(DISTINCT src_ip) FROM honeypot_credentials;        -- every attempt, all time

After the fix the record lists the numbers side by side, and they are identical: sessions 17,347, addresses 4,288, credential attempts 118,816, successful logins 12,494, commands 3,359, downloads 198. Only then was the rewrite real.

The general lesson is older than computers. When you replace a measuring instrument, you measure the same thing with both until they agree, and you find out why they do not.

Switching off #

Version 1 did not die on the day of the rewrite. It was taken apart in pieces:

• on 6 June, its 45 security and intelligence tools were moved into my main MCP server, so AI agents could keep using them from one place;
• on 7 June, version 1 was shut down: its report job suspended, every deployment scaled to zero, 11 old jobs deleted. In the same clean-up the cluster lost 525 stale replica sets and 33 GB of old container images;
• on 7 July, the v1 SQLite database was archived to the NAS with its checksum verified on transfer, frozen with data up to 23 May.

I did not delete the old data. I made it read-only and put it somewhere safe. That turned out to matter.

What it cost #

A rewrite moves data, and moved data carries its history with it.

On 30 July, while building a census of attack bursts, an observation marked critical stopped the work: one attacking address had a “first seen” date of 21 May in one table, while the timeline gave it 535 sessions on 29 March. The leading explanation was the migration itself: the legacy v1 corpus had been loaded into the sessions table, while the first-seen dates had been computed only over rows written after the move. The instruction in the record is clear: “do NOT publish burst chronology until closed”. Every claim older than late May now carries a caveat.

It also cost the comfort of a single place. Six services and a gateway are easier to change and harder to see whole. In August the same kind of defect turned up three times in one day, and the record gave it a name, the inert-control archetype: “A control that is ACCEPTED, VALIDATED, STORED, DISPLAYED — AND INERT.” A ban threshold that could be set per rule but was collapsed to the minimum for the whole category. A table of rules that one component read and another ignored. Nothing ever looked unconfigured, so nothing prompted a check.

The same note has the line I find most useful: “The classifier itself measured 3,826/3,826 clean. Defects concentrate in the layer that MEASURES and CONFIGURES, not the layer that decides.”

The after: what changed #

The rewrite made the lab boring to change, which is the best thing you can say about a system you depend on.

The roadmap for version 2 holds more than 740 tasks, most of them from June and July, when the platform grew from a honeypot dashboard into a web-threat analysis system, an enrichment pipeline, an entity graph and a dossier publisher. Those dossiers, built on v2’s data, are now more than 800 public threat reports, each with a seal anyone can check, as described in Signing threat intelligence.

The AI side changed too. The agent that needed 48 seconds to say hello was a first step. Later I tried to let a local model drive a full autonomous security agent and had to shelve it; the reasons are in Why local LLMs could not drive an autonomous pentest agent.

And the lab’s direction settled. Version 1 was a toolbox for looking outwards. Version 2 is an instrument for looking at what comes in, measuring it honestly, and sharing it so others can defend themselves.

What changed #

• Architecture: one process, one database file, one huge page became six services behind a gateway, PostgreSQL, and a modular single-page app.
• Truth: new numbers were not trusted until they matched the old ones exactly.
• Retirement: tools moved first, then the service was scaled to zero, then the data was archived read-only with a checksum.
• Cost: a provenance caveat on everything before the migration, and a new class of quiet configuration bugs that only a distributed system can have.
• Purpose: from tools that probe to an instrument that watches, records and defends.

Questions people ask me #

Why rewrite instead of refactoring the old system? #

Because the problems were in the boundaries, not in the code. I refactored version 1 that same morning and the pods stayed healthy, but every part still shared one process and one database file. Splitting files does not split failure. Separate services with their own jobs do.

How do you know a rewrite did not lose data? #

Run both systems side by side and ask them the same question, over the same time window, counting from the same table. My first v2 showed about half the attackers of v1 because of a shorter default window and a different denominator. Only when the counts matched exactly did I switch the old one off.

Is a home security lab like this legal and safe? #

Watching attacks on your own network is ordinary defensive work: a honeypot is a decoy server you own, and the traffic comes to you. Scanning or attacking systems you do not own is a different matter and usually needs written permission. My lab is built to observe, block and report, and the public reports stay at a level that helps defenders, not attackers.

What would you do differently? #

Start with PostgreSQL and separate services for anything that writes data. A single SQLite file is perfect for one writer and one reader. A honeypot and an enrichment job writing at the same time is already two writers too many.

The closing #

I once wrote a short sentence that sits well next to this story: stay humble if you can. Version 1 was built quickly and with real pride, and some of what it found still stands. But the most valuable thing it did was disagree with its replacement by a factor of two, on the day I was ready to believe the new one.

If you are about to replace something that works, keep the old one running a little longer than feels necessary. Ask both the same question. The difference between their answers is where you will learn what your new system actually measures.