My SSH honeypot accepted every password, so it never banned anyone

For three months, 194 attacker IPs walked straight past the automatic ban on my SSH honeypot. One of them opened 1,638 sessions. Nothing was broken in the usual sense: the honeypot logged every one of them, the database stored them, the dashboards drew them. The ban simply never fired, because the code waited for a failed login, and a honeypot is built to let attackers in.

I run a Cowrie SSH honeypot at home. It pretends to be a badly configured Linux server, records what people type, keeps the files they try to download, and feeds the addresses into my blocklist. This post is about the part nobody writes tutorials for: the plumbing between the honeypot and everything that acts on its data, and the five quiet ways that plumbing failed.

Contents · 12 sections

TL;DR #

• A honeypot that accepts logins makes “ban after N failed logins” useless. Treat the first connection as the signal.
• The shipper that turns honeypot logs into bans is a single point of failure. Mine went blind five different ways: log rotation, NUL bytes, a cached database address, a swallowed exception, and a deleted home folder.
• Every failure looked the same from the outside: the honeypot kept running and nothing got banned.
• Fixes: ban on connect with a threshold of 1, detect log rotation, sanitise input, resolve services by name, always advance past a bad line, and keep the shipper in Git.
• Not every visitor is hostile. A research scanner showed up as a “cluster” and turned out to be a well-known benign project.

How the pipeline works #

The chain is simple on paper:

1. Cowrie runs in a Docker container and writes one JSON event per line: connect, login attempt, command, download, disconnect.
2. A small Python shipper, run by systemd, tails that file. It pushes each event to Loki for dashboards and inserts it into PostgreSQL for the threat-intelligence side.
3. For each attacker address, the shipper asks CrowdSec for a 24-hour ban, with a 2-hour cooldown so one persistent address does not create a new decision every minute.
4. CrowdSec decisions are pushed to the network edge by a bouncer.

The shipper is the only writer of the honeypot tables and the only thing that issues the instant bans. If it stops, the honeypot keeps collecting but the lab stops learning from it.

Failure 1: waiting for a failure that never comes #

The ban logic was written the way you would write it for a real server: count failed logins, ban when the count crosses a threshold. The call to check_and_ban() lived inside the handler for cowrie.login.failed.

But Cowrie accepts logins by design. Its user database lets root in with almost any password, and admin:admin works too. That is the point: you want the attacker inside the fake shell, so you can see what they do.

So an attacker whose first guess “worked” never produced a failed login, and never got banned. When I finally asked why one address from a hosting provider had 1,638 sessions and no ban, the answer was in a single if. Counting back, 194 IPs had escaped the same way between March and June 2026.

The fix:

# Any connection to a honeypot is hostile. Ban on the earliest event.
BAN_THRESHOLD = 1

HANDLERS = {
    "cowrie.session.connect": on_connect,
    "cowrie.login.success":   on_login,
    "cowrie.login.failed":    on_login,
}

def on_connect(event):
    record_session(event)
    check_and_ban(event["src_ip"])

I also banned the 194 addresses retroactively. The lesson is simple: a honeypot has no legitimate users, so there is nothing to wait for.

Failure 2: log rotation and a seek past the end of the file #

In April the Grafana dashboard for the honeypot went empty. The shipper process was running. It just was not reading anything.

The shipper stores its file offset so a restart resumes where it stopped. Log rotation had reset cowrie.json to a fresh file of about 1.7 MB, while the stored offset was over 11 MB. In Python, seek() past the end of a file is not an error. It silently reads nothing, forever.

size = os.path.getsize(path)
if offset > size:          # the file was rotated or truncated
    log.warning("offset %d > size %d, restarting at 0", offset, size)
    offset = 0
fh.seek(offset)

Any tailer that persists an offset needs this check. CrowdSec itself hit the same class of bug on my web logs in June: a copytruncate rotation left its read position beyond the new file size, and it missed about 7 hours of events every night until I switched to rename-and-reload rotation.

Failure 3: NUL bytes from a TLS probe #

In May the shipper got stuck in a 10-second error loop. No new bans for about nine hours.

Some scanners do not speak SSH at all. They throw a TLS handshake at every open port. Cowrie logged those connections faithfully, including the client “version” string, which contained \u0000. After json.loads() that became a real NUL byte, and PostgreSQL refuses NUL bytes in text columns.

The worse half of the bug was the error handling. The generic except printed the error without advancing the offset, so the shipper read the same poisoned line again, failed again, and never got past it.

def _sanitize(value):
    if isinstance(value, str):
        return value.replace("\x00", "")
    if isinstance(value, dict):
        return {k: _sanitize(v) for k, v in value.items()}
    if isinstance(value, list):
        return [_sanitize(v) for v in value]
    return value

And the rule I now apply to every log consumer: a line that cannot be processed is logged, counted and skipped. It never blocks the lines behind it. When the fix went in, the shipper chewed through the nine-hour backlog and banned dozens of addresses that had been waiting.

Failure 4: a database address cached at startup #

In late June, PostgreSQL restarted after a storage change and came back with a new internal service address. The shipper had looked the address up once at startup and cached it. It spent hours in a loop of timeouts and “cursor already closed” errors.

Two bugs in one:

• The cached address was never cleared on reconnect, and there was even a hard-coded fallback address.
• The error handler checked whether the connection was closed, but the cursor can be dead while the connection looks alive.

The fix was to remove the fallback, resolve the database by its cluster DNS name every time it reconnects, and recreate the cursor on any database error. The general rule, which I now apply across the lab: refer to services by name, never by an address you looked up earlier.

Failure 5: the shipper lived in one place only #

In October an accident deleted my whole home folder (I wrote about that in “The test that deleted my home folder”). The honeypot container kept running on its in-memory configuration, but the shipper’s source code had lived in that home folder, outside Git. The running shipper went blind at the moment its files disappeared.

Recovery was unusual. The Python process was still alive, so the code objects were still in memory. I dumped them from the running process with a debugger and rebuilt the source until every function compiled to byte-for-byte identical bytecode. The still-open, already-deleted log files were rescued through /proc/<pid>/fd. Then I replayed the 1,243 events that had arrived during the outage through the normal processing function, which banned 26 attackers who would otherwise have been missed.

One detail worth knowing: Loki rejected the replayed entries older than about an hour, because of its out-of-order window. The dashboards have a gap for that morning. PostgreSQL, which accepts late writes, is complete.

The shipper now lives in Git, and the path the service runs is a symlink into the repository.

Not every visitor is an attacker #

“Ban on first contact” is the right default, but it is not a law of nature. One fingerprint cluster grew by 44% in a few weeks and looked like a new campaign: 26 addresses, one probe a day each, with clockwork regularity. It turned out to be the Shadowserver Foundation, a well-known non-profit that scans the internet and publishes reports for network owners. Its reverse DNS says so, GreyNoise classifies it as benign, and it never tried a password or typed a command.

An abuse database scored those addresses 100 out of 100, which is mass-reporting noise. The lesson for anyone turning honeypot data into intelligence: behaviour first, reputation second, and check the obvious benign explanations before you write “new botnet” in a report.

What a honeypot is good for #

Once the plumbing was fixed, the data paid off. Two examples from the record:

• During one targeted SSH campaign, all 42 attacking addresses were already on the CrowdSec community blocklist. Shared intelligence works: the community had flagged them before, or at the same time as, my honeypot saw them.
• Captured downloads are classified by a separate service that reads them as files: hashes, strings, antivirus family labels. That is how a “cryptominer” guess was corrected to what it really was: a classic Linux IRC-bot worm that spreads through default Raspberry Pi credentials.

Checklist #

• Ban on the first event (connect), not on failed logins. Threshold 1.
• The shipper detects log rotation (offset larger than file size) and starts over.
• Sanitise every string field before it reaches the database; NUL bytes are common.
• A bad line is logged and skipped; the offset always moves forward.
• Services are reached by DNS name, re-resolved on every reconnect. No cached or hard-coded addresses.
• Recreate the cursor on any database error, not only when the connection is closed.
• The shipper’s source, configuration and service unit live in Git.
• Alert on “no new honeypot events for N minutes”. A honeypot on the internet is never quiet, so silence means the pipeline is broken.
• Store captured files for analysis in a separate service; never execute them on the host.
• Before calling a cluster hostile, check reverse DNS and benign-scanner classifications.

Frequently asked questions #

Should a honeypot ban attackers at all? #

I think so, if it is part of a wider defence. The address that probes the honeypot is probing your real services too. Banning it on first contact, for a limited time, protects the rest of the network while the honeypot still records the session.

Why not just count failed logins like fail2ban? #

Because a honeypot accepts logins on purpose. Many attackers succeed on their first guess and never generate a failure. On a honeypot, the connection itself is the evidence.

How do I know my honeypot pipeline is still working? #

Watch the output, not the process. An internet-facing SSH honeypot receives connections every few minutes. Alert when the newest stored session is older than a short threshold, and when no ban has been issued for an unusual period. Every failure in this post left the processes “running”.

Is it safe to keep the malware attackers download? #

Keep it as inert data: stored with no execute permission, analysed by tools that read files rather than run them, and kept away from the host’s real environment. The value is in hashes, strings and family attribution, not in running it.

The lesson #

The honeypot never failed. The code around it did, five times, and each time the symptom was silence. A security pipeline that fails quietly is worse than one that fails loudly, because it lets you believe you are protected. Design the honeypot’s plumbing like any production data pipeline: tolerate bad input, survive restarts, keep the code in Git, and alarm on the absence of data.