Why my long CrowdSec bans vanished after two days

I had a tool that gave repeat attackers longer and longer bans: a day, then weeks, then most of a year. Its ledger said 1,040 addresses were inside an active long ban. CrowdSec, the engine that was supposed to enforce them, held 2.

The bans were being created correctly. They were being deleted about two days later, by a database housekeeping setting I had never looked at. Then, when I fixed that, the fix doubled the size of the block list and knocked a second component into a loop that hammered my router for an evening. This post is both stories, and the mechanisms behind them, because both are easy to hit with a default CrowdSec install.

Contents · 10 sections

TL;DR #

• CrowdSec’s local API flushes old alerts when it reaches max_items, and it deletes their decisions with them, whatever --duration you set. With ~2,500 alerts a day and a limit of 5,000, every ban older than about 48 hours disappeared.
• My escalation timer had also been dead for 24 days, because of a systemd timer pattern that never fires again after an interrupted run.
• cscli decisions list shows 100 results by default and only the newest decision per IP. Do not count bans with it.
• After the fix, the block list grew from ~1,500 to ~2,250 entries. A bouncer that treated a failed read as an empty list re-added everything, over and over: 18 storms in one evening.
• Fixes: re-assert long bans every run, use OnCalendar timers, lock with flock, and make a failed read abort the run.

The setup #

Three layers, all defensive:

1. CrowdSec reads logs from the web proxy, the SSH honeypot, the mail server and the SSO portal, and creates decisions (bans). It also imports a daily community blocklist of about 22,700 addresses.
2. A bouncer script runs from cron and mirrors CrowdSec’s decisions into an address list on the edge router, so banned traffic is dropped before it reaches any service.
3. An escalator I wrote keeps a ledger of every address that comes back after a ban. Each return moves it up a tier, and each tier means a longer ban. The ledger held 8,338 IPs and 6,329 escalations.

On 11 September 2026, after moving the machine to a new location, I ran a routine health check. CrowdSec had been upgraded from 1.7.8 to 1.8.1 and was healthy. The escalator had not run since 18 August.

Bug 1: a systemd timer that never fires again #

The escalator’s timer looked like this:

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
Persistent=true

On 18 August a run was cut off mid-way: the timer stamp was written, the log stopped one second later, and the machine rebooted. After the reboot, systemd loaded the stale stamp. A monotonic OnBootSec elapse that lies in the past is only triggered if the timer has never fired, so it was skipped. OnUnitActiveSec counts from the last time the service became active, and it never did. The timer sat there “active (elapsed)” with no next run, on every boot, for 24 days.

The symptom is easy to recognise once you know it:

systemctl list-timers --all | grep <name>
# NEXT = -        LAST = an old date

The fix is to use calendar time for periodic jobs:

[Timer]
OnBootSec=2min
OnCalendar=*:0/5
RandomizedDelaySec=30
Persistent=true

The first run after the change processed 193 decisions and escalated one address. Then I audited the other timers in the lab for the same pattern.

Bug 2: the long bans were never really long #

With the timer fixed, I read the ledger against what CrowdSec actually held. The numbers did not fit:

• 1,040 IPs and 33 subnets were inside an open escalation window. CrowdSec held 2 escalated decisions.
• 828 addresses at the top tier had zero active decisions.
• 3,243 of 6,214 IP escalations (52%) had happened while the previous long ban should still have been in force.
• The router’s own log showed entries with a 243-day timeout being removed after 3 to 6 days.

The CrowdSec log had the answer, repeated every minute:

flushed N/5018 alerts because the max number of alerts has been reached

CrowdSec’s local API keeps alerts in its database and has a flush policy, db_config.flush.max_items, set here to 5,000. This host produces around 2,500 alerts a day, most of them instant bans from the SSH honeypot. So every minute CrowdSec deleted the alerts older than about two days, and deleting an alert deletes its decisions. A ban added with cscli decisions add --duration 8760h was gone after about 48 hours. The bouncer mirrors the local API faithfully, so it removed the address from the router too.

The address came back, hit the honeypot, got a fresh 24-hour ban, and the escalator counted a “return” and climbed another tier. Half of my escalation history was an artefact of the flush.

Why not just raise the limit? #

At 5,000 alerts the CrowdSec database was already about 200 MB, and two programs read every decision with --limit 0 every few minutes. Raising the limit would make both slower and the database larger, and the next growth spurt would bring the problem back. The robust answer is that anything that issues long bans must re-assert them, or keep them outside the alert database.

So the escalator now does a renewal pass on every run. For each ledger entry with an open window, if CrowdSec no longer holds a decision for it, or the alert behind it is older than 18 hours, the escalator re-creates the ban, 200 per run. Two more guards stop the false escalations:

• an alert for an address that is already under an active escalated ban is not a return;
• the escalator ignores alerts created by its own scenarios.

The first run with the new version found 9 real returns, renewed 200 bans and left a backlog of 839 for the following runs.

Two cscli traps on the way #

While counting, I tripped over the tool itself twice.

cscli decisions list returns 100 results by default. In June that meant the bouncer synced 100 of 229 decisions to the router. Any script that reads the list must pass --limit 0.

It also returns one decision per value, the newest. An address with an active 8,158-hour escalated ban and a newer 24-hour honeypot ban shows only the 24-hour one. To see everything for one address:

cscli alerts list -i <ip>

Never count active bans per IP with decisions list.

Bug 3: a failed read is not an empty list #

The renewal pass worked. Within hours the router’s block list grew from about 1,500 to 2,250 entries. That evening my local monitoring agent raised an alert: router CPU at 14–17%, against a normal 0–3%.

The bouncer log showed 18 sync runs that each found 0 entries on the router and re-added about 2,260. Five copies of the bouncer were running at once. The chain:

1. The bouncer reads the router’s address list with curl --max-time 10. If the request fails, the script falls back to [].
2. The list was now larger. Under load, the read took longer than 10 seconds.
3. The bouncer concluded the router held nothing and re-added every entry, which takes about 8 minutes of API calls.
4. Its lock was a “stale after 300 seconds” heuristic, so the next cron run decided the lock was abandoned and started on top.
5. More concurrent runs meant more load, slower reads, more timeouts: a feedback loop.

I had multiplied the rows in a shared list without reading the consumers of that list. The fallback “failed read means empty” turned every timeout into a mass re-add.

The patched bouncer:

exec 9>/run/lock/<bouncer>.lock
flock -n 9 || { echo "already running"; exit 0; }

if ! current=$(curl -sf --max-time 60 "$ROUTER_API/<list>?.proplist=address,timeout"); then
    echo "router_list_read_failed" >&2
    exit 1                       # abort; never assume the list is empty
fi

flock releases the lock when the process dies, so there is no “stale” guess. Asking the router only for the fields the script uses (.proplist) makes the read much lighter. With the fix, router CPU sat at 2–8% with the bouncer running every minute.

This was not the first storm. In June, an earlier version without a lock or any throttling had made 2,131 add calls in a few minutes and crashed the router. The fixes then were a lock, a 200 ms pause between writes, a cap of 10 notifications per run, and tolerance for small timeout differences so the bouncer stops re-writing entries that are already right. Embedded devices have small API budgets. Treat them gently.

A smaller trap: the notification channel #

The escalator posts to Discord. The record showed 6,433 failed notifications. Discord’s edge answers HTTP 403 with “error code: 1010” to Python’s default urllib User-Agent, while the same webhook works from curl. Setting an explicit User-Agent header fixed it. While I was there, the bot credentials moved out of the script and into a root-only environment file loaded by the systemd unit.

Checklist #

• Know your CrowdSec flush.max_items and your daily alert rate. Divide one by the other: that is the real maximum age of any ban.
• Anything that issues long bans re-asserts them on a schedule.
• Periodic systemd jobs use OnCalendar, not OnUnitActiveSec with Persistent=true.
• Scripts reading decisions use --limit 0; per-IP questions use cscli alerts list -i.
• Before a change that multiplies rows in a shared list, read every consumer: timeouts, fallbacks, locks.
• A failed read aborts the run. It never means “empty”.
• Locks use flock, not age heuristics.
• Throttle writes to routers and other embedded APIs, and watch router CPU after any change to ban volume.
• Alert on “escalator last run” and “bouncer last sync” timestamps, not just on errors.

Frequently asked questions #

Why do my long CrowdSec bans disappear early? #

Check the local API’s flush settings. When the number of alerts exceeds db_config.flush.max_items, CrowdSec deletes the oldest alerts together with their decisions, regardless of the ban duration. On a busy host that can be a matter of days.

Is the CrowdSec community blocklist worth importing? #

In my data, yes. During one targeted SSH campaign, every one of the 42 attacking addresses was already on the community list. Expect the daily re-import to collide with other jobs that read decisions at the same minute; mine logs one harmless error at midnight and catches up five minutes later.

Why did my systemd timer stop firing? #

If it combines OnBootSec, OnUnitActiveSec and Persistent=true, an interrupted run followed by a reboot can leave it with no next elapse. systemctl list-timers shows NEXT as -. Use OnCalendar for periodic work.

How should a bouncer handle a router API timeout? #

Abort the run and try again next time. Treating a failed read as an empty list makes the bouncer re-add everything, which increases load and causes more timeouts.

The lesson #

Every component here did what it was configured to do. CrowdSec flushed what its limit told it to, systemd honoured a stale stamp, and the bouncer believed its fallback. The bugs lived in the gaps between them, in assumptions nobody had written down: that a ban lasts as long as its duration, that a timer keeps ticking, that “no answer” means “nothing there”. Measure what the enforcement layer actually holds, compare it with what you think it holds, and make every failure path fail closed.