A chat-ops bot is a doorway, not a convenience

For six hours one morning my infrastructure chat bot went silent. No crash, no restart, no error I could see from the outside: the service was up, running for three weeks straight, zero restarts. The cause was not in the bot at all. My own router’s firewall had added the single IP address that the messaging API resolves to on my network to a block list, and one entry took the whole bot offline.

That outage, and a self-healing feature that turned out to be decorative for weeks, are the two lessons at the centre of this post: a chat-ops bot is only as reliable as the path it reaches you on, and detection without remediation is theatre.

Contents · 11 sections

TL;DR #

• A chat bot is a thin control surface over your infrastructure. Its hard problems are reachability, authorisation and testing, not the commands.
• When the bot goes silent, check the network path before the bot. The process can be perfectly healthy while a firewall eats its traffic.
• Detection without remediation is theatre. An alert webhook with no receiver configured did nothing for weeks while looking wired.
• Test handlers the way the service calls them: the exact argument types, and inside the service’s own sandbox, not a bare root shell.
• A root service running a user-owned script is a privilege-escalation path. Fix the ownership.

What the bot is #

The bot speaks to me over two messaging platforms and also serves a small HTTP API. It handles alerts, runs infrastructure commands and reports status. Its reach spans Kubernetes and Docker operations, router management, the security tooling, storage, and monitoring queries. It receives alert webhooks on an HTTP endpoint and exposes metrics for scraping. It grew to 77 commands across about a dozen modules.

It is a control surface. That framing is the whole security model: every capability behind it is something I could do by hand, exposed through a chat window, so the bot’s job is to be reachable when I need it, to run only what I allow, and to never run something nobody asked for.

The six-hour silence #

The symptom was the worst kind: nothing. The service showed active, zero restarts, uptime of weeks. But its log repeated [Errno 101] Network is unreachable to the messaging API every forty seconds or so.

The useful distinction is what error 101 is not. It is not DNS: the name resolved fine. It is not a timeout: a timeout is a different error and a different wait. Error 101 is a routing or firewall rejection, the network refusing to carry the packet at all.

So I looked at the router, not the bot. The messaging API resolves, on my network, to a single IP address, and that exact address was sitting in a firewall block list, marked dynamic, meaning a firewall rule had added it rather than a human. Something had scanned a monitored port carrying that source address, and the rule that auto-bans scanners of those ports had blocked the entire messaging service with it. Removing the one entry restored the bot in seconds; a test call returned success in 0.17 seconds flat.

The permanent fix is a rule ordering problem: put the messaging provider’s address ranges in an allow list that is evaluated before the auto-ban rule. Until that lands, the ban will recur, because the dynamic entry comes back whenever that port is scanned from that address.

The transferable lesson is the diagnostic order. When a bot that polls an external API goes quiet, check the network path first. A healthy-looking process with a steady stream of “network unreachable” is almost never a bug in the process.

Detection without remediation is theatre #

The bot is half of a self-healing loop. Alerts fire in the monitoring stack, get routed to the bot’s HTTP endpoint, and the bot runs a bounded remediation: restart a deployment, delete a stuck pod, drop caches. Each action is rate-limited with 30 minutes of cooldown per target so a flapping alert cannot cause a restart storm, and reactive fixes run in seconds while a separate agent does slower investigation on a schedule.

That is how it works now. For a while it did not work at all, invisibly. The alerts carried auto-fix labels, but the alert router had no receiver and no route configured to deliver them to the bot. The labels were real, the handler existed, and nothing connected the two. For weeks the auto-fix was decorative.

The rule I took from it: verify the whole pipeline end to end, every link. Alert fires, router matches a route, webhook delivers, handler executes, action happens. A label that says “auto-fix” proves only that someone intended one. Test it by making a real alert fire and watching the action occur, not by reading the configuration and believing it.

A related trap on the same loop: the monitored-services list once included services that had been retired. Because the bot could restart anything on that list, a restart action could start a disabled unit back up. Keep the list of things the bot may act on in step with what actually runs.

Test the way the service calls #

Two bugs taught me that testing a bot handler in isolation proves very little.

First, the dispatcher passes command arguments as a list of words, not a string. A new handler tested by calling it with a plain string crashed on first real use with “list has no attribute strip”. Test a handler with the exact argument shape the dispatcher delivers: an empty list, and a list with one element.

Second, the service runs under a hardened systemd unit with ProtectSystem=strict, so its child processes see most of the filesystem read-only. A handler tested in a plain root shell wrote its state file happily; the same handler under the unit failed with “read-only file system”. Test a handler inside the unit’s own sandbox, for example with a transient unit that applies the same protections, so anything that needs to write outside the allowed paths fails in the test and not in production.

Both bugs share a root: the environment that runs the code is part of the code’s contract. A test that does not reproduce the dispatcher’s argument types and the unit’s sandbox is testing a different program.

The privilege-escalation path #

One audit finding is worth stating plainly because it is common. The bot’s service runs as root (it needs to manage system units and the cluster), but it executed a script file owned by an unprivileged user and group-writable. Anything that could write that file could run code as root the next time the service started. A root service must run code that only root can modify. The same applies to the hardened unit itself: it is a copy that has to be re-installed and reloaded when the source changes, and every new path it writes to must be added explicitly to the sandbox’s allowed list.

Secrets live in one place now #

The bot’s early versions had credentials hard-coded in the service file and the source. The overhaul moved them to a single environment file, mode 600, with a config class that validates them at startup and a shell command allowlist. Later, the whole lab moved to a central notification relay so that no sender holds its own copy of a messaging token at all, which I covered in a separate post. A control surface that can touch everything should carry as few long-lived secrets as possible, and none in its code.

Talking to the local agent through chat #

The newest capability lets me hold a plain-text conversation with my local AI agent from a private chat. Free text in an allowlisted chat is forwarded to the agent’s session protocol, and the agent’s approval requests come back as Approve and Deny buttons; a code-gated action asks for the code in the same thread. The first live test denied an approval and confirmed the command did not run. The chat is a convenient front end, but the authorisation stays in the agent’s permission gate, where it belongs, not in the bot.

Checklist #

• Treat the bot as a control surface: reachability, authorisation, and least privilege first.
• On silence, diagnose the network path before the process; distinguish firewall rejection from DNS and timeouts.
• Put your messaging provider’s ranges in an allow list ahead of any auto-ban rule.
• Test the full alert pipeline end to end, not the configuration.
• Keep the “may act on” list in step with what actually runs.
• Test handlers with the real argument types and inside the service sandbox.
• Never let a root service execute a user-writable script.
• Keep secrets in one validated place, out of code, and preferably behind a relay.

Frequently asked questions #

My chat bot is running but silent. Where do I look first? #

At the network path to the messaging API, not at the bot. A process can be healthy and still have every outbound packet refused. Look for a “network unreachable” error, which means a routing or firewall rejection, and check whether your firewall or an auto-ban rule has blocked the provider’s address.

How do I keep an auto-ban rule from blocking my own messaging bot? #

Resolve the provider’s address ranges and put them in an allow list that is evaluated before the rule that auto-bans scanners. Otherwise a single scan from one of those addresses can get the whole service blocked, and a dynamic ban will keep recurring.

How should I test a chat-bot command handler? #

With the exact argument type the dispatcher passes (usually a list, not a string) and inside the service’s own sandbox, so filesystem and permission restrictions that apply in production also apply in the test. A bare root shell hides both classes of bug.

Is it safe for a root service to run my script? #

Only if the script is writable solely by root. If it is owned by an unprivileged user or group-writable, anyone who can modify it can run code as root. Fix the ownership and permissions before anything else.

The lesson #

A chat-ops bot feels like a convenience, but it is a privileged doorway into your whole system, and its reliability is decided by things that are not in its code: the network path to you, the receiver on the far side of a webhook, the sandbox its handlers run in, the ownership of the file it executes. Build the doorway to be reachable, prove every link actually carries what it claims, test the code in the environment that will run it, and keep the real authorisation somewhere stricter than the chat window.