One notification relay instead of 40 copies of a token

I once counted the places in my home lab that held a copy of the same chat-bot token: about 40 copies in all. Scripts, Kubernetes manifests, a bot’s config, cron jobs, a canary script. When that token needed rotating, I had 40 edits to find and get right, and the ones I missed would fail silently, days later. Worse, several of those copies were committed to git in plain text.

So I built a notification relay: one small service that every sender talks to, holding the real credentials in exactly one place. This post is its design, the credential model behind it, and why a boring internal relay is one of the best security cleanups a self-hoster can do.

Contents · 9 sections

TL;DR #

• A notification token copied into 40 places is 40 things to rotate and 40 chances to leak. Centralise it.
• Senders call one relay with a topic, a message and a per-sender token. They never hold a platform credential.
• Use a per-publisher token, not the shared platform secret, so you can revoke one sender without touching the rest.
• Keep the same channel names across every platform, and send to a channel, not to a platform.
• Give every delivery path a fallback that does not depend on the thing that is most likely down.
• Migrate senders one at a time, and rotate the old shared secret only after the last one has moved.

The problem: a secret with 40 homes #

Notifications had grown organically. Each new script that wanted to message me got the bot token pasted into it. Over years that produced dozens of copies in dozens of formats. Three things were wrong with that, in increasing order of seriousness:

• Rotation was a manual hunt. Changing the token meant finding every copy. Audits kept finding more: a token hard-coded in several manifest files, the same token with a different value in a never-deployed manifest (already rotated in some places, not others), a literal fallback value inside a canary script.
• Failures were silent and delayed. A stale copy does not error at edit time. It fails the next time that particular script tries to send, which might be days later, as a missed alert, the worst kind of failure for a notification system.
• Secrets were in git. Several copies were committed in plain text. Git history keeps them, so the only real fix is to rotate the secret, not to delete the line.

A notification token is low-glamour and high-reach: it can message me, and in some setups it can be abused to spam or phish. Forty copies of it is forty times the exposure for zero benefit.

The design: one relay, topics not platforms #

The relay is a small service with a fixed internal address. A sender makes one HTTP call:

POST /   (relay, internal only)
Authorization: Bearer <per-publisher-token>
Content-Type: application/json

{ "topic": "backups", "level": "warning",
  "title": "Nightly backup stale",
  "text": "No successful run in 26 hours." }

The sender names a topic (a channel such as backups, security, infra, reports) and a severity. It does not know or care which platforms that topic maps to, or what the platform credentials are. The relay looks up the channel, fans the message out to every platform configured for it, and owns all the secrets.

That inversion is the whole point. Adding a platform, changing a webhook, pinning a new certificate: all of that happens once, in the relay, and every sender benefits without changing a line.

I kept the old minimal message shape (text plus level) working as well, so existing senders could move without being rewritten. A migration where the callers must change in lockstep with the service is a migration that does not happen.

Channels across platforms #

The relay delivers the same set of channel names on every platform: security, infra, backups, reports and a few more. A sender targets backups; whether that reaches a self-hosted chat server, a messaging bot, a chat webhook, or all three, is the relay’s decision.

My platform choices, and the reasoning:

• A self-hosted chat server as primary. It runs on my own storage appliance, inside my network, so a normal notification never leaves the building. Its certificate is pinned by hash in the relay’s config rather than trusted through the public certificate system, because it is a known internal endpoint and pinning removes a whole class of trust problems.
• A messaging bot as the out-of-band fallback, one bot per channel. This is the path that still works when my network or internet link is down, so it must not depend on anything inside the house. One bot per channel means a leak or a problem with one channel’s bot is contained to that channel.
• Chat webhooks for a third platform, because incoming webhooks need no bot process to keep running.

The principle under the choices: a notification system must have a path to you that does not share a failure mode with what it is trying to warn you about. If backups and alerts both flow only through an internal server, the day that server dies is the day you stop hearing that it died.

I wrote the direct senders myself instead of pulling in a general notification library, to keep the dependency surface of a privileged internal service small. For a handful of platforms, the code is less risk than the dependency tree.

The credential model #

Each value the relay needs is one entry in a credential store: a platform token for a channel, a server URL, a webhook URL, and each sender’s own publisher token. There is no shared blob of all secrets; each is addressed by a stable key.

The relay holds its platform credentials; each sender holds only its own publisher token, in its own secret. The same credential’s two locations (the store and the sender) are declared once, so a rotation knows both ends. Rotation itself has a “apply the value I edited in the vault” mode: I change the value in the password manager’s web vault, and the system applies it, without ever having the secret typed into a terminal or a file by hand. A drift check runs when the vault is unlocked, so a value that has quietly diverged is caught.

The design goal in one sentence: I change a token in one place, the vault or the keyring app, and I am done. No hunt, no missed copy.

Migrating without a flag day #

You cannot move 40 senders at once and expect to sleep. The migration is one sender at a time:

1. Declare the sender as a publisher and issue it a token.
2. Point that one sender at the relay; leave the rest alone.
3. Confirm its messages arrive on every platform for its channel.
4. Repeat.
5. Only after the last sender has moved, rotate the old shared platform token, which also closes off every copy still sitting in git history.

2 senders were pre-declared as the first movers precisely because they were safe to test with. The order matters: rotating the shared secret first would break every not-yet-migrated sender at once. Rotating it last turns the cleanup into the final, deliberate step.

Checklist #

• Count how many copies of your notification secret exist. If it is more than one, centralise.
• Put the platform credentials in the relay only; senders get a per-sender token.
• Address messages to a channel, with the same names on every platform.
• Pin the certificate of a known internal endpoint by hash instead of trusting it through the public system.
• Keep a fallback path that does not share a failure mode with your infrastructure.
• Keep the old message shape working so senders can migrate unchanged.
• Migrate one sender at a time; rotate the shared secret last.
• Remember that a secret in git history is only fixed by rotation, not deletion.

Frequently asked questions #

Why route notifications through a relay instead of calling the platform directly? #

Because every direct caller then holds the platform credential. Centralising it means one place to rotate, one place to pin certificates, one place to add a platform, and senders that hold only a revocable token of their own.

What token should a sender use? #

Its own per-publisher token, never the shared platform secret. Then you can revoke or rotate one sender without affecting the others, and a leaked sender token cannot be used to reconfigure the channel.

How do I make notifications survive an outage? #

Give the system a delivery path that does not depend on what might be down. My primary path is a self-hosted server inside the network; my fallback is an external messaging bot that still works when the network or link is down.

A notification token leaked into git. Is deleting the line enough? #

No. Git history retains the old content, so the credential must be rotated. Deleting the line only stops it spreading further. Centralising first makes that rotation a single change.

The lesson #

Notifications are the least glamorous part of a system and one of the most security-relevant, because the secret that sends them is small, powerful and tempting to copy. The fix is not clever: put the credential in one place, give every sender a revocable token of its own, send to channels rather than platforms, keep a fallback that fails independently, and migrate slowly. The reward is that the next time a token has to change, it changes once.