Secrets in GitOps history: scan by key name, then rotate

The first secret scan of my Kubernetes manifests repository returned about 350 hits. It was useless. Most of them were not my secrets at all, and the scan still missed real ones: a password with hyphens in it, and an HMAC secret two lines above a private key it did catch.

This post is about credentials that end up in a GitOps repository: how they get there, why deleting them from the files does not help, how to scan for them in a way that finds real problems, and what to do once you have found them. It comes from my own repository, which had secrets in it from the very first commit.

Contents · 10 sections

TL;DR #

• 350 raw hits from a naive regex scan; almost all of them noise.
• 15 genuine findings after triage; zero false positives.
• Scanning by value shape misses human-written passwords and short hex secrets. Scanning by key name (client_secret, hmac_secret, api_key, password bound to a long value) found 8 more real secrets.
• Removing a secret from the working tree does not revoke it. It is still in history, in every clone and in every backup. Only rotation does that.
• Rewriting history is hygiene, not remediation. Do it after rotating, if at all.
• Verification code must print fingerprints, never values. My own check printed a whole private key into a log.

How secrets got into the repository #

Nobody decides to commit a password. It happens one convenient step at a time:

• a CronJob script that sends a notification, with the bot token and webhook URL written inline “for now”;
• a Helm values file with an OIDC client secret, because the chart example had one;
• a reference copy of an application’s configuration, committed “for documentation”, including its signing key, session secret and SMTP password;
• database passwords in container args, environment variables and even in health probes (mysqladmin ping -p...);
• Git remote URLs with credentials embedded, http://user:token@host/..., which sit in .git/config and print with every git remote -v.

In my repository, one notification token appeared in 12 tracked files and had been in history since the initial commit. A private signing key was duplicated in two reference copies of a configuration file, and neither copy was even used at runtime: the application read its live configuration from a mounted host directory. Those files were pure leak vectors with no operational value.

Why a naive scan fails #

My first scan used the usual value-shaped patterns: long base64 strings, 40-character hex, -----BEGIN ... KEY-----, things that look like tokens. About 350 matches came back.

Most were noise of two kinds:

• Evidence, not secrets. My threat-intelligence reports quote attackers’ credentials, leaked cloud keys and bot tokens as evidence. A dossier documenting a stolen key is the product; it is not my leak.
• Hashes. 40- and 64-character hex strings are mostly commit IDs and sample hashes.

And the scan still missed real secrets:

• an OIDC client secret written by a human, with hyphens in it, did not match a pattern tuned for base64-looking values;
• an HMAC secret, bare hex, sat two lines above a private key the scanner did flag. Bare hex cannot be flagged by shape without drowning in hashes.

Scanning by key name #

The fix is to scan for the field name as well as the value. A secret almost always sits next to a key that says what it is. Match those names bound to a non-trivial value:

import re

KEYED = re.compile(
    r'(?i)\b(password|passwd|secret|client_secret|hmac_secret|jwt_secret|'
    r'session_secret|encryption_key|api_key|token|webhook)\b'
    r'\s*[:=]\s*["\']?([^\s"\'#]{12,})'
)

PLACEHOLDER = re.compile(r'(?i)^(\$\{.*\}|<.*>|changeme|example|xxx+|\*+)$')

Then make triage part of the scanner, not a manual step:

• exclude evidence paths where third-party credentials are quoted on purpose;
• ignore hash-shaped values that are not bound to a secret-like key;
• ignore placeholders and references such as ${VAR}, <secret>, valueFrom;
• report a fingerprint, for example a short prefix of the value’s SHA-256, never the value itself.

With those rules my scanner went from about 350 matches to 15 genuine findings with no false positives. Adding the key-name rule surfaced 8 more real secrets that the value patterns had missed. It also has a --staged mode for a pre-commit hook, so new secrets are caught before they enter history.

Lesson: secret scanning in a real repository is a triage problem more than a detection problem. A scan nobody can act on is the same as no scan.

Deleting is not revoking #

When I first moved credentials out of a CronJob into a Kubernetes Secret, the working tree became clean. That felt like a fix. It was not.

The values were still:

• in every past commit, readable by anyone with read access to the Git server;
• in every clone, on every machine that had pulled the repository;
• in filesystem backups of those clones;
• possibly in logs, shell history and screenshots.

There is exactly one remedy for an exposed credential: rotate it. Issue a new token, regenerate the webhook, change the password, and revoke the old one where the provider allows it. Until then, assume anyone who has ever had read access has it.

The same applies to secrets printed anywhere. Once a value has been echoed to a terminal or a log, including an AI assistant’s session transcript, treat it as disclosed.

Rewrite history or not? #

Tools like git filter-repo can remove a value from every commit. That is useful, but it is often misunderstood:

• it changes every commit ID from the first affected commit onwards, so every clone, CI system and automation that pushes to the repository must be re-cloned or reset;
• it does not remove the value from existing clones, forks, mirrors or backups;
• the Git server may keep unreachable objects until garbage collection runs.

So rewriting history never replaces rotation. The right order is: rotate first, so the old value is worthless; then decide whether a rewrite is worth the disruption. For a private repository whose old secrets are all rotated, many people reasonably choose not to rewrite. For anything public, rewriting is worth doing for hygiene, but the rotation is what protects you.

Moving secrets to where they belong #

The working pattern now:

• Secrets live in Kubernetes Secrets, created from a vault or from the command line, never from a committed manifest:

kubectl -n <namespace> create secret generic <app>-notify \
  --from-literal=BOT_TOKEN="$(cat /dev/stdin)" \
  --dry-run=client -o yaml | kubectl apply -f -

• Workloads consume them with envFrom or secretKeyRef, so manifests contain only references:

envFrom:
  - secretRef:
      name: <app>-notify

• Commands that need a password read it from the environment at runtime, for example redis-server --requirepass "$(REDIS_PASSWORD)", instead of having it in args.
• Health probes are credential-free: a TCP check or a ping that needs no password.
• Git remotes use a scoped token in a credential helper, not in the URL.
• Reference copies of configuration that are not used at runtime are redacted or deleted, with a pointer to the Secret.

Two practical traps from the migration:

• A private key copied out of a YAML block scalar had a blank line between every 64-character line. openssl refused to parse it. Reflowed to normal PEM it was valid. Validate a key before storing it (openssl pkey -check -noout -in key.pem), or the Secret stores an unusable blob.
• My own verification step, written to confirm a key had been removed from a file, printed the parsed value: the entire private key went into a session log. The step that checks a secret is gone is the step that still holds it. Assert on derived facts: length, a boolean, a fingerprint.

import hashlib
v = load_value()
print("present:", "PRIVATE KEY" in v, "len:", len(v),
      "sha256:", hashlib.sha256(v.encode()).hexdigest()[:12])

Checklist #

• Scan by key name as well as by value shape.
• Build triage into the scanner: evidence paths, hashes, placeholders.
• Report fingerprints, never values, in scanners and in verification code.
• Add a pre-commit hook so new secrets never reach history.
• Rotate every credential that ever appeared in a commit, a URL or a log.
• Rewrite history only after rotation, and only if the disruption is worth it.
• Keep manifests to references: envFrom, secretKeyRef, credential-free probes.
• Delete or redact reference copies of configs that are not used at runtime.
• Validate keys before writing them into Secrets.

Frequently asked questions #

Is it enough to delete a password from my Git repository? #

No. The value stays in history, in every clone and in backups. Rotate the credential; deleting it from the files only stops new copies.

Should I rewrite Git history after leaking a secret? #

Only after rotating the secret. A rewrite removes it from future clones of that repository, but not from existing clones, forks, mirrors or backups, and it forces everyone to re-clone. Rotation is what makes the leaked value useless.

Why does my secret scanner miss real passwords? #

Most patterns look for token-shaped values such as long base64 or hex strings. Human-written passwords and short secrets do not match. Add rules that look for secret-like key names, such as password, client_secret or api_key, bound to a value.

How do I stop a secret scanner from flagging hashes and examples? #

Ignore hash-shaped values that are not bound to a secret-like key, skip placeholders and variable references, and exclude directories that quote third-party credentials on purpose, such as security reports. Report fingerprints so triage never re-exposes a value.

The lesson #

Every secret in my repository arrived as a shortcut, and every one of them outlived the shortcut by months. The scan that mattered was not the one that found the most strings; it was the one that understood the repository well enough to report fifteen real problems. And the fix that mattered was not deleting lines. Assume a committed secret is public, rotate it, and change the habit that put it there.