How my Kubernetes dashboard locked me out of my own SSO

One evening in August I could not log in to my Kubernetes dashboard. The SSO page said “temporarily unavailable, too many requests”. I had not made too many requests. My dashboard had: 19,843 token requests in 48 hours, 99.8 % of them rejected by the rate limiter, and every rejection answered with another retry five seconds later.

The dashboard was Headlamp, the identity provider was Authelia, and the loop had no way out on its own. This post walks through how a background token refresh can lock you out of your own login, the wrong theory I held for a day, the upstream bug that explained it, and the small watchdog that keeps it under five minutes until the fix ships.

Contents · 11 sections

TL;DR #

• 12,871 token requests in one hour at the peak; Headlamp’s backend kept refreshing a dead OIDC refresh token every 5 seconds, with no backoff, including retrying on the rate limiter’s own rejections.
• Authelia’s rate limiter is keyed on source IP. Background refreshes and my interactive login both came from the same Headlamp pod IP, so the background loop blocked my login.
• 6 hours 23 minutes; that is how long the first fix held. Restarting Headlamp cleared the dead token and stopped the flood, then it came back.
• The root cause is an upstream race: concurrent refreshes of the same token with no de-duplication, against a provider that rotates refresh tokens and detects reuse.
• A CronJob watchdog restarts Headlamp only on the exact flood signature, and never a pod that has only just started.
• Before any Helm upgrade, I found the values file had drifted from the live config on three fields. Upgrading would have broken login on its own.

The setup #

Headlamp is a web UI for Kubernetes. In my cluster it authenticates users with OpenID Connect against Authelia, and the Kubernetes API server trusts Authelia’s tokens directly, so each admin acts on the cluster as themselves. Kubernetes RBAC maps an SSO group to cluster-admin.

The API server trusts two issuers, because the identity provider is reachable under two domains. That uses Kubernetes’ structured authentication configuration, which allows several JWT issuers with their own CA, audience and claim mappings, and replaces the old --oidc-* flags:

apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
jwt:
  - issuer:
      url: https://auth.example.com
      audiences: [<client-id>]
    claimMappings:
      username: { claim: preferred_username, prefix: "oidc:" }
      groups: { claim: groups, prefix: "oidc:" }
  - issuer:
      url: https://auth.internal.example
      audiences: [<client-id>]
      certificateAuthority: |
        <internal CA certificate>
    claimMappings:
      username: { claim: preferred_username, prefix: "oidc:" }
      groups: { claim: groups, prefix: "oidc:" }

That part has worked since May. The trouble was in refresh.

The lockout #

On 5 August the login failed with an error along these lines:

Failed to exchange token -> temporarily_unavailable (Too many requests)

Authelia’s logs were full of rate-limit entries on the token endpoint, all from one remote IP: the Headlamp pod. Headlamp’s logs showed “failed to refresh token” every five seconds. The numbers from the logs:

• 19,843 token POSTs in 48 hours;
• 19,808 of them rate-limited;
• 12,871 in the single worst hour.

What was going on:

1. Headlamp’s backend refreshes the stored OIDC token on a fixed five-second timer.
2. The stored refresh token had become invalid. Authelia rejected it.
3. Headlamp retried five seconds later. And again. With no backoff.
4. Authelia’s rate limiter escalated through its buckets, and Headlamp kept retrying on the rate limiter’s rejections too, so it never drained.
5. My fresh login is an authorization-code exchange, which Headlamp’s backend performs. It came from the same pod IP, hit the same limiter, and failed.

My login error was a symptom of the background loop, not a problem with the login.

I checked the OIDC client configuration in Authelia first, because a wrong client config would be the obvious cause: offline_access in the scopes, refresh_token in the grant types, code as the response type. All correct. So I did not touch Authelia. A Headlamp restart cleared its token store; afterwards the timer still ran but failed locally with “refresh token: key not found” and never contacted Authelia. Token requests dropped from 94 per minute to 0, and a probe from a different source IP got a normal invalid_request instead of temporarily_unavailable, proving the buckets had drained.

It came back, and my first theory #

The next morning it happened again, 6 hours 23 minutes after the pod had started and I had logged in: 5,616 rate-limit hits in 30 minutes.

Authelia’s session inactivity timeout was 6 hours. Headlamp’s silent background refresh does not count as user activity in the session. So my theory was: six hours after the last interactive login, the session lapses, the refresh token dies with it, and the loop starts. The numbers fit nicely.

It rested on one coincidence.

The real root cause, upstream #

When I looked into whether upgrading Headlamp would help, I found an issue opened upstream on 1 August, kubernetes-sigs/headlamp#6793, that described the mechanism better.

Headlamp’s token-refresh middleware refreshes the OIDC token when a request arrives close to expiry, once per incoming request, with no locking or de-duplication. The frontend fires many parallel API calls. Several land inside the refresh window at the same moment, and each one reads the same cached refresh token and exchanges it independently.

Authelia, like many providers following current OAuth security guidance, rotates refresh tokens and detects reuse. The first exchange succeeds and invalidates the old token. The others present an already-used token, get invalid_grant, and the error is logged and swallowed. Reuse detection can then revoke the whole token family. From then on, every refresh fails, and the backoff-free timer turns that into a flood.

A fix existed as an open pull request, kubernetes-sigs/headlamp#6795, de-duplicating concurrent refreshes with a singleflight group. It was not merged, and the latest release had shipped three days before the issue was even filed. So upgrading could not help yet.

The landmine I found while checking the upgrade #

Before planning any upgrade, I compared the Helm values file in Git with what the pod actually runs on. The chart injects the OIDC settings from a Secret, so the Secret, not the values file, is the truth. Three fields disagreed:

• the issuer URL pointed at the other domain;
• the callback URL pointed at an old hostname;
• the scopes were missing offline_access.

A helm upgrade -f values.yaml would have silently reverted all three: login on the internal domain would break, and without offline_access there would be no refresh tokens at all. I reconciled the values file to the live settings, with the verification command in a comment.

Lesson: before any helm upgrade, diff the values against what is live. Drift is invisible until the upgrade applies it.

The mitigation: a careful watchdog #

Until the upstream fix ships, I chose an automatic restart on detection. The tricky part is detecting the right thing. Headlamp logs “failed to refresh token” in two states:

1. the harmful flood: the error contains temporarily_unavailable, and Headlamp is hammering Authelia;
2. the harmless idle state after any restart: the error says the refresh token was not found, and Headlamp never contacts Authelia. It repeats every five seconds until someone logs in.

A watchdog that matched the generic message would restart Headlamp every five minutes forever. So it matches only the first. The logic, simplified:

#!/bin/sh
pod=$(kubectl -n <namespace> get pod -l app.kubernetes.io/name=headlamp \
  -o jsonpath='{.items[0].metadata.name}')
start=$(kubectl -n <namespace> get pod "$pod" -o jsonpath='{.status.startTime}')
age=$(( $(date +%s) - $(date -d "$start" +%s) ))
[ "$age" -lt 600 ] && { echo "pod younger than 600s: skip"; exit 0; }

hits=$(kubectl -n <namespace> logs "$pod" --since=5m | grep -c 'temporarily_unavailable')
echo "flood hits in last 5m: $hits"
[ "$hits" -gt 0 ] && kubectl -n <namespace> rollout restart deploy/headlamp
exit 0

It runs as a CronJob every five minutes with a namespaced Role that can read pods and logs and patch Deployments, nothing more. It writes to stdout only, so there is no chat token to embed in a manifest. I tested both paths: a real flood line matched once; the idle line matched zero times. The lockout window went from hours to under five minutes, unattended.

The alternative I rejected #

Newer Headlamp releases can skip OIDC entirely: put Headlamp behind the proxy’s forward auth and let it act with its own service account token. No refresh tokens, no race, no flood.

I decided against it. Today there are two independent checks: SSO authenticates the user with two factors, and Kubernetes RBAC separately checks the user’s group in the token. With the service account mode there would be one: anyone who passes the forward-auth middleware acts as the pod’s service account, which needs broad cluster rights for the dashboard to be useful. One missing middleware reference would expose the cluster with no login prompt to signal it. And every action in the Kubernetes audit log would show the service account instead of the person. A watchdog that caps the problem at five minutes is a better trade than that.

Checklist #

• If a login fails with “too many requests”, find who is actually calling the endpoint before changing the identity provider.
• Remember that a backend which performs the code exchange shares its source IP with its background traffic.
• Clients of rotating refresh tokens must serialize refreshes; concurrent use triggers reuse detection.
• Any retry loop needs backoff and must stop retrying on rate-limit responses.
• Watchdogs must key on the exact harmful signature, and include a minimum pod age.
• Before helm upgrade, diff values against the live Secret or config the chart injects.
• Do not trade two independent authorization checks for one to fix an availability problem.

Frequently asked questions #

Why does Headlamp say “temporarily_unavailable” when I log in with OIDC? #

Usually because Headlamp’s backend is flooding the provider’s token endpoint with failing refreshes, and the provider rate-limits by source IP. Your login comes from the same backend, so it is rejected too. Restarting Headlamp clears the stored token and stops the flood.

What is refresh token reuse detection? #

With refresh token rotation, each refresh returns a new refresh token and invalidates the old one. If an old token is used again, the provider treats it as possible theft and can revoke the entire token family. A client that refreshes the same token from several threads at once triggers it by accident.

Can Kubernetes trust more than one OIDC issuer? #

Yes. The structured authentication configuration (apiserver.config.k8s.io/v1) supports several JWT issuers, each with its own URL, audiences, CA and claim mappings. It replaces the single-issuer --oidc-* flags, which cannot be combined with it.

How do I know when a Helm upgrade will change live settings? #

Render or diff the chart with your values and compare against the live objects, including Secrets the chart injects into the pod. helm diff or helm template plus kubectl diff shows what an upgrade would change before it does.

The lesson #

The error appeared at my login, but the cause was a background loop I could not see from the login page. The first fix worked, the first theory fit the numbers, and both were incomplete. What actually helped was reading the caller’s logs, finding the upstream mechanism, and then choosing a mitigation that changes as little as possible: a narrow watchdog, not a weaker security model. When a control fails safely and visibly, a small guard around it is often better than redesigning it.