The one-second race that hid my nginx config change

I changed a web server’s configuration, applied it, watched the rollout finish, and saw a brand-new pod come up healthy. The site still served the old behaviour. Nothing was broken, nothing was logged, and every dashboard was green.

The cause was a race that lasted 1 second: the new pod started at 10:50:38, and the configuration it was supposed to read was updated at 10:50:39. The pod loaded the old file and kept it.

This post explains exactly how that happens in Kubernetes, why it is so easy to miss, and the 3 small habits that make it impossible.

Contents · 10 sections

TL;DR #

• kubectl apply -f manifest.yaml applies objects in the order they appear in the file.
• If a Deployment comes before the ConfigMap it uses, the Deployment’s rollout can start a new pod before the ConfigMap is updated.
• If the ConfigMap is mounted with subPath, the file inside the container is never refreshed afterwards, not even when the ConfigMap changes a second later.
• The fix: put ConfigMaps first, add a checksum annotation to the pod template, and verify the running configuration, not just the rollout status.

The setup #

My public website is a static site served by nginx in a single Kubernetes Deployment. Its nginx configuration lives in a ConfigMap with two keys, the main nginx.conf and a server.conf with the virtual host. Both are mounted into the container as single files:

volumeMounts:
  - name: nginx-config
    mountPath: /etc/nginx/nginx.conf
    subPath: nginx.conf
  - name: nginx-config
    mountPath: /etc/nginx/conf.d/default.conf
    subPath: server.conf

Mounting individual files with subPath is a common pattern. It lets you drop one file into a directory that already contains other files from the image, without hiding them. Remember that detail; it is half of the problem.

I was adding a new section to the site, a blog under /blog/. That needed new nginx rules: a redirect from /blog to /blog/, extensionless URLs for posts, and correct content types for the feeds. I also changed a small sidecar container in the same Deployment, so the Deployment’s pod template changed too.

The manifest file looked like this, in this order:

1. Namespace
2. Deployment
3. Service
4. ConfigMap

I validated the new nginx configuration with nginx -t in the same nginx image, against the real site files. It passed. The change was applied, the rollout reported success, and the new pod was running.

The symptom #

/blog redirected to http:// instead of https://. The new rule I had written returned an explicit https:// redirect, so seeing http:// meant my rule was not running at all. nginx’s own built-in “add a trailing slash to a directory” behaviour was answering instead.

Everything else looked perfect:

• the rollout was complete and the pod was Running and Ready;
• the ConfigMap in the cluster contained the new rules;
• nginx started without errors, so there was nothing in the logs.

The configuration was correct in the API and wrong inside the container.

Finding the one second #

Two numbers settled it.

The pod’s creation time:

kubectl -n <namespace> get pods -l app=<app> \
  -o jsonpath='{.items[0].metadata.creationTimestamp}'
# 2026-11-04T10:50:38Z

The ConfigMap’s last modification time, visible in its managed fields:

kubectl -n <namespace> get configmap <name> \
  -o jsonpath='{.metadata.managedFields[*].time}'
# ... 2026-11-04T10:50:39Z

The pod was one second older than its configuration. Then the file inside the running container confirmed it: it had 122 lines, while the ConfigMap’s version had 131 and 149 lines for its two keys. The container was running yesterday’s configuration with today’s code.

Why it happens #

Two separate behaviours combine.

1. kubectl apply goes in file order #

When you apply a multi-document YAML file, kubectl sends the objects to the API server one after another, in the order they appear. It does not sort them by dependency. My Deployment came second and the ConfigMap fourth.

Because the Deployment’s pod template had changed (the sidecar edit), applying it immediately triggered a rollout. The controller created a new ReplicaSet and a new pod while kubectl was still working through the rest of the file. The pod was scheduled and its volumes were set up from the ConfigMap as it existed at that moment, which was the old version. One second later the ConfigMap was updated. Too late for that pod.

If only the ConfigMap had changed, there would have been no rollout at all, and the result would have been the same: an old configuration in a running pod, just for a different reason.

2. subPath mounts never update #

Kubernetes does update ConfigMap volumes in running pods: the kubelet syncs a changed ConfigMap into mounted volumes after a short delay. But that only works for whole-volume mounts, where the ConfigMap is mounted as a directory. The kubelet updates a symlinked directory atomically.

A subPath mount is a copy taken when the container starts. It is never refreshed. The Kubernetes documentation says it in one line that is easy to skim past: a container using a ConfigMap as a subPath volume mount will not receive ConfigMap updates.

So the pod that started one second too early would keep the old configuration for its entire life, no matter how long I waited.

Why nothing complained #

Nothing was wrong from any component’s point of view:

• the API server stored the new ConfigMap correctly;
• the Deployment rolled out a pod that was healthy;
• nginx loaded a valid configuration, just an older one;
• health checks tested that the site answered, and it did.

Silent drift like this is the most expensive kind of failure, because every check you normally rely on says “fine”.

The fixes #

Put configuration before the things that use it #

The simplest fix is ordering. In any manifest, list ConfigMaps and Secrets before the Deployments that mount them:

1. Namespace
2. ConfigMaps and Secrets
3. Services
4. Deployments

That alone would have prevented this incident. It is not a full guarantee, though: someone can still apply one object at a time, or change only the ConfigMap. So add the next habit too.

Make the pod template depend on the configuration #

Helm charts solve this with a well-known trick: a checksum annotation on the pod template. The annotation holds a hash of the configuration, so any configuration change also changes the pod template and forces a fresh rollout, after the new ConfigMap exists:

spec:
  template:
    metadata:
      annotations:
        checksum/nginx-config: "<sha256 of the ConfigMap data>"

Without Helm, generate the hash in whatever applies your manifests, for example a small script that hashes the ConfigMap data and writes it into the annotation before kubectl apply. Kustomize gets the same effect with configMapGenerator, which appends a content hash to the ConfigMap’s name, so a changed configuration is a new ConfigMap and the Deployment must point to it.

Prefer whole-directory mounts where you can #

If the configuration can live in its own directory, mount the ConfigMap as a directory instead of with subPath. Then running pods receive updates, and you can reload the application (nginx supports nginx -s reload) without restarting the pod. My site already used this for a generated redirect map: a small sidecar watches the file’s hash and sends nginx a reload signal when it changes.

subPath is still fine for files that must sit next to files from the image. Just treat it as “read once at start”, because that is what it is.

Verify the running configuration, not the rollout #

This is the habit that would have caught it in seconds, whatever the cause. After a configuration change, check what the running process actually loaded, and test the behaviour you changed:

# what the container really has
kubectl -n <namespace> exec deploy/<app> -c nginx -- \
  grep -c "location = /blog" /etc/nginx/conf.d/default.conf

# what the site really does
curl -sI https://www.example.com/blog | grep -i '^location'

A rollout status tells you that a pod exists. Only a request tells you that your change is live.

How I fixed it in the moment #

The immediate fix was one command: restart the Deployment, so a new pod starts after the ConfigMap had already been updated.

kubectl -n <namespace> rollout restart deploy/<app>

The new pod read the new configuration, /blog redirected to https://, and the feeds came back with their proper content types. Then I moved the ConfigMap above the Deployment in the manifest so the next change cannot race.

Checklist for any configuration change in Kubernetes #

1. Configuration objects come before the workloads that use them, in every manifest.
2. Pod templates carry a checksum of their configuration, or use content-hashed ConfigMap names.
3. Know which mounts use subPath: they are read once, at container start.
4. Validate the new configuration offline first (nginx -t in the same image is cheap).
5. After applying, check the file inside the container and test the changed behaviour with a real request.
6. Compare timestamps when something is “applied but not live”: pod creation time vs configuration modification time answers it in one minute.

Frequently asked questions #

Does kubectl apply apply resources in a specific order? #

It applies them in the order they appear in the file or directory listing. It does not sort by dependency. Put ConfigMaps and Secrets first.

Why didn’t my pod pick up the new ConfigMap? #

Either the pod started before the ConfigMap was updated, or the ConfigMap is mounted with subPath, which is never refreshed. Often both, as here. Environment variables taken from a ConfigMap are also only read at container start.

How long does Kubernetes take to update a mounted ConfigMap? #

For whole-volume mounts, usually within a minute or so, depending on the kubelet’s sync period and cache. For subPath mounts and environment variables: never, until the container restarts.

What is the safest way to roll out configuration changes? #

Make the configuration part of the pod template’s identity, with a checksum annotation or a hashed ConfigMap name, so every configuration change produces a clean rollout. Then verify the behaviour with a real request.

The lesson #

A configuration change is not done when it is applied. It is done when the running process uses it. The gap between the two can be a single second, and nothing will warn you. Order your manifests, tie your pods to their configuration, and always end a change with a request that proves it works.