On single-node Kubernetes, RollingUpdate can mount your database twice

Raising the memory limit on my vector database meant asking Kubernetes to run two copies of it on the same data directory, for as long as the rollout took. Nothing in Kubernetes stopped it. On a single-node cluster, a ReadWriteOnce volume does not mean “one pod”. It means “one node”.

That find on 26 July 2026 turned into an audit of every Deployment in my k3s cluster. 15 of 33 stateful single-replica Deployments had the same hazard, including two PostgreSQL instances, three SQLite-backed apps and a pair of services sharing private keys. The fix was a one-line strategy change that, it turns out, needs zero downtime to apply.

This post explains the arithmetic behind the hazard, how to audit for it, and, since it started with Qdrant, what actually made my Qdrant indexing slow.

Contents · 9 sections

TL;DR #

• 2 pods on one ReadWriteOnce volume; on a single node that is allowed, because RWO is enforced per node, not per pod.
• With a single replica, the default RollingUpdate’s maxSurge: 25% rounds up to one pod and maxUnavailable: 25% rounds down to zero. Kubernetes always starts the new pod before stopping the old one.
• 15 of 33 Deployments at risk; those were the stateful single-replica ones on node-bound volumes not yet using Recreate (18 already did).
• Changing spec.strategy does not restart pods. It is not part of the pod template. I fixed every one of them with no downtime.
• 28 seconds for a Qdrant optimisation; it used to take over 10 minutes, until I raised its CPU limit from 1 to 8 cores. The real bottleneck was embedding, not Qdrant.

How the hazard works #

A Deployment’s default update strategy is RollingUpdate with maxSurge: 25% and maxUnavailable: 25%. Kubernetes converts percentages to pod counts: surge rounds up, unavailability rounds down. With one replica:

• maxSurge = ceil(0.25 × 1) = 1: one extra pod may exist during the update;
• maxUnavailable = floor(0.25 × 1) = 0: the old pod may not go away until the new one is ready.

So the controller always creates the new pod first, waits for it to become ready, and only then terminates the old one. For a stateless web app, that is exactly what you want.

For a single-replica database on a ReadWriteOnce volume, you would hope the volume stops it. On a multi-node cluster with network storage it often does: the new pod lands elsewhere and cannot attach the volume until the old pod releases it. But ReadWriteOnce means one node may mount the volume read-write. On a single-node cluster with local volumes, both pods are on the same node, both mount the same directory, and both start.

What happens next depends on the app:

• Qdrant and many embedded databases take a storage lock; the second instance fails or contends for it;
• PostgreSQL has a postmaster PID file check, but you are relying on it to catch a situation it was not designed for;
• SQLite-backed apps (Grafana, Open WebUI, ChromaDB) can have two writers on one file;
• services that share secret material, such as onion-service private keys, briefly gain a third concurrent reader.

Kubernetes has an access mode for exactly this, ReadWriteOncePod, which is enforced per pod. It needs CSI volume support. My local PVs do not have it, so the strategy has to do the job.

How I found it #

I was raising Qdrant’s memory limit. That changes the pod template, so it triggers a rollout, and with the default strategy the rollout starts a second Qdrant pod on the same node, bound to the same local PV, where the two instances contend for Qdrant’s storage lock. I switched Qdrant to Recreate that day. Then the obvious question: where else does this happen?

The audit #

The question is mechanical, so I wrote a script. Dump the cluster state:

kubectl get deploy -A -o json > deploy.json
kubectl get pvc -A -o json > pvc.json
kubectl get pv -o json > pv.json

Then for each Deployment: follow its pod template volumes to PVCs, PVCs to PVs, check whether the PV is node-bound (local, hostPath, or has nodeAffinity) and ReadWriteOnce, and compute the effective surge for its replica count and strategy. Flag anything with surge ≥ 1 on such a volume.

Result across 123 Deployments, 62 PVs and 61 PVCs:

• 33 single-replica Deployments sat on node-bound RWO volumes;
• 18 of those already used Recreate, which made it the house convention;
• 15 were at risk.

I ranked them by blast radius:

1. Real datastores: two PostgreSQL instances, Grafana’s SQLite database, Open WebUI’s SQLite database plus documents, and a ChromaDB store.
2. Secret material: two Deployments that both mount the volume with onion-service private keys. A surge would add a third concurrent mounter of the service’s identity.
3. Config and cache: eight services such as a dashboard, a webmail client, search engines and document tools. Low risk, still wrong.

The fix, and why it needs no downtime #

For each Deployment:

kubectl -n <namespace> patch deploy <app> --type=merge \
  -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'

rollingUpdate: null is needed because a Recreate strategy may not carry rollingUpdate parameters.

I expected restarts. There were none. spec.strategy is a field of the Deployment, not of the pod template. The controller only creates a new ReplicaSet when the template changes, so patching the strategy leaves running pods alone; the new strategy applies to the next rollout. I verified it: all patched pods kept their names and restart counts.

The one exception that day was Qdrant itself, because I changed its resource limits in the same apply, and that is a template change.

Two follow-ups:

• Make it durable. A live patch is lost on the next apply of an old manifest. I edited every manifest to match.
• Helm-managed apps need the change in their values. Grafana came from a Helm chart, so a bare patch would be reverted by the next helm upgrade. The durable key was the chart’s deployment strategy value. Check before patching:

kubectl -n <namespace> get deploy <app> \
  -o jsonpath='{.metadata.labels.app\.kubernetes\.io/managed-by}'

The final audit run: 33 Deployments on node-bound RWO volumes, 0 at risk. The convention is now written down: every single-replica Deployment on a node-bound RWO volume declares strategy: type: Recreate. Re-run the audit after adding any stateful service.

The trade-off is honest: with Recreate, an update means a few seconds of downtime while the old pod stops and the new one starts. For a single-replica database, that downtime is the price of never having two writers.

Meanwhile, why Qdrant was “slow” #

The Qdrant work that led here had its own lesson. Earlier in July, a honeypot IP collection stayed yellow (optimising) for more than ten minutes after each indexing batch, and the question was whether to shard it or tune it.

Measuring each stage of the pipeline told a different story:

• embedding with a local model took about 2 hours at 2.9 items per second, because the client sent one input per request;
• the Qdrant upsert took about 36 seconds, roughly 800 points per second;
• the Elasticsearch side indexed documents one by one, about 2.3 minutes;
• set_payload with wait=true took about 4 minutes; with wait=false, about 0.7;
• the post-batch HNSW optimisation took more than 10 minutes.

The optimisation was slow because Qdrant had a CPU limit of 1 core on an 88-thread machine that was 84 % idle. Re-upserting most of a 33,000-point collection triggers a vacuum and a full index rebuild, which is CPU-bound. MAX_OPTIMIZATION_THREADS was 4, but the cgroup only allowed one core’s worth of time.

The fix: CPU limit 1 → 8, request 100m → 2, memory 2 GiB → 4 GiB, optimisation threads 4 → 8, and default_segment_number: 6 on that collection so segments can be optimised in parallel. A controlled re-upsert of 9,000 points then went from yellow to green in 28 seconds.

Sharding was never the answer: a single node, a 33,000 × 1024 collection. And the biggest win was elsewhere: sending embedding inputs as arrays instead of one per call was 8.3 times faster (2.6 → 21.7 per second).

Checklist #

• On a single-node cluster, assume ReadWriteOnce does not protect you from two pods.
• Give every single-replica Deployment on a node-bound RWO volume strategy: Recreate.
• Use ReadWriteOncePod where your storage supports it.
• Audit by joining Deployments, PVCs and PVs; do not rely on memory.
• Patch the strategy live (no restart), then update the manifest or Helm values.
• Do not bundle a strategy change with a template change if you want zero downtime.
• Before tuning a vector database, time each stage: embedding, upsert, payload, optimisation.
• Check CPU limits on anything that rebuilds indexes.

Frequently asked questions #

Can two pods mount the same ReadWriteOnce volume? #

Yes, if they run on the same node. ReadWriteOnce restricts read-write mounting to a single node, not a single pod. On a single-node cluster, a RollingUpdate surge pod will mount the same volume as the old pod. Use ReadWriteOncePod (CSI only) or the Recreate strategy.

Why does a one-replica Deployment start the new pod before stopping the old one? #

The default maxSurge and maxUnavailable are both 25%. Kubernetes rounds surge up to 1 and unavailability down to 0 for one replica, so the new pod must be ready before the old one is terminated.

Does changing a Deployment’s strategy restart its pods? #

No. spec.strategy is not part of the pod template, so changing it does not create a new ReplicaSet. The running pods are untouched and the new strategy takes effect on the next rollout.

Why is Qdrant optimisation slow after a large upsert? #

Rebuilding the HNSW index after many points change is CPU-bound. A low CPU limit throttles it no matter how many optimisation threads you configure. Raise the limit, align MAX_OPTIMIZATION_THREADS, and consider more segments for parallel optimisation.

The lesson #

Kubernetes defaults are written for clusters with many nodes and network storage. On one machine, some of their guarantees quietly change meaning: “once” becomes “once per node”, and “rolling” becomes “two writers for a moment”. The fix was small and free. Finding it took reading the arithmetic behind a default I had never questioned, and then checking every place it applied instead of fixing only the one I happened to see.