Velero said Completed. It had backed up a RAM disk.

Every night, Velero reported a completed backup of my most important PostgreSQL pod. What it actually saved was the pod’s 10 GiB RAM disk for temporary files, a nearly empty in-memory volume, while the real data directory with 17 databases was skipped.

This post is about the gap between “the backup completed” and “the data is in the backup”, on a single-node k3s cluster with local volumes. It covers what Velero’s file-system backup does and does not capture, the PV fixes it needed first, two out-of-memory crashes, and the restore that finally proved the parts that worked.

Contents · 10 sections

TL;DR #

• Velero’s file-system backup (Kopia) backs up pod volumes: PVCs, emptyDirs and similar. It does not back up hostPath volumes. My shared Postgres used hostPath, so only its RAM-disk emptyDir was “backed up”.
• An audit found 13 of 17 databases on that instance had no backup anywhere. Every backup layer used a hard-coded list of targets.
• Local PVs need nodeAffinity for Velero. I fixed 38 PV manifests across 28 files, after my first attempt with yaml.dump wrecked the formatting.
• Velero itself was OOM-killed by 730 MB of vulnerability-report custom resources, and I first raised the memory of the wrong deployment.
• The restore after a real disaster worked: volumes came back byte-for-byte, checked against the backup’s own byte counts.

The setup #

One machine, k3s, about 50 namespaces, and dozens of stateful services on local persistent volumes: directories on the host disk, bound to the single node. Velero runs in the cluster with an in-cluster MinIO as its object store and Kopia for file-system backup of pod volumes. Daily and weekly schedules, all reporting Completed.

Next to Velero there are other layers: a nightly pg_dump script, a MySQL dump, a tarball script for “critical data”, an hourly dump of the cluster datastore, and a few per-app cron jobs.

Step zero: Velero needs nodeAffinity on local PVs #

Before any of the interesting problems, Velero had a prerequisite. For file-system backup of local volumes, the PVs need nodeAffinity so the backup work is scheduled on the node that holds the data. Many of my PV manifests had been written by hand over time, and in April 2026 an audit of all 42 local and hostPath PVs found:

• 7 were of type hostPath with no nodeAffinity at all;
• most were also missing claimRef, so re-applying them could bind to the wrong claim.

The target shape for each PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: <app>-data
spec:
  capacity:
    storage: 20Gi
  accessModes: [ReadWriteOnce]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: <host-data-dir>
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values: [<node-name>]
  claimRef:
    namespace: <namespace>
    name: <app>-data

My first fix used Python to load each YAML file, add the blocks and yaml.dump it back. It worked, and it destroyed every comment, ordering and multi-document layout in the files. I reverted immediately and wrote a surgical script that inserts only the missing lines at the right indentation. Final result: 38 PVs fixed across 28 files, 7 converted from hostPath to local, and all of them safe to kubectl apply repeatedly.

Lesson: never round-trip hand-maintained YAML through a serializer. Insert text.

The blind spot: hostPath is not a pod volume #

In July 2026, after an incident where a table of web-attack data was deleted, I audited every backup layer against an actual census of databases instead of against the backup scripts’ own lists. That produced the most uncomfortable finding of the year.

My shared PostgreSQL instance, the one holding most of my research and threat-intelligence data, kept its data directory on a hostPath volume. It also had an emptyDir with medium: Memory, a RAM disk I had added for temporary files after a runaway query filled the root disk.

Velero’s file-system backup works on pod volumes it can access through the kubelet. hostPath volumes are explicitly not supported. So every night Velero dutifully backed up the one volume it could see, the empty RAM disk, and reported success. The 17 databases on the real data directory were not in it.

And the other layers did not catch it either:

• the nightly pg_dump script dumped a hard-coded list of databases, written before most of the newer ones existed;
• the “critical data” tarball covered the MongoDB and Redis hostPath directories, but not the PostgreSQL one.

The audit’s conclusion: 13 of 17 databases on that instance, about 7.4 GB, had no backup anywhere. The root cause was the same in every layer: each one enumerated its targets from a list, and every database created after the list was written was silently unprotected.

The fix was to rebuild the dump script so it enumerates databases from the server itself (for example SELECT datname FROM pg_database WHERE NOT datistemplate) instead of a list, and to stop treating Velero as the backup for anything on hostPath. Velero now covers objects plus PVC-backed volumes; the big Postgres is covered by logical dumps.

How to check what Velero actually took #

velero backup describe --details lists the pod volume backups. Read it per volume, not per backup:

velero backup describe <backup> --details | sed -n '/Pod Volume Backups/,$p'
kubectl -n velero get podvolumebackups \
  -l velero.io/backup-name=<backup> \
  -o custom-columns=POD:.spec.pod.name,VOLUME:.spec.volume,BYTES:.status.progress.totalBytes

If the most valuable pod shows one small volume with a name like temp or cache, you have my problem. Warnings that say a volume was skipped because it is hostPath are not noise; they are the backup telling you what is missing.

I also found that my local AI agent’s backup check had been reporting “Kopia unavailable” on every run, because it looked for Kopia repositories in a host directory that never existed. It now checks the backup storage location phase, the age of the latest backup per schedule, the warnings classified as benign or not, and the repository state.

Two OOM kills, one wrong fix #

In October I added a daily objects-only schedule for all namespaces. Velero’s server pod started getting OOMKilled.

The cause: the vulnerability scanner in the cluster stores its reports as custom resources, and they had grown to 730 MB. Velero loads the objects it backs up, and the reports alone exhausted its memory. I excluded those CRDs from the schedule and raised Velero’s limit to 2 GiB.

The embarrassing part: the first time, I raised the memory of velero-minio, not the Velero server. Similar names, wrong deployment. The fix is mechanical, and I now do it every time:

kubectl -n velero get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}{end}'

Find the container whose last state is OOMKilled, then patch that one.

A second, smaller trap appeared in September: the MinIO image was pinned as <tag>@sha256:<digest> with a :latest-style tag, which makes the default pull policy Always. When the registry dropped that digest, the pod could not restart even though the image was cached on the node. imagePullPolicy: IfNotPresent fixed it; mirroring the image to my own registry is the durable answer.

The restore that counted #

On 3 October 2026 a test deleted my home folder (that story has its own post). Several local-PV directories were inside it. Files owned by my user were gone; files owned by container users survived. So Qdrant, Open WebUI and its Postgres were half-deleted while still running.

This was the first real restore from Velero’s Kopia repository. I did it with the Kopia CLI directly, connected read-only per namespace to the same bucket and prefix Velero uses, with the snapshot ID taken from each PodVolumeBackup’s status. Per volume:

kubectl -n <namespace> scale deploy/<app> --replicas=0
mv <volume-dir> <volume-dir>.partial-<date>      # keep the damaged copy
mkdir <volume-dir> && chown <uid>:<gid> <volume-dir> && chmod <mode> <volume-dir>
kopia snapshot restore <snapshot-id> <volume-dir>
du -sb <volume-dir>                               # must equal PodVolumeBackup totalBytes
kubectl -n <namespace> scale deploy/<app> --replicas=1

The results:

• Qdrant: 16.1 GB restored with an exact byte match (16,114,689,676 bytes), all 26 collections green;
• Open WebUI’s Postgres: exact byte match, clean WAL recovery, users, chats and files all present;
• Open WebUI’s data volume: exact match, healthy.

Writes made after that morning’s backup were lost for those services. I kept the partial directories for comparison. Several other PVC-backed apps came back the same way, with byte counts matching the PodVolumeBackup totals. Services outside Velero’s reach came back from separate copies on the NAS.

Checklist #

• Know which of your volumes are hostPath. Velero’s file-system backup will not take them.
• Audit backups against a census of what exists, never against the backup’s own target list.
• Make dump scripts enumerate databases instead of listing them.
• Read per-volume results in velero backup describe --details; treat skip warnings as findings.
• Give local PVs nodeAffinity and claimRef; edit YAML surgically, not with a serializer.
• Exclude huge, regenerable CRDs (scanner reports) from object backups.
• Before raising a memory limit, confirm the OOMKilled container from lastState.
• Do a real restore and compare bytes or row counts, not just “Completed”.

Frequently asked questions #

Does Velero back up hostPath volumes? #

No. Velero’s file-system backup (Kopia or Restic) supports pod volumes such as PVC-backed volumes and emptyDir, but not hostPath. If a pod mixes hostPath data with an emptyDir, only the emptyDir is backed up, and the backup still completes.

Why does Velero need nodeAffinity on local PersistentVolumes? #

Local PVs are tied to one node, and Velero needs to know which node holds the data so the node agent there can read it. Without nodeAffinity, backups of those volumes can fail or be skipped. Converting hostPath PVs to the local type with nodeAffinity fixes it.

Why is Velero getting OOMKilled? #

A common cause is a very large set of objects in one resource type, such as security scanner reports stored as custom resources. Exclude those resources from the schedule, then size the memory limit to what remains. Confirm first that the Velero server, not another pod, is the one being killed.

How do I verify a Velero file-system restore? #

Compare the restored directory’s size with the PodVolumeBackup’s totalBytes, then check the application itself: database integrity, collection status, row counts. A matching byte count proves the copy; the application check proves it is usable.

The lesson #

Velero never lied to me. It backed up exactly what it supports and warned about the rest. I read “Completed” as “safe”, and I checked the backup against its own list instead of against reality. The fix was not a better tool. It was asking, for each thing I could not afford to lose, “show me the bytes”, and then restoring them once to see them come back.