Prometheus WAL on a RAM disk: the trade-offs, and my empty tmpfs

My Prometheus configuration puts the write-ahead log on a 30 GB RAM disk, “for fast writes (volatile, acceptable)”. My post-reboot checklist warned that every reboot would lose about two hours of metrics because of it. While writing this post I finally looked inside the pod. The RAM disk was mounted and held 0 bytes. The real write-ahead log, 785 MB of it, was on the SSD where it had always been.

So this post has two parts. The first is how a carefully commented storage design never took effect, and how to check yours. The second is the honest trade-off: when a RAM disk helps a monitoring or database stack, when it does not, and what it costs. I use RAM disks in four places in my lab, and they have earned their keep in two of them.

Contents · 9 sections

TL;DR #

• Prometheus keeps its write-ahead log (WAL) in the wal folder inside its data directory. Mounting a RAM disk at some other path does nothing unless Prometheus actually writes there.
• Measure before you optimise: look at where the bytes are (du, mount) and whether disk writes are a bottleneck at all.
• The WAL is Prometheus’s crash-recovery log for the last couple of hours of samples. On a RAM disk, every reboot or power loss loses that data by design.
• RAM disks shine for caches and scratch space that can be rebuilt: query caches, temporary tables, transcoding. They are a poor fit for anything you would be sad to lose.
• Every RAM disk needs a size limit, because tmpfs uses memory, and a full tmpfs fails like a full disk.

The design on paper #

The lab runs kube-prometheus-stack on a single-node k3s cluster. Prometheus keeps 30 days of metrics, capped at 15 GB, and scrapes every 30 seconds. The values file starts with a neat comment describing three storage tiers:

# Storage strategy:
#   Local SSD    : TSDB blocks, 30-day retention, fast queries
#   Ramdisk      : Prometheus WAL for fast writes (volatile, acceptable)
#   NFS (CronJob): TSDB snapshots for backup

And further down, the part that was supposed to make the second line true:

prometheus:
  prometheusSpec:
    walCompression: true
    volumes:
      - name: ramdisk-wal
        hostPath:
          path: /mnt/ramdisk/prometheus-wal
          type: DirectoryOrCreate
    volumeMounts:
      - name: ramdisk-wal
        mountPath: /prometheus-wal

The host has a 30 GB tmpfs at /mnt/ramdisk. The pod gets a folder from it, mounted at /prometheus-wal. Everything applies cleanly, the pod is healthy, and the dashboards work.

What was actually happening #

Two commands inside the Prometheus container tell the whole story:

kubectl -n <monitoring-namespace> exec <prometheus-pod> -c prometheus -- \
  sh -c 'du -sh /prometheus/wal /prometheus-wal; mount | grep prometheus'

785.0M  /prometheus/wal
0       /prometheus-wal
/dev/mapper/<vg>-<lv> on /prometheus type ext4 (rw,relatime)
tmpfs on /prometheus-wal type tmpfs (rw,relatime,size=31457280k)

The WAL is in /prometheus/wal, on the SSD. The RAM disk is mounted, and nothing writes to it.

The reason is simple once you know it. A Prometheus server has one storage location, --storage.tsdb.path, and keeps everything under it: the compacted blocks, the chunks_head folder for memory-mapped head chunks, and the wal folder. The Prometheus Operator sets that path to /prometheus, backed by the persistent volume. There is no server option that places the WAL somewhere else. A volume mounted at /prometheus-wal is just an empty folder that nobody asked for.

Nothing failed, so nothing told me. The comment, the volume, and a line in my post-reboot checklist (“Prometheus loses about 2 hours of WAL, by design”) all described a system that did not exist. In practice my Prometheus does not lose its WAL on a reboot; it replays it from the SSD like any default install.

Would it have been a good idea? #

Here is the part worth learning from, because it applies to anyone tempted by the same idea.

What the WAL is for #

Prometheus keeps the newest samples in an in-memory “head block”. Every two hours or so the head is compacted into an immutable block on disk. Until then, the only durable copy of those samples is the WAL: every incoming sample is appended to it, in segments of 128 MB. After a crash or a restart, Prometheus replays the WAL to rebuild the head.

That makes the WAL the opposite of a cache. It exists precisely so that a restart does not lose recent data. Putting it on a RAM disk keeps the cost (every sample is still written) and removes the benefit (the copy no longer survives a reboot).

Was disk write speed a problem? #

Not for this workload. The WAL is written sequentially, compressed (walCompression: true), from a 30-second scrape interval. For a single node, that is a light, steady stream that any SSD handles without noticing. Speeding up something that is not the bottleneck buys nothing.

When a disk was saturated in my lab, it was a database. A batch job writing heavily to PostgreSQL drove a checkpoint roughly every 19 seconds at about 28 MB/s of PostgreSQL WAL, and the disk-queue alert fired. Prometheus was never in that picture.

If you really want it #

If you accept losing recent metrics on every reboot, the honest designs are:

• put the whole TSDB on tmpfs and accept that metrics live only until the next reboot (sometimes fine for a throwaway dev cluster);
• or run Prometheus in agent mode, which keeps only a WAL and forwards samples to remote storage, and size that for a RAM disk deliberately.

Moving just the wal folder with symlinks or init containers, behind the server’s back, is fragile, and it keeps the cost while losing the safety.

Where RAM disks did help #

The same lab uses RAM-backed storage in other places, and the difference is instructive.

Loki’s caches: yes #

In April, Grafana dashboards that queried Loki were slow and some panels timed out. Loki had only a 100 MB results cache, no chunk cache and no index cache, and a 2 GiB memory limit on a host with hundreds of gigabytes free. The fix added three in-process caches (1 GB chunk cache, 512 MB results cache, 256 MB index-query cache, each with a TTL), a 1 GB tmpfs for the index cache folder, and a 6 GiB memory limit. Repeated queries went from 102 ms cold to 51 ms warm.

That works because everything there is a cache: if the pod restarts, the caches refill from the real data on disk. In Kubernetes, the clean way to get such a folder is an emptyDir with medium: Memory and a sizeLimit, which disappears with the pod:

volumes:
  - name: cache
    emptyDir:
      medium: Memory
      sizeLimit: 1Gi

PostgreSQL temporary files: yes, with a gotcha #

After one runaway query wrote 376 GB of temporary files to the root disk in June, my shared PostgreSQL moved its temp_tablespaces to a 10 GiB memory-backed emptyDir, with temp_file_limit as a cap. Temporary files are scratch space by definition, so that is a good fit. The catch: a tablespace contains a version-specific subfolder that PostgreSQL creates only once, so after the first pod restart the empty RAM disk broke temporary tables until an init container recreated it. If a tablespace lives on tmpfs, plan for its folders to vanish.

Media transcoding: yes #

The media server writes its temporary transcode files to the same host RAM disk. Those files are thrown away after playback, so tmpfs is a natural home for them, and it spares the SSD a steady stream of writes.

Large model files: careful #

The local model server copies about 30 GB of model files into a RAM disk at boot for fast loading. One lesson from that: deleting a file from tmpfs while a process still has it mapped does not free the memory. A replacement copy then failed with “No space left on device”. Unload the model first, then delete, then copy.

How to check yours #

# 1. Where does the application really write?
kubectl exec <pod> -- sh -c 'du -sh <data-dir>/* <ramdisk-mount>; mount | grep -E "<data-dir>|<ramdisk-mount>"'
# 2. How full is the host RAM disk, and who uses it?
df -h /mnt/ramdisk; du -sh /mnt/ramdisk/*
# 3. Is disk I/O a bottleneck at all? (queue size per device)
rate(node_disk_io_time_weighted_seconds_total[5m])

If the RAM disk folder is empty after a day of normal running, it is decoration.

Checklist #

• Confirm that the application writes where you mounted the RAM disk, with du, not by reading the config.
• Prove that disk I/O is a bottleneck before moving anything to memory.
• Use RAM disks for rebuildable data only: caches, temporary files, transcodes.
• Never put a durability mechanism (a WAL, a journal) on a RAM disk unless losing it is truly acceptable.
• Set a size limit on every tmpfs and memory-backed emptyDir.
• Plan for the RAM disk to be empty after every restart, including folders the app created once.
• Re-check comments and runbooks against the running system from time to time.

Frequently asked questions #

Can I put the Prometheus WAL on a separate disk? #

Not with a server option. Prometheus stores the WAL in the wal folder under --storage.tsdb.path. Mounting a volume elsewhere has no effect. Agent mode has its own storage path, which holds only a WAL.

What happens if the Prometheus WAL is lost? #

You lose the samples that were not yet compacted into a block, typically the last few hours. Older data in completed blocks is unaffected.

Does walCompression help? #

It makes the WAL smaller on disk at a small CPU cost. For a single-node setup it is a sensible default and keeps write volume low.

Is a memory-backed emptyDir a good Kubernetes cache? #

Yes, for data that can be rebuilt. It is fast, limited by sizeLimit, counted against the pod’s memory, and wiped when the pod goes away.

The lesson #

The most interesting thing about my “WAL on a RAM disk” was not the trade-off, it was that nobody had looked. The comment was clear, the manifest applied, the pod was green, and all of it was wrong from the start. RAM disks are a good tool for things you can afford to lose. Before choosing one, check what the bytes are for. After choosing one, check that the bytes actually go there.