A 20-second power cut, an 8-hour outage: LUKS on a home server

At 00:31 on 18 August 2026 my server lost power for about 20 seconds, and the outage that followed lasted all night. It came back on by itself, ran its firmware checks, and then sat at a disk-encryption passphrase prompt, behind a splash screen, for 8 hours 25 minutes until I walked up to it in the morning.

The reboot itself would have cost about five minutes. Full-disk encryption with an interactive passphrase turned it into a night-long outage of everything: about fifty Kubernetes namespaces, mail, VPN, honeypots and the bot that is supposed to tell me when something is down.

This post covers how I reconstructed what happened from the logs, why the journal made a passphrase wait look like a slow kernel, the 20-minute wave of false alarms that every cold boot produces, and the options for unattended unlock with their honest trade-offs.

Contents · 8 sections

TL;DR #

• 8 hours 25 minutes at the LUKS passphrase prompt; that was the outage, not the power cut. Root was encrypted with an interactive passphrase and nothing could unlock it unattended.
• The journal hides the wait. Boot timestamps start when journald starts, after the unlock, so it looks like the machine booted in the morning and the “kernel” phase was slow.
• The truth is in systemd’s KernelTimestamp vs UserspaceTimestamp and in /proc/uptime, which counts boot time and ignores clock changes.
• 37 failed Jobs after the boot were three different populations stacked in one alert digest; only three were real defects.
• Unattended unlock options (the TPM chip, a network key server, remote SSH unlock) each trade off something. Whatever you choose, alert on “box is waiting at the prompt”, because nothing else will.

What happened #

The last line of the previous boot’s journal ends mid-sentence at 00:31:38. There is no shutdown sequence, no kernel panic, no machine-check, thermal or memory error, and no out-of-memory event. A disk health check half an hour earlier had reported everything clean. On the next boot, every filesystem logged that it was recovering its journal, and the EFI partition had its dirty bit set.

That is the signature of power disappearing, not of a crash.

The firmware phase on the next boot measured 4 minutes 32 seconds (this machine trains a lot of memory at POST), so POST began around 00:31:58. A dead interval of about 20 seconds between the last log line and POST fits loss of mains and the BIOS powering back on after AC recovery, not a warm reset. Software logs cannot prove that on their own. A week later I checked the kernel’s persistent crash store (/sys/fs/pstore): empty, so no panic record was ever written. That is weak evidence, since pstore only captures panics if its backend works, but it points the same way.

Then the machine reached the initramfs, asked for the disk passphrase, and waited.

Why it looked like something else #

When I first looked, it seemed the machine had rebooted at about 09:02 and simply been slow. systemd-analyze blamed the kernel phase for most of the boot time. Nothing in the journal mentioned waiting.

The reason: journald starts after the root filesystem is unlocked and mounted. The current boot’s journal therefore begins at the unlock time. Everything before it, including hours at the prompt, is invisible in the journal and gets lumped into the “kernel” phase.

The right instruments are these:

# Kernel start vs switch to the real root (both in microseconds since boot)
systemctl show -p KernelTimestampMonotonic -p UserspaceTimestampMonotonic

# Time since boot, from CLOCK_BOOTTIME (unaffected by NTP steps)
cat /proc/uptime

# Boot phases
systemd-analyze

The difference between the kernel start and the userspace (switch-root) timestamp is the time spent in the initramfs, which here means at the prompt. And if /proc/uptime is much larger than the age of the journal’s first entry, you were not looking at a clock jump; the machine was running and waiting before the journal existed.

Two more tells: the kernel ring buffer is silent for the whole window, and the first message after the silence is the mount of the filesystem inside the encrypted container.

Total: about 5 minutes of reboot and 8 h 26 min of passphrase prompt, an outage of 8 h 31 min. The prompt was hidden behind the boot splash (quiet splash, no GRUB timeout), so even a monitor plugged in would have shown a logo.

The morning after: a wave of false alarms #

When the node came back at 09:02, my alert digest exploded. Every CronJob whose schedule had passed during the night fired at once, against dependencies that were not up yet:

• backup-repository maintenance jobs failed because the object store was not ready;
• a whole data pipeline failed on connection refused to PostgreSQL;
• several jobs got ImagePullBackOff because the in-cluster image registry had not started yet.

Every one of them succeeded on its next scheduled run without help. I now budget about 6 minutes to first pod and 25 minutes to steady state after a cold boot before treating alerts as real.

The digest showed 37 failed Job objects. Sorting them properly found three populations:

1. 28 boot-storm jobs, all superseded by later successful runs. They kept alerting only because the failed-job alert fires on the existence of a failed Job object, not on ongoing failure.
2. 5 jobs that were running at 00:00–00:30 and were killed by the power cut.
3. 3 genuine defects that the noise had been hiding for up to two weeks.

I wrote a small checker that maps each failed Job to its owning CronJob and marks it deletable only if the CronJob’s lastSuccessfulTime is newer than the failure. It independently found the same three real problems I had found by hand:

• a weekly vulnerability report that was OOM-killed at 256 MiB because listing every vulnerability report in the cluster now pulled 40.8 MiB of JSON across 240 objects, which kubectl buffers in full. The symptom was exit code 137 and no log output at all. The diagnosis was two runs: the same image without a memory limit (works), then with the real limit (killed). Raised to 1 GiB, verified with a real run in 17 seconds;
• two monitoring backup jobs, also at 256 MiB, raised to 512 MiB based on measured peaks;
• a job whose retry limit I lowered from 1 to 0, so one failure is reported once.

Then I deleted all 37 failed Job objects, leaving zero failed Jobs and zero not-ready pods.

A lesson inside the lesson: a bare 256 MiB limit on any job that lists a growing resource is a time bomb. It works until the list grows.

Unattended unlock: the options #

The root cause of the long outage is a deliberate choice: an interactive passphrase means the secret is in my head, and a stolen machine is a brick. The cost is that any unplanned reboot waits for me. The common alternatives:

• TPM2 auto-unlock (systemd-cryptenroll --tpm2-device=auto). Fully unattended. But if the whole machine is stolen, it unlocks itself for the thief, which defeats much of the point of full-disk encryption against physical theft.
• Network-bound unlock (Clevis with a Tang server on the LAN). The disk unlocks only when it can reach the key server at home. Stolen and powered up elsewhere, it stays locked. It needs a second always-on device, and that device must not live in the same box.
• Remote SSH unlock (Dropbear in the initramfs). The passphrase stays in my head, and I can type it from my phone. It still needs me awake and reachable, but it turns “until I am physically there” into “a few minutes after I get an alert”.

The decision is a trade between availability and what physical theft should cost, and it deserves to be made deliberately rather than by default. What I am sure about is the second half: nothing told me the machine was waiting, because the alerting ran on the same machine. If you rely on an interactive unlock, monitor from outside: a ping or HTTP check from another device or a cheap external service, so “the box is at the prompt” becomes an alert instead of a surprise.

And if power cuts are the trigger, the decisive instrument is a UPS with logging, or at least a smart plug that records power events. My router’s logs rotate within about 20 minutes, so they could not confirm anything after the fact.

Checklist #

• Know what your server does after power returns: BIOS AC recovery setting, and what it waits for next.
• If root is encrypted with a passphrase, decide deliberately between TPM2, network-bound or remote SSH unlock, or accept the wait.
• Remove quiet splash on servers so a prompt is visible on a console.
• Diagnose boot gaps with KernelTimestamp vs UserspaceTimestamp and /proc/uptime, not with journal timestamps.
• Monitor from outside the machine; the box cannot report that it is not running.
• After a cold boot, wait for steady state before trusting job alerts, and triage failed Jobs against their CronJob’s last success.
• Treat small memory limits on list-everything jobs as time bombs.
• Log power events with a UPS or smart plug.

Frequently asked questions #

Why does my encrypted Linux server not come back after a power cut? #

If the root filesystem uses LUKS with an interactive passphrase, the initramfs waits at the prompt until someone types it. With a boot splash the prompt may not even be visible. Use TPM2, Clevis/Tang or Dropbear SSH unlock if you need unattended or remote recovery.

How can I tell how long a machine waited at the LUKS prompt? #

Compare systemd’s KernelTimestampMonotonic with UserspaceTimestampMonotonic from systemctl show. The difference is the time spent in the initramfs. Cross-check with /proc/uptime, which counts from boot and is not affected by clock changes.

Was it a power cut or a kernel panic? #

A journal that ends mid-line with no shutdown, no panic and no hardware errors, followed by filesystem journal recovery and a short dead interval before POST, points to power loss. An empty /sys/fs/pstore is consistent with that, though not proof. A UPS with logging gives you a definite answer next time.

Why do so many CronJobs fail right after a reboot? #

The CronJob controller starts missed jobs as soon as the node is back, often before databases, object stores and registries accept connections. Check job start times against the boot time before investigating, and expect them to succeed on their next run.

The lesson #

The power cut cost twenty seconds. The rest of the outage came from a security choice I had made on purpose and never connected to availability: a passphrase only I know is also a passphrase only I can type. Encrypting a server is right; doing it without a plan for unattended recovery and without monitoring from outside turns every power blip into a long silence. Decide what a reboot at midnight should look like before it happens.