A post-reboot check script for a one-machine home lab

On 18 August 2026 one reboot of my home lab left 37 failed Job objects and a wall of alerts. Thirty-four of them were noise. Three were real defects that the noise had been hiding for up to two weeks.

The lab is one workstation running a Kubernetes cluster, local language models, a research library and a security pipeline. When it reboots, firmware memory training alone takes 4 minutes 32 seconds, and about 25 minutes pass before everything is steady. In that window, dozens of things look broken that are not, and a few things are broken that look fine. I used to check by hand, and I was bad at it. Since October, one read-only script answers “did everything come back?” in a single run, with OK or FAIL per line. This post is about that script, what it checks, and the hidden failures it has found.

Contents · 9 sections

TL;DR #

• A reboot is the only test of everything that “worked until the restart”: stale passwords, deleted files still held open, renamed network interfaces, images that can no longer be pulled.
• After a cold boot, expect a wave of false alarms from scheduled jobs that fire against services that are not up yet. Wait before you triage.
• Write one read-only check script with plain OK/FAIL lines, grouped by layer, and run it after every reboot.
• Add a check every time a reboot surprises you. The script is a list of past incidents.
• Print the expected differences at the end, so known gaps do not look like new failures.

Why reboots hide problems #

A running system carries a lot of state that its configuration does not describe. A database keeps working on files that were deleted, because it still has them open. A service keeps a connection that was authenticated with an old password. A container keeps an image that the registry no longer serves. A network policy matches an IP address that a container happened to get last time.

None of that is visible while the machine runs. All of it is tested at once on the next boot.

My lab has had every one of these in 2026:

• On 26 September: after a graphics card was reseated, the onboard network card enumerated at a different PCI address and got a new interface name. The video bridge of my self-hosted Jitsi was pinned to the old name, found no allowed address and crash-looped. In the same boot, the object storage behind my cluster backups could not start, because its deployment pinned an image digest with “always pull”, and Docker Hub no longer served that digest. The exact image was still in the local cache.
• On 4 October: the first reboot after my home folder was deleted. Three MySQL databases had been running for a day on data files that no longer existed. They failed to start and were restored from the nightly dumps.
• On 30 October: two services had a pre-rotation password hard-coded as a default and had survived only on old connections (the full story is in passwords that only fail after a reboot). And a container’s bridge IP moved by one address, which broke a network policy that allowed exactly one IP.

The boot storm is not a fault #

The 18 August boot taught me to wait. When the node came back, every CronJob that had missed its schedule fired at once, against databases, registries and APIs that were still starting. Backup maintenance jobs failed. The whole threat-intelligence pipeline failed. Several pods could not pull images because the in-cluster registry was not up yet.

Every one of them succeeded on its next scheduled run, with no help.

The alert digest that morning mixed three populations that looked identical:

1. 28 jobs from the boot storm, all already superseded by a successful run;
2. 5 jobs killed mid-run by the power cut that caused the reboot;
3. 3 genuine defects hidden in the noise for up to two weeks.

The reason the first group kept alerting is worth knowing. The common KubeJobFailed alert fires while a failed Job object exists, not while the CronJob keeps failing. A CronJob that failed once at boot and succeeded ten times since still alerts until someone deletes the old Job.

So I wrote a small checker: for each failed Job, find its owning CronJob and mark the Job deletable only if the CronJob’s lastSuccessfulTime is newer than the failure.

kubectl get jobs -A -o json | jq -r '
  .items[] | select(.status.failed // 0 > 0)
  | [.metadata.namespace, .metadata.name,
     (.metadata.ownerReferences[0].name // "-"),
     .status.startTime] | @tsv' |
while IFS=$'\t' read -r ns job cj started; do
  [ "$cj" = "-" ] && { echo "REVIEW $ns/$job (no owner)"; continue; }
  last=$(kubectl -n "$ns" get cronjob "$cj" -o jsonpath='{.status.lastSuccessfulTime}')
  if [[ -n "$last" && "$last" > "$started" ]]; then echo "STALE  $ns/$job"; else echo "REAL   $ns/$job"; fi
done

It isolated the same three real problems I had found by hand. One was a weekly vulnerability report that had been killed for running out of memory since early August: the list it fetched had grown to 40.8 MiB across 240 objects, and kubectl buffers the whole list. A memory limit raise fixed it, and the report ran in 17 seconds.

My rule now: after a cold boot, give the machine about 25 minutes before treating alerts as real, then run the check script, then triage only what it reports.

What the script checks #

The script is about 70 lines of Bash. It changes nothing. Three helpers do all the work:

#!/bin/bash
# post-reboot-check.sh: did the host and cluster come back? (read-only)
set -u
OK=0; BAD=0
ok()  { printf '  OK   %s\n' "$1"; OK=$((OK+1)); }
bad() { printf '  FAIL %s\n' "$1"; BAD=$((BAD+1)); }
chk() { if eval "$2" >/dev/null 2>&1; then ok "$1"; else bad "$1"; fi; }
hdr() { printf '\n== %s\n' "$1"; }

Then it walks the stack from the bottom up, one section per layer.

Boot. Is this the kernel I expect? Any failed systemd units? Is swap on? Is every data disk mounted, and does the NAS automount answer?

hdr "Boot"
chk "no failed system units" '[ -z "$(systemctl --failed --no-legend)" ]'
for m in /mnt/data1 /mnt/data2 /mnt/ramdisk; do chk "mounted $m" "mountpoint -q $m"; done
chk "NAS answers" 'timeout 20 stat /mnt/nas >/dev/null'

Core services and timers. One line per systemd unit that matters: container runtime, Kubernetes, the intrusion-prevention agent, the model server, the backup timers. A timer that is not active is a backup that will not run.

Containers. The few things that run in plain Docker next to the cluster.

AI stack. Real health requests, not “is the process up”: the model router must list its models, the GPU must be visible.

Kubernetes. The node is Ready, no pod is outside Running/Completed, and no deployment is below its desired replicas:

hdr "Kubernetes"
chk "node Ready" 'kubectl get nodes --no-headers | grep -q " Ready"'
NOTOK=$(kubectl get pods -A --no-headers | awk '$4!="Running" && $4!="Completed"' | wc -l)
[ "$NOTOK" -eq 0 ] && ok "all pods Running/Completed" || bad "$NOTOK pods not Running/Completed"
chk "no deployment below desired replicas" \
  '[ -z "$(kubectl get deploy -A --no-headers | awk "{split(\$3,a,\"/\"); if (a[1]!=a[2]) print}")" ]'

Security pipeline. Has the firewall bouncer synced decisions in the last 30 minutes, with a plausible number of entries? A security component that silently stopped after boot is worse than one that crashed loudly.

Incident-specific checks. After the bridge IP moved on 30 October, the script compares the container’s current IP with the address the network policy allows. It is a very specific check, and that is fine: it encodes a failure that already happened once.

Backups. Was there a host snapshot in the last two hours? Does the backup freshness check report zero stale sets?

The script ends with a count and an exit code, so it can be run by a timer or an agent as well as by me:

printf '\n%d OK, %d FAIL\n' "$OK" "$BAD"
exit $([ $BAD -eq 0 ] && echo 0 || echo 1)

Expected differences, written down #

The last part of the output is a plain list of things that are different on purpose after this reboot: a desktop sync client that was not re-paired, a temporary recovery mount that is gone, old debug port-forwards that are gone. Without that list, every run starts with the same question for each line: “is this new?”

The list also needs reviewing. Mine said that Prometheus would lose about two hours of data on reboot, because its write-ahead log was on a RAM disk. When I checked for this blog, the RAM disk was mounted but empty: the log had always been on the SSD. A written expectation is still a claim, and claims need checking.

What it has caught #

On 4 October the script reported 57 checks OK and one false FAIL (a check that needed sudo without a terminal). The real damage that morning, three dead MySQL instances, was right there in the pod section of the same run. The 26 September network rename, the stale image digest and the 30 October bridge IP all show up as FAIL lines now, because each of them became a check after it happened.

Checklist #

• Wait for the boot storm to pass (measure your own; mine is about 25 minutes) before triaging alerts.
• Clear failed Job objects that a newer successful run has superseded; investigate the rest.
• Keep one read-only check script, grouped by layer, with OK/FAIL lines and an exit code.
• Use real health requests, not process checks, for anything with an API.
• Check mounts, timers and backups, not only services.
• Turn every reboot surprise into a new check.
• End the output with a reviewed list of expected differences.
• After a big change or an incident, reboot on purpose and run the script while you still remember what changed.

Frequently asked questions #

Why do so many Kubernetes jobs fail after a reboot? #

Missed CronJobs start as soon as the node is back, often before the databases, registries and services they need are ready. They fail once and then succeed on their next run. The failure alerts remain because they are tied to the failed Job objects.

Why does KubeJobFailed keep firing after the job succeeded? #

The usual rule is based on kube-state-metrics’ failed-job metric, which exists as long as the failed Job object exists. Check whether a newer run of the same CronJob succeeded, and delete the old Job if it did. Deleting it clears the alert; it does not fix anything.

What should a post-reboot check script include? #

Kernel and failed units, disk and network mounts, core services and timers, container state, real health checks of key APIs, Kubernetes node, pod and deployment state, security components, and the age of the latest backups. Then add every failure you have seen after a reboot.

Why did my network interface name change after a hardware change? #

Predictable interface names are derived from the PCI location. Adding or removing a card can renumber the bus, which gives the same network card a new name. Anything pinned to the old name, like an allowed-interfaces setting, breaks.

The lesson #

A reboot is a free integration test of every assumption a running system has quietly stopped checking. You get its results whether you want them or not. The only choice is whether you read them all at once, from one script, or discover them over the following week, one broken service at a time. Write the script before the next reboot, and make it longer after every surprise.