9,151 critical CVEs, mostly in images I wasn't running

The weekly vulnerability report for my home lab arrived on a Sunday morning in October with a number nobody can act on: 9,151 critical and 229,975 high CVEs across 1,925 container images.

It was not a security disaster. It was a counting disaster. Only 11 of those images were actually used by running Docker containers. The scanner was dutifully scanning every image I had ever pulled or built and never deleted, 203 GB of them. Once I scanned what was really running, the critical count dropped to 987, of which 759 had a fix available. That is still a lot, but it is a list you can work through.

This post is about making container scanning results useful: scoping, cleaning, sorting by “fixable”, and what happened four days later, when a real critical PHP vulnerability landed and the triage method got its first test.

Contents · 9 sections

TL;DR #

• Scan what runs, not what is stored. My report shrank from 1,925 images to 152 running ones, and from 9,151 to 987 criticals.
• Clean local images safely: keep the newest tags, anything a container uses, and anything any workload spec references, including suspended jobs and scaled-to-zero deployments. Use docker rmi without -f.
• Split every report into fixable and not fixable. Fixable criticals in running, exposed images come first.
• For a named, urgent vulnerability, inventory by asking the containers (php -v in each), not by guessing from image names.
• Pin patched images by digest and verify the version inside the new pods.

Where the number came from #

The weekly job runs Trivy against local Docker images and sends a summary to my security notification channel, with the full report by email. What it scanned was every image in the local Docker store:

• 1,822 tagged images, 203 GB;
• hundreds of old tags of my own images, because every release was built locally and pushed to my registry;
• pulled images for experiments that were long gone.

Every old tag of every image contributed its full set of CVEs. A vulnerability in a base image that I had fixed twenty releases ago was still counted twenty times. The report was accurate and useless.

Step 1: clean the local store, carefully #

Deleting old images sounds trivial. The risk is deleting something that is not running right now but will be needed: a CronJob that runs weekly, a Job that runs once after a restore, a deployment scaled to zero for the winter.

So the rule for what to keep was:

1. the 2 newest tags of each repository;
2. every image used by an existing container, running or stopped;
3. every image referenced by any Kubernetes workload spec: Deployments, StatefulSets, DaemonSets, CronJobs and Jobs, including suspended and zero-replica ones. That was 179 references.

Everything else was a candidate. I removed candidates with plain docker rmi, never -f, so Docker itself refuses to delete anything still in use:

# keep-list built from containers + every workload spec in the cluster
kubectl get deploy,sts,ds,cronjob,job -A -o json \
  | jq -r '.. | .image? // empty' | sort -u > keep.txt
docker ps -a --format '{{.Image}}' >> keep.txt

# candidates = all tags minus keep-list minus the 2 newest per repo
# then remove without force; refusals are a signal, not an obstacle
xargs -r -n1 docker rmi < candidates.txt
docker image prune -f      # dangling layers only

The result: 1,645 tags removed, 0 refused, then 12.5 GB of dangling layers pruned. The local store went from 1,956 images to 165, and from 203 GB to 142 GB. The registry was not touched, so every deleted tag could still be pulled again.

Step 2: scan what runs #

The reworked weekly scan builds its list from what is actually deployed and groups images into running and parked (referenced by a workload that currently runs nothing). Its first run, the same day:

• running: 152 images, 987 critical (759 fixable), 20,279 high (12,916 fixable);
• parked: 32 images, 249 critical, 4,423 high;
• 186 unique images in total, 2 scans failed.

That run also exposed a bug in the scan script itself: with set -o pipefail, a pipeline whose reader closed early returned exit status 141 (SIGPIPE), and the script stopped before sending its notification. A security job that dies quietly before reporting is worse than no job, because you assume silence means “nothing new”. Watch the exit code of the reporter, not just of the scanner.

Step 3: sort by what you can actually fix #

The most useful column in a Trivy report is Fixed Version. A critical CVE with a fixed version means: a newer package exists, and upgrading the image will remove it. A critical CVE without one means: nobody can fix it yet, and the work is mitigation, not patching.

trivy image --severity CRITICAL --ignore-unfixed --format json <image> \
  | jq '[.Results[]?.Vulnerabilities[]?] | length'

--ignore-unfixed gives you the actionable count. Run it without the flag too, so you know what you are carrying.

The worst running images were all large applications that bundle a whole operating system, a language runtime and hundreds of libraries: a webmail client, AI inference and pipeline servers, a video platform, a document question-answering app, and an older version of my own MCP server that two side components still used after the main deployment had moved on. That last one is a classic: one image, several workloads, and only some of them upgraded. A scan grouped by image, not by deployment, is what reveals it.

The webmail image had 78 criticals, 66 of them fixable. It was at the top of my list on Sunday. On Thursday, events made the decision for me.

A real test: a critical PHP vulnerability #

On 22 October 2026, two critical remote code execution vulnerabilities in PHP were published, unauthenticated, rated CVSS v4 9.2, affecting PHP-FPM, mod_php and CGI setups. Fixed versions had been released the day before: 8.5.12, 8.4.27, 8.3.36 and 8.2.35.

The first question in any incident like this is “where do I run PHP?”. Image names are not a reliable answer: PHP hides inside webmail clients, wikis, database admin tools and paste services. So I asked the containers themselves:

for pod in $(kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'); do
  ns=${pod%/*}; name=${pod#*/}
  v=$(kubectl exec -n "$ns" "$name" -- php -v 2>/dev/null | head -1) && echo "$pod: $v"
done

I checked the host and host Docker containers the same way. The answer was five workloads: a WordPress site, a paste service, the webmail client (on an end-of-life PHP 8.1), a database admin tool and a small CMS. No PHP on the host itself.

Each got an image that carried a fixed PHP version, pinned by digest so the tag could not move under me:

image: <registry>/<image>@sha256:<digest>   # tag noted in a comment

The webmail client went from 1.6.10 to 1.6.20, moving from the end-of-life PHP 8.1 to 8.4.27. After each rollout I ran php -v in the new pod and checked that the site answered. All five were done the same day.

The weekly report had already told me the webmail image was the worst one. The emergency simply moved it from “this week” to “this morning”, and the inventory method meant I did not have to wonder whether I had missed a sixth PHP.

Checklist #

• Scan running and referenced images, not the whole local store.
• Clean images with a keep-list built from containers and all workload specs, including suspended and zero-replica ones.
• Remove with docker rmi without -f; leave the registry intact.
• Report fixable and unfixable counts separately; act on fixable ones in exposed services first.
• Group findings by image to catch old versions still used by side workloads.
• Check the reporter’s exit code; pipefail plus an early-closing reader returns 141.
• For named incidents, inventory by executing version commands in every container.
• Pin patched images by digest and verify the version inside the new pod.

Frequently asked questions #

Why does Trivy report so many critical vulnerabilities? #

Often because it scans more than you run. Every stored image and every old tag counts. Scope the scan to images referenced by your running workloads, and separate fixable from unfixable findings. My critical count dropped by about 90% from scoping alone.

Which container vulnerabilities should I fix first? #

Fixable critical vulnerabilities in images that run and are reachable from the internet, especially without authentication. Then fixable ones in internal services. Unfixed vulnerabilities need mitigation, such as limiting exposure, until a fix exists.

Is it safe to delete old Docker images? #

Yes, if you keep everything that any container or workload spec references, including scheduled and scaled-down ones, and you delete without force. Docker refuses to remove images still in use, and images that exist in a registry can be pulled again.

How do I find every container running a vulnerable runtime like PHP? #

Ask the containers. Run the runtime’s version command in every running pod and container and collect the answers. Image names miss runtimes bundled inside applications.

The lesson #

A vulnerability count is only as useful as its denominator. Mine was counting a museum of old images, so it measured my storage habits, not my risk. Once the scan matched what really ran, and the report separated what I could fix from what I could not, it turned into a short, ranked to-do list, and when a real emergency arrived, the first item on it was already the right one.