Falco on Linux 7.0: 222 restarts and a BPF verifier error

My runtime security sensor restarted 222 times in four weeks. Each crash lasted about a second, the pod came back on its own, and the dashboard stayed mostly green. That is exactly what made it dangerous: a security tool that blinks nine times a day is a security tool with nine gaps a day, and nobody notices gaps.

The sensor is Falco, which watches system calls on my Kubernetes node and raises alerts for things like a shell spawning in a container or a process reading credential files. The crashes came from a kernel upgrade, but getting to the fix took me through memory limits, dropped events, a Helm release that had been “failed” for six months, and the eBPF verifier. Here is the whole path.

Contents · 9 sections

TL;DR #

• On Linux 7.0, Falco 0.43 crashed several times a day with “could not parse param” errors. The diagnosis: one of the driver’s eBPF programs was incompatible with a kernel change in how a signal tracepoint is typed, and the kernel’s BPF verifier rejected it.
• The upstream fix shipped in driver 11.0.1 / Falco 0.45.1. After the upgrade: 0 restarts.
• Earlier, high syscall volume caused OOM kills and up to 281,000 dropped events per second, which produced similar parse crashes. Memory went from 2 GiB to 8 GiB, plus base_syscalls.repair.
• The Helm release had been failed since April, because kubectl set edits conflict with Helm 4’s server-side apply. Pin every live value in the values file, then upgrade with --force-conflicts.
• Monitor restart counts of security agents, not just “is it running”.

The symptoms #

A sanity check on 25 October listed Falco as healthy. The pod was Running. But its restart counter told a different story: 64 restarts in seven days, about nine a day, 222 since late September. The log line before each crash was a variant of:

Error: could not parse param 1 (exe) for event type 223 (clone)

Earlier crashes had said the same about openat. Falco’s userspace libraries read raw events from a kernel driver, decode each parameter, and refuse to continue when the bytes do not match the expected layout. Crash, restart, one second of blindness, repeat.

First suspect: load and memory #

This was not the first time I had seen parse errors, and the earlier cause was real, so it was the first thing I checked.

In September I had scaled a document-processing workload to 40 pods on the same node. Falco was OOM-killed at 1.7 GiB against a 2 GiB limit. I raised it to 4 GiB. Eleven days later it was OOM-killed again at 3.8 GiB, and the replacement pod restarted four times in under four hours with the same “could not parse param” error. Meanwhile the driver reported between 20,000 and 281,000 dropped syscalls per second.

That is a lot of events. Falco’s driver writes them into a ring buffer shared with userspace. When userspace cannot keep up, events are dropped, and the parser can end up with incomplete state for a process, which surfaces as parse failures. Two changes helped:

falco:
  base_syscalls:
    repair: true          # let Falco add the syscalls it needs to keep process state consistent
resources:
  limits:
    cpu: 2
    memory: 8Gi
  requests:
    cpu: 200m
    memory: 2Gi

The OOM kills stopped. The parse crashes did not, which meant memory was a second problem, not the main one.

The real cause: the verifier said no #

The crashes were tied to the Linux 7.0 kernel the node was running. Following the error upstream led to the explanation: Falco’s modern eBPF driver has a program attached to the signal_deliver tracepoint, and a kernel commit changed how that tracepoint’s arguments are typed. On kernels with that change, the BPF verifier rejects the program with:

R9 invalid mem access 'scalar'

What the verifier is doing #

eBPF lets you run small programs inside the kernel. Because they run with kernel privileges, the kernel does not simply trust them. Before a program is loaded, the verifier walks every possible path through it and tracks what each register holds: a pointer to the stack, a pointer to a map value, a pointer to context data, or just a number (a “scalar”). It only allows a memory read through a register it can prove is a valid pointer.

“R9 invalid mem access ‘scalar’” means: on some path, register 9 is used as an address, but the verifier can only prove that it holds a plain number. The program might be perfectly safe in practice. The verifier cannot prove it, so it refuses.

Modern eBPF tools are compiled once and adapt to each kernel through type information the kernel publishes (BTF). That is what makes them portable, and it is also why a change in a tracepoint’s declared types can break a program that was fine the day before: the same code now reads a field the verifier sees with a different type. This is not a Falco bug in the usual sense. It is the price of running code inside a kernel whose internal interfaces are allowed to change.

The fix #

The Falco maintainers fixed the program in the driver (version 11.0.1), and the fix shipped in Falco 0.45.1, chart 9.2.1. The upgrade on 30 October went in cleanly. Falco 0.45.1 loaded its modern BPF probe on the 7.0 kernel and ran with 0 restarts. My custom exception rules still loaded, with deprecation warnings for the old append syntax that I will migrate.

If you hit this class of error, the checklist is short: find your Falco version and driver version, find your exact kernel build, and check the Falco and falcosecurity/libs release notes for verifier or “invalid mem access” fixes. Upgrading Falco is usually the answer. Downgrading the kernel is the emergency brake.

The other problem: a Helm release that had been failed for six months #

Upgrading should have been one command. It was not, because helm status falco said failed, and had said so since 24 April.

Back in April the Falcosidekick image and the rules ConfigMap had been changed by hand with kubectl, outside Helm. Later, because the release was already failed, I changed resources with kubectl set resources too. Each of those edits made kubectl the owner of those fields. Helm 4 uses server-side apply, which tracks a field manager for every field. When Helm tries to change a field that another manager owns, server-side apply reports a conflict and Helm gives up. Every later emergency change went in by hand too, which made the drift worse.

There was a second trap: two values files existed. One was live; the other was a stale copy from April. Upgrading with the wrong one would have reverted months of changes.

The repair, in September:

1. Confirm which values file is live: helm get values falco -n <namespace> and compare.
2. Mark the stale copy clearly as stale.
3. Put every hand-made live change into the live values file: pinned image tags, resources, the repair setting. If you skip this, --force-conflicts will happily overwrite your pins with chart defaults.
4. Server-side dry run, then:

helm upgrade falco falcosecurity/falco -n <namespace> \
  -f falco-values.yaml --version <chart> --force-conflicts --wait

The release went back to deployed. The October upgrade to chart 9.2.1 hit the same conflict again on fields owned by before-first-apply, a manager left over from the very first install, and the same flag resolved it. The rule I follow now: no kubectl set on a Helm-managed workload, except in an emergency, and then the same change goes into the values file in the same hour.

Tuning the noise #

A security sensor that cries wolf is ignored, which is as bad as one that crashes. A weekly report in April showed 40 Critical and 16 Warning events, and every one was a false positive in my environment:

• package installation inside my security-testing containers;
• the container runtime using memfd_create when it starts pods;
• CrowdSec clearing its own rotated logs;
• my own user reading PAM files during normal logins.

Each became a narrow exception in a separate rules file, scoped to the namespace, process or user involved, never a global disable. Narrow exceptions keep the rule alive for everything else.

Checklist #

• Alert on restart count of security agents, not only on “pod not running”.
• After any kernel upgrade, check that your eBPF tools load and stay up for a day.
• Read the verifier message. “Invalid mem access ‘scalar’” points to a program/kernel type mismatch, usually fixed by upgrading the tool.
• Size Falco for your busiest day; watch the dropped-events counter.
• Enable base_syscalls.repair if you trim syscalls or see state-related parse errors.
• One live values file. Delete or clearly mark any copy.
• Never leave a Helm release in failed state. Pin live changes in values, then --force-conflicts.
• Write narrow exceptions for false positives; never disable a rule globally.

Frequently asked questions #

What does “R9 invalid mem access ‘scalar’” mean in eBPF? #

The kernel’s BPF verifier could not prove that register 9 holds a valid pointer at the point where the program reads memory through it; it only knows the register holds a number. The verifier rejects the program. With tools like Falco, this usually means the program was written for kernel type information that has changed, and a newer version of the tool fixes it.

Why does Falco crash with “could not parse param”? #

Falco’s userspace failed to decode an event from the kernel driver. Causes I saw were a driver that did not match the kernel, and heavy event drops under load. Check the Falco version against your kernel and look at the dropped-events statistics.

Why does helm upgrade fail with field manager conflicts? #

Someone changed fields of the release’s resources with kubectl set or kubectl apply. Helm 4 uses server-side apply, which refuses to take over fields owned by another manager. Put those changes into your values file, then upgrade with --force-conflicts.

Is a short Falco restart a security problem? #

Yes. During the restart nothing is watched, and a crash that repeats nine times a day is a predictable gap. Treat repeated restarts of a security agent as an incident, even when each one is short.

The lesson #

The sensor never looked down, only tired. I found the real fault because I looked at the restart counter instead of the status column, and I could fix it quickly only after I had repaired the deployment tooling I had broken with “temporary” manual edits. Kernel-level security tools live on the edge of a moving interface. Upgrade them with the kernel, watch how often they restart, and keep their deployment boring enough that the upgrade is one command.