NetworkPolicy gotchas: hostNetwork Traefik and Docker DNAT

I wrote a NetworkPolicy that allowed exactly one destination: my LLM server on the host, one IP, one port. The pod got “connection refused”. I widened the rule until it worked, and a later probe showed the widened rule now reached every Docker container on the machine and a game server’s admin panel too.

That was one of three NetworkPolicy surprises on my single-node k3s cluster in October 2026. None of them is a bug. Each one is a place where the network does something before, or around, the policy you wrote. This post explains the mechanisms so you can write policies that mean what you think they mean.

Contents · 10 sections

TL;DR #

• 3 surprises on one cluster, and none of them is a bug: each is something the network does before or around the policy.
• 1 forged header (Remote-Groups: admins) sent from the node was enough to create an admin account in one of my apps, because the policy could not tell Traefik from the rest of the host.
• Traefik on hostNetwork has the node’s IP as its pod IP. A podSelector for Traefik never matches. You have to allow the node IP, and then every other hostNetwork pod and every host process is allowed too.
• So identity headers are not proof. An app that trusts Remote-User from “anything the policy admits” can be fooled from the node. I added a secret edge-proof header that only Traefik sends.
• Docker DNAT happens before the policy sees the packet. Egress to a Docker-published host port must allow the container’s bridge IP and container port, not the host IP and published port.
• Be exact when you allow that: one container IP per rule, never the whole bridge subnet. The bridge IP is not stable, so check it after every reboot.
• Policies take a few seconds to apply. Retest before you conclude anything.

Background: the cluster #

One machine, k3s with its embedded network policy controller (kube-router), Traefik as the ingress controller, and a mix of Kubernetes workloads and plain Docker containers on the same host. Some services, like my local LLM server, run in Docker and publish a port on the host. Pods reach them through the host.

That mix is common in home labs, and it is exactly where NetworkPolicy gets subtle.

Surprise 1: Traefik is not a pod you can select #

Traefik in this cluster runs with hostNetwork: true. Its pod IP is the node’s IP. When it forwards a request to a backend pod, the backend sees the connection coming from the node (or, after SNAT, from the CNI gateway address).

So this policy, which looks right, admits nothing from Traefik:

# Looks right, does nothing for hostNetwork Traefik
ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: kube-system
        podSelector:
          matchLabels:
            app.kubernetes.io/name: traefik

Policies match pod IPs. Traefik’s traffic does not come from a pod IP. What works is an ipBlock for the node address, plus the CNI gateway address that backends see:

ingress:
  - from:
      - ipBlock:
          cidr: <node-ip>/32
      - ipBlock:
          cidr: <cni-gateway-ip>/32

That works, and it has a cost you must understand: the node IP is shared. Every hostNetwork pod on the node uses it (in my case several others besides Traefik, such as a video bridge and a TURN server), and so does every process running on the host itself. At layer 3, “allow Traefik” and “allow the whole machine” are the same rule.

The same mechanism affects anything that trusts a proxy by address. An nginx set_real_ip_from for the gateway address trusts “anything on the workstation”, not “Traefik”. And PROXY protocol cannot fix it, because Traefik supports it only for TCP backends, not HTTP. The one client address that cannot be forged is the one Traefik records in its own access log from the TCP peer.

Surprise 2: what that means for identity headers #

Many self-hosted apps support “forward auth”: the SSO proxy authenticates the user and the app trusts a header such as Remote-User or Remote-Groups. That is safe only if the header can come only from the proxy.

My research library’s API gateway and auth service believed identity headers from any caller that reached them, and their NetworkPolicies admitted the node IP, as they had to for Traefik. On 31 October 2026 I proved that a direct call with a forged Remote-Groups: admins header auto-provisioned an admin account. (The test account was disabled immediately.) Several other backends had no NetworkPolicy at all.

A NetworkPolicy cannot close this, for the reason above. The fix has to be at layer 7:

1. An edge proof. A Traefik middleware, placed after the forward-auth middleware, adds a header carrying a secret value. The backend believes Remote-* headers only when that header is present and correct. Any other caller, even from the node, does not know the secret.
2. A service token for internal service-to-service calls, required on every route except health probes.
3. Drop alternative identity headers like X-Forwarded-User that the app does not need, so they cannot be smuggled in.
4. Add NetworkPolicies to every backend anyway, so the set of callers is as small as it can be.

After that, the spoofed admin request returned 403, the auth service without a token returned 401, and normal browser access still went through SSO.

Surprise 3: Docker rewrites the destination first #

Back to the opening story. A pod needed to reach the LLM server, a Docker container that publishes its port on the host. The egress rule allowed the host IP and the published port:

egress:
  - to:
      - ipBlock:
          cidr: <node-ip>/32
    ports:
      - protocol: TCP
        port: <published-port>

Connection refused.

The reason is ordering in the kernel’s packet path. Docker publishes ports with DNAT rules: a packet to <node-ip>:<published-port> is rewritten to <container-bridge-ip>:<container-port> in the NAT table. That rewrite happens before kube-router’s egress filter evaluates the packet. By the time the policy looks, the destination is the container’s bridge address and its internal port, which the policy does not allow.

So the egress rule must allow the destination after DNAT:

egress:
  - to:
      - ipBlock:
          cidr: <node-ip>/32
    ports:
      - protocol: TCP
        port: <published-port>
  - to:
      - ipBlock:
          cidr: <container-bridge-ip>/32   # exactly one container
    ports:
      - protocol: TCP
        port: <container-port>

Find both values before writing the rule:

docker inspect <container> --format '{{json .HostConfig.PortBindings}}'
docker inspect <container> --format '{{.NetworkSettings.Networks.bridge.IPAddress}}'

(On my Docker version the old top-level .NetworkSettings.IPAddress template field does not exist; the per-network path does.)

How “making it work” made it worse #

My first working version allowed the whole Docker bridge subnet on the container port, and the host IP on that port number too. A live probe from the pod then showed what that really permitted:

• the whole bridge subnet on that port reached every Docker container listening on it, not just the LLM server;
• the host IP on that port number reached a game server’s admin panel that happens to listen on all interfaces.

The policy now allows exactly one bridge IP and one port, plus the published port on the host. The catch is that Docker does not pin a container’s bridge IP; it can change after a reboot. So my post-reboot check script now verifies the LLM container’s bridge IP matches what the policies expect. When a second service hit the same “connection refused” a couple of weeks later, the cause was recognised at once because the mechanism was written down.

Two smaller gotchas #

Policies are not instant. New or changed policies took a few seconds to apply. A test run straight after kubectl apply can fail even when the rule is right, and pass a few seconds later. Retest before you conclude a rule is wrong.

A podSelector rule inside one namespace did not work for me. On k3s v1.34.4 with the embedded controller, a policy allowing ingress from podSelector: {app: X} in the same namespace still rejected the traffic (connection refused, not a timeout). Deleting the policy made it work. I did not find the cause; I worked around it. Rules based on namespace selectors and IP blocks behaved as expected. If you see the same thing, test the simple case in isolation before trusting a more complex policy.

Checklist #

• Before writing a policy for an ingress controller, check whether it runs on hostNetwork. If it does, you must allow the node IP, and you are allowing the whole node.
• Never let an app trust identity headers based on network position alone. Add an edge-proof secret or a signed token.
• For egress to Docker-published ports, allow the post-DNAT destination: container bridge IP and container port.
• Allow one IP per rule (a single-address block), never a whole bridge subnet, and probe what the policy actually permits.
• Track unpinned container IPs in a post-reboot check.
• Wait a few seconds and retest after changing a policy.
• Test policies with a real connection from a real pod; YAML that applies cleanly proves nothing.

Frequently asked questions #

Why doesn’t my NetworkPolicy podSelector match Traefik? #

If Traefik runs with hostNetwork: true, its traffic comes from the node’s IP, not from a pod IP, and NetworkPolicy selectors only match pod IPs. Use an ipBlock for the node address (and the CNI gateway address your backends see) instead, and remember that this admits every process on the node.

Why does a pod get “connection refused” to a Docker container on the host even though egress allows the host IP? #

Docker’s port publishing uses DNAT, which rewrites the destination to the container’s bridge IP and internal port before the network policy controller filters egress. Allow that post-DNAT address and port as well, ideally as a single /32.

Can a NetworkPolicy stop header spoofing behind a reverse proxy? #

Only partly. It can reduce who can reach the backend, but if the proxy shares the node IP with other workloads, the policy cannot tell them apart. Have the proxy add a secret header or token that the backend verifies, and accept identity headers only when it is present.

Do Kubernetes NetworkPolicies apply immediately? #

Not always. With kube-router in k3s, changes took a few seconds to take effect in my tests. Wait and retest before deciding a rule is wrong.

The lesson #

A NetworkPolicy describes packets at one point in their journey, and on a real machine other things touch those packets first: host networking hides which workload sent them, and Docker rewrites where they are going. Every surprise here came from reasoning about the packet I imagined instead of the packet the filter actually sees. Find out what the source and destination look like at the moment of filtering, allow exactly that, and never let network position stand in for identity.