Scanners in the Mirror: Reading Intent Through the Auth Proxy

TI-2026-059D · Series: The Front Door · Classification: MEDIUM · Confidence: HIGH

Fourth in The Front Door series, and the one about our own instruments. The previous three read clean signals. This one is about a dirty signal — the noisiest intent class we have — and the honest work of separating a threat from a search engine when both knock on the same locked door.

Executive Summary

scanner_fingerprint is the intent class we trust least, and this dossier explains why — and how to read it anyway.

The problem is architectural. Most of our services sit behind Authelia, a forward-authentication proxy. When anything hits a protected host — a browser, a crawler, or a scanner probing /.env — Authelia does not return the 404 an unprotected server would. It returns 200: the login page. Every probe, benign or hostile, produces the same success code. The oldest heuristic in a web analyst's toolkit — 200 means served, 404 means probed — is inverted into uselessness.

The intent survives in one place: the ?rd= redirect parameter, where Authelia records where the request was actually trying to go. Decode ?rd=, add User-Agent honesty, request cadence, origin ASN and reputation, and the muddy scanner_fingerprint bucket resolves into three sharp populations: verifiable crawlers, plain scanners, and impostors.

The Instrument Problem

Here is what the auth proxy does to a scanner's probe:

Unprotected host:   GET /.env                 -> 404   (intent obvious)
Authelia host:      GET /.env
                    -> 302 -> /?rd=%2F.env     -> 200   (intent hidden in ?rd=)

A real example, from IP 104.244.77.108:

GET /?rd=https%3A%2F%2Fweb-offline.shuffle-on.com%2Fstatic%2Fcss%2Fmanifest.json&rm=GET  -> 200

To a status-code-only view, that is a healthy 200. Decoded, it is a scanner reading the target host's asset tree through the login redirect. The ?rd= decode is the instrument that makes protected-surface traffic legible — the same instrument that surfaced the AI-secret paths in 059A. Without it, scanner_fingerprint would be an undifferentiated wall of 200s.

Three Populations, One Bucket

Population 1 — Verifiable Crawlers (false positives)

One consistent identity. One polite /robots.txt fetch. Threat near zero. 213.180.203.0/24 is Yandex's own space; 114.119.x is Huawei's PetalBot. These are search engines doing their job, swept into the class only because they touched a protected host. Blocking them is a mistake — a documented negative that the class must be taught to exclude.

Population 2 — Plain Scanners

Datacenter IPs wearing consumer-browser User-Agents, firing in synchronized bursts. A residential browser does not live in a China Telecom IDC rack; a crawler does not disguise itself as Chrome-on-Windows. Spoofed-UA + datacenter origin + burst cadence = scanner. These are correctly classed.

Population 3 — The Impostor

104.244.77.108 (AS53667 PONYNET / FranTech Solutions — BuyVM, geo Switzerland, abuse 100) is the interesting one. In a single session it presented five different crawler identities:

Mozilla/5.0 (compatible; Baiduspider/2.0; ...)
Mozilla/5.0 (compatible; Applebot/0.1; ...)
Mozilla/5.0 (compatible; GoogleOther; ...)
Mozilla/5.0 (compatible; YandexBot/3.0; ...)
Mozilla/5.0 (compatible; Amazonbot/0.1; ...)

No legitimate crawler is Baidu and Apple and Google and Yandex and Amazon. A User-Agent is unauthenticated text; this IP is trying on the entire wardrobe of trusted bots to find one your WAF allowlists. It then followed the Authelia ?rd= redirect into /static/css/ and /static/js/, harvesting the login SPA's real asset tree. CrowdSec autobanned it (scanner_fingerprint:6); MikroTik banned the /32. The five-identity rotation is the definitive impostor signature.

How to Tell Them Apart

The load-bearing test is the one an attacker cannot forge: reverse DNS followed by forward DNS. An impostor can copy Googlebot's User-Agent in seconds; it cannot make 104.244.77.108 resolve to googlebot.com, because it does not own Google's addresses. Imperva measures roughly 4% of all "Googlebot" traffic as fake — the impostor is a population, not a curiosity — and AI-crawler spoofing (GPTBot, ChatGPT-User) now runs as high as 16.7%.

Read Between the Lines

Q: Why not just drop the whole scanner_fingerprint class if it's this noisy? Because two of its three populations are real threats, and the third is exactly the traffic you must not block. Dropping the class blinds you to the impostors; blanket-banning it blocks Yandex and Google and quietly wrecks your SEO. The noise is not a bug to be discarded — it is a triage bucket that demands a second, cheap verification step (reverse DNS) before any verdict. The class is honest about its own uncertainty; the analyst has to finish the job.

Q: If the impostor got a 200 login page and no further, what did it actually gain? More than nothing. It confirmed the host exists, learned it runs an Authelia-protected SPA, and — by following ?rd= — enumerated the front-end asset tree (manifest.json, hashed CSS/JS bundles). That is reconnaissance: a map of the application's shape, its framework fingerprints, and its resource naming, all harvested from behind a login it never passed. Auth protects the data; it does not hide the architecture.

Q: Doesn't the auth proxy returning 200 to everything actually help attackers? It is a genuine trade-off. Returning 200 to a probe denies the scanner the 404-vs-200 oracle it uses to map unprotected servers — that is a defensive win. The cost is that our own logs lose that oracle too, which is why the ?rd= decode exists. We deliberately blinded the attacker and then rebuilt our own sight from the redirect parameter. The proxy is a good bet; the decode is the price of it.

The Alternative Interpretation

Steelman the sceptic: maybe 104.244.77.108 really is a distributed crawler infrastructure that legitimately fronts multiple search products.

It fails on ownership. Real multi-product crawlers (Google runs several) still originate from the operator's own ASN and pass reverse-DNS to the operator's domain. Baidu, Apple, Yandex and Amazon do not share a crawler and would never route it through FranTech/BuyVM, a budget VPS reseller in the United States. The identities span four competing companies; the origin belongs to none of them; the address is on a CrowdSec blacklist and a MikroTik ban list. The benign story cannot survive the reverse-DNS test. The evidence wins.

Verdict

scanner_fingerprint is not a verdict — it is a triage bucket produced by the auth proxy, and its honesty about its own noise is a feature. Decoded through ?rd=, User-Agent scrutiny, cadence, ASN and reverse DNS, it resolves into verifiable crawlers (exclude), plain scanners (ban), and crawler-impersonating impostors (ban and fingerprint). The instrument that makes the protected surface legible is the same ?rd= decode that powers the rest of this series. Confidence: HIGH on the disambiguation method and the impostor call; the class label itself is deliberately MEDIUM — that is the point.

Defensive Response

• Never allowlist by User-Agent. Verify crawler claims with reverse-DNS + forward-DNS to the search engine's domain and an ASN check. This single test defeats the impostor.
• Decode the redirect, not the status. On forward-auth surfaces, reconstruct intent from ?rd=; treat 200 as uninformative.
• Ban populations 2 and 3, allowlist population 1. CrowdSec's scanner_fingerprint autoban is safe for scanners and impostors; add a verified-crawler allowlist so Yandex/Google/Huawei are never caught.
• Do not leak architecture in assets. Impostors that follow ?rd= enumerate your SPA's asset tree; keep secrets and meaningful structure out of front-end bundle paths.

Sources

• LSN web-threats platform — scanner_fingerprint IP dossiers for 104.244.77.108, 213.180.203.178, 114.119.151.179, 106.75.144.250, 106.63.26.148 (Authelia ?rd= decode + CrowdSec + MikroTik).
• Imperva — Was That Really a Googlebot Crawling My Site? (≈4% of Googlebot traffic fake; reverse+forward DNS verification) — https://www.imperva.com/blog/was-that-really-a-google-bot-crawling-my-site/
• Search Engine Land — What is Googlebot Fraud (fake-crawler abuse; AI-crawler spoofing rates) — https://searchengineland.com/guide/what-is-googlebot-fraud
• Google Search Central — verifying Googlebot via reverse DNS (canonical method referenced throughout).

Investigation TI-2026-059D · The Front Door series · The disambiguation of a noisy class through the auth proxy · Verifiable crawlers vs plain scanners vs five-identity impostors · Cross-referenced with Imperva fake-crawler research · Confidence: HIGH on method.