The robots.txt Tell: The Handshake a Crawler Declares and an Impostor Mimics

TI-2026-086Q · Series: The Wardrobe · Case 17 · Classification: HIGH · Confidence: HIGH

There is one file every legitimate crawler fetches before it does anything else: /robots.txt, the crawler protocol's handshake, where a site declares what may and may not be crawled. A real Googlebot reads it first and obeys it. So a natural question runs through the whole series: do the masks read robots.txt? The answer is more interesting than yes or no. It sorts the wardrobe into three tiers — the crude impostor that skips the handshake, the careful impostor that performs it, and the genuinely verified crawler that performs it and is still somewhere it should not be. And it teaches the series' subtlest lesson about trust: a declared virtue is not a verified one.

Executive Summary

Across the estate, /robots.txt drew 3,354 hits from 788 IPs on 102 hosts, and its fetchers are, overwhelmingly, the verifiable crawler population fetching from their real networks:

The real Googlebot fetches robots.txt from AS15169 — Google's own crawler network. And the impostor home, GCP tenant space AS396982, contributes just 17 hits. Meanwhile the crude .env-sweep fleets of 086B–D — the Fake-Googlebot, Fake-Yandex, Fake-ia_archiver, Fake-Baidu nodes — fetch robots.txt never; they dive straight for /.env and Joomla exploit paths.

So the honest handshake correlates strongly with being real. But correlation is not proof, and one node proves it: 216.73.216.198, the fake ClaudeBot on AWS (SPOOFED in 086E), does fetch robots.txt — and is convicted anyway, because provenance does not care about the handshake. That splits the finding into three tiers, and the middle one is the important one: robots.txt sorts the crude from the careful; it does not convict.

Tier One: The Crude Impostor Skips the Handshake

The GCP credential-harvest fleets are, for all their fleet coordination, behaviourally lazy in one revealing way: they do not pretend to be crawlers first. 34.125.48.170 (Fake-Yandex) opened with 151 .env paths. 34.95.201.0 (Fake-ia_archiver) with 79. 136.107.160.133 (Fake-Baidu) with 97, straight at the Gitea host. 34.87.136.84 (Fake-Googlebot) went directly to a Joomla SP Page Builder asset. Not one of them fetched /robots.txt.

That absence is a cheap, high-value first-pass sort. A client that announces itself as Googlebot and then, on its very first request, asks for /.env instead of /robots.txt has violated the crawler contract in the one place every real crawler honours it. Real Googlebot does not skip the handshake; it is the handshake, first, every time. So the pairing "claims to be a crawler" + "never read robots.txt" + "went straight for secrets" is a near-certain crude impostor, catchable before any provenance lookup or fleet clustering. It is the same shape as the behavioural-envelope verifier of 086F (a preview bot doing 327 requests): the identity's own defining first act is one the impostor did not bother to perform.

The crude tier is the majority of the wardrobe's volume, and robots.txt filters it almost for free. Which is exactly why the careful tier matters.

Tier Two: The Careful Impostor Performs the Handshake — and Is Still Caught

216.73.216.198 is the counterexample that keeps the case honest. It is a fake ClaudeBot on AWS — 086E convicted it SPOOFED, because Anthropic's real ClaudeBot egresses AS399358 and this node is on Amazon's AS16509. And it appears in the /robots.txt fetcher list. It performed the handshake. It read the rules first, like a good crawler. It dressed the costume all the way down to the accessory that most says I am legitimate.

It did not matter. Run it through the provenance check and the verdict is unchanged: SPOOFED. Fetching robots.txt is a request the impostor makes; being on Anthropic's network is a fact it cannot forge. So the handshake bought it nothing against the check that matters — it only would have fooled a defender who read "fetches robots.txt" as a legitimacy signal.

This is the load-bearing lesson of the case, and it generalises past robots.txt to every self-declared virtue. A visitor telling you it honours the rules is a visitor claiming a virtue, and a claim is exactly the class of thing this entire series has shown to be forgeable. robots.txt-compliance is not in the same category as provenance or behaviour; it is in the same category as the User-Agent — a thing the client asserts about itself, useful because most honest clients assert it truthfully and most crude liars do not bother, but worthless the moment a careful liar decides to assert it too. A declared virtue narrows the suspect pool. It never empties it. So robots.txt sorts — crude out, careful in — and hands the survivors to provenance (086E) and behaviour (086I), which convict.

Tier Three: Compliant, and Still Where It Should Not Be

The third tier is not about impostors at all, and it is the subtlest thing robots.txt reveals. In 085C, a genuinely verified OAI-SearchBot — provenance confirmed against OpenAI's published ranges — fetched /robots.txt seven times on lsn-git.shuffle-on.com, a private, authentication-gated Gitea that was never published, never linked, never meant to be seen by any crawler. It was real. It was compliant. It read the rules first, exactly as it should. And it was standing at the door of a host it had no business knowing existed.

robots.txt answers "do you follow the crawler protocol?" It does not answer "were you invited?" or "how did you learn this private host exists?" A crawler can be perfectly compliant and still be somewhere it should not be, because compliance and invitation are orthogonal — one is about how you crawl, the other about whether you should be here at all. The verified overstepper honours robots.txt on a host it should never have discovered, and the honouring makes the presence more unsettling, not less: it is the behaviour of something that believes it has the right to be there. Verified is not the same as welcome, and robots.txt-compliance, far from resolving that, is precisely the signal that cannot see the difference.

Weaponising the Handshake: The Disallow Canary

There is a defensive twist that turns the impostor's mimicry against it. robots.txt contains a Disallow list — the paths a site asks crawlers not to visit. A legitimate crawler reads Disallow in order to avoid those paths. A bad actor reads Disallow in order to find them — because "the paths the site does not want crawled" is a curated list of things worth probing.

So a defender can seed a honeypot path in Disallow and watch who fetches it. No honest crawler ever will, by definition — the whole point of Disallow is that they skip it. Any client that reads robots.txt and then requests the Disallowed honeypot path has performed a disqualifying act that reveals it was mining robots.txt for targets, not obeying it. And this catches exactly the Tier Two careful impostor — the one that performs the handshake to look legitimate — because the act it added to seem real (reading robots.txt) is the act the canary tests. The mimicry of virtue creates the surface on which the virtue is checked: perform the handshake, and you can be handed a poisoned rule.

(One honest note: this estate does not exempt robots.txt from its L7 controls — of the 3,354 fetches, 1,513 returned 403 on block_bot/auth-guarded routes. The handshake is not sacred here; on protected hosts it is intercepted like any other path.)

Steelmanning: "robots.txt is noise — real crawlers and fakes both fetch it, so it tells you nothing"

The sharp objection reads the data the opposite way: since a careful impostor (the fake ClaudeBot) fetches robots.txt just like the real crawlers do, the signal is contaminated and useless — you cannot sort on a field both classes populate.

The objection over-reads Tier Two into the whole picture. Three points hold the signal's value at its true, limited size. First, the base rates are lopsided: robots.txt is fetched overwhelmingly by verifiable crawlers from real networks (thousands of hits from Google/Apple/Meta/Semrush/OpenAI ASNs) and barely at all from the impostor home (17 from AS396982), so absence of the handshake in a self-declared crawler is a strong crude-impostor signal even though presence is weak. Second, the case never claims robots.txt convicts — it claims it sorts, explicitly handing the survivors to provenance and behaviour; a sorting signal is not refuted by the existence of items that pass the sort. Third, the Disallow canary converts the contaminated presence-signal into a clean behavioural one: it does not ask did you fetch robots.txt (both classes do) but did you fetch a Disallowed honeypot (only bad actors do). The objection correctly identifies that presence-of-fetch is a weak signal; the case already treats it as weak, uses absence-of-fetch as the strong one, and weaponises the canary for the rest.

What Holds, and What Is Bounded

Documented, HIGH confidence: the /robots.txt fetcher population (verifiable crawlers from real ASNs, real Google on AS15169, minimal AS396982); the crude fleets' total absence of robots.txt fetches; the fake ClaudeBot's robots.txt fetch coexisting with its SPOOFED provenance verdict; the 085C verified-overstepper; the 403-heavy status profile.

Inferred, MEDIUM confidence: the Disallow-canary's effectiveness against careful impostors is a reasoned defensive proposal, not a deployed-and-measured control on this estate; and the strength of "absence of robots.txt" as a standalone signal is asserted from the crude fleets' path profiles rather than a full population-level fetch-vs-no-fetch contingency table (a natural follow-up).

Why It Matters

The robots.txt Tell refines the series' central discipline with a case that looks, at first, like it should hand the defender an easy win — surely the crawler protocol's own handshake tells you who is real — and instead teaches why easy wins in identity are illusions. robots.txt is a declared signal: the crawler saying I honour the rules. Crude impostors do not bother to say it, so its absence sorts them out cheaply; but a careful impostor says it too, so its presence proves nothing; and a real crawler can say it truthfully and still be at a door it was never given. Declared virtue, mimicked virtue, and compliant trespass — three different things the same fetch can mean.

The resolution is the one the whole series converges on, stated one more way: do not trust what a visitor declares about itself — not its name, not its compliance — trust only what it cannot control (its provenance) and what it actually did (its behaviour). robots.txt is a self-declaration, so it lives with the User-Agent in the category of useful-but-forgeable, not with provenance in the category of convicting. Use it to thin the field and to bait a canary; never to clear a suspect. The one file every crawler reads first turns out to be one more thing a crawler says, and this series has been, from the beginning, about the difference between what a visitor says and what is true.

Cross-references

• TI-2026-085C — Verified, and Still Where It Shouldn't Be — the verified overstepper fetching robots.txt on a private Gitea; verified is not welcome.
• TI-2026-086E — The Newest Trust — the fake ClaudeBot whose robots.txt fetch is defeated by the operator-ASN provenance check.
• TI-2026-086F — The Social Unfurlers — the behavioural-envelope verifier (defining first act) that robots.txt-absence is a species of.
• TI-2026-086I — The Costume Fits the Crime — robots.txt sorts; provenance and behaviour convict.

Next in the series — TI-2026-086R, The Wordlist as a Blueprint: what the attackers' .env dictionary and DevOps credential list reveal about their model of a modern estate.

TLP:WHITE. Attacker fleet behaviour and impersonated-crawler telemetry only. No internal addresses or secrets disclosed.