The Counterfeit Crawlers: Forging Googlebot, ClaudeBot, and the New Identity Theft

The Recidivists — Part 9

Eight letters have sorted attackers by their relationship to persistence. The web recidivists of Parts 2 through 7 persist — same address, again and again, until the ladder makes it unaffordable. The SSH flood of Part 8 does not — it rotates through a thousand disposable faces and never climbs at all. This final letter is about a third mode that is neither: an attacker who does not hide and does not persist, but impersonates — who arrives wearing the name and the User-Agent of a crawler the perimeter is supposed to trust, betting that a stolen identity will open a door that its own would not.

The perimeter catches roughly a hundred and sixty of them, sorted into fleets by the name each forges: a Fake-Googlebot fleet of 74, a Fake-ClaudeBot fleet of 23, Fake-ia_archiver (the Internet Archive) at 22, Fake-Baiduspider at 10, Fake-ChatGPT-User at 7, and smaller fleets wearing the faces of Yandex, Semrush, and Twitter. They are, individually, mostly one-time offenders — banned once and gone, like the SSH majority — but they are banned for a different reason than anyone else in this series. Every other attacker here is banned for what it did. The counterfeit crawlers are banned for who they claimed to be, because the claim was a lie, and the lie was the whole point.

A verification, shown before it is used

This letter makes an unusual accusation for the series — that specific traffic from Google's own cloud, from Amazon, from Cloudflare, is fraudulent — and that accusation carries an obvious failure mode I was obliged to rule out before writing a word of it. So the check comes first, in the open.

The danger is a false positive. If this perimeter sat behind a reverse proxy like Cloudflare, then every request — including legitimate crawler traffic — would arrive bearing the proxy's IP address, and a detector that checked "is this Cloudflare address a real ChatGPT crawler?" would answer no and brand thousands of real bots as fakes. A "Fake-ChatGPT" fleet on Cloudflare IPs is exactly what that mistake would look like. Before trusting the label, I had to know whether the fronting existed.

It does not. The domain resolves directly to the server's own address; there is no Cloudflare, no AWS, no proxy in front of it. Traffic is ingested from the edge router's own access logs, so the client_ip recorded is the true TCP originator — the machine that actually opened the connection. That single fact collapses the false-positive scenario entirely: a Cloudflare address in these logs is not a proxy standing in front of a real crawler; it is a machine on Cloudflare's network that dialed this server directly. And real crawlers do not do that. OpenAI's ChatGPT-User egresses from OpenAI's own published ranges; Anthropic's ClaudeBot from Anthropic's; neither routes its fetches through Cloudflare Workers or rents AWS instances to crawl from. So an address on Cloudflare or AWS, connecting directly, announcing itself as ChatGPT or ClaudeBot, is not a mislabeled real bot. It is precisely what the label says.

That is how the accusation earns the right to be made: the one reading of the data that would have made the detector wrong was tested against the DNS and the ingest path, and it failed. What remains is real.

Why "it is on Google's cloud" is not "it is Googlebot"

The detector's method is the reason it survives that scrutiny, and it is worth stating precisely because it is the crux of the whole subject. It does not trust the User-Agent string — anyone can type Googlebot into a header. It does not trust reverse DNS alone — a PTR record can be forged or coincidental. It verifies against the authoritative IP ranges each operator publishes: Google's googlebot.json, OpenAI's chatgpt-user.json, and the equivalent range files or BGP announcements for the others. An address claiming to be Googlebot is checked against the list of addresses Google says Googlebot actually uses. In the list, it is verified and never touched. Absent from the list, with the name in its User-Agent, it is spoofed — a proven verdict, because a verifier ran and the claim failed.

This is what defeats the cleverest version of the trick, and the data shows the trick in its cleverest form. Of the 74 addresses in the Fake-Googlebot fleet, 69 are hosted on Google Cloud Platform. The impersonators rent virtual machines from Google to forge Google's crawler — because a lazy filter that checks only "does the ASN belong to Google?" sees GOOGLE-CLOUD-PLATFORM, Google LLC and waves the traffic through as obviously legitimate. It is a genuinely good bet against a naive defender. It fails here for one reason only: Google's crawler does not run from Google's cloud. Real Googlebot originates from Google's dedicated crawler ranges, which are published, and which do not include the GCP customer space that anyone with a credit card can rent. Being on Google's network is not being in the range Google reserves for its crawler — and the published-range check is the difference between those two facts. The 69 GCP impostors are Google-hosted machines that are not Googlebot, and the verifier knows the list.

The remaining five Fake-Googlebot addresses are on Cloudflare, and the same logic disposes of them: Cloudflare is not in Google's published crawler ranges either. Impersonators stage from whichever cloud is cheap and superficially plausible; the verification does not care which, only whether the address is on the operator's own list.

The roster, by forged name and staging ground

Laid out in full, the counterfeit fleets sort by the identity each steals and the cloud each rents to steal it from:

Two families sit inside that list. The search and archive impersonations — Googlebot, Baiduspider, YandexBot, ia_archiver — are the classical forgeries, stealing the names that have meant "let me in so you can be found" for twenty years. The AI-crawler impersonations — ClaudeBot, ChatGPT-User, and the GPTBot machinery — are the new arrivals, and the smaller SemrushBot and Twitterbot fleets round out a third motive: SEO and social-preview crawlers, whose names are trusted by sites hungry for ranking and link-preview coverage. Every forged name on the list is a name some category of site decided to trust; the fleet sizes are, roughly, a map of how valuable each stolen key currently is.

The multi-brand rotators: caught by contradiction

One small fleet on that roster is detected by a different and rather elegant rule, and it deserves its own note because it needs no published range at all. The identity-rotation cluster is made of addresses that, over their short lives, claimed to be several different crawlers — the classifier's canonical example is a single address rotating through Baidu, Apple, Google, Yandex, and Amazon identities in turn. No published-range lookup is required to know that address is lying, because the lie is internal: those are five competing companies on five separate networks, and no real crawler is all of them. An address presenting three or more distinct crawler brands has convicted itself by contradiction — a bot cannot be simultaneously Googlebot and Yandex, so an address that claims both is neither.

This rotation pattern is the impersonator caught over-reaching. A fleet that forges one name consistently must at least be checked against that name's published range; a source that cycles through five borrowed names is trying every key on the ring and, in doing so, proves it holds none of them legitimately. The detection is cheap, certain, and forgery-proof: you cannot fake your way past a rule that keys on the inconsistency of your own claims. It is the one impersonation verdict in this letter that requires no external list to reach — the address impeaches its own testimony.

What the borrowed name is for

Impersonation is a means, not an end, and a handful of the fleet members reveal the end by failing to stop at the identity. The perimeter's records include impostors banned not only for the forged name but for what they did while wearing it — vhost_enum, the enumeration of guessed subdomains, and credential_harvest, the hunt for exposed secrets that recurs throughout this series. Those are the same intents the honest-identity attackers of Parts 2 through 7 pursue; the only difference is the disguise. The counterfeit crawler is not a distinct goal so much as a distinct entry technique bolted onto the same goals — a way to get the credential-harvesting request past a filter that would have dropped it from an unknown IP but waves it through from "Googlebot."

That is the practical danger the forged name is bought to exploit, and it is worth being concrete about the mechanism. Many sites maintain bot-specific rules: a request from a verified crawler is exempted from rate limits, served content that anonymous visitors are challenged for, or allowed past a WAF rule that would block a stranger. Those exemptions are an access-control decision keyed entirely on identity — and identity, unverified, is just a string in a header. The impostor's whole play is to inherit an allow-listed crawler's exemptions and then do, from inside them, exactly what it came to do. The forged Googlebot that enumerates subdomains is using Google's borrowed authority to scan more freely than its own name would permit. The stolen identity is not the crime's disguise; it is the crime's skeleton key.

The new identity worth stealing

The most telling fleet is not the Googlebot one — crawler impersonation is old, and Googlebot has been the forged name of choice for a decade. The telling fleet is Fake-ClaudeBot: 23 addresses, all inside a single Amazon 216.73.217.0/24, all announcing themselves as Anthropic's crawler — joined by seven Fake-ChatGPT-User addresses on Cloudflare and the machinery for GPTBot behind them. These are new names. ClaudeBot and ChatGPT-User did not exist as identities to steal until very recently, and their sudden appearance as forged identities is a small, precise economic signal about how the web changed.

The logic is straightforward once stated. A crawler identity is only worth forging if wearing it grants access that your own identity would not — and for most of the web's history the only identities worth that were the search engines, because being crawlable by Google meant being found. But over the last stretch, a great many sites began explicitly allow-listing the AI crawlers — ClaudeBot, GPTBot, ChatGPT-User — either to court inclusion in AI answers or simply because the operators added them to the same trusted-bot rules as Googlebot. The instant an identity lands on enough allow-lists, it becomes a key; and the instant it becomes a key, it gets counterfeited. The Fake-ClaudeBot fleet on Amazon is the direct consequence: attackers rent AWS, paint on the newest trusted name, and try the doors that name now opens. The impersonation economy tracks the trust economy with almost no lag — the moment the web decided to trust AI crawlers, someone started forging them.

That is the quiet importance of this fleet. It is not large — twenty-three addresses — but it is a leading indicator. Crawler-identity theft migrates to whichever names have most recently become valuable, and the appearance of forged ClaudeBot and ChatGPT identities in a perimeter's logs is the fingerprint of AI crawlers having crossed the threshold into default trust. The counterfeiters found the new keys about as fast as the locks were re-cut to accept them.

The economics of a borrowed name

There is a reason the counterfeit crawlers stage from Google Cloud, Amazon, and Cloudflare rather than from the bulletproof and grey hosts that housed the recidivists of Parts 4 through 7, and it completes the picture. Impersonation and persistence want opposite things from their infrastructure. The sit-and-hammer credential-harvester needs an address it can keep — a stable perch on abuse-tolerant hosting that will not evict it mid-campaign. The impostor needs the opposite: an address that is cheap, disposable, and superficially reputable, held only long enough to spend a stolen identity before the ban lands and a fresh instance takes over. Mainstream cloud is perfect for that. A GCP or AWS instance costs cents an hour, spins up in seconds, and — crucially — carries the reputation of the cloud, which for a naive filter reading only the ASN is a reputation of "Google" or "Amazon," not of the anonymous tenant actually renting it.

So the same cloud reputation-dampening that Part 5 identified as a shield for the persistent cloud recidivist is, for the impostor, a second stolen credential layered under the first. The forged Googlebot on Google Cloud wears two borrowed authorities at once: the identity of Google's crawler in its User-Agent, and the reputation of Google's network in its ASN. A filter that trusts either one lets it through; only a filter that checks the specific published range — that refuses to conflate "hosted by Google" with "is Googlebot," or "on a major cloud" with "is trustworthy" — defeats both borrowings together. The impostor's economics are the mirror image of the recidivist's: where the recidivist paid in the effort of not rotating to keep a foothold, the impostor pays almost nothing precisely because it intends to be caught and cycled, spending each disposable instance the way the SSH botnet spends a bot. It is Part 8's rotate-fast disposability again, wearing Part 5's cloud reputation, aimed through a stolen identity — the three threads of the whole series braided into one small, cheap, and revealingly modern attack.

Identity as the offense, and why the ladder need not wait

The counterfeit crawlers connect back to Part 8's central point from an unexpected angle. Like the SSH flood, they mostly do not recidivate — they are cloud-hosted, rented cheap, banned once, and rotated. Judged by address persistence, they would vanish exactly as the SSH majority does. But the perimeter does not judge them by persistence at all, and does not have to, because of how they are banned: a confirmed impostor is banned on the impersonation itself, at a threshold of one, with no need for any further malicious act. For everyone else in this series, identity is a clue to intent; for the counterfeit crawler, the identity fraud is the intent, complete in itself. A machine on AWS lying about being Anthropic's crawler has already done the thing worth banning before it requests a single page.

This is why the ladder does not need the impostors to come back. The whole apparatus of escalating tiers exists to raise the cost of a return — but a return is only necessary to establish guilt when the first visit was ambiguous. The impostor's first visit is not ambiguous; the forged identity is a completed offense, verified against a published list, and the immediate ban is the whole and correct response. It is the same immediacy the honeypot applies to SSH, arrived at from the opposite direction: the honeypot bans instantly because the SSH source will never return to be judged later, while the perimeter bans the impostor instantly because it has already been judged — its lie was checkable, and it was checked, and it failed.

The one honest residual

No verifier is perfect, and the discipline of this series requires naming where this one could still err, even after the false-positive scenario was ruled out. The spoofed verdict depends on the operators' published range files being current at the moment an address is checked. Crawler operators rotate their ranges; a legitimate bot that moved to a new address before the published feed caught up could, in a narrow window, be checked against a stale list and briefly mislabeled. The system is built to fail safe against exactly this — DNS ambiguity resolves to "claimed," never "spoofed"; two specific shared operator ASNs (Azure, which really does host Bing, and Apple's) are exempted from identity-only bans and require corroborating malicious behaviour; and the range feeds are re-pulled on a schedule to chase operator rotation. Those guards make a stale-range false positive rare and short-lived, not impossible. The honest confidence is therefore high but not absolute: the design is sound and was verified sound, the fronting false positive is ruled out by the direct-exposure fact, and the residual risk is a thin, self-correcting window around range rotation — worth stating, not worth doubting the fleets over.

Three faces of one adversary

The Recidivists set out to name the attackers who would not stop, and it ends having sorted the entire inbound tide into three modes, distinguished not by their tools but by their relationship to identity and time. The persistent attacker of Parts 2 through 7 reuses one address until the ladder buries it — visible because it holds still. The ephemeral SSH flood of Part 8 reuses nothing, wearing a thousand disposable faces — invisible to the per-address lens, and answered with instant judgement on first contact. The counterfeit crawler of this letter wears a borrowed face, a name it has no right to — and is answered by the one check the other two never needed, a verification of the identity itself against the list of who is really allowed to carry it.

Persistence, absence, and impersonation: reuse the address, rotate the address, or steal a trusted one. Three strategies against a single perimeter, and three matched answers from a system that, for all its blunt simplicity, turns out to have a correct response to each — patience for the one that stays, immediacy for the one that flees, and verification for the one that lies. The ladder counted the first. The honeypot kept the second's payload. And the published-range check unmasked the third. What began as a ledger of repeat offenders closes as something broader: a working taxonomy of how hostility presents itself to a door that is simply paying attention — and a demonstration that paying attention, kept up long enough and honestly enough, is most of the defense.