120 fake Googlebots, 9 real ones, and the week I banned Google

When I started checking the crawlers that visit my sites, 120 addresses claiming to be Google’s crawlers turned out to be fakes. Only 9 were real. One fleet of 77 impostors ran on Google Cloud, so a quick “is this Google’s network?” check would have waved every one of them through.

Then, a few months later, I made the opposite mistake and banned the real Googlebot for a week.

This post is about telling real crawlers from people wearing their costume, and about the place where my own logs could not show me that I was blocking the one visitor I wanted.

Contents · 10 sections

TL;DR #

• The User-Agent header is a claim, not an identity. Anyone can send Googlebot.
• ASN checks are unsound for crawlers that run on shared clouds. Fake Googlebots live on Google Cloud; real Bing and OpenAI crawlers live on Azure.
• Verify with what the operator publishes: published IP ranges first, then forward-confirmed reverse DNS. If an operator publishes neither, the honest answer is “unverifiable”.
• Impostors are worth banning on identity alone: impersonating a crawler is itself the intent. I batch-banned 171 proven impostors on the first run.
• A drop at the router never reaches the web server’s logs. When Search Console says it cannot connect and your access logs show nothing, look upstream.

Why I needed this #

My web traffic analysis service classifies every request by intent: credential harvesting, CMS probing, shell uploads, and so on. Addresses that show enough bad intent are banned automatically for a week.

In July 2026 I found it was about to ban Bingbot. The old classifier treated loose path fragments as suspicious and relied on status codes. Status codes are useless on a site where an SSO portal or a single-page app answers almost everything with HTTP 200: 3,938 requests for /.env had “succeeded” that way. And the crawler exemption was based on the network (ASN) alone, which exempted attackers who happened to rent machines on the same cloud as a real crawler.

I rewrote the classification to be path-based and status-independent, and then built a second classifier for identity: who is this visitor claiming to be, and can I confirm it?

Four answers, not two #

Each request’s User-Agent is matched against a rule set (it started with 90 rules covering search, AI, social, SEO, monitoring and feed bots) and then checked. The result is one of four values:

• verified: the claim matches evidence the operator controls;
• spoofed: the claim is checkable, and the evidence says no;
• claimed: a checkable claim that could not be checked this time (for example, a lookup failed);
• unverifiable: the operator publishes nothing that would let anyone check.

That last category matters. Pretending you can verify something you cannot is how you end up whitelisting attackers.

Why ASN checks fail #

The first verifier compared the source network with the crawler operator’s network. That is wrong in both directions:

• False “spoofed”: OpenAI’s crawlers run on Microsoft Azure. Before I added published ranges, OpenAI showed 760 spoofed hits in 48 hours. With ranges, 67, and those were genuine fakes.
• False “verified”: the largest impostor fleet, 77 fake Googlebots, ran on Google Cloud’s network, which is not the network Googlebot crawls from. Renting a VM from Google does not make you Google Search.

Another variant: Anthropic’s crawler traffic arrives from an address block registered to Anthropic but announced by Amazon. An ASN check rejected the real crawler. The tempting fix, “also accept Amazon’s network”, would verify every rented EC2 instance as that crawler, which is far worse than the bug. I added only the specific block.

And a quieter one: an offline IP-to-ASN database did not know a young network at all, so verification silently degraded to “claimed”. Matching the address against published CIDR ranges has no such dependency.

What does work #

1. Published IP ranges #

Google, Bing and OpenAI publish JSON files listing the ranges their crawlers use. So do several monitoring services and Telegram’s link-preview bot. Matching against these is fast and exact:

import ipaddress, json

def load_ranges(path):
    data = json.load(open(path))
    return [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
            for p in data["prefixes"]]

def in_ranges(ip, nets):
    addr = ipaddress.ip_address(ip)
    return any(addr in n for n in nets)

After adding Google and Bing ranges: Bing 238 verified / 7 spoofed, Google 9 verified / 120 spoofed. Refresh the files on a schedule; ranges change.

2. Forward-confirmed reverse DNS #

Google also documents the classic method: look up the PTR record of the address, check that the name ends in an official domain, then resolve that name forward and check you get the same address back. Both steps matter. Anyone who controls the reverse zone of their own block can set a PTR record that says crawl-1.googlebot.com. They cannot make Google’s forward zone point back at them.

import socket

def fcrdns(ip, suffixes):
    try:
        host = socket.gethostbyaddr(ip)[0].rstrip(".").lower()
    except OSError:
        return "claimed"                 # lookup failed; do not guess
    if not host.endswith(tuple(suffixes)):
        return "spoofed"
    try:
        forward = socket.gethostbyname_ex(host)[2]
    except OSError:
        return "spoofed"
    return "verified" if ip in forward else "spoofed"

Some operators need care. One SEO crawler publishes PTR names that have no forward record at all; its documented method is a PTR-suffix match, so its rule carries a “PTR only” flag. Amazon’s crawler forward-confirms properly. Another large crawler only has generic cloud PTR names; accepting the cloud’s domain would verify any rented VM, so it stays unverifiable.

3. Admit what cannot be verified #

When I went through every bot without a verifier, many were genuinely uncatchable: HTTP libraries, federated software, peer-to-peer crawlers, bots running on shared cloud addresses with no published list. Those stay “unverifiable”, and they are judged on behaviour alone.

From identity to campaigns #

With identity on every request, a new pattern appeared. A group of five or more addresses on the same network presenting the same spoofed identity is a fleet. A single /24 rotating through several spoofed identities is a costume rack. The first run found the 77-address fake Googlebot fleet, plus fake versions of the Internet Archive’s crawler, AI crawlers, Baidu, Yandex, Twitter and an SEO bot.

I decided that impersonation is the intent. Members of proven impostor fleets are banned without waiting for malicious paths: 171 on the first run. Confirmed impostors also ban on any malicious request at a threshold of 1. The first one caught that way was a fake Googlebot on Google Cloud doing a single round of virtual-host enumeration, well below the normal threshold.

Across one analysis of my traffic, impostors claimed 14.6% of all crawler identities. Forged desktop browsers on datacenter addresses and empty User-Agents were even more common, and 86% of the empty-UA addresses were hostile. The doctrine I took from it: score the request, not the sender. Reputation can inform a decision; it must never be the gate.

The week I banned the real Googlebot #

Now the mistake. In late October 2026, Google Search Console started reporting “Server connection error” and my sitemaps “Couldn’t fetch”. I checked the web proxy logs. Nothing: no errors, no blocked requests from Google, no requests from Google at all.

That absence was the clue. A broad automatic ban rule on the router, meant for scanners, had caught two genuine Googlebot addresses and put them on a drop list for a week. The drop happened at the router, before the traffic reached the web proxy. So every log I usually read said, truthfully, that nothing on the web side was blocking Google.

The fix was an explicit exemption for the published crawler ranges of Google and Bing, evaluated before the automatic bans. Not by User-Agent, which an attacker sets, but by address range, which they cannot.

It was not my first collateral damage either. A CrowdSec scenario for crawlers that request too many non-static pages banned ordinary readers of my library site, because its search results load dozens of cover images from URLs with no file extension. And an old CrowdSec parser that whitelisted “search engines” by IP prefix only, with no reverse DNS check, let scanners from a search company’s address space through. I disabled it in favour of the rDNS-based whitelist.

Checklist #

• Treat User-Agent as a claim. Verify, then classify.
• Verify with published ranges first, forward-confirmed rDNS second. Never ASN alone.
• Keep four outcomes: verified, spoofed, claimed, unverifiable. Do not force a yes/no.
• Never widen a verifier to a whole cloud provider’s network.
• Refresh published range files on a schedule.
• Ban proven impostor fleets on identity; impersonation is the intent.
• Exempt real crawlers by address range at every layer that bans, including the router.
• When an external service reports connection errors and your logs are empty, check the layers before the logs: router, firewall, ban lists.
• Test new detection scenarios against your own legitimate traffic patterns, such as pages that load many images.

Frequently asked questions #

How do I verify that a visitor is really Googlebot? #

Use Google’s published crawler IP ranges, or do a reverse DNS lookup, check that the name ends in googlebot.com, google.com or googleusercontent.com as Google documents for each crawler type, then resolve that name forward and confirm it returns the same IP. Do not rely on the User-Agent or on the address belonging to a Google network.

Why can’t I just check that the IP belongs to Google’s ASN? #

Because Google Cloud customers use Google’s networks too. In my data the largest fake-Googlebot fleet, 77 addresses, ran on Google Cloud. The same problem hits crawlers on Azure and AWS.

Should I block fake crawlers? #

I do. Someone who forges a search engine’s identity is trying to get through filters that trust that identity. In my traffic many impostors also probed for vulnerabilities.

Why does Search Console report errors when my server logs show nothing? #

Something in front of the server is dropping the requests: a router rule, a firewall, a CDN or a ban list. Search for Google’s addresses in those layers, not in the web server logs.

The lesson #

Identity on the internet is something you check, not something you are told. The same rule cut both ways for me: it unmasked 120 fake Googlebots, and once I had stopped trusting names, it reminded me to make sure my real visitors could prove who they were at every layer that might turn them away. The logs you read are only as complete as the point where they are written.