Search Console said 'connection error'. My router had banned Googlebot

For days, Google Search Console reported a “Server connection error” for my site and said it “Couldn’t fetch” my sitemaps. My web server’s access log showed no failed request from Google at all. Both were telling the truth. The fetches that failed never reached the web server, because my own router had put 2 genuine Googlebot addresses on its attack list.

This is the story of how a defensive auto-ban rule caught a search engine, why web-log analysis could not see it, and the two-step check I now run whenever a search engine reports fetch errors.

Contents · 10 sections

TL;DR #

• If a search engine reports connection errors and the requests are missing from your web server logs, look at the firewall in front of it before anything else.
• Automatic ban lists catch anything that matches the pattern, including legitimate crawlers.
• Verify a crawler by reverse and forward DNS, or by the operator’s published IP ranges. Never by user agent, and never by the “Google LLC” organisation name.
• Put verified crawler ranges in an allowlist that is evaluated before the auto-ban rule, and keep every other filter rule for them.
• Search Console verification can fail for reasons unrelated to your site; a meta tag is a robust alternative to the HTML file.

The setup #

My lab serves two public sites from one machine: www.shuffle-on.com, a static site about the lab, and a WordPress blog on another domain. Both sit behind Traefik, and in front of Traefik is the home router.

The router does more than forward ports. It runs an automatic defence: any source on the internet that touches certain sensitive ports is added to an address list, and everything from that list is dropped very early, in the “raw” stage of the firewall, before connection tracking. That stage is cheap, and it is the right place to drop known scanners: nothing they send costs the rest of the system anything.

It also means that a dropped packet leaves no trace in any log behind the router. Traefik never sees the connection, so the access log has nothing to show.

The investigation, in two rounds #

Round 1: a real 404, quickly fixed #

On 24 October Search Console reported that “Googlebot gets 404” on the blog. The Traefik access logs, kept in Loki, made that easy to check: Google’s crawlers had made 27 requests in 14 days, 26 of 27 returned 200, and the single 404 was for a new Search Console verification file. The old verification file from February still worked. Somebody (me) had generated a new token in Search Console and never put the file on the server.

I wrote the file into the WordPress web root, checked it from outside, and clicked “Verify” again. One side note from that round: Google crawls a small blog very little. 27 requests in two weeks means that indexing depends heavily on a verified property and a submitted sitemap.

Round 2: the errors the logs could not see #

Four days later, verification of the blog “failed in a way that indicates your site might have been hacked”. Google’s verification service had fetched both verification files and the homepage, all with 200, in the same minutes. Safe Browsing showed nothing unsafe for either domain. I switched to the meta-tag verification method, which worked.

But on www.shuffle-on.com, URL inspection kept reporting “Server connection error” and sitemaps “Couldn’t fetch”. The access log showed no failed request at all from Google for those fetches. A request that fails without appearing in the log has not failed at the web server. It failed somewhere in front of it.

So I looked at the router’s address lists for Google’s crawler addresses. Two Googlebot addresses were there, on the attack list, added in the early hours of 27 October. Both were genuine: their reverse DNS names were crawl-*.googlebot.com, and those names resolved forward to the same addresses.

Why a search engine touched one of the watched ports, the record does not show. A crawler fetches every URL it discovers, and a single link with an unusual port is enough. From the router’s point of view, it did exactly what a scanner does.

Part of that crawl traffic had been silently refused. I cannot prove how much of my ranking drop in those weeks came from it, but a search engine that cannot fetch your sitemap is not helping you.

Verifying a crawler properly #

The obvious fix is “never ban Googlebot”. The hard part is knowing what Googlebot is, because the user agent proves nothing: in my web-threat data, fake Googlebot traffic was 13 times the real one. Impersonators wear that name precisely because operators are afraid to block it. The organisation name of the address is no better, because Google’s crawl network and Google Cloud’s customer ranges both say “Google LLC”.

Google documents two reliable methods. The first is reverse and forward DNS, which is how I confirmed the two banned addresses:

ip=66.249.66.1                       # example address from Google's documentation
name=$(dig +short -x "$ip")          # expect crawl-....googlebot.com. or ....google.com.
echo "$name"
dig +short "${name%.}"               # must return the same IP

The second is the list of crawler IP ranges that Google and Bing publish as JSON. That one is better for firewall rules, because it can be loaded into an address list instead of doing DNS lookups per packet.

The fix #

The fix has three properties, and each one matters.

1. A dedicated allowlist of verified crawler ranges: Googlebot’s published range and Bingbot’s published ranges.
2. Evaluated before the auto-ban rule, so a verified crawler touching a watched port is not added to the attack list.
3. Narrow: it only exempts crawlers from the automatic ban. Every other filter rule still applies to them, so a compromised or misused range cannot bypass the rest of the firewall.

In generic terms, for a router that processes rules in order:

raw: accept   src in "verified-crawlers"   (only for the auto-ban trigger)
raw: add src to "attack-list"              (anyone else touching watched ports)
raw: drop     src in "attack-list"
filter: ... normal rules apply to everyone, crawlers included

Existing ban entries for crawler addresses also have to be removed by hand: an allowlist prevents new entries, it does not delete old ones.

Helping the crawl recover #

While the crawler was blocked, the blog had never been properly indexed at all. Two small changes help a search engine find it again:

• a crawlable link from the indexed site to the not-yet-indexed one, in an ordinary HTML section, not just in a footer;
• meta-tag verification for both properties, which does not depend on a file in a web root.

A note on testing from inside #

Testing your public entry point from the LAN can mislead you. Hairpin connections to the public address may time out, and pointing curl --resolve at the wrong internal port can make WordPress redirect to its canonical URL, which looks like an error. Use a proper outside view (a remote fetch service or a phone on mobile data) when you need to know what a crawler sees.

Checklist #

• When a search engine reports fetch or connection errors, first check whether the requests appear in your access logs at all.
• If they do not, check every ban list between the internet and the server: router, CrowdSec-style bouncers, cloud firewalls.
• Verify crawler addresses by reverse plus forward DNS, or match them against published ranges.
• Never trust the user agent or the organisation name of an address.
• Put verified crawlers in an allowlist before auto-ban rules, and keep other rules for them.
• After adding the allowlist, remove the existing ban entries.
• Use meta-tag verification when file verification behaves strangely.

Frequently asked questions #

Why does Search Console show “Server connection error” when my server is up? #

Because Google could not open a connection, which can happen before your web server is involved: a firewall drop, a ban list, a DNS problem. If the failing fetches are not in your access log, the cause is in front of the server.

How do I verify that an IP really is Googlebot? #

Do a reverse DNS lookup; the name must end in googlebot.com, google.com or googleusercontent.com for the matching crawler type. Then resolve that name forward; it must return the same IP. Or check the address against Google’s published crawler IP ranges.

Can I allowlist Googlebot by user agent? #

No. Anyone can send a Googlebot user agent, and in my data fake Googlebot traffic was 13 times the real crawler’s. Allowlist by verified address only.

Will a few days of blocked crawling hurt my ranking? #

Pages that cannot be fetched cannot be refreshed or discovered, and repeated connection errors can slow crawling. Once access is restored, the crawler recovers on its own schedule; a submitted sitemap and good internal links help.

The lesson #

Logs only show what reaches them. My web logs were complete and correct, and they still told the wrong story, because the failures happened one hop earlier. A defence that works silently and early is a good defence, but it needs its own visibility and its own exceptions. When something important says “I can’t reach you” and your logs say “I never saw you”, believe both, and look in between.