The Door That Answers: How Our Blocklist Taught the Fleet Which Names to Wear

Letter B ended on a guess.

A single address had worn fifty-six identities in five point zero two seconds — every major AI crawler, plus thirty-five browser strings — and the guess was that this could not be disguise, because nothing disguises itself as fifty-six things at once. It had to be a question. "Somewhere there is a filter that lets ClaudeBot past and blocks Go-http-client, and this is the tool that maps it."

The filter has now been found. It is ours. It answered every question the fleet asked, honestly and completely, and the fleet acted on the answer.

This letter is about what a blocklist is when the person reading it is the person you are blocking.

The shape of the thing

The rule is not exotic and it is not badly written. It is the single most common piece of anti-scraper configuration on the internet: a list of user-agent substrings, matched case-insensitively, returning 403. Some version of it runs in front of a very large fraction of the web.

That is why it is worth taking apart. The defect is not in this rule. It is in the shape of the idea.

What the carousel asked

The fleet from letter B is 23 addresses on AS396982 — Google Cloud customer egress — running a 233-path credential sweep. Over three weeks it issued 2,839 requests through this site's edge while cycling identities.

Twenty-one of those identities were AI crawlers. Two of them, Bytespider and Diffbot, are on the blocklist above. The other nineteen are not.

Read that arrangement as a query and it becomes legible. The tool is not trying to be Bytespider. It is asking whether Bytespider works here. It asks the same question about ClaudeBot, GPTBot, CCBot, Applebot, cohere-ai, Perplexity, Mistral, Amazon, and the rest, and it asks all of them inside a five-second window from one address, because the answers are independent and there is no reason to wait.

What the door answered

Every request the fleet made, grouped by the identity it wore, and split by whether the edge refused it:

Two rows are absolute. Diffbot: forty-five asked, forty-five refused, none passed. Bytespider: forty-two asked, forty-two refused, none passed. One hundred per cent, no exceptions, no ambiguity.

Every other row is a door. The scattered 403s elsewhere — seven here, eleven there — are the ordinary background of a defended host: CrowdSec decisions, host-based rules, rate limits catching a burst mid-flight. They are noise. They are not a pattern, and nothing in them would tell an observer that a name was forbidden rather than merely unlucky.

So the answer the fleet received was not fuzzy. It was a clean partition: two names are walls, fifty-four are doors.

And then they stopped asking

This is the part that turns an interesting observation into a finding.

The two blocked names, by date:

Eighty-seven requests wearing the two blocked names across the first three appearances. Then nothing — not one, across 2,752 subsequent requests and six further sessions spanning nineteen days.

They asked three times, got the same answer three times, wrote it down, and never asked again.

That is not a tool cycling a static list. A hardcoded carousel would keep emitting Bytespider forever, because a hardcoded carousel does not know what happened to the last request. This one changed its behaviour in response to our replies. Somewhere between the 23rd and the 27th of July, two entries were removed from a file, and the removals correspond exactly to the two entries this site refuses.

The fleet did not discover our blocklist by reading it. We read it out to them, one 403 at a time, and they took notes.

A blocklist is an oracle

The failure here is structural, and stating it plainly is the point of this letter.

A user-agent blocklist is a function. You hand it a name, it returns allowed or forbidden. It is stateless, it is free to query, it is available to anyone who can make an HTTP request, and — this is the whole problem — it cannot see who is asking.

It answers a researcher and an adversary identically, because from inside the rule there is no difference between them. Both send a header. Both get a status code.

Which means every deployment of this pattern ships with a lookup service attached. The blocklist is not a secret that the rule protects; the rule is the disclosure mechanism. Fifty-six requests — five seconds, one address, no sophistication beyond a loop — extracted its complete contents.

And the extracted answer is durable in a way the block is not. The 403 stops one request. The knowledge that Bytespider is refused here stops that name being used against this host ever again. We spent two entries of blocklist coverage to purchase 87 blocked requests, and in exchange we told an active adversary precisely which fifty-four identities to prefer.

The exchange rate is bad. It is bad in a way that gets worse the more names you add, because each additional name is another bit of information available for free to anyone who asks, and it only ever blocks the subset of scrapers honest enough to keep announcing themselves after being refused once.

The arithmetic of the disclosure

It is worth putting numbers on the exchange, because they are not close.

What it cost them to learn. One address. Five point zero two seconds. Fifty-six requests, of which twenty-one were the ones that mattered. No credentials, no vulnerability, no tooling beyond an HTTP client and a list — and the list of names to test is itself public, published by the vendors whose crawlers they are. Total investment: a loop.

What it cost us to teach. Eighty-seven requests refused, spread over three sessions. Set against 2,839 requests the fleet made in total, that is 3.1% — and every one of the other 96.9% proceeded to ask this site where it keeps its keys.

What they retain. Permanent knowledge that two specific names fail here and fifty-four do not. That knowledge does not expire, does not need re-verification, transfers to any other target running the same commodity list, and is small enough to write on a napkin.

What we retain. Two blocklist entries that will never fire against this operator again.

The asymmetry is not marginal, and it is not a matter of tuning. A rule that answers questions costs the asker almost nothing and costs the answerer the entire contents of the rule. Adding more names does not improve the ratio — it enlarges the disclosure while leaving the query cost unchanged, because fifty-six requests reads an eleven-entry list exactly as easily as it reads a hundred-entry one.

Meanwhile it was blocking real readers

Corrected after publication. The first version of this section reported one victim: genuine ClaudeBot, refused 220 times. That figure was right and the account was incomplete. There were three.

The instrumentation this letter's closing section calls for was built immediately after publication — a job that counts refusals by name and by date, which is the one thing nobody was doing. Pointed at a window covering the historical period, it returned this:

One thousand six hundred and fifty-seven refusals of crawlers verified by source network, across three of the four major operators. The largest was OpenAI's search crawler at 1,172 — five point three times the ClaudeBot figure this letter originally reported, and it went unmentioned because ClaudeBot was simply the stream that happened to be noticed by hand.

All three end on or about 2026-07-08, which is when the rule's pattern was amended to allow AI assistant crawlers. Apple alone was never on the list, which is also why Applebot's near-perfect read pattern in letter C stands out: it was the only one of the four not being turned away while the measurements were taken.

The character of the block does not change with the numbers. ClaudeBot's entire footprint here is fourteen paths, topped by /robots.txt (134 requests) and / (112) — over half its total requests were compliance checks. It arrived, asked for the rules, asked for the front door, was refused, and came back later to ask again. Since the fix it has made five requests, all 302 redirects from an unrelated ingress, and has not returned to read the dossiers.

Set the two outcomes beside each other, because together they are the argument, and the correction sharpens it rather than softening it:

• Against the impostors, the rule blocked 87 requests, and then taught them which names to stop using.
• Against genuine crawlers, it blocked 1,657 requests over five weeks, and at least one of them has not come back.

Nineteen refusals of an honest reader for every one of an adversary — and the adversary kept the knowledge, while the honest reader may simply have kept away. That is not a bug in the regular expression. It is what happens when identity is self-declared: the only entities a name-based rule can reliably act on are the ones telling the truth.

There is one more thing worth saying about how this correction arrived. It was not found by reading the logs again more carefully. It was found because the closing recommendation of this letter — instrument the refusals — was implemented, and the instrument reported it on its first run. The gap between what a careful manual investigation saw and what a five-line query saw was 1,437 refusals.

Seven names that have never matched anything

A smaller finding, kept because it bears on how these lists get maintained.

Of the eleven patterns in the rule, seven have never matched a single request in the site's history: DataForSeoBot, webz.io, Timpibot, img2dataset, ICC-Crawler, Omgili, Seekr.

Four have ever fired: Bytespider (291 requests), PetalBot (166), Scrapy (49), Diffbot (45).

The seven are not harmful. They cost a few microseconds of regex per request and they may catch something one day. But they are worth noticing, because a blocklist that is two-thirds aspirational looks, to the person maintaining it, exactly like a blocklist that is working. Nothing in the interface distinguishes this name is blocked and never appears from this name is blocked and would otherwise be a problem. Both render as silence.

Meanwhile two of the four names that do fire, Scrapy and img2dataset, are not organisations at all — they are libraries. Anyone can run them from anywhere, and anyone who does not want to be identified simply sets a different string. They are on the list because they are polite by default.

What we changed, and what we deliberately did not

The obvious response is to stop matching names and start matching networks. A network is not self-declared: an address is where the packets actually come from, and it cannot be edited by the sender.

That is now in place, but only where the evidence supports it. Measured across every address that has ever presented these user-agents here:

So 114.119.0.0/16 is now blocked by network. PetalBot cannot escape it by changing its user-agent, because the range is where PetalBot actually lives. That range was checked before it was added: 184 requests from it in the site's entire history, 166 of them PetalBot, the remaining 18 a datacenter string from Huawei Cloud Hong Kong. No human traffic.

What was not done is the version of this that sounds tidier. Converting the whole rule to network matching would require blocking AS396982 — all of Google Cloud customer egress — because that is where the Bytespider and Diffbot traffic actually originates. That would break a very large amount of legitimate traffic in order to catch requests that are impersonation anyway, and which are already handled correctly one layer down: the autoban path bans on behaviour, and every one of those 23 addresses was banned within seconds of starting, on the strength of what it asked for rather than what it called itself.

The name rule is therefore kept, deliberately, for the cases network matching cannot reach: libraries with no home network, and scrapers on rented infrastructure that announce themselves honestly.

Steelmanning the boring explanation

The comfortable reading is that this is a great deal of analysis about a 403, that blocklists are known to be imperfect, that everybody runs one anyway, and that 87 blocked requests is 87 blocked requests.

Most of that stands. The rule did what it was asked to do. Nothing was compromised, nothing leaked that was not already public — a blocklist's contents are, in the end, a list of names anybody could guess. And the specific fleet documented here was banned at layers 3 and 7 within seconds regardless, so in operational terms the blocklist's contribution was marginal in both directions.

Two things resist it.

The adaptation is measured, not inferred. The interesting claim in this letter is not "blocklists can be enumerated" — that is obvious and has been said for twenty years. It is that we watched a specific adversary enumerate one, in five seconds, and then observed it permanently change behaviour in response, with the change visible as an exact zero across 2,752 subsequent requests and nineteen days. Theory says this is possible. The log says it happened here, on this date, to this rule.

The asymmetry is the actual cost. If a blocklist only failed to stop adversaries, it would be merely useless. This one also spent five weeks refusing verified crawlers 1,657 times, and at least one of them has not returned. A control that is weak against its target and effective against everyone else is not a partial success. It is inverted.

The negative that matters most

There is no evidence in these logs that anybody noticed any of it.

Not the block: nothing recorded when the refusals began on the 3rd of June. Not the fix: nothing recorded when the pattern was amended on or about the 8th of July, ending them. Not the enumeration: the 56-identity burst on the 21st of July passed through the edge as 484 ordinary requests, and the three days on which the fleet tested our blocklist produced no alert of any kind.

Every one of those events was fully present in data we already had. The 403s were in the request log with the router name attached, which is what eventually identified the rule in a single query. The carousel was in the same table, one row per identity. The adaptation — 87 requests, then a hard zero — required nothing more than grouping by date.

The information was not missing. Nobody asked.

That is the same failure as letters B and C in a third form. Letter B: filtering by a derived, adversary-supplied field hid a quarter of a fleet. Letter C: the platform records what every crawler said it was and never what it is. Here: the platform records every refusal it issues and nothing watches the pattern of them. In each case the data was complete and the question went unasked, and in each case the gap was found only by taking the other side's point of view — reading our own logs as the adversary would.

A defence you do not instrument is a defence you cannot evaluate. This one ran for five months, blocked 87 adversary requests, taught that adversary its complete contents, refused verified crawlers 1,657 times across three operators, and carried seven patterns that have never matched anything — and none of that was visible from inside, because a blocklist's only output is silence and silence looks identical whether it is working or not.

What to take from it

Assume your blocklist is public. Not because it will leak, but because it answers. Anyone willing to spend fifty-six requests can read it end to end, and the cost of asking is a rounding error against the value of knowing. Write it on the assumption that its contents are already in the hands of whoever cares most.

Match on what the sender cannot choose. A user-agent is a claim. An address is a fact. Where a crawler genuinely owns its range — PetalBot in 114.119.0.0/16 — network matching cannot be evaded by editing a header. Where it does not, do not force it: blocking a shared cloud to catch an impersonator costs more than it saves, and behaviour-based banning already covers that case.

Instrument the refusals. Count what you are blocking, by name and by date. The two signals in this letter — a name that goes to zero, and a legitimate crawler that starts failing — are both trivially detectable from data every reverse proxy already writes, and both went unnoticed for weeks.

Audit who you are refusing, and whether they came back. The block was lifted a month before anyone looked, and one of the three readers it excluded has not returned. Removing a wall does not restore what stood behind it. And note how the fuller picture arrived: not from re-reading, but from counting.

And the thing underneath all four.

The fleet in letter B wore fifty-six faces in five seconds and the reading then was that this was absurd — a machine impersonating everyone at once, fooling nobody. That reading was wrong. It was not trying to fool anything. It was conducting an interview, and the questions were well designed, and it got a complete and truthful answer to every one of them inside five seconds.

We were not being deceived. We were being asked. And a door that answers honestly, to anyone, about which names it will open for — that is not a lock. It is a directory.

Series: The Census. Preceded by TI-2026-102A — "The Census: A Scanner That Breaks In, Asks What You Are, and Leaves", TI-2026-102B — "The Wardrobe: Fifty-Six Identities in Five Seconds", and TI-2026-102C — "The Readers: Who Is Counting the Counters."

All figures drawn from the web-threats request corpus. Crawler identity determined by source network against vendor-published ranges, never by user-agent string. Rule contents read from the live Traefik IngressRoute. This letter was corrected after publication; the correction is printed in place rather than applied silently.