The Readers: Who Is Counting the Counters

Two letters in this series have been about machines that came here to count things.

The first counted hardware — 101 addresses, 997 sessions, two commands, uname -s -m, and a wordlist that turned out to be a hardware manifest. The second counted doors — 23 addresses wearing 56 identities, asking 233 times whether a key was lying on the floor.

This letter is about the machines that came to count us.

While those two operations were running, four other fleets were reading this site. Not probing it. Reading it — the dossiers, the findings, the evidence pages, the attestations. Between them they have fetched 24,712 pages. One of them has taken 4,744 distinct URLs, which is materially the entire published corpus, repeatedly.

They are the real ones. And what they do here is the cleanest available answer to the question letter B raised: what does an honest crawler actually look like?

The shape of the thing

Three of these four rows need a caveat, and this letter originally gave it to only one. A rule this site ran between 2026-06-03 and 2026-07-08 refused verified crawler traffic 1,657 times: OpenAI 1,172, Meta 265, Anthropic 220. Apple was never on the list. So ClaudeBot's 4.2% is dominated by that block, and GPTBot's 63.2% and Meta's 83.2% are both depressed by it as well — only Applebot's 99.8% is clean. The rule is gone. The section below traces which layer produced it, and corrects two earlier versions of this letter: the first got the cause wrong, and the second reported one victim instead of three.

Every genuine crawler in that table asks for robots.txt. GPTBot has asked 3,237 times. ClaudeBot, which has made only 261 requests in total, spent 134 of them — more than half its entire footprint — on that one file.

The counterfeit fleet asked zero times, across 2,318 requests.

That is the whole discriminator, stated as plainly as it can be stated. It costs nothing to implement, requires no threat feed, no reputation service, and no cryptographic verification. A crawler that has never asked for your rules is not a crawler. Letter B established that the impostor never read robots.txt while wearing the name of a robots.txt directive. This letter establishes the other half of the claim: every real one does, without exception — including the ones this site spent five weeks refusing at the door.

GPTBot took the whole library

4,744 distinct paths. There are roughly 820 published dossiers on this site plus their verification pages, evidence bundles, index pages and static assets. GPTBot has been through effectively all of it, at a 63.2% success rate across 9,812 requests.

The 36.8% that were not 200s are worth reading, because they are not failures in the sense letter B's impostor produced failures. A large crawler works from an index it built earlier. Slugs get renamed, series get reordered, a dossier gets corrected and re-published under a cleaner URL. The crawler asks for what it remembers and takes the 404 as an instruction to forget. Its miss rate is a measure of how much this site has changed since it was last indexed — a property of us, not a defect in it. Part of it, as the correction below records, is also a wall this site was putting in front of it.

Compare the impostor: 233 paths, 0% success, no index at all, because it was never reading. It was reciting.

That is the difference between a crawler and a scanner, and it is visible without inspecting a single user-agent string. One of them has a model of your site that decays. The other has a list that never had anything to do with your site in the first place.

Applebot reads differently, and better

Applebot's numbers are the most disciplined in the table: 5,611 requests, 353 distinct paths, 99.8% success.

Three hundred and fifty-three paths is not the whole corpus. It is a fraction of what GPTBot took. But Applebot asked for those 353 pages 5,611 times — an average of sixteen visits per page — and almost never asked for something that was not there.

That is a crawler that maintains its index rather than rebuilding it. It knows what exists, it returns to check whether it changed, and it does not speculate. Where GPTBot's pattern says fetch everything, sort it out later, Applebot's says know exactly what is here, and revisit it.

Neither is wrong. They are different bets about where the cost lives. But for a defender the operational point is that both patterns produce the same signature — a high 200-rate and a robots.txt request — and no amount of scale or aggression moves either crawler toward looking like the fleet in letter B.

One caveat belongs here rather than in a footnote: Applebot is also the only one of the four that was never blocked by this site. Some of the distance between its 99.8% and the others' figures is its own discipline, and some of it is simply that nobody put a wall in front of it.

Meta reads the most and checks the least

Meta's row is the one that complicates the tidy version of this letter.

By volume it is the second-largest reader here: 9,028 requests, 3,264 distinct paths, an 83.2% success rate — the second-highest in the table, and a sign of a well-maintained index. On every behavioural measure that separates a crawler from a scanner, Meta passes comfortably.

Except one. It requested robots.txt thirteen times.

Thirteen, against GPTBot's 3,237 and ClaudeBot's 134. Meta has made roughly the same number of requests as GPTBot and consulted the rules roughly 0.4% as often. Proportionally it is the least compliant genuine crawler observed, by two orders of magnitude.

That is not the same offence as the fleet in letter B, and the difference matters. Zero is a categorical statement: this agent has never acknowledged that rules exist. Thirteen is a cache — a crawler that fetched the policy, kept it, and did not re-check it often. Every large crawler caches robots.txt; the question is only for how long. GPTBot's 3,237 fetches suggest it re-reads the file constantly, plausibly per-worker or per-session. Meta's thirteen suggest a long TTL and a fleet that shares one answer.

Both are defensible engineering. But it means the robots.txt discriminator is not a threshold test — you cannot say "fewer than N fetches implies impostor," because a legitimate operator sits at thirteen. It is a presence test. The signal is not the count. It is whether the number is zero.

There is a coda to Meta's row that belongs here because it was found while investigating something else entirely. During the same period, Meta's renderer generated 70 requests for /threat-intel/null and /threat-intel/verify/null — a bare relative null resolved against whatever directory it was on. No such link exists anywhere in this site's markup; the string was manufactured on their side. Forty-seven distinct addresses produced it, and 46 of them sent ordinary desktop browser user-agents rather than a crawler string, because Meta renders pages with a headless browser.

Which is a second illustration of this letter's thesis from the opposite direction. Identify that traffic by user-agent and you conclude that dozens of human visitors clicked a broken link. Identify it by address — all inside one /24 belonging to Meta — and you conclude correctly that it is one crawler with a null-coercion bug. The user-agent said browser. The network said Meta. Only one of them was speaking from knowledge.

They were turned away, and that was our doing

ClaudeBot's row is the strange one, and the explanation is not about Anthropic.

261 requests from Anthropic's range. 220 of them returned 403. Eleven returned 200. Its entire path list is fourteen entries long, and the top two are /robots.txt (134 requests) and / (112).

So the picture is: it arrives, asks for the rules, asks for the front door, gets refused, and comes back later to ask again. It was not blocked because it misbehaved. It never got far enough to misbehave.

First correction, printed rather than quietly fixed. An earlier version of this letter reported the block in the present tense and attributed it to emergent complexity — "nobody decided that… the compound result of layered defences." Both halves were wrong, and tracing the request path settled it in one query.

The refusals came from a single Traefik router:

websecure-kube-system-block-ai-bots-3ddb9c0aa0c5aa42856a@kubernetescrd

That is an IngressRoute named block-ai-bots, in kube-system, priority 999, matching on a User-Agent regular expression and dispatching matches to a middleware called block-all. It is not a forward-auth side effect, not a CrowdSec decision inherited from a shared range, and not a WAF heuristic. It is a deliberate, purpose-built rule with an unambiguous name. Somebody decided exactly that.

And the block is already over. The refusals stop on 2026-07-08, when the pattern was amended to allow AI assistant crawlers. Since then there have been zero 403s from Anthropic's range.

Second correction, larger than the first. This section originally named one victim. Instrumentation built after publication — a job that simply counts refusals by name and by date, which nobody had been doing — found three:

One thousand six hundred and fifty-seven refusals of crawlers verified by source network. The largest was OpenAI's search crawler — 5.3 times the ClaudeBot figure originally reported here, unmentioned only because ClaudeBot happened to be the stream noticed by hand.

What survives is that the block was real, that it lasted five weeks, and that one of its victims was the most robots.txt-compliant crawler observed here — over half of ClaudeBot's total requests were compliance checks. A site published on a transparency argument spent June refusing three of the four organisations most likely to read it, and it did so by explicit rule rather than by accident. Apple's uninterrupted 99.8% read pattern, which reads as exemplary crawler discipline elsewhere in this letter, is partly just the absence of a wall the other three were meeting.

What replaces the original account is quieter and less comfortable. Since the rule was fixed, genuine ClaudeBot has made five requests — all in August, all 302 redirects from the Kiwix ingress. It has not come back to read the corpus. Whether a crawler that was refused for five weeks simply de-prioritised the domain is not answerable from this side of the connection; what is observable is that removing a block does not summon back the reader it excluded.

Meanwhile the name ClaudeBot remains busy here: 116 successful fetches and 58 more refusals in August alone — none of them from Anthropic's range. Those are letter B's impostors, still wearing the identity of a crawler this site actually locked out.

What it means that they are reading this

The recursion here is not decorative, so it is worth being precise about.

This site publishes forensic dossiers naming criminal infrastructure — ASNs, addresses, hosting providers, wordlists, the operator who added stfu_and_be_quite to a file on the 29th of March, misspelled. Those documents are now inside the ingestion pipelines of at least four large model operators, several of which will use them for training, retrieval, or both.

Three consequences follow, and only the first is comfortable.

The intelligence propagates. A defender asking a model about AS396982 or about Applebot-Extended spoofing has some chance of receiving an answer shaped by this corpus rather than by nothing. That is the argument for publishing in the first place, and it now has a delivery mechanism it did not have three years ago.

The methodology propagates too. These dossiers do not just name infrastructure; they explain how the attribution was reached, including which techniques failed. Letter A documented that a shared HASSH fingerprint identifies a compiler rather than an operator. Letter B documented that a crawler which never receives a 200 is not a crawler. Anyone can read those. That includes the people whose operations they describe.

And the corpus is a target. Letter B's fleet was hunting /.codex/config.toml — an AI coding agent's configuration — alongside AWS keys and SSH keys. The same ecosystem that ingests this intelligence is itself becoming an attack surface, and this site documents both halves of that from inside.

They are reading the proofs, not just the prose

Break the successful fetches down by what part of the site they landed on, and something unexpected appears.

Eight hundred and four distinct verification pages.

Every dossier on this site carries a cryptographic attestation — a Merkle root over its evidence, actors, findings, connections and infrastructure records, signed with Ed25519 and anchored into Bitcoin via OpenTimestamps. The verify page is where that commitment is exposed for independent checking: the leaf count, the root, the signature, and the specific claims each leaf covers.

Those pages exist for humans who want to confirm that a dossier's evidence has not been edited since publication. They are the least narrative thing on the site. They are tables of hashes.

The crawlers are taking them anyway — 2,216 times, across 804 distinct URLs.

There is no way to know from an access log what an ingestion pipeline does with a Merkle root. But the structural fact is worth recording plainly: the machines reading this corpus are pulling down not only the claims but the apparatus that makes the claims checkable. Whatever ends up in a training set or a retrieval index, it includes the evidence commitments alongside the assertions.

That is either meaningless — a crawler follows links, and the verify pages are linked from every article — or it is the most interesting thing in this letter. The honest position is that we cannot tell which, and the reason we cannot tell is that the question is about what happens on their side, in systems that publish nothing about their internals. We can observe the fetch. We cannot observe the use.

What can be said is narrower and still worth saying. A corpus that publishes its evidence in a machine-checkable form is being ingested with that form intact. If any of these pipelines ever chooses to weight sourced, attested claims differently from unsourced ones, the material to do it with is already inside them.

The weather they read against

None of this happened in quiet conditions. The ten days spanning this investigation:

Between six hundred and one thousand distinct addresses per day, and roughly a third of all traffic terminating in a 404. That baseline is the ambient condition of an internet-facing host that publishes, and it is the noise floor against which letter B's fleet had to be found — 2,318 requests spread across three weeks, inside a corpus of tens of thousands.

The 6th of August is the interesting row: 8,073 requests, the highest volume of the period, at a 9.3% 404-rate — a quarter of every other day. High volume with a low miss rate is the signature of something reading rather than probing. That day was crawler traffic, not scanner traffic, and the 404-rate alone separates it from its neighbours without any inspection of who sent it.

This is the same discriminator as before, applied at the level of a whole day instead of a single client. The site's own traffic can be sorted into reading and interrogating by outcome, and the sort survives every disguise on either side of it.

Steelmanning the boring explanation

The comfortable reading is that none of this is interesting: large crawlers crawl large sites, a 403 is a misconfiguration rather than a finding, and reading 24,712 pages of a public website is what public websites are for.

Most of that is correct and should be conceded plainly. The crawlers here are behaving exactly as documented. Nothing in their traffic is adversarial, none of it is unusually aggressive, and none of it required any defensive response. And the block turned out to be neither mysterious nor permanent: a named rule, since amended.

Two things resist the dismissal.

The robots.txt figure is not a triviality, it is a control. Letter B's conclusion — that identity should be verified by network and behaviour rather than by name — was argued from one side only: the impostor's failure to read robots.txt. A single negative result is weak. This letter supplies the positive control: four independent crawler operators, four different crawling philosophies, four different scales, and all four ask for the rules. The discriminator holds in both directions, which is what makes it usable.

A finding against yourself is still a finding — including when it corrects an earlier draft of itself, twice. This letter first reported the block as ongoing and blamed emergent complexity. One query against the router name disproved both claims. It then reported a single victim; instrumentation built afterwards found three, and the largest was 5.3 times the one named. Both errors are left visible above rather than silently amended, because a corpus that asks to be checked has to survive being checked.

The negative that matters most

Letter B's negative was that filtering by a derived, attacker-supplied field hid a quarter of a fleet. This letter's negative is smaller and closer to home.

Every figure in the table above was reconstructed by matching source addresses against published vendor ranges — 17.0.0.0/8 for Apple, 74.7.x for OpenAI, 216.73.x for Anthropic, 57.141.x for Meta. That reconstruction was done by hand, in a query, for this article.

The platform does not do it automatically. There is no verified_crawler column. The web-threats classifier records agent_class, agent_vendor, agent_product and agent_confidence — and every one of those fields is derived from the user-agent header, which is to say from a string the client chose. The classifier does mark agent_confidence: spoofed when the claim and the network disagree, and that flag fired correctly on the impostor fleet. But the authoritative fact — this address is inside a range the vendor publishes — is not stored anywhere. It is recomputed by whoever thinks to ask.

Which means the platform can tell you, accurately and at scale, what every crawler on the internet said it was. To find out what any of them actually is, you have to leave the index and go check the address yourself.

The cost of that gap is now measurable twice over. In August, traffic bearing the ClaudeBot user-agent runs to 116 successful fetches and 58 refusals — and none of it is Anthropic. Genuine ClaudeBot contributed five requests in the same month. And the 1,437 refusals this letter missed on first writing were missed for the same reason: nothing counted refusals by verified network, so the only victim visible was the one somebody happened to notice.

That gap is the same shape as the one letter B found, and it did not get closed by finding it. It is recorded here so that the next investigation does not have to rediscover it.

What to take from it

Three things, in descending order of how much they will save you.

Verify by network, and confirm with robots.txt. Every genuine crawler observed here — across four vendors, three orders of magnitude of scale and two opposite indexing philosophies — requests robots.txt. The counterfeit fleet, across 2,318 requests, never did. Combined with published vendor ranges, that is a two-field check that defeats an entire wardrobe of forged identities, and it requires nothing you do not already have in your access log.

A high 404-rate is the tell, not the user-agent. GPTBot misses 36.8% of the time because its index is stale. The impostor missed 100% of the time because it had no index. Applebot misses 0.2% because it maintains one. Read the outcome, not the claim.

Audit who you are refusing, and count it. A wall stood in front of three of the four crawlers in this letter's own table for five weeks, and the first version of this letter — written by someone looking straight at the data — named one of them. The full figure came from a job that does nothing cleverer than group refusals by network and by date. If you publish anything on the argument that it ought to be read, it is worth knowing periodically whether it can be, and by whom.

And one thing that is not operational at all.

The series is called The Census because the first letter caught a machine taking an inventory — logging in, asking what architecture the host ran, and leaving without doing anything with the answer. Inventory taken and not yet used, as that letter put it, is not a failure; it is a stock-take.

The machines in this letter are doing the same thing to the dossiers themselves. They arrive, they read what has been written about the people who broke in, they take a copy, and they leave. Nothing visible happens next.

Somewhere in a training corpus or a retrieval index there is now a record of a wordlist curated in March, a fleet that wore 56 faces in five seconds, and an operator who misspelled quiet. The counting goes in both directions, and only one side of it gets to choose whether it is counted.

Series: The Census. Preceded by TI-2026-102A — "The Census: A Scanner That Breaks In, Asks What You Are, and Leaves" and TI-2026-102B — "The Wardrobe: Fifty-Six Identities in Five Seconds." Followed by TI-2026-102D — "The Door That Answers."

All figures drawn from the web-threats request corpus. Crawler attribution performed by matching source addresses against vendor-published ranges, not by user-agent string. This letter has been corrected twice after publication; both corrections are printed in place rather than applied silently.