What we measured
OpenAI runs three crawlers with three different jobs, controlled separately in robots.txt. GPTBot collects training data. OAI-SearchBot builds the index behind ChatGPT search results and citations. ChatGPT-User fetches a page live when someone asks.
Blocking the first is a training-consent decision. Blocking the second removes you from ChatGPT search entirely. Most published advice treats them as one thing, so we wanted to know what sites are actually doing.
The sample
We drew 100 domains from each of four Tranco rank bands — 1–1k, 1k–10k, 10k–100k and 100k–1m — using a deterministic seed so the sample can be reproduced exactly.
Of 400 domains, 233 returned a robots.txt. The rest are mostly CDN and DNS infrastructure that Tranco includes but which are not websites (awsdns-*, akamai.net, rbxcdn.com), plus a set that answered our request with a 403 — itself worth noting, since a crawler blocked at the edge never gets to read robots.txt at all.
All percentages below are of the 233 reachable domains.
Headline: the opt-out is blunter than intended
| Domains | Share of blockers | |
|---|---|---|
| Block GPTBot | 43 | — |
| …and also block OAI-SearchBot | 22 | 51% |
| …but keep OAI-SearchBot open | 21 | 48% |
Roughly half of the sites that decided not to feed model training also, in the same robots.txt, removed themselves from ChatGPT search citations.
Those are separable decisions. A site can refuse training and still be cited when someone is actively looking for what it knows — that combination is three lines of configuration. The fact that half of blockers do not make that distinction suggests the advice circulating about "blocking AI crawlers" is being applied wholesale.
Blocking rates by crawler
| Crawler | Blocked | Named explicitly | What blocking costs |
|---|---|---|---|
| GPTBot | 18% | 17% | OpenAI model training |
| ClaudeBot | 16% | 13% | Anthropic crawling |
| Google-Extended | 15% | 11% | Gemini model training |
| CCBot | 15% | 14% | Common Crawl |
| Bytespider | 15% | 14% | ByteDance crawling |
| ChatGPT-User | 11% | 7% | ChatGPT live fetches |
| PerplexityBot | 10% | 8% | Perplexity citations |
| OAI-SearchBot | 9% | 6% | ChatGPT search citations |
The training-oriented crawlers are blocked roughly twice as often as the retrieval-oriented ones, which is the shape you would expect if the decisions were deliberate. The 51% overlap above is what complicates that reading.
Note the gap between "blocked" and "named explicitly": some sites block a crawler through a catch-all User-agent: * rule without ever naming it. Those sites have not made a decision about AI crawlers at all — they have a restrictive robots.txt and AI crawlers are caught by it.
By rank band
| Band | Reachable | Blocks GPTBot | Blocks OAI-SearchBot | Has llms.txt |
|---|---|---|---|---|
| 1–1k | 55 | 18% | 10% | 20% |
| 1k–10k | 61 | 19% | 8% | 9% |
| 10k–100k | 51 | 11% | 5% | 9% |
| 100k–1m | 66 | 22% | 12% | 9% |
Blocking is not concentrated among large publishers the way the coverage of this topic implies. The long tail blocks at a similar or higher rate — often, we suspect, through inherited restrictive templates rather than a considered position.
llms.txt adoption is higher than we expected
28 of 233 reachable domains (12%) serve a real llms.txt, rising to 20% in the top 1,000.
We expected low single digits. This is a proposed convention that no major AI provider has publicly confirmed using for retrieval, and yet one in eight sampled sites publishes one.
Two caveats keep this honest:
- A 200 response is not a file. 42 domains returned HTML at
/llms.txt— their SPA - Some adoption is platform-driven. Several hits were small e-commerce sites on the same
shell rather than a real document. Counting those would have roughly tripled the number. We required markdown structure, and excluded anything that looked like HTML.
platform, which suggests an app or theme generating the file rather than a deliberate choice by the operator.
So: adoption is real and measurable, but it is not evidence that llms.txt works. Those are different claims, and the second one still has no public support.
What we would tell you to do
- Decide training and retrieval separately. If you want to opt out of training but stay
- Read the file you actually serve.
curl -s https://yoursite.com/robots.txt. CDNs, - Check for 403s at the edge. A bot-protection rule that rejects unfamiliar user agents
- Treat llms.txt as optional. Cheap to add, no confirmed effect. Do robots.txt properly
findable, block GPTBot and allow OAI-SearchBot. Check which of the two you are currently doing rather than assuming.
hosting platforms and security products all rewrite or generate this file.
overrides everything in robots.txt, because the crawler never reads it.
first.
Method and reproducibility
The collector and the sampler are both in the repository, and the full dataset is linked at the top of this page.
node scripts/sample-tranco.mjs top-1m.csv 400 > domains.txt
node scripts/crawler-access-study.mjs domains.txt out.json
robots.txt is evaluated for the site root only: longest match wins, Allow beats Disallow at equal specificity, an agent's own group beats *, and the absence of a matching group means permitted. The parser was checked against known cases — a site with an explicit User-agent: GPTBot / Disallow: / reads as blocked, and a site that never mentions GPTBot reads as allowed.
This is a 400-domain sample, not a census. Treat the percentages as indicative with a margin of several points, and re-run it yourself if a decision depends on the exact figure.