The three crawlers, and what blocking each one costs
The names matter because the consequences are completely different, and the most common mistake in this area is blocking all three when you only meant to opt out of training.
| Crawler | Job | What you lose by blocking it |
|---|---|---|
| OAI-SearchBot | Builds the index behind ChatGPT search | You disappear from ChatGPT search citations |
| ChatGPT-User | Fetches a page live on user request | ChatGPT cannot open your page even when asked directly |
| GPTBot | Collects public web data for model training | Your content is excluded from future training data |
Only the third of those is about training. The first two are about whether people can find and read you through ChatGPT at all.
What to put in robots.txt
To stay visible in ChatGPT search while opting out of training:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /
To allow everything, you need no rules at all — the absence of a matching block is permission. Explicit Allow lines are only useful for readability, or to carve an exception out of a broader Disallow.
The mistake that costs the most
A site decides it does not want its content training a model, finds a "block AI crawlers" snippet, and pastes in a list of every AI user agent. Training is blocked — and so is every path by which an assistant could cite the site to a person actively looking for it.
Those are separable decisions. Treat them separately.
Check what you are actually serving
Read the live file, not the one in your repo, because CDNs and platform layers sometimes inject their own rules:
curl -s https://example.com/robots.txt
Also check that a WAF or bot-protection rule is not blocking these agents above the robots.txt layer. A crawler that gets a 403 never reads your robots.txt at all, so a perfectly written file can sit behind a rule that makes it irrelevant. If you have server logs, confirm you are seeing successful fetches from these user agents rather than assuming.
Other crawlers in the same category
The same allow-or-block reasoning applies to PerplexityBot (Perplexity), ClaudeBot (Anthropic), and Google-Extended. Google-Extended is the one most often misunderstood: it governs whether your content is used for Gemini model training, and it does not control whether you appear in Google Search or in AI Overviews. Those are governed by ordinary Googlebot access and by your snippet settings.
Crawler names and behaviour change. Verify against OpenAI's published crawler documentation before relying on any specific string.