Skip to content

Crawlers

Does blocking GPTBot affect ChatGPT citations?

Short answer

No. GPTBot governs whether your content is used for model training. ChatGPT search citations depend on OAI-SearchBot, a separate crawler controlled by a separate robots.txt rule. Blocking GPTBot while allowing OAI-SearchBot keeps you fully citable in ChatGPT search — and that combination is what most publishers actually want.

By The EchoVerse teamUpdated 3 min read

Why the two get confused

Both crawlers belong to OpenAI, both appear in the same robots.txt file, and most "block AI bots" advice online treats them as one thing. They are not. They feed different systems and blocking them has opposite consequences for your visibility.

Training is about whether a future model absorbs your writing. Search is about whether a person asking a question today is shown your page as a source. Conflating them means opting out of the second while intending only the first.

The combination most publishers want


User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

That says: do not train on me, but do cite me when someone is looking for what I know.

When blocking everything is the right call

There are legitimate reasons to block all three — content licensing, paywalled archives, a contractual obligation, or a considered position on training consent. That is a valid choice. Make it deliberately, with the traffic consequence understood, rather than inheriting it from a copied snippet.

What blocking does not do

Blocking GPTBot does not remove knowledge a model already has. Training runs happen at intervals; if your content was crawled before you added the rule, it may already be reflected in a deployed model's parametric knowledge. A robots.txt change is forward-looking. It does not function as a deletion request, and it will not stop a model describing your product from what it already learned.

If a model states something incorrect about you and cites nothing, that is parametric knowledge, not retrieval, and robots.txt is the wrong tool for it entirely.

Verify rather than assume

After changing robots.txt, confirm the live file is what you think it is, and check server logs for successful fetches from OAI-SearchBot. A bot-protection rule that returns 403 to unfamiliar agents will silently override anything robots.txt says, because a crawler blocked at the edge never gets far enough to read it.

Common questions

Will unblocking GPTBot get me cited faster?
Not directly. GPTBot feeds training, which moves on the timescale of model releases. Citations come from retrieval via OAI-SearchBot, which responds far more quickly. If your goal is citations, allowing OAI-SearchBot is the change that matters.
Does blocking GPTBot remove my content from models already trained on it?
No. Blocking is forward-looking and applies to future crawls. Content already used in a deployed model is unaffected, and robots.txt is not a deletion mechanism.