The Most Expensive Mistake in AI SEO: Blocking the Wrong Crawler
Trying to block GPTBot (training) but accidentally blocking OAI-SearchBot (retrieval)? That's the mistake that quietly removes you from ChatGPT answers while doing nothing to protect your content.
In 2023, "AI is stealing our content" was the panic. Thousands of sites added blanket blocks for every AI-looking user agent. In 2026, many of those same sites wonder why they never appear in ChatGPT or Perplexity answers.
The expensive mistake
OAI-SearchBot sounds like it might gather training data. It does not. It is the crawler that lets ChatGPT cite you. Blocking it removes you from ChatGPT answers — and does nothing about training. The training crawler you actually meant to block is GPTBot. They are independent:
- Blocking GPTBot → opts you out of OpenAI training, leaves your ChatGPT citations untouched.
- Blocking OAI-SearchBot → removes you from ChatGPT answers, does nothing about training.
Other silent traps
- Empty Disallow: a bare
Disallow:line means allow, not block. A common typo that "blocks" nothing while you think it blocks everything. - Wildcard blocking:
User-agent: *+Disallow: /blocks every crawler including the retrieval bots that would cite you. - Forgetting exceptions: a broad block plus per-bot allows is fragile; one typo quietly drops you out of answers.
The 2026 default
For most marketing sites: allow retrieval and user-triggered crawlers, decide training per your data policy. You can't be cited if you're not crawled.
Audit your robots.txt today: run a free AI crawler check and see exactly which crawlers you're blocking — before it costs you AI visibility.