What Is OAI-SearchBot? The Crawler That Gets You Cited in ChatGPT
OAI-SearchBot is OpenAI's retrieval crawler — the one that decides whether your site gets cited in ChatGPT answers. Blocking it costs you citations. Here's what it is and how to check yours.
If you've ever looked at your robots.txt and wondered what OAI-SearchBot is — you're not alone. It's one of the most misunderstood crawlers in the AI ecosystem, and getting it wrong quietly costs sites their ChatGPT visibility.
OAI-SearchBot ≠ GPTBot
OpenAI runs several crawlers, and they do very different jobs:
- GPTBot — the training crawler. It harvests content to train OpenAI's foundation models. Blocking it opts you out of training.
- OAI-SearchBot — the retrieval crawler. It indexes pages for ChatGPT Search answers. Blocking it removes you from ChatGPT answers.
- ChatGPT-User — the user-triggered fetcher. It grabs a page in real time when someone clicks a citation in chat.
Why blocking OAI-SearchBot is the expensive mistake
In 2023, many sites added blanket AI blocks. A common pattern: User-agent: GPTBot Disallow: / — then someone "helpfully" added OAI-SearchBot to the block list too, thinking it was another training crawler. The result: the site opted out of ChatGPT citations while still being crawled for training. Exactly the opposite of what they wanted.
Check whether you're blocking it
curl -A "OAI-SearchBot/1.0" -I https://yoursite.com200 OK = you're visible to ChatGPT Search. 403/4xx = you're blocking the crawler that gets you cited.
What to do
If you want ChatGPT visibility, make sure OAI-SearchBot and ChatGPT-User are allowed — even if you keep blocking GPTBot for training. These are independent decisions:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /Not sure of your current status? Run a free check against all 24 AI crawlers and see exactly which rules are blocking you.