AI Crawlers 101: What's Crawling Your Site in 2026
GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot... there are now dozens of AI crawlers, and they do very different jobs. Here's what each type does and why it matters for your site.
In 2025, "what crawls your site" was a question only SEO nerds asked. In 2026, it's a business question: 42% of buyers use AI search as part of their evaluation process, and an LLM cites only 2-7 domains per answer. If the crawlers that feed AI answers can't reach you, you don't exist in AI search.
Three types of AI crawlers
Not all AI crawlers are the same. Blocking one type can cost you citations; blocking another only opts you out of model training. The three jobs:
- Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider) harvest your content to train future models. Blocking them = opting out of training. It does not remove you from AI answers.
- Retrieval / search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot) build the index an assistant cites in live answers. Blocking these removes you from AI answers.
- User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) grab a page in real time when a person clicks your link in a chat. Blocking them breaks citation clicks.
Why it matters
58.5% of Google searches are now zero-click. Organic traffic is down 15-25%. Traffic is migrating from "link networks" to "answer networks" — and the crawler list is growing (150+ user agents and counting, changing weekly).
The single most common mistake? Blocking a retrieval crawler while trying to block training. That's the subject of our next post.
Not sure where your site stands? Check your site against 24 major AI crawlers — free, no signup.