AI Crawlers 101: What's Crawling Your Site in 2026

In 2025, "what crawls your site" was a question only SEO nerds asked. In 2026, it's a business question: research on AI-assisted buying suggests a large and growing share of buyers now use AI search somewhere in their evaluation, and an LLM typically cites only a handful of domains per answer. If the crawlers that feed AI answers can't reach you, you don't exist in AI search.

Three types of AI crawlers

Not all AI crawlers are the same. Blocking one type can cost you citations; blocking another only opts you out of model training. The three jobs:

• Training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider) harvest your content to train future models. Blocking them = opting out of training. For these, it does not remove you from AI answers.
• Retrieval / search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot) build the index an assistant cites in live answers. Blocking these removes you from AI answers.
• User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) grab a page in real time when a person clicks your link in a chat. Blocking them breaks citation clicks.

There is also a fourth category worth knowing: control tokens like Google-Extended and Applebot-Extended. They have no user-agent of their own — they only tell the operator how content it already crawled may be used, and blocking them does not stop any request.

Why it matters

Traffic is migrating from "link networks" to "answer networks". Two widely cited industry findings frame the shift: roughly 58.5% of US Google searches ended without a click in 2024 (SparkToro / Datos), and in G2's 2025 buyer research, using AI search as a research source ranged from about 35% at small companies to 55%+ at larger ones (G2). Treat these as directional third-party research, not measurements of your own traffic — your analytics are the only source for that.

The single most common mistake? Blocking a retrieval crawler while trying to block training. That's the subject of our next post.

Not sure where your site stands? Check your site against 40 major AI crawlers — free, no signup.