Every AI crawler we check — who runs it, what it does, and whether you should allow it. 40 crawlers across three jobs.
Build the index AI answers cite from. Blocking these removes you from AI answers — the expensive mistake.
ChatGPT Search retrieval; blocking removes you from ChatGPT answers
ChatGPT ads landing-page validation
Claude answer retrieval; blocking stops Claude from citing you
Perplexity indexing + live retrieval with citation backlinks; blocking is costly
Google's main search crawler; almost never block
Bing + Copilot retrieval; almost never block
Siri / Spotlight / Safari suggestions
DuckDuckGo AI summaries
Meta AI search index; blocking removes you from Meta AI answers
Petal Search / Celia assistant index
Hive's AI image search index
iAsk.ai AI search engine index
Fetch a page in real time when a user opens your link in a chat. Safe to allow.
Real-time fetch when ChatGPT users click your links
Real-time fetch when Claude users click your links
Perplexity user-triggered fetch (sometimes ignores robots.txt)
Real-time fetch when Le Chat users click your links
Real-time fetch when Meta AI users open links
Real-time fetch when DeepSeek users open links
Harvest content to train models or build datasets. Blocking opts you out of training — a legitimate choice, not a visibility loss.
Trains GPT models; blocking opts out of training, does not affect ChatGPT citations
Trains Claude models; blocking opts out of training
Anthropic legacy training crawler
Legacy Anthropic crawler (deprecated); many sites still block it explicitly
Gemini training opt-out; blocking opts out of Gemini training & AI Overviews
Google's generic crawler for research & datasets (not Search indexing); often blocked alongside AI bots
Apple Intelligence training opt-out
Open dataset feeding most open-source model training
Doubao / TikTok training; often ignores robots.txt
Meta AI (Llama) training & data collection
Meta's older LLM training crawler (Llama datasets)
Alexa / Rufus training & retrieval
Cohere model training
Cohere legacy training-data crawler
Knowledge-graph data broker; blocking prevents content resale
Mistral model training crawler
you.com index + training crawler
Commercial data broker licensing content for AI training
Academic research crawler (Allen AI); builds open datasets (Dolma)
Huawei Pangu model training
Timpi independent web index (search & AI data)
DeepSeek model training / data collection
Check your site against all 40 →