Legacy Anthropic training token, superseded by ClaudeBot. Still found in older robots.txt blocks.
Your call — blocking is a legitimate choice
anthropic-ai is Anthropic's training/data crawler. Blocking it opts your content out of Anthropic's model training and datasets. For this crawler, training and citation are separate decisions — citations are governed by the operator's retrieval crawler.
Common reasons to block: copyright or licensing policy, competitive sensitivity, paywalled content.
Common reasons to allow: maximum AI reach, research visibility, and being part of the datasets future models learn from.
User-agent: anthropic-ai Allow: /
User-agent: anthropic-ai Disallow: /
Community source Community-maintained list — last reviewed 2026-09-12
https://github.com/ai-robots-txt/ai.robots.txt
We have not verified this entry against the operator's own documentation. Treat the description as indicative, and check the operator's site before relying on it.
"Should you block it?" below is our own interpretation, not the operator's recommendation.
See whether anthropic-ai — and 39 other AI crawlers — can reach your site today.