Legacy Anthropic crawler (deprecated); many sites still block it explicitly
Your call — blocking is a legitimate choice
Claude-Web is Anthropic's training/data crawler. Blocking it opts your content out of Anthropic's model training and datasets. It does not remove you from AI answers — citations are governed by separate retrieval crawlers.
Common reasons to block: copyright or licensing policy, competitive sensitivity, paywalled content.
Common reasons to allow: maximum AI reach, research visibility, and being part of the datasets future models learn from.
User-agent: Claude-Web Allow: /
User-agent: Claude-Web Disallow: /
See whether Claude-Web — and 39 other AI crawlers — can reach your site today.