ByteDance data collection. Frequently reported to ignore robots.txt — community-reported; ByteDance publishes limited crawler documentation.
Blocking is common — and only partly effective
Bytespider is ByteDance's data collection crawler. ByteDance publishes limited crawler documentation, so this entry is community-sourced rather than vendor-confirmed.
Blocking it is a common and legitimate choice, but it has frequently been reported to ignore robots.txt. If you must stop it, enforce at the network layer (User-Agent + IP rules) rather than relying on robots.txt alone.
User-agent: Bytespider Allow: /
User-agent: Bytespider Disallow: /
Community source Community-maintained list — last reviewed 2026-09-12
https://github.com/ai-robots-txt/ai.robots.txt
We have not verified this entry against the operator's own documentation. Treat the description as indicative, and check the operator's site before relying on it.
"Should you block it?" below is our own interpretation, not the operator's recommendation.
See whether Bytespider — and 39 other AI crawlers — can reach your site today.