Cloudflare Blocks AI Crawlers by Default on September 15 — What It Means for Your Site

On September 15, 2026, Cloudflare starts applying a new default on its network: AI training crawlers and AI agent crawlers are blocked by default on ad-supported pages of newly registered domains. Search crawlers stay allowed. It's the latest step in Cloudflare's "content independence" push — over a million Cloudflare customers had already opted into some form of AI crawler blocking, and a pay-per-crawl model is rolling out that lets publishers charge AI companies for access.

If your site runs on Cloudflare, this quietly changes the relationship between your robots.txt and what actually reaches your server.

Your robots.txt is no longer the whole story

Cloudflare's block happens at the network edge — it does not edit your robots.txt. Your file can say Allow: / for GPTBot while Cloudflare returns a 403 before the request ever reaches your application.

That creates a failure mode robots.txt checkers cannot see: the file says one thing, the edge does another. It's exactly why we built real-fetch verification into our checker — it sends actual requests with GPTBot, OAI-SearchBot and PerplexityBot user-agents and compares them against a browser baseline.

The pattern to watch: training blocked, search allowed

Cloudflare's default maps almost one-to-one onto the training/retrieval distinction:

• Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot…) → blocked by default
• Search crawlers (OAI-SearchBot, PerplexityBot…) → still allowed by default

For most sites that's the right default — you keep getting cited while opting out of training. But be aware it produces a pattern that looks like the classic "expensive mistake" in reverse: training crawlers blocked, retrieval crawlers allowed. The difference is intent, and edge-level evidence.

A note on Googlebot: parts of the SEO community worried that Googlebot could be swept up by the new defaults. Cloudflare classifies crawlers as Search, Agent and Training, and Google's main crawler (Googlebot) falls under Search. Don't take anyone's word for it, including ours — verify with real requests against your own domain.

What to do before September 15

• Baseline now. Run a free check and note today's result — which crawlers are allowed at the robots.txt level, and which actually reach your server.
• Watch for the flip. Turn on free monitoring; we re-check daily and email you when any crawler's access changes. A Cloudflare default silently flipping your visibility is exactly the change you want to catch.
• Decide deliberately. If you want AI citations, keep retrieval crawlers allowed (the default doesn't touch them). If you want to opt out entirely, Cloudflare's AI Crawl Control now gives you enforcement that robots.txt never could.

Not sure where your site stands? Run the free check — it covers all 40 AI crawlers we track, at both the robots.txt and the network level.