Practical guides on AI search visibility, AEO, and the crawlers that decide who gets cited.
Starting September 15, 2026, Cloudflare begins blocking AI training and agent crawlers by default on new domains — even if your robots.txt says allow. Here's what changes, what it means for your AI citations, and how to check which side your site is on.
Read article →OAI-SearchBot is OpenAI's retrieval crawler — the one that decides whether your site gets cited in ChatGPT answers. Blocking it costs you citations. Here's what it is and how to check yours.
Read article →If your robots.txt blocks GPTBot, you've opted out of OpenAI model training — which may be intentional. The real danger is accidentally blocking the wrong crawler along the way. Here's how to fix it the right way.
Read article →Three fast checks to see whether ChatGPT can access and cite your site — and what to fix if it can't. No tools needed for the first two.
Read article →Want to know if GPTBot, ClaudeBot or PerplexityBot are visiting your site? Three practical methods: server logs, robots.txt checks, and a crawler visibility scan.
Read article →GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot... there are now dozens of AI crawlers, and they do very different jobs. Here's what each type does and why it matters for your site.
Read article →Trying to block GPTBot (training) but accidentally blocking OAI-SearchBot (retrieval)? That's the mistake that quietly removes you from ChatGPT answers while doing nothing to protect your content.
Read article →Three ways to check whether your site is visible to AI search: the quick robots.txt check, the UA-level fetch check, and what to do when you find a problem.
Read article →