ChatGPT, Perplexity & Claude only cite what their crawlers can reach. Check your robots.txt against 40 major AI crawlers and see who can — and who can't — see you.
Search is shifting from link networks to answer networks. If AI crawlers can't reach you, you don't exist in AI search.
58.5% of US Google searches ended without a click (SparkToro / Datos, 2024). In G2's 2025 buyer survey, using AI search as a research source ranged from about 35% at small companies to 55%+ at larger ones.
Blocking a retrieval crawler while trying to block training silently removes you from AI answers.
This checker covers 40 named crawlers; the wider ecosystem runs to well over a hundred user agents and keeps growing, so a robots.txt that's fine today can break tomorrow.
Sources: SparkToro / Datos zero-click study (2024) · G2 buyer behavior survey (2025). Figures are third-party research, not measurements of your site.
One check, three things revealed.
Any public website — yours, a client's, a competitor's.
Fetch and parse it against the full AI crawler roster.
Per-crawler access, the rule that caused it, and fix suggestions.
Training, retrieval and user-triggered — we tell them apart so you don't make the expensive mistake.
Run a check, then subscribe to daily monitoring — we'll email you the moment your AI visibility changes.
Check your site nowPractical guides on AI search, AEO and the crawlers that decide who gets cited.
Everything you need to know before you check your site.
Yes. Single checks are completely free with no signup. You only need an account if you want continuous monitoring and email alerts.
We check 40 major AI crawlers across three types: training (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, FacebookBot…), retrieval (OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, Applebot…) and user-triggered (ChatGPT-User, Claude-User, Perplexity-User, DeepSeek-User).
GPTBot is OpenAI's training crawler — blocking it opts you out of model training. OAI-SearchBot is the retrieval crawler that gets you cited in ChatGPT answers — blocking it removes you from ChatGPT results without affecting training. Blocking the wrong one is the most common (and most expensive) mistake.
robots.txt is a cooperative protocol — well-behaved crawlers (most major AI companies) respect it. Two caveats we surface in the results: some operators state robots.txt may not apply to their user-triggered fetchers (for example Perplexity-User and ChatGPT-User), and some entries in our list are robots.txt control tokens rather than crawlers (for example Google-Extended and Applebot-Extended have no HTTP user-agent of their own). Also, a robots.txt that says "allow" doesn't guarantee access: CDNs and WAFs can still block at the HTTP layer — that's what the UA-level fetch check below looks for.
Beyond reading robots.txt, we send HTTP requests to your page with the user-agents of GPTBot, OAI-SearchBot and PerplexityBot, plus a browser baseline. If the browser request succeeds but an AI user-agent gets 403 or a mitigation-style 5xx, that points to a CDN/WAF rule blocking by user-agent — something a robots.txt check alone would never reveal. Be clear on the limit: this tests user-agent handling only. It does not prove that the real crawler can or cannot reach you, because real crawlers may also be evaluated by verified-bot identity, published IP ranges and behaviour.
Our check shows the exact robots.txt rule blocking each crawler, plus a suggested fix (usually an explicit Allow for the retrieval crawler you need). Edit your robots.txt, then re-run the check to confirm the fix took effect.
Because it still belongs in your robots.txt decision. Google-Extended is a control token with no HTTP user-agent — Google uses it to decide whether crawled content may be used for Gemini training and grounding, and states it does not affect Search inclusion or ranking. We label it as a token so you don't mistake it for a bot you can test with a user-agent request.
No — three separate things have to line up, and they are reported as separate layers. (1) Crawl permission: your robots.txt lets the crawler in. (2) Indexing permission: the page must not carry a page-level noindex or an X-Robots-Tag that excludes it — we read both from the response headers and the first 64 KB of HTML, and a crawlable page can still be marked not-indexable. (3) Citation: the platform must then choose to reference you, which depends on content quality and relevance, not on crawler access. Treat a passing check as "nothing is blocking the crawler", not as a promise of citations.
www.example.com and example.com can be different hosts with different robots.txt files — sometimes one exists and the other doesn't. We fetch the other host's robots.txt and compare the resulting Allow/Block decisions across all crawlers we track, so a mismatch you'd otherwise miss shows up as a table of differing crawlers. It costs one extra robots.txt request and does not affect the rest of the check.
Once subscribed, we check your site once a day, keep recent check history (up to 90 checks, with the last 30 shown on your history page), and email you after a daily check finds that an AI crawler's robots.txt access changed — so an accidental block doesn't stay silent. Alerts run at the daily check, not in real time.
Anonymous free checks are processed and shown instantly; we do not store those results. Three things are stored only when you explicitly act: subscribing to monitoring saves your site's history; creating a shareable report saves that result for 30 days; and unsubscribing deletes your subscription and its entire history.
Yes. The checker works on any public website. Many SEO professionals use it to audit client sites and compare their AI visibility against competitors.