Robots.txt Checker
Use our free robots.txt checker and syntax validator to test your robots.txt file online. Verify crawler permissions across Googlebot, Bingbot, and 40+ leading AI crawlers (GPTBot, ClaudeBot, PerplexityBot), catch syntax errors, and prevent costly search and AI visibility disasters.
1. What is a Robots.txt Checker and Why is It Critical?
A robots.txt checker is a specialized developer and SEO testing utility that audits your website's /robots.txt file to ensure search engines and artificial intelligence crawlers can crawl your content according to your exact specifications. The robots.txt file serves as the universal gateway to your website, dictating which directories automated bots are allowed to fetch and which sections must remain off-limits.
Because robots.txt directives are processed automatically by thousands of web spiders every day, a single typographical error or misplaced character can have catastrophic repercussions. A stray slash in a Disallow: line can de-index an entire multi-million dollar e-commerce store overnight. Conversely, an incomplete rule can leave sensitive internal endpoints exposed to aggressive scraping. Employing an automated robots.txt checker helps prevent these disastrous misconfigurations by validating rules prior to deployment.
In 2026, the rise of answer engines—such as ChatGPT Search, Perplexity AI, Claude, and Google AI Overviews—has made a reliable robots.txt checker indispensable. Today, webmasters must manage not only traditional search crawlers like Googlebot and Bingbot, but also dozens of foundation model training bots and real-time retrieval agents. Using this testing suite ensures that your site stays protected without accidentally cutting off valuable AI search citation traffic.
2. Robots.txt Syntax Validation & RFC 9309 Rules Explained
In September 2022, the Internet Engineering Task Force (IETF) officially standardized the Robots Exclusion Protocol under RFC 9309. Our robots.txt checker evaluates your directives strictly against this RFC 9309 standard. Understanding the foundational syntax principles will help you interpret your validation results:
User-Agent Grouping
Directives in robots.txt apply to specific User-agents. Crawlers look for a block matching their specific name first (e.g., User-agent: GPTBot). If no matching group exists, they fall back to the generic wildcard group (User-agent: *). The tool verifies that your user-agent groupings are separated cleanly by blank lines.
Longest Match Precedence
When both Allow: and Disallow: directives match a given URL path, RFC 9309 mandates that the longest matching path string takes precedence. If two matching rules have identical length, the Allow directive wins. The parser accurately calculates string lengths to resolve directive conflicts.
Pattern Matching: Wildcards & Anchors
RFC 9309 formally codified two pattern matching characters: the asterisk (*), which designates any sequence of zero or more characters, and the dollar sign ($), which anchors the match to the exact end of the URL path (such as Disallow: /*.pdf$). Our validator tests these patterns against real URL paths.
Additionally, RFC 9309 requires that the file be named exactly robots.txt (in lowercase) and reside at the root of the domain. Placing directives in subfolders or using uppercase characters (like ROBOTS.TXT) causes crawlers to ignore the file entirely.
3. The 7 Most Common Robots.txt Mistakes (Checklist & Fixes)
Our diagnostic engine has analyzed tens of thousands of robots.txt files. Below is our definitive checklist of the seven most common syntax errors and misconfigurations identified by our robots.txt checker:
-
1. Missing Leading Slash in Disallow Paths: Writing
Disallow: admin/instead ofDisallow: /admin/violates RFC 9309 syntax. Relative paths without a leading forward slash cause unpredictable behavior across different crawler parsers. The scanner flags missing slashes immediately. -
2. The Empty Disallow Directive Misunderstanding: Site owners often write
User-agent: * Disallow:thinking it blocks all crawlers. In reality, an empty Disallow directive means "nothing is disallowed"—it functions as an explicit ALLOW for all URLs! To block crawlers, you must include a forward slash:Disallow: /. Our validator prevents this critical misconception. -
3. Consecutive User-Agent Lines Without Separation: Under RFC 9309, consecutive
User-agent:lines form a single combined group applying to all listed agents. However, legacy parsers often drop the first agent and only apply rules to the last one. The tool validates multi-agent blocks to guarantee parser compatibility. -
4. Using "Noindex" in Robots.txt: In the past, some webmasters included non-standard
Noindex: /path/directives in robots.txt. Google formally discontinued support for noindex in robots.txt in September 2019, and RFC 9309 does not support it. Directives containing noindex are completely ignored. Use HTTP headers or HTML meta tags instead, as recommended by our diagnostic guide. -
5. Case-Sensitivity Pitfalls: URL paths in robots.txt are case-sensitive. If your rule specifies
Disallow: /private/, web crawlers are still permitted to crawl/Private/or/PRIVATE/. Ensure your path casing matches your server's routing architecture. -
6. Blanket AI Blocking Hurting Search Visibility: Placing
User-agent: * Disallow: /blocks not only training scrapers, but also AI retrieval crawlers likeOAI-SearchBotandPerplexityBot. This eliminates your brand from ChatGPT Search answers and Perplexity summaries. Our robots.txt checker helps you separate training from retrieval. - 7. Stale CDN Caching of Robots.txt: CDNs and reverse proxies often cache robots.txt responses with aggressive TTLs. If you deploy an emergency fix to your robots.txt file, edge caching may continue serving the old blocked rules for days. The live scanner issues cache-bypassing requests to verify edge propagation.
4. How Robots.txt Affects Modern AI Crawlers (Search vs Training)
The greatest challenge in modern web crawling is distinguishing between different categories of artificial intelligence crawlers. A conventional audit only tests traditional search engines. Our advanced robots.txt checker categorizes AI bots into three distinct functional groups:
| AI Crawler Type | Prominent Crawlers | Primary Purpose | Consequence of Disallowing |
|---|---|---|---|
| Training Crawlers | GPTBot, ClaudeBot, CCBot, Bytespider |
Scrapes web content to train generative foundation models | Protects content from AI training; NO impact on search |
| Search & Retrieval Crawlers | OAI-SearchBot, PerplexityBot, Claude-SearchBot |
Indexes content for real-time search queries & user citations | Removes your site from AI answers & direct referral clicks |
| User-Triggered Fetchers | ChatGPT-User, Perplexity-User |
Fetches specific links requested by human chat users | Breaks link sharing & URL summarization in chat interfaces |
This distinction represents the single most consequential strategic decision in modern SEO. If an organization disallows GPTBot, it protects its intellectual property from being digested into OpenAI's foundation models. However, if it also disallows OAI-SearchBot, users asking ChatGPT for recommendations or product comparisons will never see the organization's website cited.
Furthermore, major content delivery networks have introduced automated AI blocking. In September 2024, Cloudflare began blocking AI training and agent crawlers by default on newly registered domains. This means your server might return an HTTP 403 Forbidden at the CDN edge before the crawler even reaches your robots.txt file! Our robots.txt checker detects these edge-level blocks, providing diagnostic clarity that standard syntax validators miss.
5. Recommended Robots.txt Templates for 2026
Based on RFC 9309 standards and verified by our engineering team, here are three best-practice templates:
Template A: Balanced AI Visibility (Block Training, Allow Search Retrieval)
# Allow search engines and AI retrieval bots to index content
User-agent: *
Allow: /
# Block AI foundation model training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# Keep AI search retrieval crawlers open for citation traffic
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# XML Sitemap declaration
Sitemap: https://yourdomain.com/sitemap.xmlTemplate B: Open Access (Maximum Search & AI Visibility)
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Sitemap: https://yourdomain.com/sitemap.xmlTemplate C: Complete Crawl Restriction (Staging / Development Environments)
User-agent: *
Disallow: /6. Frequently Asked Questions About Robots.txt Checker & Protocol Rules
What does an online robots.txt checker do?
An online robots.txt checker is an automated diagnostic utility that fetches and parses your website's /robots.txt file. It checks your file against the official RFC 9309 Robots Exclusion Protocol, tests syntax compliance, evaluates Allow and Disallow directive matching, validates User-agent groupings, checks XML sitemap declarations, and tests access permissions across search engines (like Googlebot and Bingbot) and modern AI crawlers (like GPTBot, ClaudeBot, and PerplexityBot).
Where must the robots.txt file be located on a web server?
The robots.txt file must always be placed at the top-level root directory of your website's domain or subdomain (e.g. https://example.com/robots.txt). Search engine crawlers and testing utilities will never look for robots.txt files in subdirectories such as https://example.com/blog/robots.txt or https://example.com/assets/robots.txt. Furthermore, robots.txt is host-specific: example.com and www.example.com each require their own valid robots.txt file.
Does an empty Disallow: directive allow or block web crawlers?
An empty Disallow: directive with no path specified (e.g. 'Disallow:') tells web crawlers that there are NO disallowed paths on your site — meaning it functions as an explicit ALLOW for all URLs! This is one of the most common points of confusion identified by our tool. If you intend to block all crawlers from your entire domain, the correct syntax is 'Disallow: /' with a forward slash.
Can I use robots.txt to remove private or sensitive pages from Google Search?
No! Robots.txt is a crawl prevention mechanism, not an indexing prevention mechanism. If an external website links to a URL that is disallowed in your robots.txt, Googlebot may still index that URL in search results without crawling its content (often displaying the URL with the snippet 'No information is available for this page'). To guarantee that a page is never indexed in search results, allow Googlebot to crawl the page and include a '<meta name="robots" content="noindex">' tag in the HTML head or an 'X-Robots-Tag: noindex' HTTP response header.
Why do AI answers still cite my content after blocking GPTBot in robots.txt?
OpenAI operates separate crawlers for distinct purposes. GPTBot is OpenAI's training crawler used to collect datasets for future foundation models (GPT-4, GPT-5). In contrast, OAI-SearchBot is the retrieval crawler that indexes content for ChatGPT Search and real-time citations. If you only blocked GPTBot in your robots.txt file, OAI-SearchBot and ChatGPT-User can still fetch and cite your pages in real-time answers. A modern robots.txt checker helps you verify permissions for both training and retrieval bots independently.
How does this robots.txt checker detect CDN and WAF edge blocks?
Our testing utility goes beyond static text parsing. In addition to analyzing the syntax of your /robots.txt file, our tool issues real-time HTTP requests simulating various crawler user agents (including Googlebot, GPTBot, and browser baselines). If your robots.txt specifies 'Allow: /' but your CDN (such as Cloudflare, Fastly, or AWS CloudFront) returns an HTTP 403 Forbidden due to an automated AI bot mitigation policy, our robots.txt checker highlights this hidden conflict immediately.