Not a crawler — a robots.txt control token with no separate HTTP user-agent. Controls whether crawled content may be used for Gemini training and Gemini grounding. Google states it does not affect Search inclusion or ranking.
| Operating Entity | |
| Primary Purpose | Training / Model Pre-training Crawler |
| Exact User-Agent String | Google-Extended |
| Robots Control Token | Yes (robots.txt policy token, no standalone HTTP user-agent) |
| Robots Protocol Status | No (strictly conforms to robots.txt RFC 9309 rules) |
| Official Documentation | https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers |
| Cloudflare Edge Default | Not applicable. Cloudflare and network firewalls cannot detect or block Google-Extended because it generates no network traffic. Enforcement occurs entirely within Google's indexing infrastructure. |
| Real Blocking Impact | Disallowing Google-Extended prevents your published content from being used to train Gemini models or assist in Gemini grounding. Google officially confirms that Google-Extended has no effect on Google Search inclusion, rankings, or snippet presentation. |
| Related Portfolio Bots | Googlebot, GoogleOther |
This entry has no HTTP user-agent of its own, so a user-agent fetch test cannot verify it — only your robots.txt directive applies.
Google-Extended is a robots.txt standalone control token introduced by Google in September 2023. It is NOT a web crawler and has NO standalone HTTP User-Agent. Instead, it serves as a policy declaration lever that instructs Google whether content crawled by Googlebot may be utilized for training Gemini models and powering Gemini grounding. As artificial intelligence engines transition from closed training runs toward live, real-time knowledge retrieval, understanding the mechanical distinctions between background model pre-training and conversational inference is paramount for modern digital publishers and web engineering teams.
Google-Extended does NOT generate HTTP web requests and possesses no distinct HTTP User-Agent. Crawling is performed exclusively by Googlebot or GoogleOther. During back-end index processing, Google evaluates the presence of Google-Extended in the target domain's robots.txt. If disallowed, Google omits the crawled content from Gemini training datasets and real-time Gemini grounding pipelines. When web servers receive incoming connections from Google-Extended, the underlying networking stacks evaluate whether the request adheres to documented IP subnets, whether DNS reverse-lookups validate operator provenance, and whether the targeted URLs are cleared by domain governance policies.
Data Processing & Ingestion Architecture: Content gathered by Googlebot is inspected against the Google-Extended directive. Disallowing it ensures text is strictly excluded from Gemini LLM pre-training, fine-tuning, and grounding queries, while retaining standard search indexing. For website operators, this architecture dictates whether your intellectual property is processed into static parameters of large language models or referenced dynamically during conversational question-answering sessions.
Cloudflare and Edge Firewall Status: Not applicable. Cloudflare and network firewalls cannot detect or block Google-Extended because it generates no network traffic. Enforcement occurs entirely within Google's indexing infrastructure. On September 15, 2026, Cloudflare introduced foundational changes to its default bot management system across all ad-supported tiers of newly created domains. If your origin server relies entirely on robots.txt files to govern Google-Extended, you may experience discrepancies where the edge firewall terminates the crawler connection with an HTTP 403 Forbidden code before your origin application or robots.txt parser is even invoked.
Verifying whether Google-Extended can successfully reach and process your web pages requires looking across three separate, non-overlapping architectural layers: the robots.txt policy layer, the network-level User-Agent HTTP probe layer, and the page-level indexability layer. Assuming that a passing result in one layer guarantees full visibility in another is a widespread engineering trap.
Step 1: Test the User-Agent via HTTP curl emulation
Execute the following command in your terminal to inspect how your web server, CDN, and WAF respond when a client identifies with the Google-Extended user agent string:
curl -A "Google-Extended/1.0" -I https://yoursite.com
Analyze the HTTP response headers carefully:
Step 2: Inspect Page-Level Index Directives
Even if Google-Extended is fully allowed to crawl a URL over HTTP, it cannot index or cite the page if the document carries a noindex directive in its response headers (X-Robots-Tag: noindex) or in its HTML header (<meta name="robots" content="noindex">). A crawlable page can still be completely invisible to search indexes.
Step 3: Run the Free Automated AI Crawler Visibility Check
Instead of manually running command-line probes and inspecting headers across multiple hosts, use our instant free AI crawler visibility checker on checkaibots.com. It simultaneously audits your website against Google-Extended and 39 other major AI crawlers, compares www versus naked domain robots.txt configurations, performs UA-level HTTP baseline probes, and flags accidental misconfigurations in seconds.
To grant Google-Extended full permission to crawl your website in accordance with the official RFC 9309 protocol, place an explicit User-agent declaration at the root of your domain (https://yoursite.com/robots.txt).
Standard Full-Site Access Rule:
User-agent: Google-Extended Allow: /
Selective Access (Protecting Sensitive Subdirectories):
If your website contains private administrative portals, checkout flows, internal search results, or user account dashboards, you can protect those sensitive paths while granting Google-Extended unrestricted access to public articles, blog posts, documentation, and product catalogs:
User-agent: Google-Extended Allow: / Disallow: /admin/ Disallow: /checkout/ Disallow: /internal/ Disallow: /api/ Disallow: /*?token=*
RFC 9309 Rules and Precedence: Under official robots.txt protocol rules, when multiple path rules match a given URL, the rule with the longest matching character length takes absolute precedence. In the event of a tie where both an Allow and a Disallow rule have identical string lengths, the Allow directive wins. Furthermore, avoid the common error of leaving an empty Disallow statement (Disallow:), which according to the specification acts as a complete permit for all paths.
If your organization's copyright policy, competitive licensing stance, or infrastructure budget requires restricting Google-Extended, you can signal an explicit block by adding the following directive to your robots.txt file:
User-agent: Google-Extended Disallow: /
Google-Extended is not a standalone web crawler and possesses no independent HTTP user-agent header. Adding a Disallow: / rule informs Google's backend systems not to utilize content already crawled by primary fetchers for the specific training or grounding purpose this token governs.
Because it generates no HTTP traffic, no firewall or curl test can block or test Google-Extended over the network. Google states this token does not affect inclusion in Google Search or search rankings.
Crawl Budget and Cache Invalidation: Once your robots.txt file is updated, automated crawlers typically require between 24 and 48 hours to refresh their cached copy of your domain's directives. Existing data already ingested into previously trained neural weights cannot be retroactively purged via robots.txt, but future training iterations and live indexing updates will respect the updated boundary.
A primary failure mode in modern search optimization is treating all crawlers from a given artificial intelligence operator as a monolithic entity. Operators such as OpenAI, Anthropic, Google, and Meta partition their crawling infrastructure into distinct functional categories: bulk model pre-training crawlers, real-time search citation indexers, and user-directed live fetchers.
The table below illustrates the exact architectural distinctions, operational objectives, and visibility implications across Google's crawler fleet and comparable industry counterparts:
| Crawler Token | Functional Role | Affects Citations? | Affects Training? | Cloudflare Default |
|---|---|---|---|---|
| Google-Extended (Current) | Training / Model Pre-training Crawler | No | Yes (Directly) | N/A |
| GoogleOther | Training Crawler | No | Yes (Training) | Blocked |
| Googlebot | Search Indexer | Yes (Citations) | No | Allowed |
The Expensive Mistake in AI SEO: When organizations implement blanket robots.txt disallows aiming to protect copyright, they frequently copy-paste blocklists that include both training crawlers and search indexers (such as blocking OAI-SearchBot while intending to block GPTBot). Disallowing a search indexer silently excises the domain from AI answers, preventing brand citations, referral clicks, and audience discovery while leaving model training opt-outs completely unaffected. Keep your training policies and search indexing permissions strictly decoupled.
Because Google-Extended is a robots.txt control token with no standalone HTTP User-Agent, attempting to verify it with network-level HTTP fetch tests will fail. Verification relies entirely on robots.txt syntax analysis.
Google has explicitly stated that Google-Extended governs Gemini products, not Google Search features like AI Overviews. AI Overviews are managed through Googlebot indexing and page-level directives such as nosnippet.
Placing Disallow rules under Googlebot instead of Google-Extended will de-index your site from Google Search entirely.
Google has stated clearly in its webmaster documentation that Google-Extended is not a ranking signal in Google Search.
Many sites serve completely different robots.txt files on www.yoursite.com and the bare domain yoursite.com. Automated crawlers treat these as separate origins; if you update one host but omit the other, crawlers will continue encountering old directives.
Because Google-Extended is a control token, not an HTTP user agent. Google crawls pages using Googlebot and checks the Google-Extended rule in your robots.txt during backend processing.
No. Google has clarified that Google-Extended governs Gemini training and grounding, but does not control AI Overviews in Google Search. AI Overviews are managed through Googlebot indexing and page-level directives like nosnippet.
Google's official documentation states unambiguously that Google-Extended is not used as a ranking signal and has no effect on your presence in Google Search.
Add 'User-agent: Google-Extended Disallow: /'. Ensure that you do not accidentally block Googlebot itself.
Google fetches your robots.txt, parses the Google-Extended directive, and stores that policy alongside your domain's index records in its internal database.