User-initiated fetch. Perplexity states this fetcher generally ignores robots.txt rules; it is not a crawler and is not used for training.
| Operating Entity | Perplexity |
| Primary Purpose | User-triggered Interactive Fetcher |
| Exact User-Agent String | Perplexity-User |
| Robots Control Token | No (active HTTP web crawler) |
| Robots Protocol Status | Yes (operator states robots.txt directives may not apply to user-directed fetches) |
| Official Documentation | https://docs.perplexity.ai/docs/resources/perplexity-crawlers |
| Cloudflare Edge Default | Handled under AI Agent traffic and manageable via Cloudflare WAF User-Agent and IP rules. |
| Real Blocking Impact | Blocking Perplexity-User prevents Perplexity from inspecting your web pages in real time when users ask questions about your links. If you must block it, you must use WAF rules, not robots.txt alone. |
| Related Portfolio Bots | PerplexityBot |
Perplexity states robots.txt may not apply to this user-triggered fetch, so a Disallow here is not a guarantee of blocked access.
Perplexity-User is Perplexity's user-initiated fetching agent, dispatched when an end-user explicitly submits a query or clicks a link that requires real-time page inspection within Perplexity. As artificial intelligence engines transition from closed training runs toward live, real-time knowledge retrieval, understanding the mechanical distinctions between background model pre-training and conversational inference is paramount for modern digital publishers and web engineering teams.
Perplexity's official documentation states explicitly that because requests are initiated directly by human users, Perplexity-User generally ignores robots.txt rules. Therefore, a robots.txt Disallow directive is NOT a reliable way to block it. When web servers receive incoming connections from Perplexity-User, the underlying networking stacks evaluate whether the request adheres to documented IP subnets, whether DNS reverse-lookups validate operator provenance, and whether the targeted URLs are cleared by domain governance policies.
Data Processing & Ingestion Architecture: Downloaded web pages are rendered into the user's active search context to extract answers to their immediate prompt. The text is not permanently archived into training datasets. For website operators, this architecture dictates whether your intellectual property is processed into static parameters of large language models or referenced dynamically during conversational question-answering sessions.
Cloudflare and Edge Firewall Status: Handled under AI Agent traffic and manageable via Cloudflare WAF User-Agent and IP rules. On September 15, 2026, Cloudflare introduced foundational changes to its default bot management system across all ad-supported tiers of newly created domains. If your origin server relies entirely on robots.txt files to govern Perplexity-User, you may experience discrepancies where the edge firewall terminates the crawler connection with an HTTP 403 Forbidden code before your origin application or robots.txt parser is even invoked.
Verifying whether Perplexity-User can successfully reach and process your web pages requires looking across three separate, non-overlapping architectural layers: the robots.txt policy layer, the network-level User-Agent HTTP probe layer, and the page-level indexability layer. Assuming that a passing result in one layer guarantees full visibility in another is a widespread engineering trap.
Step 1: Test the User-Agent via HTTP curl emulation
Execute the following command in your terminal to inspect how your web server, CDN, and WAF respond when a client identifies with the Perplexity-User user agent string:
curl -A "Perplexity-User/1.0" -I https://yoursite.com
Analyze the HTTP response headers carefully:
Step 2: Inspect Page-Level Index Directives
Even if Perplexity-User is fully allowed to crawl a URL over HTTP, it cannot index or cite the page if the document carries a noindex directive in its response headers (X-Robots-Tag: noindex) or in its HTML header (<meta name="robots" content="noindex">). A crawlable page can still be completely invisible to search indexes.
Step 3: Run the Free Automated AI Crawler Visibility Check
Instead of manually running command-line probes and inspecting headers across multiple hosts, use our instant free AI crawler visibility checker on checkaibots.com. It simultaneously audits your website against Perplexity-User and 39 other major AI crawlers, compares www versus naked domain robots.txt configurations, performs UA-level HTTP baseline probes, and flags accidental misconfigurations in seconds.
To grant Perplexity-User full permission to crawl your website in accordance with the official RFC 9309 protocol, place an explicit User-agent declaration at the root of your domain (https://yoursite.com/robots.txt).
Standard Full-Site Access Rule:
User-agent: Perplexity-User Allow: /
Selective Access (Protecting Sensitive Subdirectories):
If your website contains private administrative portals, checkout flows, internal search results, or user account dashboards, you can protect those sensitive paths while granting Perplexity-User unrestricted access to public articles, blog posts, documentation, and product catalogs:
User-agent: Perplexity-User Allow: / Disallow: /admin/ Disallow: /checkout/ Disallow: /internal/ Disallow: /api/ Disallow: /*?token=*
RFC 9309 Rules and Precedence: Under official robots.txt protocol rules, when multiple path rules match a given URL, the rule with the longest matching character length takes absolute precedence. In the event of a tie where both an Allow and a Disallow rule have identical string lengths, the Allow directive wins. Furthermore, avoid the common error of leaving an empty Disallow statement (Disallow:), which according to the specification acts as a complete permit for all paths.
If your organization's copyright policy, competitive licensing stance, or infrastructure budget requires restricting Perplexity-User, you can signal an explicit block by adding the following directive to your robots.txt file:
User-agent: Perplexity-User Disallow: /
Perplexity officially documents that because Perplexity-User is triggered directly by end-user requests, standard robots.txt rules may not be observed as a hard barrier. Disallowing it in robots.txt alone is not a reliable way to block it.
To reliably restrict Perplexity-User, enforce network-layer access control policies combining User-Agent header inspection and Perplexity's published IP CIDR ranges in your CDN or web application firewall (Cloudflare WAF, AWS WAF, or Nginx).
Crawl Budget and Cache Invalidation: Once your robots.txt file is updated, automated crawlers typically require between 24 and 48 hours to refresh their cached copy of your domain's directives. Existing data already ingested into previously trained neural weights cannot be retroactively purged via robots.txt, but future training iterations and live indexing updates will respect the updated boundary.
A primary failure mode in modern search optimization is treating all crawlers from a given artificial intelligence operator as a monolithic entity. Operators such as OpenAI, Anthropic, Google, and Meta partition their crawling infrastructure into distinct functional categories: bulk model pre-training crawlers, real-time search citation indexers, and user-directed live fetchers.
The table below illustrates the exact architectural distinctions, operational objectives, and visibility implications across Perplexity's crawler fleet and comparable industry counterparts:
| Crawler Token | Functional Role | Affects Citations? | Affects Training? | Cloudflare Default |
|---|---|---|---|---|
| Perplexity-User (Current) | User-triggered Interactive Fetcher | No | No | Blocked |
| PerplexityBot | Search Indexer | Yes (Citations) | No | Allowed |
The Expensive Mistake in AI SEO: When organizations implement blanket robots.txt disallows aiming to protect copyright, they frequently copy-paste blocklists that include both training crawlers and search indexers (such as blocking OAI-SearchBot while intending to block GPTBot). Disallowing a search indexer silently excises the domain from AI answers, preventing brand citations, referral clicks, and audience discovery while leaving model training opt-outs completely unaffected. Keep your training policies and search indexing permissions strictly decoupled.
Perplexity officially documents that Perplexity-User generally ignores robots.txt rules. You must enforce blocking at the network/WAF layer.
Blocking PerplexityBot stops search indexation, whereas Perplexity-User handles single-user link lookups.
Configure firewall rules matching both User-Agent and Perplexity's published IP blocks for reliable enforcement.
Blocking user-directed fetches breaks citations when users ask Perplexity to summarize your documentation.
Many sites serve completely different robots.txt files on www.yoursite.com and the bare domain yoursite.com. Automated crawlers treat these as separate origins; if you update one host but omit the other, crawlers will continue encountering old directives.
Because Perplexity-User acts on the direct instruction of an end-user viewing a page, functioning like a browser proxy rather than an autonomous crawler.
Create an edge WAF rule in Cloudflare, AWS WAF, or Nginx matching 'http.user_agent contains "Perplexity-User"' and block the request with HTTP 403.
No. Perplexity documents that it is not a crawler and is not used to train foundation models.
Yes. Allow PerplexityBot in robots.txt for search visibility while creating a WAF rule to block Perplexity-User.
Return an HTTP 403 Forbidden or 401 Unauthorized code at the edge firewall.