- ✓Decide policy by purpose before copying bot names into a file.
- ✓Validate exact tokens against current official documentation.
- ✓Use authentication and server controls for sensitive or transactional paths.
Write the decision table first
The robots file should implement a policy the organization can explain.
For each actor, document provider, purpose, desired access, affected paths, business rationale, owner, review date, and enforcement beyond robots.txt. Consider search visibility, model development, real-time retrieval, licensing, capacity, and user-directed tasks separately.
Apply the minimum necessary restriction. Blocking a search crawler and a training crawler can have different consequences according to provider documentation.
| Purpose | Question to decide |
|---|---|
| Search / answer indexing | Should public pages be discoverable and cited? |
| Model development | May content be collected for future model training? |
| User retrieval | May an assistant fetch public content for a user now? |
| Delegated action | Which authenticated tasks may an agent perform? |
Use documented tokens and standard syntax
Token spelling and rule placement matter.
Place the file at the root of each hostname and use groups that match the exact robots user-agent token. Test important public and restricted paths after deployment. Keep comments concise and maintain the source documentation outside the file.
Google documents Google-Extended as a control token without a separate request user agent. Perplexity and Anthropic document purpose-specific tokens. Follow their current pages rather than an old copied list.
- One root file per hostname
- Exact documented token
- Normalized paths
- Explicit allow / disallow
- Versioned policy owner
- Post-deployment request monitoring
User-agent: ExampleAICrawler
Disallow: /private/
Allow: /public/Know what robots.txt cannot do
Robots rules are public crawl preferences, not a security boundary.
A non-compliant client can ignore the file, a spoofed identity can claim another token, and a user-requested fetcher may have provider-specific behavior. Do not expose secrets or rely on robots.txt to protect authenticated content.
Enforce sensitive paths with access control, rate limits, WAF rules, and application authorization. Measure allowed, denied, and failed requests so policy behavior can be audited.
Evidence noteReview legal and content-licensing implications with qualified counsel; this guide is technical and analytical, not legal advice.
Methodology and verification.
Last verified August 17, 2026. The page is updated when the underlying analytics or provider documentation changes materially.
- 01
Reviewed the linked primary documentation and separated provider claims from observations a website can verify.
- 02
Kept human referrals, sampled answer visibility, machine requests, and modeled influence in separate evidence classes.
- 03
Marked limitations wherever the available source or request data cannot support a provider-level conclusion.
Verify the evidence.
Provider behavior and analytics definitions change. These are the primary references reviewed for this page.
Questions teams ask.
Does robots.txt block every AI crawler?+
No. It communicates policy to compliant clients. Security and authorization require server-side controls.
Should every AI bot be blocked?+
There is no universal answer. Decide separately for search, training, retrieval, and delegated tasks based on business, legal, visibility, and capacity goals.
Does blocking Google-Extended remove a site from Google Search?+
Google states that Google-Extended does not affect inclusion or ranking in Google Search.