- ✓A user-agent token states an identity; stronger policy decisions may require provider-supported verification.
- ✓Classify search, model-development, user-retrieval, and control tokens separately.
- ✓Keep automated requests out of human visitor, engagement, and conversion metrics.
Documented machine identities and control tokens
The same provider can publish several tokens because the requests serve different purposes.
OpenAI, Perplexity, Anthropic, and Google document search crawlers, model-development crawlers, user-requested retrieval, generic fetchers, or robots product controls. The declared purpose should be retained because it affects analytics, policy, and the business decision.
Google-Extended is included as a policy control rather than an HTTP user-agent identity. Always follow the linked current provider documentation before writing a WAF, robots, or legal policy.
| Token | Provider | Purpose class | Verification / classification note |
|---|---|---|---|
| GPTBot | OpenAI | Model development | Use current OpenAI documentation |
| OAI-SearchBot | OpenAI | Search | Use current OpenAI documentation |
| ChatGPT-User | OpenAI | User-requested retrieval | Automated request, not a human visit |
| PerplexityBot | Perplexity | Search | Provider publishes live IP data |
| Perplexity-User | Perplexity | User-requested retrieval | Provider publishes live IP data |
| ClaudeBot | Anthropic | Model development | Token documented by Anthropic |
| Claude-SearchBot | Anthropic | Search | Token documented by Anthropic |
| Claude-User | Anthropic | User-requested retrieval | Automated request, not a human visit |
| Google-Extended | Robots product control | No separate HTTP user agent | |
| GoogleOther | Generic product crawler | Follow Google crawler verification |
Classify purpose before volume
Ten thousand search-crawler requests and ten thousand attempted agent tasks do not describe the same activity.
Use purpose classes such as search or answer indexing, model development, user-triggered retrieval, monitoring, delegated action, and unknown. Store the provider’s stated purpose and the site’s observed behavior separately.
Unknown must remain a valid class. A browser-like user agent, spoofed token, or undocumented system should not inherit a trusted provider classification because its requests look similar.
| Purpose | Useful question |
|---|---|
| Search / answer indexing | Which public pages are being discovered? |
| Model development | Which content is being collected under current policy? |
| User retrieval | Which resources are fetched in response to a user? |
| Delegated action | Which authorized tasks are attempted and completed? |
| Unknown | What requires investigation or restriction? |
Use layered identity verification
A user-agent match is useful evidence but can be spoofed.
Begin with the full raw user-agent string and exact token. Add provider-supported IP, reverse DNS, signature, or published-range verification where available and appropriate. Record the verification method, result, timestamp, and rule version.
Do not invent a network verification method when the provider does not publish one. In that case, retain lower confidence and combine the declared token with behavior, rate, authentication, and policy evidence for analysis.
- Exact token match
- Full raw user agent
- Provider-supported network evidence
- Behavior and request sequence
- Authentication / authorization
- Confidence and timestamp
Store a policy-ready machine record
A bot-name column cannot explain what happened or whether the request was allowed.
Record provider, token, purpose, raw user agent, source network evidence, verification result, requested host and path, method, response status, bytes, latency, robots decision, WAF decision, authentication state, and time under an approved retention policy.
Aggregate for routine reporting and restrict access to detailed logs. Redact query strings, credentials, payloads, and personal data that are not necessary for the stated operational purpose.
provider · token · purpose · confidence · verification_method · verification_result
host · path_class · method · status · bytes · latency · policy_decision · timestampReport machine activity with machine denominators
Machine requests belong beside human analytics, not inside it.
Show verified and declared actors, requests, machine sessions, pages requested, purpose, status, denial reason, latency, retries, and rate limits. For agents, add discovery, authorization, attempt, completion, and failure stages where the site can observe them.
Do not calculate human engagement, bounce, conversion, or audience share from machine requests. Compare page-level machine access with citations or referrals only as contextual evidence.
| Machine metric | Denominator |
|---|---|
| Request share | Automated requests |
| Successful fetch rate | Eligible machine requests |
| Denied request rate | Policy-evaluated requests |
| Agent completion rate | Authorized attempted tasks |
Match the control to the risk
robots.txt communicates crawl preferences; it is not authentication or a security boundary.
Use robots rules for compliant public crawl preferences, WAF and rate limits for capacity and abuse control, authentication for identity, authorization for permitted actions, and application checks for consequential tasks. Keep the business rationale and owner beside every rule.
Monitor allowed, denied, challenged, and failed requests after changes. A policy that cannot be observed and reviewed will drift as provider identities and product behavior change.
- robots.txt preference
- WAF and rate limits
- Authentication
- Action authorization
- Sensitive-path protection
- Post-change monitoring
Change log and correction policy
A maintained directory should show what changed and how the correction was verified.
Version 2.0 adds layered verification, a policy-ready log schema, purpose-specific analytics, and control guidance. The visible table and downloadable CSV share the same last-verified date.
Send corrections to hello@wandered.io with the provider documentation, exact token, stated purpose, verification method, and date. Do not submit credentials, private IP logs, personal data, or sensitive request payloads.
Evidence noteThis directory supports classification and policy review; it must not be the sole control protecting sensitive resources.
Methodology and verification.
Last verified August 17, 2026. The page is updated when the underlying analytics or provider documentation changes materially.
- 01
Used provider-owned crawler documentation and the Robots Exclusion Protocol as primary references.
- 02
Separated declared identity, stated purpose, observed behavior, and independent verification instead of treating a token as proof.
- 03
Mapped each purpose to an analytics denominator and appropriate control so the directory supports real operational decisions.
Added purpose taxonomy, layered verification, machine log schema, analytics denominators, control guidance, and correction requirements.
Initial major-provider token directory and downloadable CSV.
Verify the evidence.
Provider behavior and analytics definitions change. These are the primary references reviewed for this page.
Questions teams ask.
Is a user-agent match enough to verify an AI bot?+
No. It identifies a declared token. Use provider-supported network or signature evidence when available and retain confidence.
Why is Google-Extended listed without a user agent?+
It is a robots.txt product control token, not a separate HTTP request identity.
Are human AI referrals included?+
No. This directory covers machine identities. Human referral sources are maintained separately.
Should ChatGPT-User or Claude-User count as a visitor?+
No. They are documented automated fetchers used in response to a user request, not human browser sessions.
How should unknown automation be classified?+
Keep it Unknown with the available request, behavior, and confidence evidence. Do not assign a trusted provider solely from similarity.