- ✓Normalize the declared token but keep the raw value for audit.
- ✓Use official IP, DNS, or signature verification when the provider publishes it.
- ✓Never invent certainty for undocumented or browser-like machine traffic.
Use an evidence hierarchy
Identity becomes more defensible as independent signals agree.
Begin with the raw user agent and match documented tokens. Look up the provider’s current verification method: published IP ranges, forward-confirmed reverse DNS, or another signed mechanism. Record whether verification succeeded, failed, or was unavailable.
Request patterns can support classification but should not replace identity. A crawler-like sequence can also be a scraper, monitor, or malicious actor.
| Signal | Strength | Use |
|---|---|---|
| User-agent token | Low alone | Candidate identity |
| Official live IP range | Strong when current | Network verification |
| Provider-supported DNS | Strong when validated | Network verification |
| Request behavior | Supporting | Purpose and anomaly context |
| Authentication / signature | Strong | Authorized agent identity |
Build a versioned classifier
Provider identities and verification endpoints change over time.
Store actor, provider, purpose, evidence method, confidence, documentation URL, rule version, and last verified date. Reprocess aggregates when a rule changes, while preserving the original raw evidence within retention limits.
Use Unknown, Suspected, Declared, and Verified states rather than a binary bot flag. High-impact authorization should require stronger identity than low-risk public retrieval.
- Raw and normalized user agent
- Provider and actor
- Purpose class
- Verification result
- Confidence
- Rule and documentation version
Separate analytics classification from access control
Knowing who sent a request does not automatically decide whether it should be allowed.
Access policy may depend on purpose, path, rate, contract, authorization, robots directives, and risk. Keep the identity result and policy decision as distinct fields.
Monitor spoofing, unexpected rate, identity failures, and requests to sensitive paths. Never grant broad access solely because a request claims a well-known bot token.
Evidence notePerplexity publishes live IP endpoints; Anthropic currently documents bot tokens but says it does not publish IP ranges. Verification must follow each provider’s actual guidance.
Methodology and verification.
Last verified August 17, 2026. The page is updated when the underlying analytics or provider documentation changes materially.
- 01
Reviewed the linked primary documentation and separated provider claims from observations a website can verify.
- 02
Kept human referrals, sampled answer visibility, machine requests, and modeled influence in separate evidence classes.
- 03
Marked limitations wherever the available source or request data cannot support a provider-level conclusion.
Verify the evidence.
Provider behavior and analytics definitions change. These are the primary references reviewed for this page.
Questions teams ask.
Can an AI bot user agent be spoofed?+
Yes. The User-Agent header is client supplied. Use current provider-supported verification when stronger confidence is needed.
What if the provider publishes no IP range?+
Record the identity as declared or suspected, apply policy appropriate to that confidence, and do not manufacture a verified label.
Should unknown bots be blocked?+
That depends on path sensitivity, rate, business policy, and risk. Unknown identity is an input to the decision, not a universal block rule.