- ✓Collect server-side request evidence before classifying the actor.
- ✓Verify identity with provider-supported network or signature evidence where available.
- ✓Measure access, response, rate, content type, and task outcome—not request count alone.
Collect the minimum useful request fields
The log needs enough context to classify and diagnose a request without retaining unnecessary sensitive data.
Record time, hostname, method, normalized path, response status, bytes, latency, declared user agent, verified network result where available, and the policy decision. Redact sensitive query values and define a short raw-log retention period.
Do not rely on the user-agent string alone for high-impact access decisions. It is self-declared and can be spoofed.
- Time and host
- Method and normalized path
- Status, bytes, and latency
- Declared user agent
- Verification result
- Policy and authorization result
Classify purpose before interpreting volume
Training, search indexing, real-time retrieval, and user-delegated action are different activities.
Map current documented tokens into purpose classes and retain Unknown for requests that cannot be verified. Keep provider, actor name, purpose, confidence, and documentation version as separate fields.
A user-triggered fetch can have business relevance without becoming a human visit. A high crawler count can show access or interest, but not recommendation, citation, or conversion.
| Class | Example purpose | Primary metric |
|---|---|---|
| Training crawler | Collect model-development material | Pages and bytes fetched |
| Search crawler | Build answer/search index | Freshness and successful fetches |
| User fetcher | Retrieve a page for a prompt | Requests and response success |
| Delegated agent | Attempt a permitted task | Task completion and failure |
Report behavior and impact
Request volume is useful only with the pages, outcomes, cost, and policy around it.
Show unique verified actors, requests, successful responses, pages fetched, content categories, bandwidth, error rates, blocked requests, and rate trends. For agents, add discovery, authorization, attempt, completion, and failure stages.
Compare machine access with citations or human referrals only as separate series by page and time. Investigate correlations without turning them into a blended audience count.
Evidence noteRetain the raw evidence needed to audit classification, but minimize and expire it according to a documented purpose.
Methodology and verification.
Last verified August 17, 2026. The page is updated when the underlying analytics or provider documentation changes materially.
- 01
Reviewed the linked primary documentation and separated provider claims from observations a website can verify.
- 02
Kept human referrals, sampled answer visibility, machine requests, and modeled influence in separate evidence classes.
- 03
Marked limitations wherever the available source or request data cannot support a provider-level conclusion.
Verify the evidence.
Provider behavior and analytics definitions change. These are the primary references reviewed for this page.
Questions teams ask.
Why not measure AI bots with a browser analytics tag?+
Many bots do not execute client JavaScript, and client tags are designed around visitor events. Server or CDN logs capture the underlying HTTP requests.
Can user agents be trusted?+
Not by themselves. Use official IP lists, reverse DNS, signatures, or other provider-supported verification where available, and retain an Unknown class.
Should bots be included in pageviews?+
No. Report automated requests separately from human pageviews, sessions, engagement, and conversion.