- ✓Classify the actor before interpreting the volume.
- ✓Measure a task funnel from discovery through completion.
- ✓Treat identity and intent claims as evidence with confidence, not unquestioned fact.
Crawler, retrieval bot, or agent?
A request count says very little until the system knows what kind of machine made it and why.
Traditional crawlers primarily discover and index public pages. Retrieval systems may fetch current content to support an answer. Training collectors gather material under their own policies. A user-delegated agent attempts a task for a person, such as checking availability, gathering product details, or completing a permitted workflow.
These categories can overlap in implementation, and self-declared user agents can be incomplete or misleading. Classification should combine declared identity, verified network evidence where available, request behavior, authentication state, and the site policy that applied at the time.
- Declared actor or user agent
- Verified network or signature evidence
- Requested resources and sequence
- Authentication and authorization context
- Applicable robots or agent policy
A useful agent task funnel
For an agent that is trying to do something, task progress is more informative than raw request volume.
A practical funnel starts with capability discovery: did the agent find the relevant interface, document, or action? Selection records whether it chose a supported capability. Authorization shows whether required identity or permission was established. Attempted means the task began. Completed means the intended, permitted result was achieved.
The drop between stages is often the most useful signal. Discovery failures point to machine-readable navigation or documentation. Authorization failures indicate identity and permission friction. Attempt failures can reveal ambiguous inputs or unsupported states. Completion failures should be tied to a reason that a human operator can investigate.
Evidence noteA high completion rate is only positive when the completed action was authorized and aligned with user intent.
Metrics for agent operations
Separate operational health from business value and from simple crawler activity.
Report sessions or tasks by verified actor class, then show discovery, authorization, attempt, completion, and failure counts. Latency, retry behavior, rate limits, and failure reasons help engineering teams improve reliability. Sensitive or consequential actions also need auditability: who delegated the task, what permission applied, and what result was produced.
Crawler requests belong in a neighboring view with their own purpose categories—search indexing, AI retrieval, training, monitoring, and unknown. This prevents a spike in indexing requests from being mistaken for customer demand or successful agent adoption.
- Distinct actors and machine sessions
- Task attempts and completion rate
- Authorization success and denial reasons
- Failure stage and reason
- Latency, retries, and rate-limit events
- Crawler volume by purpose and policy
Analytics should reinforce agent governance
Measurement is part of the control surface, not a reason to weaken it.
Agent analytics should make permitted and denied behavior visible without recording unnecessary sensitive content. Logs need a defined retention period, access controls, and redaction rules. High-impact actions should have stronger evidence than low-risk public retrieval.
Teams should be able to answer four questions: which actor made the request, on whose behalf, under which policy, and with what result. When any answer is unavailable, the interface should say so. The objective is understandable automation, not a false appearance of certainty.
Methodology and verification.
Last verified August 17, 2026. The page is updated when the underlying analytics or provider documentation changes materially.
- 01
Reviewed the linked primary documentation and separated provider claims from observations a website can verify.
- 02
Kept human referrals, sampled answer visibility, machine requests, and modeled influence in separate evidence classes.
- 03
Marked limitations wherever the available source or request data cannot support a provider-level conclusion.
Verify the evidence.
Provider behavior and analytics definitions change. These are the primary references reviewed for this page.
Questions teams ask.
What is the difference between an AI crawler and an AI agent?+
A crawler mainly discovers or retrieves content. An agent attempts a task on behalf of a user or system. The boundary can be imperfect, so classification should use identity, behavior, authorization, and policy evidence.
Should bot traffic appear in normal website visitor counts?+
No. Automated requests and human visits represent different activity and should be reported separately. Blending them can distort traffic, engagement, and conversion metrics.
What is agent task completion rate?+
It is the share of attempted agent tasks that reach the defined successful result. The denominator and completion definition must be explicit, and authorization failures should remain visible as a separate stage.
Can every AI agent be identified reliably?+
No. Some actors provide strong identity evidence; others only self-declare or resemble ordinary browser traffic. Good reporting includes an unknown class and communicates confidence.