wandered
Tell me when it’s live →
Home/Resources/AI agents
Wandered field notes · 5 resources

Understand the machines using your site.

References for AI crawlers, retrieval bots, user-delegated agents, server logs, identification, and access policy.

Choose your next task

Move from question to evidence.

01

Identify

Classify bots using layered evidence.

→02

Measure

Analyze machine traffic in server logs.

→03

Govern

Write policy by actor and purpose.

→04

Reference

Use the maintained crawler directory.

→
Start here01
Field note 02

AI agent analytics: measure tasks, not just bot requests

Learn how to distinguish AI crawlers, assistants, and user-delegated agents, then measure discovery, authorization, task completion, and failure.

8 minuteRead →
Implementation guides03
How-to

How to measure AI bot traffic with server and CDN logs

Measure AI crawler and retrieval activity by actor, verified identity, requested page, response, policy, rate, and purpose—without counting bots as people.

10 minuteRead →
How-to

How to identify AI bots without trusting the user agent alone

Classify AI crawlers and fetchers using declared tokens, provider documentation, IP or DNS verification, request behavior, and confidence.

9 minuteRead →
Policy guide

robots.txt for AI crawlers: write policy by actor and purpose

Create a documented AI crawler policy for training, search, retrieval, and user agents using current provider tokens and server-side enforcement.

10 minuteRead →
Maintained references01
Maintained reference

AI crawler user-agent directory: purpose, policy, and verification

A maintained reference for major AI crawler, search, retrieval, and control tokens with provider documentation and verification guidance.

15 minute referenceRead →
Private development, public field notes

One email when Wandered goes live.

One launch email. No account, no weekly drip, no noise.