Verified Bots Directory

Good Bot
Practices

The standard this directory measures every bot against. Each practice maps to a field in the bot schema, so every directory entry doubles as a conformance scorecard.

  1. 01

    Identify honestly

    Send a stable User-Agent token: product name, version, and an info URL. Example: ExampleBot/2.1 (+https://example.com/bot).

    schema: user_agents

  2. 02

    Be verifiable

    Publish machine-readable IP ranges at a stable HTTPS URL (the Google/OpenAI prefixes JSON shape is the de-facto standard), support forward/reverse DNS on a domain you control, or implement Web Bot Auth (IETF HTTP Message Signatures) and host a key directory at /.well-known/http-message-signatures-directory. Implement with Cloudflare's open-source libraries (github.com/cloudflare/web-bot-auth); self-test free at fingerprint.com/web-bot-auth/test/.

    schema: verification

  3. 03

    Respect robots.txt

    Honor robots.txt and document your token. Service traffic that is not a crawler (webhooks, monitors the site owner configured) is exempt by nature.

    schema: behavior.respects_robots_txt

  4. 04

    Behave

    Sane request rates. Back off on 429 and 503. Honor cache headers.

    schema: behavior

  5. 05

    Be reachable

    Publish operator identity, documentation, and a contact channel for site owners.

    schema: operator, docs

Ready to be listed? Submit your bot →