AI assistants 22

Aleksma Ai, Inc. (Lyrenth) · AI assistants

Lyrenth's on-demand fetcher. It identifies itself separately from the AIWebIndex crawler when a customer asks the index for a URL that is not in it yet, so site owners can tell scheduled crawling and on-demand fetches apart in their logs. The operator states both user agents honour robots.txt the same way, and the crawler's address feed covers this user agent too.

Amazon · AI assistants

Amazon's on-demand fetcher supporting user actions, such as responding to Alexa queries that need up-to-date information; Amazon states it does not crawl content for generative AI model training. Amazon publishes its live-crawl source addresses on a dedicated page as a dated snapshot; they are recorded here verbatim.

OpenAI · AI assistants

OpenAI's user-triggered fetcher, used when a ChatGPT user or a GPT Action asks the assistant to visit a specific page. It does not crawl autonomously.

Anthropic · AI assistants

Anthropic's user-triggered fetcher, used when a person asks Claude to visit a specific web page (e.g. via tool use in a conversation).

Diffbot · AI assistants

Diffbot's user-triggered fetcher, which Diffbot documents as being used by requests originating on behalf of a human user browsing a URL through Diffbot software, in response to that person's input. Diffbot's robots.txt guidance covers its web crawls; it makes no separate statement about these user-initiated fetches, and publishes no IP list, ASN or reverse-DNS pattern.

DuckDuckGo · AI assistants

DuckDuckGo's real-time fetcher for DuckDuckGo Search's AI-assisted answers, which cite their sources. DuckDuckGo states the data is not used to train AI models, and publishes a machine-readable list of the bot's IP addresses.

ElevenLabs · AI assistants

ElevenLabs' knowledge-base fetcher. It retrieves the pages that ElevenLabs customers add to their conversational agents' knowledge bases, both when a page is added and when it is refreshed. ElevenLabs documents that it honors robots.txt and that it is configured independently of the training crawler, so a site can allow this token while disallowing ElevenlabsCrawler.

Exa · AI assistants

Exa's user-triggered fetcher, which retrieves a page live when a person, or an application built on the Exa API, asks for its current content. Exa documents it as separate from ExaSearchBot: it does not crawl automatically, it does no scheduled or background fetching, and it is not used to decide whether content appears in Exa search. Requests are signed with Web Bot Auth.

Google · AI assistants

The fetcher used by AI agents hosted on Google infrastructure to navigate the web and perform actions on behalf of a user who asked for them. Google publishes a dedicated IP range list for it and signs a subset of its requests with Web Bot Auth under the agent.bot.goog identity. As a user-triggered fetcher it generally ignores robots.txt rules.

Google · AI assistants

Google tools and product features that fetch a specific page because an end user asked for it (Google Read Aloud, Site Verifier, Gemini Notebook, Chrome Web Store, Google Messages, Pinpoint, Publisher Center, and similar), rather than autonomous crawling for search indexing. They generally ignore robots.txt because a human requested the fetch.

Iframely · AI assistants

Iframely's processing agent, a separate identity for the automated-processing requests its customers make through the Iframely API — indexing, search, AI-assisted retrieval and AI input. Iframely documents it as request-driven, one URL at a time on behalf of a specific customer request, and explicitly scoped against training use, which goes out as the Iframely Training Agent instead. Every request is signed with Web Bot Auth and carries a Forwarded header naming the customer behind it.

PT Epithre Teknologi Indonesia · AI assistants

Epithre's on-demand fetcher for IsonAI, the assistant built on Ison Search. It visits a page in real time when a person using IsonAI asks about that page or asks the assistant to read, summarise or analyse a link, and the answer then cites the source. The operator states plainly that it is not used to crawl the web automatically and not used to build or rank the Ison Search index, so it is listed separately from IsonSearchBot. Because the fetches are started by a user, Epithre says robots.txt rules may not apply and that site owners who want to stop it should block on the user agent or ask. It has no published address range by design, and the operator directs verifiers to its signature instead.

Moonshot AI · AI assistants

Moonshot AI's user-triggered fetcher for Kimi, which retrieves a page when a person asks Kimi to summarise it or answers a question needing live web retrieval. Moonshot states it is not used for automated bulk crawling and that, because the actions are user-triggered, robots.txt rules may not directly apply.

Liner · AI assistants

Liner's on-demand fetcher. It retrieves a page in real time when a Liner user asks a question, and the answer cites that page as its source. Liner documents it alongside LinerBot as the second of two agents, each with its own user agent, robots.txt token and signing key, and states explicitly that rules written for LinerBot do not apply to Liner-User. Liner publishes no address ranges for either agent and points site owners at the HTTP Message Signatures key directory instead.

Meta · AI assistants

Meta's on-demand fetcher that retrieves a single link at a user's request to support agentic AI features (e.g. an AI assistant navigating a page a user asked about), rather than broad indexing. Because its fetches are requested by a user, Meta documents that this crawler may bypass robots.txt rules. Verified by ASN lookup (AS32934).

Mistral AI · AI assistants

Mistral AI's user-triggered fetcher. When someone asks Vibe a question, it may visit a web page to help answer and link to the source in its response. Mistral documents that it is not used for automatic crawling of the web, nor to collect content for generative AI training, and publishes the addresses it fetches from.

Mozilla (Tabstack) · AI assistants

Fetcher for Tabstack, Mozilla's developer-facing platform for programmatic, AI-driven interaction with web content. Every request carries a dedicated user agent, and the operator documents that Tabstack respects robots.txt rules addressed to it, stops immediately on a disallowed path, fails fast rather than retrying, and caches robots.txt results to reduce follow-up requests.

Perplexity · AI assistants

Perplexity's user-triggered fetcher, used when a user's question requires visiting a specific web page to produce an accurate answer.

SB Intuitions Corp. · AI assistants

SB Intuitions' user-triggered fetcher. The operator documents that when someone asks its Sarashina service a question, this agent visits websites on that person's behalf to improve the quality of the search results it reasons over, that answers may include links to the pages it visited, and that what it retrieves is not used for AI development. That last point distinguishes it from SBIntuitionsBot, which the same page says is used for AI development and information analysis.

Parallel Web Systems · AI assistants

Parallel's user-triggered fetcher. It identifies itself when Parallel accesses content on behalf of a user, giving content owners visibility into user-initiated requests. The operator documents that it is not used for automatic crawling and is intended to provide visibility rather than to be managed through robots.txt, which governs ShapBot instead. Parallel signs Shap-User requests with Web Bot Auth and publishes the key directory.

Shopify · AI assistants

Shopify's cloud browser agent, which loads pages so that Shopify AI agents such as Sidekick can act on a merchant's or customer's request, and so that Shopify tools can screenshot pages and check them for display problems. It visits a page only when a Shopify tool directs it to and does not crawl or index the web on its own. Shopify documents that the Cloudsurf token is appended to the standard user agent of the browser running the session, so the browser prefix varies; requests from mobile-emulation tools carry a plain phone user agent with no Cloudsurf token at all, which is why Shopify directs site owners to verify the Web Bot Auth signature rather than the user agent.

Yandex · AI assistants

A Yandex robot that proxies user actions taken on Yandex services: it sends requests in response to button clicks and downloads pages for online translation, so its requests to a site follow from something a person just did. It does not follow the general robots.txt rules.