https://verifiedbots.dev Bots newly added to the directory, with identity, category, and verification tier. en Sat, 05 Sep 2026 04:31:23 GMT https://verifiedbots.dev/bots/fedicabot https://verifiedbots.dev/bots/fedicabot Thu, 03 Sep 2026 13:20:13 GMT The fetcher used by Fedica's social-media publishing platform. The operator documents that it does not crawl whole sites: it reads only the page or RSS feed a Fedica user has specified, downloading feed content so the user can schedule posts from it, and reading Open Graph metadata so a shared link renders as a card. Fedica publishes the robots.txt token but states no robots.txt policy and no address ranges. Operated by Fedica. https://verifiedbots.dev/bots/swiftbot https://verifiedbots.dev/bots/swiftbot Thu, 03 Sep 2026 13:20:13 GMT The web crawler behind Swiftype's site-search product, now operated by Elasticsearch B.V. Unlike a general search-engine crawler it only visits sites that Swiftype customers have asked it to crawl, in order to build a search index for those sites. The operator documents that Swiftbot obeys every restriction it finds in robots.txt, including a full-site disallow, and that it looks for the token Swiftbot. Operated by Elasticsearch B.V. (Swiftype). https://verifiedbots.dev/bots/oai-adsbot https://verifiedbots.dev/bots/oai-adsbot Tue, 01 Sep 2026 13:17:25 GMT OpenAI's fetcher for advertising on ChatGPT. When an advertiser submits an ad, OAI-AdsBot visits the submitted landing page to check that it complies with OpenAI's ad policies and to judge when the ad is relevant to show. OpenAI states it only visits pages submitted as ads and that what it collects is not used to train generative AI foundation models. It has its own published IP range list, separate from GPTBot, OAI-SearchBot and ChatGPT-User. Operated by OpenAI. https://verifiedbots.dev/bots/w3c-checklink https://verifiedbots.dev/bots/w3c-checklink Tue, 01 Sep 2026 13:17:25 GMT W3C's link checker, which follows the links on a user-submitted page to report broken and redirected references. W3C's validation services page marks it as the one validator that, being a crawling service, honors robots.txt directives, and gives the same published source addresses as the other W3C validators. Operated by World Wide Web Consortium (W3C). https://verifiedbots.dev/bots/w3c-css-validator https://verifiedbots.dev/bots/w3c-css-validator Tue, 01 Sep 2026 13:17:25 GMT W3C's CSS validator, which fetches a user-submitted page and its stylesheets to check them against the CSS specifications. It runs on W3C's Jigsaw server and identifies itself with a Jigsaw prefix followed by its own W3C_CSS_Validator_JFouffa token, from the same published validator addresses. Operated by World Wide Web Consortium (W3C). https://verifiedbots.dev/bots/w3c-feed-validator https://verifiedbots.dev/bots/w3c-feed-validator Tue, 01 Sep 2026 13:17:25 GMT W3C's feed validator, which fetches a user-submitted RSS or Atom feed to check it against the feed specifications. It is a one-off conformance check rather than a subscription fetcher, and W3C publishes its user agent and the shared validator source addresses. Operated by World Wide Web Consortium (W3C). https://verifiedbots.dev/bots/w3c-i18n-checker https://verifiedbots.dev/bots/w3c-i18n-checker Tue, 01 Sep 2026 13:17:25 GMT The fetcher behind W3C's Internationalization Checker, which retrieves a user-submitted page and reports on its language, encoding and other i18n markup. W3C's validation services page lists its user agent alongside the single IPv4 address and IPv6 prefix all W3C validators fetch from. Operated by World Wide Web Consortium (W3C). https://verifiedbots.dev/bots/w3c-validator-nu https://verifiedbots.dev/bots/w3c-validator-nu Tue, 01 Sep 2026 13:17:25 GMT The fetcher behind the Nu HTML Checker, the W3C validation service that checks a user-submitted URL against current HTML conformance rules. W3C's validation services page lists its user agent and states that all W3C validator traffic comes from one published IPv4 address and one IPv6 prefix. Operated by World Wide Web Consortium (W3C). https://verifiedbots.dev/bots/yandex-accessibility https://verifiedbots.dev/bots/yandex-accessibility Tue, 01 Sep 2026 13:17:25 GMT Yandex's accessibility checker, which downloads pages to assess how accessible they are to users. Yandex documents that it sends up to three requests per second, ignores the crawl-rate setting in Yandex Webmaster, and does not follow the general robots.txt rules written for arbitrary robots. Operated by Yandex. https://verifiedbots.dev/bots/yandex-additional https://verifiedbots.dev/bots/yandex-additional Tue, 01 Sep 2026 13:17:25 GMT Yandex's robot for processing robots.txt so that site owners can keep already indexed page content out of the AI responses shown in Yandex Search. Yandex's robot table states it applies to pages that have already been indexed by the primary crawler and that it makes no indexing requests of its own. Operated by Yandex. https://verifiedbots.dev/bots/yandex-com https://verifiedbots.dev/bots/yandex-com Tue, 01 Sep 2026 13:17:25 GMT Yandex's indexing robot for content in languages other than Russian, feeding the international yandex.com search index. Yandex's robot table documents that it can index content when there is no explicit robot-specific restriction, and lists it among the robots that do not follow the general robots.txt rules written for arbitrary robots. Operated by Yandex. https://verifiedbots.dev/bots/yandex-mobile https://verifiedbots.dev/bots/yandex-mobile Tue, 01 Sep 2026 13:17:25 GMT Yandex's mobile-layout robot, which fetches pages to determine whether their layout is suitable for mobile devices. It advertises an iPhone Safari prefix followed by its own YandexMobileBot token, and Yandex lists it among the robots that do not follow the general robots.txt rules written for arbitrary robots. Operated by Yandex. https://verifiedbots.dev/bots/yandex-pagechecker https://verifiedbots.dev/bots/yandex-pagechecker Tue, 01 Sep 2026 13:17:25 GMT The fetcher behind Yandex's structured data validator, which retrieves a page so its markup can be validated. Yandex's robot table records it as taking the general robots.txt rules into account. Operated by Yandex. https://verifiedbots.dev/bots/alertsite https://verifiedbots.dev/bots/alertsite Mon, 31 Aug 2026 13:20:25 GMT SmartBear's AlertSite runs customer-configured availability and transaction monitors against the sites and APIs their owners point it at, from a published set of monitoring locations. SmartBear tells site owners to filter the traffic by looking for "AlertSite" in the user agent string or by the monitoring location addresses. Some mobile-network locations use dynamic carrier addresses that are not in the published list. Operated by SmartBear Software. https://verifiedbots.dev/bots/feedwind-crawler https://verifiedbots.dev/bots/feedwind-crawler Mon, 31 Aug 2026 13:20:25 GMT The crawler behind FeedWind, an embeddable RSS/Atom widget. It fetches the feed sources that a widget owner has configured, every five minutes to five hours depending on their plan, so the rendered widget stays current. The operator's support page prints the crawler's full user agent and asks feed publishers to unblock it if a firewall is turning it away. Operated by Mikle KK (FeedWind). https://verifiedbots.dev/bots/ghost-inspector https://verifiedbots.dev/bots/ghost-inspector Mon, 31 Aug 2026 13:20:25 GMT Ghost Inspector runs customer-authored browser tests against the sites their owners point it at, from a fixed pool of cloud test runners in 18 regions. Its runners append the literal string "Ghost Inspector" to the browser user agent they run, and the operator publishes the full list of test-runner addresses so site owners can filter the traffic out of their analytics. Operated by Ghost Inspector. https://verifiedbots.dev/bots/repo-lookout-bot https://verifiedbots.dev/bots/repo-lookout-bot Mon, 31 Aug 2026 13:20:25 GMT Repo Lookout is a non-commercial internet-wide security scanner that looks for source-code repositories accidentally exposed to the public web and reports them to the domain's technical contact. The operator documents the scanner's user-agent prefix and offers two opt-outs: an email request, or denying every request whose user agent starts with RepoLookoutBot. Operated by Crissy Field GmbH (Repo Lookout). https://verifiedbots.dev/bots/wepch-searchengine https://verifiedbots.dev/bots/wepch-searchengine Mon, 31 Aug 2026 13:20:25 GMT The crawler for Wepch, an independent privacy-focused search engine still in development. Its operator page states the project's purpose, gives a contact address for crawl complaints, and documents the robots.txt token site owners can use to crawl-delay or disallow it. Operated by Wepch. https://verifiedbots.dev/bots/atlassian-bot https://verifiedbots.dev/bots/atlassian-bot Sat, 29 Aug 2026 13:18:16 GMT Crawler behind the Teamwork Graph "custom website" connector, which indexes a site an Atlassian customer has connected so its content is searchable in Rovo. Atlassian documents the robots.txt token and publishes the connector's egress ranges under the rovo-crawler product in its IP-ranges feed. Operated by Atlassian. https://verifiedbots.dev/bots/easycron https://verifiedbots.dev/bots/easycron Sat, 29 Aug 2026 13:18:16 GMT Hosted cron service that calls URLs its users schedule, on the interval or cron expression they configure. Requests go only to endpoints the user entered, and EasyCron publishes the IPv4 and IPv6 addresses its execution bots call from as a JSON list. Operated by EasyCron. https://verifiedbots.dev/bots/edgewatch https://verifiedbots.dev/bots/edgewatch Sat, 29 Aug 2026 13:18:16 GMT Internet-wide scanner for EdgeWatch's attack-surface and threat-intelligence platform. It makes connection attempts to public IPv4 and IPv6 addresses daily and completes protocol handshakes to fingerprint the services it finds; EdgeWatch documents the subnet its scanners use and an e-mail opt-out. Operated by EdgeWatch. https://verifiedbots.dev/bots/fedreporter-bot https://verifiedbots.dev/bots/fedreporter-bot Sat, 29 Aug 2026 13:18:16 GMT Scheduled data-retrieval bot that pulls publicly available banking-institution records from the FFIEC National Information Center for Fed Reporter's internal research dashboards. Its documentation states it targets only public data, uses no CAPTCHA bypass, and runs a separate testing agent off-peak. Operated by Fed Reporter Inc.. https://verifiedbots.dev/bots/productsup-crawler https://verifiedbots.dev/bots/productsup-crawler Sat, 29 Aug 2026 13:18:16 GMT Crawler behind Productsup's website data-import source, which reads pages of a site a Productsup customer has configured and turns them into a product data feed. Productsup documents the default user agent, and notes that customers may append a hash to it for their own filtering. Operated by Productsup. https://verifiedbots.dev/bots/rakuten-usa-image-bot https://verifiedbots.dev/bots/rakuten-usa-image-bot Sat, 29 Aug 2026 13:18:16 GMT Rakuten's image-extraction bot, which retrieves product images from merchant sites that have partnered with Rakuten. Its bot page names the identifying token that appears in the user-agent string and gives an abuse contact. Operated by Rakuten. https://verifiedbots.dev/bots/scrunchbot https://verifiedbots.dev/bots/scrunchbot Sat, 29 Aug 2026 13:18:16 GMT Fetcher for Scrunch AI's brand-visibility product. Scrunch documents that it accesses customer websites to diagnose issues with their content and configuration, and that it does not proactively crawl or spider the general internet — requests are made only in response to customer configuration. Operated by Scrunch AI. https://verifiedbots.dev/bots/sirdata-bot https://verifiedbots.dev/bots/sirdata-bot Sat, 29 Aug 2026 13:18:16 GMT Crawler for Sirdata's contextual advertising API. It fetches and categorises page content when Sirdata's on-page script cannot read the document from the parent frame. Sirdata publishes the addresses it crawls from at proxies-list.sirdata.fr, but that pool rotates continuously — it recommends re-fetching every ten minutes — so no snapshot of it stays accurate. Operated by Sirdata. https://verifiedbots.dev/bots/xy-archive-compliance https://verifiedbots.dev/bots/xy-archive-compliance Sat, 29 Aug 2026 13:18:16 GMT Website archiver for XY Archive, XYPN's compliance-archiving product for financial advisory firms. It crawls a subscribing firm's site to decide which pages to capture and then screenshots each page for the archive; XYPN documents the single address it archives from. Operated by XY Planning Network. https://verifiedbots.dev/bots/yahoo-ad-monitoring https://verifiedbots.dev/bots/yahoo-ad-monitoring Sat, 29 Aug 2026 13:18:16 GMT Page-fetch client that retrieves the landing pages of URLs listed with Yahoo advertising services. Yahoo documents that it fetches each advertiser-supplied landing page to check policy compliance and to improve the accuracy of the ad listing, and that it uses separate desktop and mobile user agents. Operated by Yahoo. https://verifiedbots.dev/bots/deskyobot https://verifiedbots.dev/bots/deskyobot Thu, 27 Aug 2026 13:23:13 GMT Link-preview fetcher for Deskyo, a service that lets its members save web pages. The operator's bot page documents that Deskyo must display information tied to a saved page — its title, description and favicon — which is what this fetcher retrieves, and gives a legal contact for questions about links saved by a member. The page states no robots.txt policy and publishes no addresses. Operated by Deskyo. https://verifiedbots.dev/bots/feeder-co https://verifiedbots.dev/bots/feeder-co Thu, 27 Aug 2026 13:23:13 GMT Crawler for Feeder, a hosted RSS reading service. The operator documents that it only fetches feeds users have actively subscribed to, that it acts as an agent on the user's behalf and therefore does not check robots.txt, and that rate limits can be arranged by contacting support. Its user agent is a browser-shaped string in which Feeder deliberately inserts the feeder.co identifier as the honest self-identification. Operated by Feeder. https://verifiedbots.dev/bots/fleebsbot https://verifiedbots.dev/bots/fleebsbot Thu, 27 Aug 2026 13:23:13 GMT Crawler for fleebs.com, a German real-time search engine. The operator's bot page documents two identities: one that searches for new information and one that analyses the pages it finds. It documents robots.txt blocking with the FleebsBot token and notes that pages indexed before a block may take time to drop out of the index. Operated by iontic GmbH (fleebs.com). https://verifiedbots.dev/bots/innguma-fetcher https://verifiedbots.dev/bots/innguma-fetcher Thu, 27 Aug 2026 13:23:13 GMT Innguma's feed fetcher. It retrieves RSS and Atom feeds that users have added to Innguma or to another application built on the Innguma cloud, and refreshes them roughly once an hour. The operator states that because the requests follow explicit action by human users rather than automated crawling, the fetcher does not follow robots.txt, and documents serving an error status to the Innguma/1.0 user agent as the opt-out. Operated by Innguma. https://verifiedbots.dev/bots/linkcheckerbot https://verifiedbots.dev/bots/linkcheckerbot Thu, 27 Aug 2026 13:23:13 GMT LinkChecker.pro's backlink-monitoring crawler. It revisits the backlinks that the service's customers track, checking whether each link is still present and how it is marked. The operator documents that the crawler strictly respects robots.txt and Crawl-delay, and publishes the addresses it crawls from. Operated by Local Profy LLC (LinkChecker.pro). https://verifiedbots.dev/bots/mcontextual-bot https://verifiedbots.dev/bots/mcontextual-bot Thu, 27 Aug 2026 13:23:13 GMT MContextual's crawler for contextual advertising. The operator documents that it reads page content so advertisers can build cookieless contextual audiences from a site's subject matter, that it can be blocked with standard robots.txt directives, and that its requests originate from AWS and GCP addresses rather than a published range. Operated by MContextual. https://verifiedbots.dev/bots/mediamonitoringbot https://verifiedbots.dev/bots/mediamonitoringbot Thu, 27 Aug 2026 13:23:13 GMT Crawler behind MediaMonitoringBot, a Ukrainian media-monitoring service that indexes news and media publisher pages and delivers mention alerts to PR and communications subscribers. The operator documents that it strictly respects robots.txt allow and disallow rules, publishes its main crawler addresses, and documents a reverse-DNS suffix for verification. Operated by MediaMonitoringBot. https://verifiedbots.dev/bots/mozilla-tabstack https://verifiedbots.dev/bots/mozilla-tabstack Thu, 27 Aug 2026 13:23:13 GMT Fetcher for Tabstack, Mozilla's developer-facing platform for programmatic, AI-driven interaction with web content. Every request carries a dedicated user agent, and the operator documents that Tabstack respects robots.txt rules addressed to it, stops immediately on a disallowed path, fails fast rather than retrying, and caches robots.txt results to reduce follow-up requests. Operated by Mozilla (Tabstack). https://verifiedbots.dev/bots/nestdaddybot https://verifiedbots.dev/bots/nestdaddybot Thu, 27 Aug 2026 13:23:13 GMT Web crawler behind the NestDaddy search engine. The operator's webmaster page documents that it extracts page title, meta description, headings, language and link structure to build the index, filters adult and abusive content out of results, and fully respects robots.txt including the Crawl-delay directive. It states that the crawler runs from a distributed network whose current ranges are available only on request. Operated by NestDaddy Technologies. https://verifiedbots.dev/bots/observepoint https://verifiedbots.dev/bots/observepoint Thu, 27 Aug 2026 13:23:13 GMT ObservePoint's web-governance platform runs customer-configured audits and user journeys against their own sites to validate analytics and marketing tags. Requests come from a published set of static addresses that the operator states are used by ObservePoint alone, and most of its browser-based user agents carry the literal name "ObservePoint". Operated by ObservePoint. https://verifiedbots.dev/bots/scour-rss-bot https://verifiedbots.dev/bots/scour-rss-bot Thu, 27 Aug 2026 13:23:13 GMT Feed fetcher for Scour, a personalized content feed service in which users subscribe to RSS, Atom and JSON feeds and topics of interest. It polls each subscribed feed once every 900 seconds regardless of subscriber count, uses conditional requests, rate-limits itself to two requests per second per registrable domain, and backs off on error responses. Operated by Scour (Evan Schwartz). https://verifiedbots.dev/bots/urlsuma https://verifiedbots.dev/bots/urlsuma Thu, 27 Aug 2026 13:23:13 GMT Crawler for urlsuma.de, a German general-purpose web search engine under construction. The operator documents that it re-reads robots.txt without caching before every single fetch, makes about two requests per visit (robots.txt plus one page), executes no JavaScript, and treats a 4xx response as a permanent block. Current crawler addresses are shared on request only. Operated by urlsuma.de (Gerhard Stöbe). https://verifiedbots.dev/bots/yahoo-mail-proxy https://verifiedbots.dev/bots/yahoo-mail-proxy Thu, 27 Aug 2026 13:23:13 GMT Yahoo Mail's content-fetch proxy. It retrieves images and other assets embedded in the URLs of messages sent to Yahoo Mail users, so that the content is served through Yahoo rather than fetched directly by the recipient's browser. Yahoo documents the user agent and publishes the address ranges the proxy fetches from. Operated by Yahoo. https://verifiedbots.dev/bots/artemis-web-reader https://verifiedbots.dev/bots/artemis-web-reader Tue, 25 Aug 2026 13:21:31 GMT The feed poller behind Artemis, a hosted web reader that follows websites and blogs on behalf of its subscribers. It updates once per day and uses HEAD and conditional GET requests with If-Modified-Since and If-None-Match to reduce bandwidth. Artemis publishes the addresses it polls feeds from as a plain-text list. Operated by Artemis (jamesg.blog). https://verifiedbots.dev/bots/classla-web https://verifiedbots.dev/bots/classla-web Tue, 25 Aug 2026 13:21:31 GMT The CLASSLA-web crawler collects web corpora for South Slavic and other languages under the CLARIN Knowledge Centre for South Slavic languages, continuing the CEF-funded MaCoCu project's crawling infrastructure. It runs the SpiderLing crawler developed at Masaryk University. The operator documents that it adheres to the robots exclusion standard and reads robots.txt on the first access of each crawl run. Operated by CLARIN.SI (CLASSLA Knowledge Centre). https://verifiedbots.dev/bots/cotoyogi https://verifiedbots.dev/bots/cotoyogi Tue, 25 Aug 2026 13:21:31 GMT A research web crawler run by the Data Lake Research and Development Center of the Joint Support-Center for Data Science Research (ROIS-DS), Research Organization of Information and Systems, Japan. It collects Japanese-language web data for research use. The operator page documents RFC 9309 robots.txt handling with Crawl-delay support, robots meta tag support, and the address range the crawler requests from. Operated by ROIS-DS Data Lake Research and Development Center. https://verifiedbots.dev/bots/findfilesbot https://verifiedbots.dev/bots/findfilesbot Tue, 25 Aug 2026 13:21:31 GMT The crawler behind the FindFiles.net file search engine. It checks that publicly accessible files users search for are still available, using multiplexed HTTP/2 HEAD requests with a minimum ten-second interval per server, and may download images, videos and executables for classification and safety checks. Purpose-specific user agents are published for its link checking, favicon fetching, virus scanning and image classification agents, which all share one crawler host and address. Operated by FindFiles.net. https://verifiedbots.dev/bots/iboubot https://verifiedbots.dev/bots/iboubot Tue, 25 Aug 2026 13:21:31 GMT The crawler behind Ibou, a French conversational search engine. It discovers publicly accessible pages and indexes them so Ibou can cite and link back to the sites it answers from. The operator documents a 5-second politeness delay per host, publishes its address ranges as JSON, and documents forward-confirmed reverse DNS as the reliable way to tell a genuine request from a spoofed user agent. Operated by Babbar (Ibou). https://verifiedbots.dev/bots/linksindexerbot https://verifiedbots.dev/bots/linksindexerbot Tue, 25 Aug 2026 13:21:31 GMT The crawler behind the Links Indexer URL indexing service. It parses third-party sites to verify their URLs and status for sitemap campaigns, querying pages for metadata and favicons and taking a homepage screenshot. The operator states it never harvests e-mail addresses or content unrelated to sitemap campaigns, runs at most four parallel requests, and publishes the single address it crawls from. Operated by Links Indexer (Kalpraj Solutions (OPC) Private Limited). https://verifiedbots.dev/bots/lyonlbot https://verifiedbots.dev/bots/lyonlbot Tue, 25 Aug 2026 13:21:31 GMT The crawler used by Lyonl Search to discover and refresh public web documents for its index, with separate identities for web, image and news crawling so site owners can control each class independently. The operator documents robots.txt and crawl-delay support, conditional fetches with ETag and Last-Modified, and that the crawler does not submit forms or bypass authentication. It states that crawler address ranges are not currently published, so no verification recipe is recorded. Operated by Lyonl Search. https://verifiedbots.dev/bots/magnetmebot https://verifiedbots.dev/bots/magnetmebot Tue, 25 Aug 2026 13:21:31 GMT Magnet.me's web crawler. It discovers and refreshes career-related pages — vacancies, events and employer information — for the Magnet.me careers platform. The operator documents that it may render JavaScript, follows the canonical version of a page, does not submit forms and adjusts its crawl speed automatically, and it publishes the addresses it crawls from as both a JSON and a plain-text list. Operated by Magnet.me. https://verifiedbots.dev/bots/motominerbot https://verifiedbots.dev/bots/motominerbot Tue, 25 Aug 2026 13:21:31 GMT MotoMiner's vehicle-listing crawler. It primarily crawls automotive dealership websites and adds their vehicle detail pages to the MotoMiner index. The operator documents robots.txt and crawl-delay support, honours bot-specific noindex meta directives, throttles outbound requests against defined traffic thresholds, and publishes the address the crawler currently requests from. Operated by MotoMiner.