FirecrawlAgent

Listed only

The fetcher behind Firecrawl's scraping and crawling API, which retrieves web pages on behalf of the developers and AI agents calling that API. Firecrawl reads robots.txt before crawling and applies the rules written for the token FirecrawlAgent as well as those for *. The operator documents FirecrawlAgent as a robots.txt token only, warns that it may not appear in the User-Agent header of the fetch itself, and states that Firecrawl does not use a fixed set of outbound IP addresses — so requests cannot be attributed to the operator.

Operated by Firecrawl · Official documentation

User agents

Patterns this directory matches, with real observed strings.

regexFirecrawlAgent
observedMozilla/5.0 (compatible; FirecrawlAgent; +https://firecrawl.dev/)

How to verify

No verification recipe published by the operator.

Good Bot Practices scorecard

Identifies honestly
Stable UA token documented (1 pattern)
Verifiable
Operator publishes no verification path
Respects robots.txt
Honors robots.txt (token: FirecrawlAgent)
Behaves
Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset
Reachable operator
Operator and documentation published

Measured against the Good Bot Practices.