AI crawler · Exa
ExaSearchBot: what it is and how to block it
ExaSearchBot crawls and indexes the public web for Exa, a search engine used by AI applications. Exa signs every request so you can verify it cryptographically.
BrandVector Editorial · Last verified Oct 8, 2026
- Operator
- Exa
- Type
- Search
- robots.txt token
ExaSearchBot- Follows robots.txt
- Yes, per Exa
- IP ranges
- Not published
- Blocked by name
- 0.4% of top sites
What is ExaSearchBot?
ExaSearchBot is Exa's search crawler: it indexes pages for search results. In Exa's words:
“Exa is a search engine. Similar to other search engines, Exa uses a robot (ExaSearchBot) to discover and index pages on the public web. This index allows users to find, retrieve, and cite your content.”
ExaSearchBot user agent string
Exa documents this user agent string for ExaSearchBot:
Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)
In robots.txt, use the token ExaSearchBot, not the full string. Crawlers match on the token, and version numbers in the full string can change.
How many websites block ExaSearchBot?
In our Oct 8, 2026 check of 817 of the most-visited websites, 0.4% block ExaSearchBot by name for their whole site (3 sites).
| robots.txt rule for ExaSearchBot | Sites | Share |
|---|---|---|
| Blocks the whole site by name | 3 | 0.4% |
| Blocks part of the site by name | 0 | 0.0% |
| Names it but doesn't block it | 0 | 0.0% |
| Blocked only by a catch-all (*) rule | 38 | 4.7% |
For comparison, 0.1% of the same sites block Googlebot by name. The sites are the top 1,000 origins in the Chrome UX Report for August 2026; see how we measured.
How to block ExaSearchBot in robots.txt
Add this group to the robots.txt file at the root of your domain to block ExaSearchBot from your whole site:
User-agent: ExaSearchBot Disallow: /
To block only part of your site, list those paths instead:
User-agent: ExaSearchBot Disallow: /private/
A group that names ExaSearchBot replaces your User-agent: * rules for it entirely (RFC 9309). If your catch-all group blocks paths that ExaSearchBot should also stay out of, repeat them in its group. To let ExaSearchBot in while a catch-all rule blocks others:
User-agent: * Disallow: / User-agent: ExaSearchBot Allow: /
“ExaSearchBot respects the Robots Exclusion Protocol. When choosing which rules to follow, it looks for a group addressed to ExaSearchBot. If you haven't provided one, it follows the rules you've set for major search engines; otherwise it follows the * group.”
What happens if you block ExaSearchBot?
“robots.txt controls crawling, not indexing.”
How to verify requests from ExaSearchBot
“Every ExaSearchBot request is cryptographically signed using HTTP Message Signatures (RFC 9421), following the Web Bot Auth scheme.”
See every AI crawler and how often top sites block it
Sources
Rather have an expert do it?
Tell us about your site and goals, and we'll match you with an AI visibility (GEO) specialist. Free, no obligation.