AI crawler · Anthropic
Claude-SearchBot: what it is and how to block it
Claude-SearchBot indexes web pages to improve the search results Claude uses in its answers. Anthropic says blocking it may reduce your site's visibility in those results.
BrandVector Editorial · Last verified Oct 8, 2026
- Operator
- Anthropic
- Type
- Search
- robots.txt token
Claude-SearchBot- Follows robots.txt
- Yes, per Anthropic
- IP ranges
- Published
- Blocked by name
- 5.1% of top sites
What is Claude-SearchBot?
Claude-SearchBot is Anthropic's search crawler: it indexes pages for search results. In Anthropic's words:
“Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.”
“Please do this for every subdomain that you wish to opt out from.”
Claude-SearchBot user agent string
Anthropic doesn't publish a full user agent string for Claude-SearchBot. In robots.txt, use the token Claude-SearchBot.
How many websites block Claude-SearchBot?
In our Oct 8, 2026 check of 817 of the most-visited websites, 5.1% block Claude-SearchBot by name for their whole site (42 sites).
| robots.txt rule for Claude-SearchBot | Sites | Share |
|---|---|---|
| Blocks the whole site by name | 42 | 5.1% |
| Blocks part of the site by name | 24 | 2.9% |
| Names it but doesn't block it | 7 | 0.9% |
| Blocked only by a catch-all (*) rule | 28 | 3.4% |
For comparison, 0.1% of the same sites block Googlebot by name. The sites are the top 1,000 origins in the Chrome UX Report for August 2026; see how we measured.
How to block Claude-SearchBot in robots.txt
Add this group to the robots.txt file at the root of your domain to block Claude-SearchBot from your whole site:
User-agent: Claude-SearchBot Disallow: /
To block only part of your site, list those paths instead:
User-agent: Claude-SearchBot Disallow: /private/
A group that names Claude-SearchBot replaces your User-agent: * rules for it entirely (RFC 9309). If your catch-all group blocks paths that Claude-SearchBot should also stay out of, repeat them in its group. To let Claude-SearchBot in while a catch-all rule blocks others:
User-agent: * Disallow: / User-agent: Claude-SearchBot Allow: /
“Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt.”
What happens if you block Claude-SearchBot?
“Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.”
How to verify requests from Claude-SearchBot
Anthropic publishes the IP addresses Claude-SearchBot uses. Check a request's IP address against that list before trusting its user agent, which any client can fake.
“If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.”
Other Anthropic crawlers
- ClaudeBot · AI training
- Claude-User · User-triggered
See every AI crawler and how often top sites block it
Sources
Rather have an expert do it?
Tell us about your site and goals, and we'll match you with an AI visibility (GEO) specialist. Free, no obligation.