AI crawler · Anthropic
ClaudeBot: what it is and how to block it
ClaudeBot is Anthropic's crawler for collecting web content that could be used to train its AI models. Blocking it signals that your future content should be left out of Anthropic's training data.
BrandVector Editorial · Last verified Oct 8, 2026
- Operator
- Anthropic
- Type
- AI training
- robots.txt token
ClaudeBot- Follows robots.txt
- Yes, per Anthropic
- IP ranges
- Published
- Blocked by name
- 12% of top sites
What is ClaudeBot?
ClaudeBot is Anthropic's AI training crawler: it collects content that may be used to train AI models. In Anthropic's words:
“ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.”
“To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt.”
“Please do this for every subdomain that you wish to opt out from.”
“Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.”
ClaudeBot user agent string
Anthropic doesn't publish a full user agent string for ClaudeBot. In robots.txt, use the token ClaudeBot.
How many websites block ClaudeBot?
In our Oct 8, 2026 check of 817 of the most-visited websites, 12% block ClaudeBot by name for their whole site (99 sites).
| robots.txt rule for ClaudeBot | Sites | Share |
|---|---|---|
| Blocks the whole site by name | 99 | 12% |
| Blocks part of the site by name | 32 | 3.9% |
| Names it but doesn't block it | 11 | 1.3% |
| Blocked only by a catch-all (*) rule | 21 | 2.6% |
For comparison, 0.1% of the same sites block Googlebot by name. The sites are the top 1,000 origins in the Chrome UX Report for August 2026; see how we measured.
How to block ClaudeBot in robots.txt
Add this group to the robots.txt file at the root of your domain to block ClaudeBot from your whole site:
User-agent: ClaudeBot Disallow: /
To block only part of your site, list those paths instead:
User-agent: ClaudeBot Disallow: /private/
A group that names ClaudeBot replaces your User-agent: * rules for it entirely (RFC 9309). If your catch-all group blocks paths that ClaudeBot should also stay out of, repeat them in its group. To let ClaudeBot in while a catch-all rule blocks others:
User-agent: * Disallow: / User-agent: ClaudeBot Allow: /
“Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt.”
What happens if you block ClaudeBot?
“When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets.”
How to verify requests from ClaudeBot
Anthropic publishes the IP addresses ClaudeBot uses. Check a request's IP address against that list before trusting its user agent, which any client can fake.
“If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.”
Other Anthropic crawlers
- Claude-SearchBot · Search
- Claude-User · User-triggered
See every AI crawler and how often top sites block it
Sources
Rather have an expert do it?
Tell us about your site and goals, and we'll match you with an AI visibility (GEO) specialist. Free, no obligation.