AI crawler · Anthropic
Claude-User: what it is and how to block it
Claude-User is the agent Claude uses to visit a page when a user asks it a question. Unlike some user-triggered agents, Anthropic says site owners can control it with robots.txt.
BrandVector Editorial · Last verified Oct 8, 2026
- Operator
- Anthropic
- Type
- User-triggered
- robots.txt token
Claude-User- Follows robots.txt
- Yes, per Anthropic
- IP ranges
- Published
- Blocked by name
- 4.9% of top sites
What is Claude-User?
Claude-User is Anthropic's user-triggered crawler: it fetches a page when a user asks an AI assistant to. In Anthropic's words:
“Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.”
“Please do this for every subdomain that you wish to opt out from.”
Claude-User user agent string
Anthropic doesn't publish a full user agent string for Claude-User. In robots.txt, use the token Claude-User.
How many websites block Claude-User?
In our Oct 8, 2026 check of 817 of the most-visited websites, 4.9% block Claude-User by name for their whole site (40 sites).
| robots.txt rule for Claude-User | Sites | Share |
|---|---|---|
| Blocks the whole site by name | 40 | 4.9% |
| Blocks part of the site by name | 21 | 2.6% |
| Names it but doesn't block it | 6 | 0.7% |
| Blocked only by a catch-all (*) rule | 31 | 3.8% |
For comparison, 0.1% of the same sites block Googlebot by name. The sites are the top 1,000 origins in the Chrome UX Report for August 2026; see how we measured.
How to block Claude-User in robots.txt
Add this group to the robots.txt file at the root of your domain to block Claude-User from your whole site:
User-agent: Claude-User Disallow: /
To block only part of your site, list those paths instead:
User-agent: Claude-User Disallow: /private/
A group that names Claude-User replaces your User-agent: * rules for it entirely (RFC 9309). If your catch-all group blocks paths that Claude-User should also stay out of, repeat them in its group. To let Claude-User in while a catch-all rule blocks others:
User-agent: * Disallow: / User-agent: Claude-User Allow: /
“Claude-User allows site owners to control which sites can be accessed through these user-initiated requests.”
What happens if you block Claude-User?
“Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.”
How to verify requests from Claude-User
Anthropic publishes the IP addresses Claude-User uses. Check a request's IP address against that list before trusting its user agent, which any client can fake.
“If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.”
Other Anthropic crawlers
- ClaudeBot · AI training
- Claude-SearchBot · Search
See every AI crawler and how often top sites block it
Sources
Rather have an expert do it?
Tell us about your site and goals, and we'll match you with an AI visibility (GEO) specialist. Free, no obligation.