AI crawler · Diffbot
Diffbot-User: what it is and how to block it
Diffbot-User is the agent Diffbot's software uses when a person browses a URL through it.
BrandVector Editorial · Last verified Oct 8, 2026
- Operator
- Diffbot
- Type
- User-triggered
- robots.txt token
Diffbot-User- Follows robots.txt
- Diffbot doesn't say
- IP ranges
- Not published
- Blocked by name
- 0.7% of top sites
What is Diffbot-User?
Diffbot-User is Diffbot's user-triggered crawler: it fetches a page when a user asks an AI assistant to. In Diffbot's words:
“This is used by requests originating on behalf of a human user browsing a URL using Diffbot software, in response to their input.”
Diffbot-User user agent string
Diffbot doesn't publish a full user agent string for Diffbot-User. In robots.txt, use the token Diffbot-User.
How many websites block Diffbot-User?
In our Oct 8, 2026 check of 817 of the most-visited websites, 0.7% block Diffbot-User by name for their whole site (6 sites).
| robots.txt rule for Diffbot-User | Sites | Share |
|---|---|---|
| Blocks the whole site by name | 6 | 0.7% |
| Blocks part of the site by name | 0 | 0.0% |
| Names it but doesn't block it | 0 | 0.0% |
| Blocked only by a catch-all (*) rule | 38 | 4.7% |
For comparison, 0.1% of the same sites block Googlebot by name. The sites are the top 1,000 origins in the Chrome UX Report for August 2026; see how we measured.
How to block Diffbot-User in robots.txt
Add this group to the robots.txt file at the root of your domain to block Diffbot-User from your whole site:
User-agent: Diffbot-User Disallow: /
To block only part of your site, list those paths instead:
User-agent: Diffbot-User Disallow: /private/
A group that names Diffbot-User replaces your User-agent: * rules for it entirely (RFC 9309). If your catch-all group blocks paths that Diffbot-User should also stay out of, repeat them in its group. To let Diffbot-User in while a catch-all rule blocks others:
User-agent: * Disallow: / User-agent: Diffbot-User Allow: /
Diffbot doesn't say whether Diffbot-User follows robots.txt, so a robots.txt block is a request it may not honor.
What happens if you block Diffbot-User?
Diffbot doesn't document what blocking Diffbot-User changes.
How to verify requests from Diffbot-User
Diffbot doesn't publish IP ranges or a verification method for Diffbot-User. Any client can send its user agent string, so treat requests that claim to be Diffbot-User as unverified.
Other Diffbot crawlers
- Diffbot · Search
See every AI crawler and how often top sites block it
Sources
Rather have an expert do it?
Tell us about your site and goals, and we'll match you with an AI visibility (GEO) specialist. Free, no obligation.