AI crawler · You.com
YouBot: what it is and how to block it
YouBot crawls and indexes the web for the You.com search engine. You.com documents three ways to verify its requests, including cryptographic signatures.
BrandVector Editorial · Last verified Oct 8, 2026
- Operator
- You.com
- Type
- Search
- robots.txt token
YouBot- Follows robots.txt
- Yes, per You.com
- IP ranges
- Not published
- Blocked by name
- 8.0% of top sites
What is YouBot?
YouBot is You.com's search crawler: it indexes pages for search results. In You.com's words:
“YouBot is the web crawler that powers the You.com search engine. It automatically discovers and indexes web pages to provide real-time, accurate search results for You.com users.”
“YouBot caches robots.txt for 30 minutes.”
YouBot user agent string
You.com documents this user agent string for YouBot:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36
In robots.txt, use the token YouBot, not the full string. Crawlers match on the token, and version numbers in the full string can change.
How many websites block YouBot?
In our Oct 8, 2026 check of 817 of the most-visited websites, 8.0% block YouBot by name for their whole site (65 sites).
| robots.txt rule for YouBot | Sites | Share |
|---|---|---|
| Blocks the whole site by name | 65 | 8.0% |
| Blocks part of the site by name | 22 | 2.7% |
| Names it but doesn't block it | 1 | 0.1% |
| Blocked only by a catch-all (*) rule | 35 | 4.3% |
For comparison, 0.1% of the same sites block Googlebot by name. The sites are the top 1,000 origins in the Chrome UX Report for August 2026; see how we measured.
How to block YouBot in robots.txt
Add this group to the robots.txt file at the root of your domain to block YouBot from your whole site:
User-agent: YouBot Disallow: /
To block only part of your site, list those paths instead:
User-agent: YouBot Disallow: /private/
A group that names YouBot replaces your User-agent: * rules for it entirely (RFC 9309). If your catch-all group blocks paths that YouBot should also stay out of, repeat them in its group. To let YouBot in while a catch-all rule blocks others:
User-agent: * Disallow: / User-agent: YouBot Allow: /
“Yes, YouBot fully respects robots.txt directives, including user-agent specific rules and crawl-delay settings.”
What happens if you block YouBot?
You.com doesn't document what blocking YouBot changes.
How to verify requests from YouBot
“You.com supports three ways to verify that requests are genuinely from YouBot: cryptographic signatures (recommended), reverse DNS checks, and IP range validation.”
See every AI crawler and how often top sites block it
Sources
Rather have an expert do it?
Tell us about your site and goals, and we'll match you with an AI visibility (GEO) specialist. Free, no obligation.