AI crawler list: every AI user agent and who blocks it

All AI crawlers

“Blocked by name” is the share of top websites whose robots.txt names the crawler and disallows their whole site. See the methodology.

Types of AI crawlers

AI Overviews and Copilot don't have their own crawlers

Some AI search features use their company's main search crawler, so there's no AI-specific token to block. Blocking Googlebot or Bingbot would remove you from regular search too, so both companies point to page-level controls instead.

How to block AI training but stay in AI search

This robots.txt group blocks the crawlers and tokens whose operators say they're used for AI training, and leaves search crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot untouched:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: MistralAI-Training
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
User-agent: AI2Bot
Disallow: /

Consecutive User-agent lines share the rules that follow them. Each operator decides what its token controls, so check the crawler's page before relying on a block, and remember that robots.txt is a request: it doesn't stop a crawler that ignores it.

Methodology

Websites. The 1,000 most-visited origins worldwide in the Chrome UX Report for August 2026, via the CrUX top lists archive. Origins are counted separately, so country versions of one site (such as Google's) each count.

Collection. On Oct 8, 2026, we requested /robots.txt once from each origin, following redirects, with a user agent that identifies BrandVector. 759 sites returned a robots.txt file and 58 had none, which under RFC 9309 means every crawler is allowed. We excluded 183 sites we couldn't read: 103 refused the request (usually bot protection), 38 were unreachable, 1 returned server errors, and 41 returned a web page instead of a robots.txt file. Percentages are of the 817 sites we could check.

Rules. We applied RFC 9309: all groups that name a crawler apply to it (matched case-insensitively on its token), a User-agent: * group applies only when none do, and the longest matching rule wins. “Blocks the whole site” means the root is disallowed with no allowed exceptions.

Limitations. robots.txt states a site's preference. It doesn't show firewall or CDN blocks, or whether crawlers comply, and sites that refused our request may block more crawlers than the ones we could read. We repeat the study monthly.