AI crawler · Meta
Meta-ExternalFetcher: what it is and how to block it
Meta-ExternalFetcher fetches individual links when a user asks, including to help Meta's AI agents complete tasks. Meta says it may bypass robots.txt because a user requested the fetch.
BrandVector Editorial · Last verified Oct 8, 2026
- Operator
- Meta
- Type
- User-triggered
- robots.txt token
Meta-ExternalFetcher- Follows robots.txt
- Partly, per Meta
- IP ranges
- Not published
- Blocked by name
- 5.1% of top sites
What is Meta-ExternalFetcher?
Meta-ExternalFetcher is Meta's user-triggered crawler: it fetches a page when a user asks an AI assistant to. In Meta's words:
“The Meta-ExternalFetcher crawler fetches individual links at a user’s request and supports product functions such as evaluating and improving agentic AI capabilities—including helping AI navigate websites to complete tasks for users.”
Meta-ExternalFetcher user agent string
Meta documents these user agent strings for Meta-ExternalFetcher:
meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers)
meta-externalfetcher/1.1
In robots.txt, use the token Meta-ExternalFetcher, not the full string. Crawlers match on the token, and version numbers in the full string can change.
How many websites block Meta-ExternalFetcher?
In our Oct 8, 2026 check of 817 of the most-visited websites, 5.1% block Meta-ExternalFetcher by name for their whole site (42 sites).
| robots.txt rule for Meta-ExternalFetcher | Sites | Share |
|---|---|---|
| Blocks the whole site by name | 42 | 5.1% |
| Blocks part of the site by name | 11 | 1.3% |
| Names it but doesn't block it | 1 | 0.1% |
| Blocked only by a catch-all (*) rule | 29 | 3.5% |
For comparison, 0.1% of the same sites block Googlebot by name. The sites are the top 1,000 origins in the Chrome UX Report for August 2026; see how we measured.
How to block Meta-ExternalFetcher in robots.txt
Add this group to the robots.txt file at the root of your domain to block Meta-ExternalFetcher from your whole site:
User-agent: Meta-ExternalFetcher Disallow: /
To block only part of your site, list those paths instead:
User-agent: Meta-ExternalFetcher Disallow: /private/
A group that names Meta-ExternalFetcher replaces your User-agent: * rules for it entirely (RFC 9309). If your catch-all group blocks paths that Meta-ExternalFetcher should also stay out of, repeat them in its group. To let Meta-ExternalFetcher in while a catch-all rule blocks others:
User-agent: * Disallow: / User-agent: Meta-ExternalFetcher Allow: /
Meta says robots.txt doesn't fully control Meta-ExternalFetcher. To stop it completely, block it at your firewall or CDN using the verification details below.
“Accordingly, this crawler may bypass robots.txt rules.”
What happens if you block Meta-ExternalFetcher?
Meta doesn't document what blocking Meta-ExternalFetcher changes.
How to verify requests from Meta-ExternalFetcher
Meta doesn't publish IP ranges or a verification method for Meta-ExternalFetcher. Any client can send its user agent string, so treat requests that claim to be Meta-ExternalFetcher as unverified.
Other Meta crawlers
- Meta-ExternalAgent · AI training
- Meta-WebIndexer · Search
See every AI crawler and how often top sites block it
Sources
Rather have an expert do it?
Tell us about your site and goals, and we'll match you with an AI visibility (GEO) specialist. Free, no obligation.