AI crawler · Meta
Meta-WebIndexer: what it is and how to block it
Meta-WebIndexer crawls the web to improve the search results Meta AI uses in its answers. Meta says allowing it helps Meta AI cite and link to your content.
BrandVector Editorial · Last verified Oct 8, 2026
- Operator
- Meta
- Type
- Search
- robots.txt token
Meta-WebIndexer- Follows robots.txt
- Yes, per Meta
- IP ranges
- Not published
- Blocked by name
- 2.8% of top sites
What is Meta-WebIndexer?
Meta-WebIndexer is Meta's search crawler: it indexes pages for search results. In Meta's words:
“The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users. In doing so, Meta analyzes online content to enhance the relevance and accuracy of Meta AI.”
“Please allow up to 24 hours for changes to robots.txt to take effect because crawlers may cache the contents of robots.txt for up to 24 hours.”
Meta-WebIndexer user agent string
Meta documents these user agent strings for Meta-WebIndexer:
meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)
meta-webindexer/1.1
In robots.txt, use the token Meta-WebIndexer, not the full string. Crawlers match on the token, and version numbers in the full string can change.
How many websites block Meta-WebIndexer?
In our Oct 8, 2026 check of 817 of the most-visited websites, 2.8% block Meta-WebIndexer by name for their whole site (23 sites).
| robots.txt rule for Meta-WebIndexer | Sites | Share |
|---|---|---|
| Blocks the whole site by name | 23 | 2.8% |
| Blocks part of the site by name | 6 | 0.7% |
| Names it but doesn't block it | 1 | 0.1% |
| Blocked only by a catch-all (*) rule | 35 | 4.3% |
For comparison, 0.1% of the same sites block Googlebot by name. The sites are the top 1,000 origins in the Chrome UX Report for August 2026; see how we measured.
How to block Meta-WebIndexer in robots.txt
Add this group to the robots.txt file at the root of your domain to block Meta-WebIndexer from your whole site:
User-agent: Meta-WebIndexer Disallow: /
To block only part of your site, list those paths instead:
User-agent: Meta-WebIndexer Disallow: /private/
A group that names Meta-WebIndexer replaces your User-agent: * rules for it entirely (RFC 9309). If your catch-all group blocks paths that Meta-WebIndexer should also stay out of, repeat them in its group. To let Meta-WebIndexer in while a catch-all rule blocks others:
User-agent: * Disallow: / User-agent: Meta-WebIndexer Allow: /
“In order to block these crawlers, add a disallow for the relevant crawler to robots.txt.”
What happens if you block Meta-WebIndexer?
“Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI’s responses.”
How to verify requests from Meta-WebIndexer
Meta doesn't publish IP ranges or a verification method for Meta-WebIndexer. Any client can send its user agent string, so treat requests that claim to be Meta-WebIndexer as unverified.
Other Meta crawlers
- Meta-ExternalAgent · AI training
- Meta-ExternalFetcher · User-triggered
See every AI crawler and how often top sites block it
Sources
Rather have an expert do it?
Tell us about your site and goals, and we'll match you with an AI visibility (GEO) specialist. Free, no obligation.