Google-Agent: what it is and how to block it

Operator

Google

Type

User-triggered

robots.txt token

None documented

Follows robots.txt

No, per Google

IP ranges

Published

What is Google-Agent?

Google-Agent is Google's user-triggered crawler: it fetches a page when a user asks an AI assistant to. In Google's words:

“Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.”

Google-Agent user agent string

Google documents these user agent strings for Google-Agent:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36

How to block Google-Agent

Google doesn't document a robots.txt token for Google-Agent, and says these fetchers generally ignore robots.txt. To block it, filter requests by its user agent or Google's published IP ranges at your firewall or CDN.

“Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.”

What happens if you block Google-Agent?

Google doesn't document what blocking Google-Agent changes.

How to verify requests from Google-Agent

Google publishes the IP addresses Google-Agent uses. Check a request's IP address against that list before trusting its user agent, which any client can fake.

“A subset of requests made by the Google-Agent are signed with Web Bot Auth; in these cases, they are authenticated as https://agent.bot.goog.”

Other Google crawlers

• Google-Extended · Control token
• Google-CloudVertexBot · Owner-requested

See every AI crawler and how often top sites block it