AI crawlers · Googlebot
Googlebot: Google's search engine crawler
Googlebot is Google's search engine crawler. What it does, whether to allow it, robots.txt rules and how to verify real Googlebot traffic.
↓ Markdown for agents- Vendor
- Type
- Search engine crawler
- Runs JavaScript
- Yes
- IP ranges published
- Yes
Googlebot is Google's search crawler. It indexes pages for Google Search, including AI Overviews and AI Mode.
Should you allow Googlebot?
Allow it. Googlebot powers Google Search, including AI Overviews and AI Mode. Blocking it removes you from that search engine, including AI Overviews and AI Mode.
robots.txt
To allow Googlebot everywhere:
User-agent: Googlebot
Allow: /To block it:
User-agent: Googlebot
Disallow: /Bots follow the most specific User-agent group that matches them. If you have a User-agent: Googlebot group, the rules under User-agent: * no longer apply to it. Matching on the token is case-insensitive.
Identifying Googlebot
Googlebot identifies itself with Googlebot in its User-Agent header. It renders JavaScript, but rendering is deferred and costly, so server-rendered content is still read faster and more reliably.
Google publishes the IP ranges Googlebot crawls from at developers.google.com/static/crawling/ipranges/common-crawlers.json. Anyone can fake a user agent, so a hit claiming to be Googlebot from outside those ranges is spoofed. LLMentions' bots_report checks this for you.
Check your site
llmentions audit example.com # is Googlebot allowed by robots.txt, and what does it see?
llmentions bots /var/log/nginx/access.log # did Googlebot really visit, and is it genuine?Or identify any user agent over REST (free): GET https://api.llmentions.com/v1/bots/lookup?ua=Googlebot.