AI crawlers · OAI-SearchBot
OAI-SearchBot: OpenAI's AI search indexer
OAI-SearchBot is OpenAI's AI search indexer. What it does, whether to allow it, robots.txt rules and how to verify real OAI-SearchBot traffic.
↓ Markdown for agents- Vendor
- OpenAI
- Type
- AI search indexer
- Runs JavaScript
- No
- IP ranges published
- Yes
OAI-SearchBot is OpenAI's crawler that builds the search index behind ChatGPT search. When someone asks a question, the answer is grounded in, and cites, pages this bot has indexed.
Should you allow OAI-SearchBot?
Allow it. If OAI-SearchBot can't crawl you, ChatGPT search can't cite you, no matter how good your content is. Blocking it is the most common mechanical reason a brand is missing from AI answers.
robots.txt
To allow OAI-SearchBot everywhere:
User-agent: OAI-SearchBot
Allow: /To block it:
User-agent: OAI-SearchBot
Disallow: /Bots follow the most specific User-agent group that matches them. If you have a User-agent: OAI-SearchBot group, the rules under User-agent: * no longer apply to it. Matching on the token is case-insensitive.
Identifying OAI-SearchBot
OAI-SearchBot identifies itself with OAI-SearchBot in its User-Agent header. It does not run JavaScript: it reads the raw HTML only. Content your site adds with JavaScript is invisible to it.
OpenAI publishes the IP ranges OAI-SearchBot crawls from at openai.com/searchbot.json. Anyone can fake a user agent, so a hit claiming to be OAI-SearchBot from outside those ranges is spoofed. LLMentions' bots_report checks this for you.
Check your site
llmentions audit example.com # is OAI-SearchBot allowed by robots.txt, and what does it see?
llmentions bots /var/log/nginx/access.log # did OAI-SearchBot really visit, and is it genuine?Or identify any user agent over REST (free): GET https://api.llmentions.com/v1/bots/lookup?ua=OAI-SearchBot.
Other OpenAI bots
- GPTBot: AI training crawler
- ChatGPT-User: live AI user fetch