AI crawlers
AI crawler directory
Every AI and search crawler that matters for AI visibility: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and more. What each does, whether to allow it, and robots.txt rules.
↓ Markdown for agentsWhich bots should you let in? It depends on what each one is for. AI search indexers and live user fetches decide whether AI answers can cite you, so blocking them removes you from answers. Training crawlers are a separate choice. None of the AI bots run JavaScript.
Check your own site with llmentions audit example.com, and see which bots really visit with llmentions bots access.log. Both are in the site audit guide.
Search
GooglebotGooglebot is Google's search engine crawler.BingbotBingbot is Microsoft's search engine crawler.ApplebotApplebot is Apple's search engine crawler.
AI search
OAI-SearchBotOAI-SearchBot is OpenAI's AI search indexer.Claude-SearchBotClaude-SearchBot is Anthropic's AI search indexer.PerplexityBotPerplexityBot is Perplexity's AI search indexer.
User fetch
ChatGPT-UserChatGPT-User is OpenAI's live AI user fetch.Claude-UserClaude-User is Anthropic's live AI user fetch.Perplexity-UserPerplexity-User is Perplexity's live AI user fetch.DuckAssistBotDuckAssistBot is DuckDuckGo's live AI user fetch.MistralAI-UserMistralAI-User is Mistral's live AI user fetch.
Training
GPTBotGPTBot is OpenAI's AI training crawler.ClaudeBotClaudeBot is Anthropic's AI training crawler.Meta-ExternalAgentMeta-ExternalAgent is Meta's AI training crawler.AmazonbotAmazonbot is Amazon's AI training crawler.BytespiderBytespider is ByteDance's AI training crawler.CCBotCCBot is Common Crawl's AI training crawler.