LLMentions
AI crawlers · ClaudeBot

ClaudeBot: Anthropic's AI training crawler

ClaudeBot is Anthropic's AI training crawler. What it does, whether to allow it, robots.txt rules and how to verify real ClaudeBot traffic.

↓ Markdown for agents
Vendor
Anthropic
Type
AI training crawler
Runs JavaScript
No
IP ranges published
No

ClaudeBot is Anthropic's crawler for collecting training data for Anthropic's Claude models. It reads public pages so future models know about them. It does not fetch pages to answer a specific question.

Should you allow ClaudeBot?

Your choice. Blocking ClaudeBot keeps your content out of future training data. It does not remove you from AI search answers, which use separate bots (Claude-SearchBot and Claude-User). Being in training data can still help models know your brand when they answer without searching.

robots.txt

To allow ClaudeBot everywhere:

text
User-agent: ClaudeBot
Allow: /

To block it:

text
User-agent: ClaudeBot
Disallow: /

Bots follow the most specific User-agent group that matches them. If you have a User-agent: ClaudeBot group, the rules under User-agent: * no longer apply to it. Matching on the token is case-insensitive.

Identifying ClaudeBot

ClaudeBot identifies itself with ClaudeBot in its User-Agent header. It does not run JavaScript: it reads the raw HTML only. Content your site adds with JavaScript is invisible to it.

Anthropic doesn't publish a machine-readable IP range list for ClaudeBot that we verify against, so treat the user agent alone as unverified. bots_report marks these hits as unverifiable rather than genuine.

Check your site

terminal
llmentions audit example.com            # is ClaudeBot allowed by robots.txt, and what does it see?
llmentions bots /var/log/nginx/access.log   # did ClaudeBot really visit, and is it genuine?

Or identify any user agent over REST (free): GET https://api.llmentions.com/v1/bots/lookup?ua=ClaudeBot.

Other Anthropic bots