LLMentions
Audit checks · ai_search_blocked

robots.txt blocks an AI search or answer bot

robots.txt blocks an AI search or answer bot: why it keeps you out of AI answers and how to fix it. Audit check ai_search_blocked (error).

↓ Markdown for agents
Check id
ai_search_blocked
Severity
error
Tool
site_audit

AI search bots (like OAI-SearchBot, Claude-SearchBot and PerplexityBot) build the index that AI answers cite, and user-fetch bots (like ChatGPT-User) read your page live when someone asks. Block them and you can not be cited, however good the page is.

How to fix it

Allow the bots listed in ai_access with class search_index or user_fetch; blocking them keeps you out of AI answers (blocking training bots alone is a separate choice).

How LLMentions checks it

site_audit crawls your site the way AI bots do (raw HTML, no JavaScript), reads robots.txt for every known AI and search bot, and renders a few pages in a real browser for comparison. This check reports as ai_search_blocked with severity error, which fails CI with --fail-on error.

terminal
llmentions audit example.com

The result lists every issue with the pages affected, most severe first, and what changed since your last audit. See the site audit guide.