Audit checks · ai_search_blocked
robots.txt blocks an AI search or answer bot
robots.txt blocks an AI search or answer bot: why it keeps you out of AI answers and how to fix it. Audit check ai_search_blocked (error).
↓ Markdown for agents- Check id
ai_search_blocked- Severity
- error
- Tool
site_audit
AI search bots (like OAI-SearchBot, Claude-SearchBot and PerplexityBot) build the index that AI answers cite, and user-fetch bots (like ChatGPT-User) read your page live when someone asks. Block them and you can not be cited, however good the page is.
How to fix it
Allow the bots listed in ai_access with class search_index or user_fetch; blocking them keeps you out of AI answers (blocking training bots alone is a separate choice).
How LLMentions checks it
site_audit crawls your site the way AI bots do (raw HTML, no JavaScript), reads robots.txt for every known AI and search bot, and renders a few pages in a real browser for comparison. This check reports as ai_search_blocked with severity error, which fails CI with --fail-on error.
llmentions audit example.comThe result lists every issue with the pages affected, most severe first, and what changed since your last audit. See the site audit guide.