# robots.txt blocks an AI search or answer bot

AI search bots (like OAI-SearchBot, Claude-SearchBot and PerplexityBot) build the index that AI answers cite, and user-fetch bots (like ChatGPT-User) read your page live when someone asks. Block them and you can not be cited, however good the page is.

## How to fix it

Allow the bots listed in ai_access with class search_index or user_fetch; blocking them keeps you out of AI answers (blocking training bots alone is a separate choice).

## How LLMentions checks it

`site_audit` crawls your site the way AI bots do (raw HTML, no JavaScript), reads robots.txt for every known AI and search bot, and renders a few pages in a real browser for comparison. This check reports as `ai_search_blocked` with severity **error**, which fails CI with `--fail-on error`.

```bash
llmentions audit example.com
```

The result lists every issue with the pages affected, most severe first, and what changed since your last audit. See the [site audit guide](/docs/site-audit/).

## Related errors

- [Site isn't served over HTTPS](/checks/not-https/)
- [Page returns an HTTP error](/checks/http-error/)
- [Almost no text without JavaScript](/checks/js-only-content/)
- [Most of the content only appears after JavaScript runs](/checks/content-needs-js/)

---
https://llmentions.com/checks/ai-search-blocked/
