---
title: Site audit
description: Check whether AI answer engines can read and cite your site. Crawl as AI bots see it, check robots.txt per bot, find JavaScript-only content, and verify real bot traffic from your logs.
group: Guides
---

# Site audit

An engine can only cite what it can read. Most sites that are invisible in AI answers have a mechanical cause: robots.txt blocks the bots that build AI search indexes, a CDN rule challenges them, or the main content only appears after JavaScript runs, which AI crawlers don't do.

`site_audit` finds those causes and tells you how to fix each one. `bots_report` then confirms, from your own server logs, which bots actually visit.

## Run an audit

```bash
llmentions audit acme.com
```

Or with MCP (`site_audit`) or REST:

```bash
curl -s https://api.llmentions.com/v1/site/audit \
  -H "authorization: Bearer $LLMENTIONS_API_KEY" \
  -H 'content-type: application/json' \
  -d '{"url": "https://acme.com", "max_pages": 100, "render": 5}'
```

What the audit does:

1. **Reads robots.txt for every known AI and search bot**, and reports which ones are allowed or blocked: training bots, AI search indexers and live user-fetch bots are listed separately, because blocking each has different consequences.
2. **Fetches your homepage as an AI bot and as a browser** and compares the two, to catch CDN and WAF rules that challenge AI user agents.
3. **Crawls up to `max_pages` pages (default 25, at most 100) without JavaScript**, the way AI crawlers see them, and records the text, title, headings, canonical, structured data and response time.
4. **Renders up to `render` pages in a real browser** (default 3, at most 10; homepage and thinnest pages first) and compares the text, to find content that only exists after JavaScript.
5. **Lists issues most-severe first**, each with the pages affected and a fix.

Audits above 25 pages run in the background; poll `audit_get` (or `llmentions audits acme.com`). Every audit is stored, and each new one reports what changed since the previous one: new issues, fixed issues and AI-bot access changes.

## Example output

```text
acme.com  · 48 pages · 5 rendered · 2.1 s
AI access   OAI-SearchBot ✓  ChatGPT-User ✓  PerplexityBot ✗  Claude-SearchBot ✓  GPTBot ✗ (training)

ERROR    ai_search_blocked      robots.txt blocks an AI search or answer bot   PerplexityBot
         → allow search_index and user_fetch bots; blocking them keeps you out of AI answers
ERROR    content_needs_js       Most of the content only appears after JavaScript runs   12 pages
         → server-render (SSR/SSG) the main content
WARNING  slow_response          Slow server response (over 800 ms)              3 pages
NOTICE   no_structured_data     No JSON-LD structured data                     48 pages

since last audit: 1 new, 2 fixed, 0 changed issue(s); 0 AI-bot access change(s)
```

Every check, what it means and how to fix it is listed in the [audit checks reference](/checks/).

## Price

$0.0002 per page crawled and $0.002 per page rendered. A 100-page audit with 5 rendered pages costs $0.03. Test keys audit for free.

## Fail CI when your site stops being readable

```bash
llmentions audit https://preview-123.acme.dev --fail-on regression
```

`--fail-on` takes `error`, `warning` or `regression`, and exits `1` when it trips. There is a ready-made GitHub Action in [CLI → In CI](/docs/cli/#in-ci-github-actions).

## Verify real bot traffic

Your robots.txt says who may crawl. Your logs say who actually did. Send a raw access log (up to 20 MB) to `bots_report`:

```bash
llmentions bots /var/log/nginx/access.log
```

You get hits, status codes and top pages per bot, the live AI fetches (a proxy for real-time citations), and **IP verification**: a hit claiming to be GPTBot from outside OpenAI's published ranges is flagged as spoofed. `bots_report` is free.

The [AI crawler directory](/bots/) lists every bot we recognise, what it's for, and whether you should allow it.
