LLMentions
Docs · Site audit

Site audit

Check whether AI answer engines can read and cite your site. Crawl as AI bots see it, check robots.txt per bot, find JavaScript-only content, and verify real bot traffic from your logs.

↓ Markdown for agents

An engine can only cite what it can read. Most sites that are invisible in AI answers have a mechanical cause: robots.txt blocks the bots that build AI search indexes, a CDN rule challenges them, or the main content only appears after JavaScript runs, which AI crawlers don't do.

site_audit finds those causes and tells you how to fix each one. bots_report then confirms, from your own server logs, which bots actually visit.

Run an audit

terminal
llmentions audit acme.com

Or with MCP (site_audit) or REST:

terminal
curl -s https://api.llmentions.com/v1/site/audit \
  -H "authorization: Bearer $LLMENTIONS_API_KEY" \
  -H 'content-type: application/json' \
  -d '{"url": "https://acme.com", "max_pages": 100, "render": 5}'

What the audit does:

  1. Reads robots.txt for every known AI and search bot, and reports which ones are allowed or blocked: training bots, AI search indexers and live user-fetch bots are listed separately, because blocking each has different consequences.
  2. Fetches your homepage as an AI bot and as a browser and compares the two, to catch CDN and WAF rules that challenge AI user agents.
  3. Crawls up to max_pages pages (default 25, at most 100) without JavaScript, the way AI crawlers see them, and records the text, title, headings, canonical, structured data and response time.
  4. Renders up to render pages in a real browser (default 3, at most 10; homepage and thinnest pages first) and compares the text, to find content that only exists after JavaScript.
  5. Lists issues most-severe first, each with the pages affected and a fix.

Audits above 25 pages run in the background; poll audit_get (or llmentions audits acme.com). Every audit is stored, and each new one reports what changed since the previous one: new issues, fixed issues and AI-bot access changes.

Example output

text
acme.com  · 48 pages · 5 rendered · 2.1 s
AI access   OAI-SearchBot ✓  ChatGPT-User ✓  PerplexityBot ✗  Claude-SearchBot ✓  GPTBot ✗ (training)

ERROR    ai_search_blocked      robots.txt blocks an AI search or answer bot   PerplexityBot
         → allow search_index and user_fetch bots; blocking them keeps you out of AI answers
ERROR    content_needs_js       Most of the content only appears after JavaScript runs   12 pages
         → server-render (SSR/SSG) the main content
WARNING  slow_response          Slow server response (over 800 ms)              3 pages
NOTICE   no_structured_data     No JSON-LD structured data                     48 pages

since last audit: 1 new, 2 fixed, 0 changed issue(s); 0 AI-bot access change(s)

Every check, what it means and how to fix it is listed in the audit checks reference.

Price

$0.0002 per page crawled and $0.002 per page rendered. A 100-page audit with 5 rendered pages costs $0.03. Test keys audit for free.

Fail CI when your site stops being readable

terminal
llmentions audit https://preview-123.acme.dev --fail-on regression

--fail-on takes error, warning or regression, and exits 1 when it trips. There is a ready-made GitHub Action in CLI → In CI.

Verify real bot traffic

Your robots.txt says who may crawl. Your logs say who actually did. Send a raw access log (up to 20 MB) to bots_report:

terminal
llmentions bots /var/log/nginx/access.log

You get hits, status codes and top pages per bot, the live AI fetches (a proxy for real-time citations), and IP verification: a hit claiming to be GPTBot from outside OpenAI's published ranges is flagged as spoofed. bots_report is free.

The AI crawler directory lists every bot we recognise, what it's for, and whether you should allow it.