Skip to content

AI readiness

The AI readiness app checks whether AI assistants and AI search engines can read and cite each crawled page: whether robots.txt lets their crawlers in, whether the text is in the HTML the server sends, whether the page forbids snippets, and whether it carries the structured data and metadata an assistant uses to judge a source. It rides the same crawl as Broken Links (no extra page requests, no separate metering) apart from one GET /llms.txt per run, and refreshes on every crawl. Only the raw HTML is read; the page is not rendered.

Install it with aiEnabled at registration (POST /v1/sites) or later (PATCH /v1/sites/{id}). When aiEnabled is omitted at registration it defaults on only if another crawl app (Broken Links, Accessibility or SEO) is on, so it never schedules a crawl by itself; send aiEnabled: true to run it on its own. The report is served at GET /v1/sites/{id}/ai-report.

Rule Severity Fires when
ai-search-bot-blocked critical robots.txt disallows the page for a major AI search bot, such as ChatGPT’s (see the table below). The finding names the product first
ai-other-search-bot-blocked moderate robots.txt disallows the page only for the smaller search bots (Applebot, DuckAssistBot, MistralAI-User)
ai-content-js-only critical the raw HTML looks like a JavaScript app shell and holds fewer than 50 words
ai-thin-html minor fewer than 100 words in the raw HTML, and no app-shell markers
ai-noindex serious a robots meta tag or X-Robots-Tag header carries noindex (or none) for all crawlers or a catalogued search bot
ai-nosnippet serious nosnippet, none or max-snippet:0 is set, which keeps the page out of quoted answers
ai-max-snippet-low moderate max-snippet is above 0 but below 50 characters
ai-data-nosnippet-heavy moderate more than half of the page’s words sit inside data-nosnippet elements
ai-no-structured-data moderate the page has no valid JSON-LD
ai-no-organization moderate, site-wide no page describes the organization (Organization or LocalBusiness) in structured data, or the description has no sameAs links
ai-article-no-author minor an Article in structured data names no author
ai-no-modified-date minor no dateModified in structured data, no article:modified_time meta tag and no lastmod in the sitemap
ai-lang-missing moderate the <html> element has no lang attribute
ai-heading-structure minor a page of 300 words or more has no H2, or heading levels are skipped

ai-no-organization is evaluated once for the whole site rather than per page; for scoring it counts as affecting every scanned page.

Robots directives are read per value: a name: prefix (for example googlebot: noindex) scopes the rest of that value to that one agent, and only the search bots in the table below count.

Overwatch reads robots.txt once per run and checks every crawled page against each bot’s product token (matching is case-insensitive). Search bots decide whether your site can appear in AI answers, so they are scored. Training bots only decide whether content may be used to train models; that is a business choice, so it is shown but never scored.

Bot Operator What it does Kind
OAI-SearchBot OpenAI Surfaces sites in ChatGPT search search, scored (major)
ChatGPT-User OpenAI Fetches pages when a ChatGPT user asks for them search, scored (major)
Claude-SearchBot Anthropic Indexes pages for Claude search results search, scored (major)
Claude-User Anthropic Fetches pages when a Claude user asks for them search, scored (major)
PerplexityBot Perplexity Indexes pages for Perplexity search results search, scored (major)
Perplexity-User Perplexity Fetches pages when a Perplexity user asks for them search, scored (major)
Googlebot Google Google Search, which feeds AI Overviews and AI Mode search, scored (major)
Bingbot Microsoft Bing index, which feeds Copilot and partner products search, scored (major)
Applebot Apple Powers Siri, Spotlight and Safari suggestions search, scored (other)
DuckAssistBot DuckDuckGo Fetches pages in real time for DuckAssist answers search, scored (other)
MistralAI-User Mistral Fetches pages when a Mistral user asks for them search, scored (other)
GPTBot OpenAI Collects content that may train OpenAI models training, noted
ClaudeBot Anthropic Collects content that may train Anthropic models training, noted
Google-Extended Google Opts content out of Gemini model training training, noted
Applebot-Extended Apple Opts content out of Apple foundation model training training, noted
CCBot Common Crawl Builds the open Common Crawl dataset, widely used for training training, noted
meta-externalagent Meta Collects content to train Meta’s AI models training, noted
Bytespider ByteDance Collects content to train ByteDance’s AI models training, noted

Google-Extended also controls whether Gemini apps can use your pages to ground their answers, not only training. Blocking it can therefore cost you more than a training opt-out suggests.

The report rolls the per-page results up into one status per bot: allowed (no page blocked), partial (some pages blocked) or blocked (every page blocked).

Some facts are useful context but not something to penalise:

  • Training bots. Training bots are shown in the report’s training-bot list with their allowed, partial or blocked status. Blocking training is a legitimate choice and does not change the score.
  • llms.txt. Whether /llms.txt exists (served as plain text or markdown; an HTML page returned with status 200 does not count). It is reported for information only; it is not a standard that assistants are known to rely on, so it never affects the score.
  • Cloudflare. Whether the front page is served through Cloudflare. If it is, blocking can also happen at the network edge, regardless of robots.txt; see Cloudflare AI Crawl Control.

Each run gets a 0–100 score using the same method as the SEO score: every rule is penalised by its worst severity (critical 10, serious 7, moderate 3, minor 1) and by the share of pages it affects, so one template-wide problem counts as one systemic fix, not hundreds of independent hits. The penalty p becomes round(100 − 100p/(p+25)), and only a completely clean run scores 100 (anything else is capped at 99). Site-wide rules count as affecting every scanned page. Rules are ranked in the report by how many score points fixing them would recover.

The overall chip is separate from the score:

Chip Shown when
Blocked from AI search a major search bot is blocked on at least a quarter of the scanned pages
Hard for AI to read otherwise, critical or serious findings (for example JavaScript-only content or noindex) affect at least a quarter of the scanned pages; a site-wide one counts as every page
Readable by AI none of the above

AI readiness is informational. It does not count toward the site’s overall health verdict.

Losing AI access is usually an accident: a deploy that adds noindex, or a robots.txt that gains a Disallow. An AI-access-lost alert (own recipient list under Notifications) fires when a crawl finds any of these after a run that found none:

  • the chip reads Blocked from AI search;
  • a major search bot is blocked on any scanned page, even just one;
  • noindex, or nosnippet, covers at least a quarter of the scanned pages (each counted on its own).

It’s transition-based: one alert when access is lost, reset silently once it’s restored. A site’s first AI readiness run only records its state, so a site that was already blocked when the app was installed isn’t alerted. Generic webhooks receive an ai.blocked event.

This score measures whether AI assistants can read and cite your site — not whether they currently mention it.

Readiness is about your side of the door: can a crawler get in, can it read what is there, and does the page carry the signals that make it a credible source. A site can score 100 and never be named in an answer, and a poorly built site can occasionally be quoted anyway.

Whether assistants actually mention or cite you is visibility, and it can only be observed by asking the assistants. That is what prompt tracking does: a tool keeps a list of questions your customers might ask (the “prompts”), runs them regularly against ChatGPT, Perplexity, Google’s AI features and others, and records whether your brand or site appears, in what position, and which sources are cited. Answers vary from run to run, so these tools report trends over many runs rather than a single truth.

Overwatch does not do this. Tools in the category include Profound, Peec AI, the Semrush AI Visibility Toolkit and Ahrefs’ Brand Radar. They are named only as examples of the category, and their coverage and pricing change often.

If you would like help choosing a visibility tool or setting one up, WEAREHEAVY can help.

robots.txt is a request, not a lock. If your site is served through Cloudflare, AI crawlers can be blocked at the edge even when robots.txt allows them, and the AI readiness report cannot see that: it reads robots.txt and the HTML it is served, not your Cloudflare configuration. When the report notes that the site appears to be behind Cloudflare, check:

  1. In the Cloudflare dashboard, open the zone and go to AI Crawl Control (previously AI Audit).
  2. On the Crawlers tab, look at each crawler’s action. Search crawlers such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot and Googlebot should be allowed if you want to appear in AI answers.
  3. Check the account’s Security and Bots settings for any blanket “block AI bots” toggle or custom rule that matches these user agents.

Blocking training crawlers there is fine and does not affect this score; it is a blocked search crawler that keeps you out of answers. Cloudflare’s menus move around, so see Cloudflare’s AI Crawl Control documentation for the current layout.

Issues are derived at read time from per-page snapshots, so changing the site’s excludePaths globs immediately updates the report. The score stored on each run feeds the trend chart and is refreshed when the globs change.

Toggling the app off stops the extraction and makes the report respond 404; past runs’ snapshots are kept, so the report returns when the app is re-installed. If AI readiness is the only crawl-consuming app on the site, turning it off also unschedules the crawl — see Apps.