AI readiness
The AI readiness app checks whether AI assistants and AI search engines
can read and cite each crawled page: whether robots.txt lets their crawlers
in, whether the text is in the HTML the server sends, whether the page
forbids snippets, and whether it carries the structured data and metadata an
assistant uses to judge a source. It rides the same crawl as Broken Links (no
extra page requests, no separate metering) apart from one GET /llms.txt per
run, and refreshes on every crawl. Only the raw HTML is read; the page is not
rendered.
Install it with aiEnabled at registration
(POST /v1/sites) or later
(PATCH /v1/sites/{id}). When aiEnabled is
omitted at registration it defaults on only if another crawl app (Broken
Links, Accessibility or SEO) is on, so it never schedules a crawl by itself;
send aiEnabled: true to run it on its own. The report is served
at GET /v1/sites/{id}/ai-report.
What is checked
Section titled “What is checked”| Rule | Severity | Fires when |
|---|---|---|
ai-search-bot-blocked |
critical | robots.txt disallows the page for a major AI search bot, such as ChatGPT’s (see the table below). The finding names the product first |
ai-other-search-bot-blocked |
moderate | robots.txt disallows the page only for the smaller search bots (Applebot, DuckAssistBot, MistralAI-User) |
ai-content-js-only |
critical | the raw HTML looks like a JavaScript app shell and holds fewer than 50 words |
ai-thin-html |
minor | fewer than 100 words in the raw HTML, and no app-shell markers |
ai-noindex |
serious | a robots meta tag or X-Robots-Tag header carries noindex (or none) for all crawlers or a catalogued search bot |
ai-nosnippet |
serious | nosnippet, none or max-snippet:0 is set, which keeps the page out of quoted answers |
ai-max-snippet-low |
moderate | max-snippet is above 0 but below 50 characters |
ai-data-nosnippet-heavy |
moderate | more than half of the page’s words sit inside data-nosnippet elements |
ai-no-structured-data |
moderate | the page has no valid JSON-LD |
ai-no-organization |
moderate, site-wide | no page describes the organization (Organization or LocalBusiness) in structured data, or the description has no sameAs links |
ai-article-no-author |
minor | an Article in structured data names no author |
ai-no-modified-date |
minor | no dateModified in structured data, no article:modified_time meta tag and no lastmod in the sitemap |
ai-lang-missing |
moderate | the <html> element has no lang attribute |
ai-heading-structure |
minor | a page of 300 words or more has no H2, or heading levels are skipped |
ai-no-organization is evaluated once for the whole site rather than per
page; for scoring it counts as affecting every scanned page.
Robots directives are read per value: a name: prefix (for example
googlebot: noindex) scopes the rest of that value to that one agent, and only
the search bots in the table below count.
Bots we check
Section titled “Bots we check”Overwatch reads robots.txt once per run and checks every crawled page
against each bot’s product token (matching is case-insensitive). Search
bots decide whether your site can appear in AI answers, so they are scored.
Training bots only decide whether content may be used to train models; that
is a business choice, so it is shown but never scored.
| Bot | Operator | What it does | Kind |
|---|---|---|---|
OAI-SearchBot |
OpenAI | Surfaces sites in ChatGPT search | search, scored (major) |
ChatGPT-User |
OpenAI | Fetches pages when a ChatGPT user asks for them | search, scored (major) |
Claude-SearchBot |
Anthropic | Indexes pages for Claude search results | search, scored (major) |
Claude-User |
Anthropic | Fetches pages when a Claude user asks for them | search, scored (major) |
PerplexityBot |
Perplexity | Indexes pages for Perplexity search results | search, scored (major) |
Perplexity-User |
Perplexity | Fetches pages when a Perplexity user asks for them | search, scored (major) |
Googlebot |
Google Search, which feeds AI Overviews and AI Mode | search, scored (major) | |
Bingbot |
Microsoft | Bing index, which feeds Copilot and partner products | search, scored (major) |
Applebot |
Apple | Powers Siri, Spotlight and Safari suggestions | search, scored (other) |
DuckAssistBot |
DuckDuckGo | Fetches pages in real time for DuckAssist answers | search, scored (other) |
MistralAI-User |
Mistral | Fetches pages when a Mistral user asks for them | search, scored (other) |
GPTBot |
OpenAI | Collects content that may train OpenAI models | training, noted |
ClaudeBot |
Anthropic | Collects content that may train Anthropic models | training, noted |
Google-Extended |
Opts content out of Gemini model training | training, noted | |
Applebot-Extended |
Apple | Opts content out of Apple foundation model training | training, noted |
CCBot |
Common Crawl | Builds the open Common Crawl dataset, widely used for training | training, noted |
meta-externalagent |
Meta | Collects content to train Meta’s AI models | training, noted |
Bytespider |
ByteDance | Collects content to train ByteDance’s AI models | training, noted |
Google-Extended also controls whether Gemini apps can use your pages to
ground their answers, not only training. Blocking it can therefore cost you
more than a training opt-out suggests.
The report rolls the per-page results up into one status per bot: allowed
(no page blocked), partial (some pages blocked) or blocked (every page
blocked).
Notes (never scored)
Section titled “Notes (never scored)”Some facts are useful context but not something to penalise:
- Training bots. Training bots are shown in the report’s training-bot list with their allowed, partial or blocked status. Blocking training is a legitimate choice and does not change the score.
llms.txt. Whether/llms.txtexists (served as plain text or markdown; an HTML page returned with status 200 does not count). It is reported for information only; it is not a standard that assistants are known to rely on, so it never affects the score.- Cloudflare. Whether the front page is served through Cloudflare. If it
is, blocking can also happen at the network edge, regardless of
robots.txt; see Cloudflare AI Crawl Control.
Score and verdict
Section titled “Score and verdict”Each run gets a 0–100 score using the same method as the
SEO score: every rule is penalised by its worst severity
(critical 10, serious 7, moderate 3, minor 1) and by the share of pages it
affects, so one template-wide problem counts as one systemic fix, not hundreds
of independent hits. The penalty p becomes round(100 − 100p/(p+25)), and
only a completely clean run scores 100 (anything else is capped at 99).
Site-wide rules count as affecting every scanned page. Rules are ranked in the
report by how many score points fixing them would recover.
The overall chip is separate from the score:
| Chip | Shown when |
|---|---|
| Blocked from AI search | a major search bot is blocked on at least a quarter of the scanned pages |
| Hard for AI to read | otherwise, critical or serious findings (for example JavaScript-only content or noindex) affect at least a quarter of the scanned pages; a site-wide one counts as every page |
| Readable by AI | none of the above |
AI readiness is informational. It does not count toward the site’s overall health verdict.
AI access alert
Section titled “AI access alert”Losing AI access is usually an accident: a deploy that adds noindex, or a
robots.txt that gains a Disallow. An AI-access-lost alert (own
recipient list under Notifications) fires when a crawl
finds any of these after a run that found none:
- the chip reads Blocked from AI search;
- a major search bot is blocked on any scanned page, even just one;
noindex, ornosnippet, covers at least a quarter of the scanned pages (each counted on its own).
It’s transition-based: one alert when access is lost, reset silently once it’s
restored. A site’s first AI readiness run only records its state, so a site
that was already blocked when the app was installed isn’t alerted. Generic
webhooks receive an ai.blocked event.
What this score doesn’t measure
Section titled “What this score doesn’t measure”This score measures whether AI assistants can read and cite your site — not whether they currently mention it.
Readiness is about your side of the door: can a crawler get in, can it read what is there, and does the page carry the signals that make it a credible source. A site can score 100 and never be named in an answer, and a poorly built site can occasionally be quoted anyway.
Whether assistants actually mention or cite you is visibility, and it can only be observed by asking the assistants. That is what prompt tracking does: a tool keeps a list of questions your customers might ask (the “prompts”), runs them regularly against ChatGPT, Perplexity, Google’s AI features and others, and records whether your brand or site appears, in what position, and which sources are cited. Answers vary from run to run, so these tools report trends over many runs rather than a single truth.
Overwatch does not do this. Tools in the category include Profound, Peec AI, the Semrush AI Visibility Toolkit and Ahrefs’ Brand Radar. They are named only as examples of the category, and their coverage and pricing change often.
If you would like help choosing a visibility tool or setting one up, WEAREHEAVY can help.
Cloudflare AI Crawl Control
Section titled “Cloudflare AI Crawl Control”robots.txt is a request, not a lock. If your site is served through
Cloudflare, AI crawlers can be blocked at the edge even when robots.txt
allows them, and the AI readiness report cannot see that: it reads
robots.txt and the HTML it is served, not your Cloudflare configuration. When
the report notes that the site appears to be behind Cloudflare, check:
- In the Cloudflare dashboard, open the zone and go to AI Crawl Control (previously AI Audit).
- On the Crawlers tab, look at each crawler’s action. Search crawlers such
as
OAI-SearchBot,ChatGPT-User,Claude-SearchBot,PerplexityBotandGooglebotshould be allowed if you want to appear in AI answers. - Check the account’s Security and Bots settings for any blanket “block AI bots” toggle or custom rule that matches these user agents.
Blocking training crawlers there is fine and does not affect this score; it is a blocked search crawler that keeps you out of answers. Cloudflare’s menus move around, so see Cloudflare’s AI Crawl Control documentation for the current layout.
Retroactive settings
Section titled “Retroactive settings”Issues are derived at read time from per-page snapshots, so changing the
site’s excludePaths globs immediately updates the report. The score stored on
each run feeds the trend chart and is refreshed when the globs change.
Uninstalling
Section titled “Uninstalling”Toggling the app off stops the extraction and makes the report respond 404;
past runs’ snapshots are kept, so the report returns when the app is
re-installed. If AI readiness is the only crawl-consuming app on the site,
turning it off also unschedules the crawl — see Apps.