Skip to content
Overwatch API

Crawl behavior

This guide explains what the crawler actually does between “trigger a crawl” and “here is your report”: how runs are scheduled, which URLs are visited, how the crawler stays polite, and — most importantly — what does and doesn’t end up in the dead-link report.

Scheduling

Section titled “Scheduling”

Every site has a re-crawl cadence, cadenceMinutes (default 10080 — weekly). Daily (1440) is the floor — sub-daily cadences are not offered; a site created before the floor keeps (or slows) its cadence but can’t go more frequent. A scheduler wakes every 15 minutes and starts a crawl for each site whose next run is due, so the effective granularity of any cadence is about 15 minutes.

The crawl is shared infrastructure for the Broken Links and Accessibility apps: it runs when either is enabled, and is unscheduled only when both are off. The accessibility render pass runs on its own clock (a11yCadenceMinutes, default monthly) over the crawl’s persisted page set — see the Accessibility guide.

Crawls also start on demand via POST /v1/sites/{id}/crawl. Both paths are idempotent per site: if a crawl is already running, nothing new starts.

A manual crawl records the site’s last_run_at but does not advance a future next_run_at — on-demand crawls don’t shift the schedule. Changing cadenceMinutes reschedules the next run from the last run, so a new frequency takes effect without waiting out the old cycle.

Scope: what gets crawled

Section titled “Scope: what gets crawled”

The site’s rootUrl hostname is the crawl boundary:

  • Internal links (same hostname) are fetched with GET, their links extracted, and newly discovered URLs added to the crawl frontier — the crawl recurses through the site.
  • External links (any other hostname) are checked once with a HEAD request. They are never recursed into.

www. and non-www. are treated as the same host: internal URLs are folded to the site’s canonical host before entering the frontier, so the same page is never crawled twice under both spellings.

Seeding and exclusion:

  • robots.txt is fetched first. Disallow rules are honored (prefix matching; */$ wildcards are a known gap), and a declared Crawl-delay further caps the crawler’s concurrency.
  • Sitemaps referenced from robots.txt seed the frontier, so pages not reachable by links are still checked.
  • excludePaths — up to 200 glob patterns (e.g. /drafts/*) for internal paths the crawler must never enqueue, applied on top of robots.txt.
  • stripParams — up to 200 query-parameter names (e.g. utm_source) dropped from URLs before they enter the frontier, so the same page isn’t crawled once per tracking-parameter value.

Both lists are edited via PATCH /v1/sites/{id}. excludePaths also applies retroactively to reads: the dead-link report, the accessibility report, and the stored accessibility score drop findings from now-excluded pages immediately, without waiting for the next crawl.

The crawler identifies itself as:

wah-overwatch/1.0 (+https://github.com/weareheavy/wah-overwatch)

What counts as broken

Section titled “What counts as broken”

Every checked URL resolves to one of four outcomes; only two of them are reported:

Outcome Meaning In the report?
ok 2xx/3xx — reachable No
broken 404/410 and other 4xx — definitively dead Yes
error Network failure, timeout, TLS error, or persistent 5xx — couldn’t verify Yes
blocked 401/403/429 — the target exists but is gated or throttling No

The distinction matters for editors: broken means “fix this link”, while error means “we couldn’t reach it — it may be down or blocking crawlers”. blocked targets are deliberately kept out of the report; a login-walled page is not a dead link.

Each reported link includes the source page, the target, the status code, any redirect chain, and best-effort locator metadata (link_text, link_selector, link_attrs) so an integration can point the editor at the exact anchor.

Two guards keep the report trustworthy:

  • Re-verification. Before a run finalizes, unverified internal error targets (5xx, timeouts, no-response — never confirmed 4xx, which are deterministic) are re-checked once more at minimum concurrency, after a short settle pause with the origin no longer under crawl load. A page that only failed because it was momentarily struggling drops out of the report.
  • Distinct targets. Report rows are fanned out one per (target × linking page) for attribution, but the headline counts are distinct broken targets — a broken footer link counts once, not once per page of the site.

Politeness and throttling

Section titled “Politeness and throttling”

The crawler keeps a bounded, adaptive number of requests in flight per site (AIMD, like TCP): it slow-starts at 8 concurrent requests, ramps up on clean responses, and halves on any 429/503 — honoring Retry-After when the origin sends one. A throttled page is re-queued and retried, never reported as dead.

maxConcurrency (set at site creation, default 100, clamped to 2–200) is only the ceiling this ramp-up may reach; a robots.txt Crawl-delay caps it further. Fragile origins therefore see gentle traffic even with a high ceiling.

Safety caps

Section titled “Safety caps”

Per run, the crawler stops at whichever comes first:

  • maxPages pages crawled (default 5000; an operator-managed limit),
  • 50,000 link checks,
  • 30 minutes of wall-clock time.

A run that hits a cap finalizes normally and reports what it found so far.