Site health crawl

A crawler walks up to 200 pages of the site and reports an on-page score with a count for every issue class — broken links, missing or duplicate titles and descriptions, 4xx and 5xx.

What it is

A two-phase audit: a crawl task is posted to DataForSEO and its id is parked on the site record, then a later pass collects the finished summary. Seven issue classes are counted as errors (broken links, duplicate title, duplicate description, no title, no description, 4xx, 5xx) and everything else in the checks map is counted as warnings. Each completed crawl is stored as its own dated row with the full checks payload.

Also called: technical SEO audit · broken links · missing meta descriptions · site errors

See it
Site health crawl
Won
38%
Avg cycle
11d
Pipeline
$580k
JanSep
The Website › Site health sub-tab: the on-page score, error and warning counts, and the issue-class breakdown from the last crawl. Sample data — no customer information appears here.
How it works
  1. 1startAudit() posts an on_page crawl task for the domain; the returned task id is stored in seo_sites.settings.pendingAuditTask.
  2. 2The daily cron — and a manual 'collect' action — call auditSummary() and only store the result once crawl_progress is 'finished'.
  3. 3Errors are summed across the seven blocking classes; the remainder of the checks become warnings.
  4. 4The row lands in seo_audits with score, page count, errors, warnings and the raw issues map.
  5. 5A non-zero error count triggers a 'health' proposal in the next weekly audit.
Why we built it

The health proposal states the stake in one line: "Crawl errors depress every ranking on the domain at once." Splitting the crawl into post-then-collect was forced by reality — a full crawl outlives any single serverless invocation, so blocking on it would either time out or burn an invocation waiting. Parking the task id and collecting later is the same outbox shape used elsewhere in the platform.

The problem
  • Technical problems degraded every ranking silently.
  • A crawl takes longer than a request, so it could not be run inline.
Sound familiar?
What you get
A dated score and error count for the whole site
Issue classes counted individually, so the biggest problem is obvious
Errors feed straight into the weekly report as a proposal

See it on your own jobs

Twenty minutes, your numbers, no slide deck. We’ll build one of your real buildings in front of you and send you the estimate link at the end — yours to keep either way.

or keep browsing features →