Crawl health & URL inspection

What the indexed count means, the three data sources, URL inspection quotas and reset rules, plus the robots.txt checker and generator.

Page overview#

The crawl health page is /dashboard/mkit/crawl; /dashboard/mkit/bots now redirects there. The overview endpoint is GET /api/v1/sites/:siteId/reports/crawl (with a top-level alias GET /api/v1/reports/crawl), and batch URL inspection is POST /api/v1/reports/crawl/inspect.

Area	Content
KPIs	Crawl overview including indexed pages
Crawl sources	GSC, Bing crawl stats, Cloudflare edge enhancements
robots.txt	Status check and generator
URL inspection	Batch inspection of index and crawl state

Note: The days parameter accepts only 7 / 14 / 28 / 90; any other value falls back to 28. The default window is 28 days ending yesterday.

What the indexed count means#

indexed.value is computed by calling GSC searchanalytics.query with dimensions=["page"] and rowLimit=25000, then taking the length of the returned rows.

This is the number of pages with impressions or clicks in GSC, not Google's total indexed pages; pages with no impressions or clicks are absent
The previous period is counted the same way to produce prev for comparison
When rows reach 25000, capped=true and the UI labels the value as a lower bound only

Note: The default window is 28 days ending yesterday; it is not a realtime index count and does not replace the Pages index report in GSC.

The three crawl sources#

Crawl data is presented per source, and the code explicitly distinguishes GSC, Bing Webmaster and Cloudflare.

Source	What it provides	Notes
GSC	Indexed page count, per-URL index/crawl state, robots.txt	Primary data
Bing Webmaster	GetCrawlStats: Bingbot daily crawl volume and error count	The crawl overview is fixed at 28 days
Cloudflare	Threat blocking, top bot user agents, cache hit rate, cache trend	Enhancement layer; null without CF and never affects GSC primary data

Note: Googlebot crawl counts and Crawl Errors are explicitly not provided: GSC has no public crawl-request API and the Crawl Errors API is retired, so the page deliberately omits both. Cloudflare top bot user agents are classified by UA regex and do not use botScore.

URL inspection quota#

Daily batch-inspection allowances are Free 0, Pro 50 and Business 200; a single request can contain at most 100 URLs; Google URL Inspection API has a hard cap of 2000 per project per day.

Note: Quota is stored in KV RATELIMIT per siteId + UTC date with a 3-day TTL and resets at UTC midnight. Free throws PlanRequiredError("pro") (403); exceeding the quota throws RateLimitError (429) with used / limit / resetsAt in meta; a Google 401/403 marks scope_required and does not consume quota.

Result rows: url, indexed, verdict, coverageState, lastCrawl, crawledAs, pageFetchState, issues, error
The lastInspection batch summary is cached in KV for 24 hours
A failed URL carries its own error without affecting the rest of the batch

robots.txt check and generator#

The robots check has the Worker fetch the site's /robots.txt directly and parse it; state is ok | blocked | missing | error, along with URL, check time, sitemaps, line count and group count. The result is cached for 4 minutes.

Note: The check has SSRF protection: targets such as localhost, private IPv4 ranges, 169.254.169.254 and ::1 are rejected.

The robots.txt generator is a pure frontend tool and does not call the backend
The six Google UA knowledge cards on the page are static explanations, not data slices
You still need to deploy the generated rules to the site root yourself

Keywords report: /keywords
GSC data source: /gsc
Data latency & metric definitions: /data-latency
Site connections & authorization: /connections