How TechDash discovers and walks XML sitemaps

Understand sitemap discovery, index traversal, and complete-tree membership.

Last checked 21 August 2026

TechDash discovers and walks XML sitemaps during the daily site-resource cycle. The walk measures the tree, checks enabled monitored URLs against it, and evaluates sitemap quality. Discovery does not add every sitemap URL to Monitored Pages. After Add website, use Sources (Search Console, Bulk URL Import, or crawl links) to choose which pages to watch. See Use Sources after adding a website.

Choose the starting files

TechDash reads case-insensitive Sitemap: lines from stored robots.txt. If at least one exists, each value becomes a starting URL and /sitemap.xml is not added. Otherwise, TechDash tries the website origin plus /sitemap.xml.

Only absolute http:// or https:// sitemap URLs enter the walk. This means a relative Sitemap: /sitemap.xml value is not walked. Publish absolute sitemap declarations when you rely on robots.txt discovery.

The daily run uses stored robots.txt text. If the current robots.txt fetch fails, the previous successful text remains available, so its sitemap declarations can still seed the walk.

Walk indexes and URL sets

Starting URLs enter a queue. TechDash requests each file without following redirects, streams the response, and recognizes sitemapindex and urlset, including namespace-prefixed forms. It then:

  1. Queues absolute child <loc> values.
  2. Counts URL entries across URL sets.
  3. Records found enabled monitored URLs.
  4. Samples up to 2,000 locations.
  5. Retains up to eight implausible <lastmod> examples.

HTTP 2xx is required during the walk. A redirect, 304, client error, server error, timeout, unreadable response, or unrecognized XML leaves that file incomplete.

TechDash waits between files, saves progress, and has a hard walk deadline. A later run resumes the checkpoint when the enabled monitored-page set has not changed. The single-worker Site resources queue rotates website ordering so one long tree does not permanently block others.

Understand limits and counts

Boundaries are 50,000 URLs per file, 50 MB per file, 50,000 child files per tree, and 2,000 stored URL samples.

Exactly 50,000 URLs is permitted. The 50,001st URL makes Sitemap limits exceeded fail. Evidence stores that file URL and observed count. Exceeding 50 MB also fails. Reaching the child-file boundary or deadline leaves the tree incomplete.

Understand the sticky successful baseline

After a complete walk, TechDash stores the combined URL count and samples as a sticky successful baseline:

  • An unavailable or unparseable later walk does not erase it.
  • A partial checkpoint does not replace it with a smaller interim count.
  • Repeated failures do not make the website look as if it never had a sitemap.

Sitemap URLs can therefore show the last successful total while the current walk is unavailable or catching up. Open Sitewide Checks for the current evaluation.

When membership is judged

TechDash normalizes sitemap locations and monitored URLs before comparing them. It removes fragments, decodes common XML entities, and removes a trailing slash from non-root paths. Membership is judged only when the walk is usable and complete.

If a file failed or work remains queued, XML sitemap membership is skipped and appears as Advisory. This is not Good and not proof that all monitored URLs are listed. TechDash does not infer missing URLs from a partial tree.

After a complete walk, URLs missing from sitemap is Warning when enabled monitored URLs are absent and a prior successful sitemap baseline exists. A site that has never had a usable sitemap stays quiet. A complete first successful walk stores the baseline without treating its initial state as a loss.

When member validity is judged

After a complete walk, Sitemap non-indexable members checks found monitored URLs against their latest snapshots. Non-indexable URLs in sitemap is Warning when a member:

  • Has noindex in its stored indexability state, robots meta, or X-Robots-Tag
  • Returns a status outside 2xx and 304

Evidence identifies each URL and reason, such as noindex or HTTP 503. An incomplete walk skips this check as Advisory.

Read other quality results

Sitemap unavailable or unparseable is Warning only after a previously complete, usable sitemap becomes unusable. Implausible sitemap lastmod is Advisory and samples unparseable dates, dates at least 10 years old, or dates more than 24 hours in the future.

Sitemap inclusion supports discovery, but it does not guarantee indexing. Keep Monitored Pages focused on URLs you deliberately want TechDash to check. Use Check now after changing declarations, indexes, or child files to queue a full resource run.

Custom Checks against sitemap XML inspect the stored text of the first discovered sitemap target, or /sitemap.xml as fallback. They do not concatenate the complete walked tree.

To verify a correction, wait for the Site resources queue and any resumed checkpoint to finish. Confirm the expected count in Site Resources, then review availability, membership, member validity, limits, and lastmod results in Sitewide Checks.

Related articles