Check robots.txt or sitemap.xml content

Apply a custom value rule to stored site-resource text.

Last checked 21 August 2026

Select robots.txt or sitemap.xml under Look in when a required or forbidden literal value belongs to one of those website resources rather than a rendered page.

These are the only site-resource targets supported by Custom Checks. They always run sitewide, so the dialog does not offer This page or a Page selector.

Create a robots.txt rule

Use robots.txt for text that should or should not occur in the website's /robots.txt file.

TechDash fetches robots.txt hourly. Saving or editing its custom check queues Site Resources. The rule uses case-insensitive literal substring matching against the fetched text.

TechDash's built-in robots checks should remain your primary way to monitor reachability, changes, monitored-page blocking, and AI crawler rules. A custom rule does not parse directive precedence or decide whether a robots policy is valid.

Create a sitemap.xml rule

Use sitemap.xml for text in the sitemap file TechDash fetches for custom matching. TechDash takes the first sitemap URL declared in robots.txt. If robots.txt does not declare one, it uses /sitemap.xml.

The custom check searches that one fetched sitemap file, limited to the first 100,000 stored characters. It does not combine child sitemap files. TechDash separately walks sitemap indexes for built-in coverage checks, but a custom value in a child sitemap is not guaranteed to match the sitemap.xml target.

Sitemap resource work normally runs daily. Saving or editing the custom rule queues Site Resources immediately.

Understand unavailable and previous content

Resource availability and custom content matching are separate results. A fetch failure can open a built-in robots or sitemap incident. When the current fetch is unusable, the custom rule may use the previously stored robots or sitemap text.

This avoids claiming that required values disappeared during a temporary fetch problem. A pass therefore does not prove that the current fetch succeeded. Review the built-in resource status alongside the custom result.

On a website with no previously stored usable text, the fallback is empty. An Is found rule will fail against that empty value, while an Isn’t found rule will pass. Again, the built-in availability result is the source of truth for whether TechDash could retrieve the file.

Choose a dependable value

Matching is case-insensitive and literal. It does not parse XML, apply robots semantics, support regex, count entries, or validate a URL set. Choose enough text to avoid unintended matches. For a URL, include its hostname and path if needed.

After saving, review the fetch in Site Resources and the custom result in Sitewide Checks or the incident. Use Check now after correcting the file. For values found only in a child sitemap, use built-in sitemap reports or move the invariant to the fetched top-level file.

Worked example: keep staging out of the sitemap

Suppose a production sitemap must never contain staging.example.com. This matters because publishing staging URLs can send search engines and customers to an unfinished or access-controlled site.

Configure the rule as follows:

Look in: sitemap.xml
Match: Isn’t found
Value: staging.example.com
Incident level: Warning
Notify: Send a notification

The target is automatically sitewide. A pass means the case-insensitive literal substring is absent from the first 100,000 stored characters of the one fetched sitemap file. It does not prove that every child sitemap is clean or that the complete sitemap is valid XML.

For a safe deliberate failure, change the rule temporarily to Is found with a unique text value that is not in the fetched top-level sitemap, such as techdash-deliberate-test-marker. Save it, then open Site Resources and select Check now. A real failure of the production rule occurs when a deployment writes a staging hostname into the fetched sitemap.

The current HTTP observation appears in Site Resources. The custom result appears in Sitewide Checks and any open Custom check failed incident. For this example, the incident is Warning because a staging hostname needs prompt investigation but does not by itself prove an immediate site outage or compliance emergency. Use Critical only when the specific assertion has documented immediate impact. With Notify set to Send a notification, the transition is eligible for enabled desktop and webhook channels. TechDash does not edit the sitemap automatically.

A failure may be benign if the hostname appears only in an XML comment or another harmless text location, because matching does not understand XML semantics. Conversely, a pass may be misleading if the value is after the storage limit, in a child sitemap, or evaluated against previous stored content after the current fetch failed. Verify the current fetch status, fetched sitemap URL, stored text, and built-in sitemap incidents before declaring the deployment safe.

Restore the production rule or remove the staging hostname from the top-level sitemap, then select Check now. A fresh pass recovers the custom incident. If the fetch still fails, fix that separate resource problem and check again so the recovery is supported by current evidence.