Understand Robots.txt status and monitoring

Learn how hourly robots.txt fetching supports content, block, and AI crawler checks.

Last checked 21 August 2026

TechDash checks /robots.txt hourly, independently of website and page frequencies. The scheduler reviews due work every minute while monitoring is running. Sleep, offline status, an unavailable license, paused monitoring, or another job in the single-worker Site resources queue can delay the actual start.

Understand the request and baseline

TechDash requests the exact website origin plus /robots.txt, does not follow redirects, and accepts 2xx or 304 as usable. It stores the latest status and checked time on every attempt. After a successful response, it stores up to 100,000 characters of body text and a short SHA-256-derived hash.

The first successful body establishes content and policy baselines. It does not open Robots.txt content changed, Robots.txt blocks monitored URLs, or AI crawler access removed. Later successful bodies are compared with the previous successful body.

If a later request fails, the previous successful body and hash remain stored. The robots.txt row can show 404 (Missing), 503 (Unavailable), or 0 (Failed) while TechDash retains the prior policy for future comparison.

Read each robots.txt check

Each hourly run evaluates four built-in checks:

  • Robots.txt availability passes as Good for a usable response. Redirects, 4xx, 5xx, timeouts, and network failures open Robots.txt unavailable, a Warning. Evidence gives the expected usable response and current HTTP label.
  • Robots.txt content changes compares successful body hashes. A changed hash opens Robots.txt content changed, a Warning, with highlighted previous and current text. Comments and formatting can trigger it even when crawl policy is unchanged.
  • Robots.txt blocks monitored URLs compares enabled monitored paths for Googlebot. Only paths that changed from allowed to blocked fail. The incident is Critical and lists those paths.
  • AI crawler access evaluates GPTBot, ClaudeBot, PerplexityBot, and Google-Extended separately. Only crawlers that changed from allowed to blocked open AI crawler access removed, a Warning. Evidence lists the affected crawler names.

Understand rule matching

For each crawler, TechDash uses explicit groups for that user-agent when present; otherwise it uses User-agent: *. It supports * wildcards and a trailing $ end anchor. Among matching Allow and Disallow rules, the longest rule wins. Allow wins an equal-length tie.

For example, Disallow: /*.pdf$ blocks /reports/annual.pdf but not /reports/annual.pdf?download=1. A longer Allow: /products/public/ can override Disallow: /products/ for that subsection.

AI crawler state is stored and compared per crawler. One crawler remaining blocked does not make another crawler look blocked. The incident reports only newly blocked crawler names, not every crawler already restricted by the prior policy.

Understand unavailable and skipped results

An unavailable fetch has two different effects:

  1. Robots.txt availability fails because the current resource is not usable.
  2. Monitored-path and AI crawler policy checks are skipped because TechDash cannot judge current rules.

A skipped result appears as Advisory, not Good. If a blocking check was already failing, TechDash preserves that prior failure instead of storing the skipped result. This prevents an outage from producing a false recovery or closing an unresolved blocking incident. Once a usable body returns, TechDash compares it with the last successful body.

A 404 is unavailable to TechDash even though search engines may handle a missing file differently from a 5xx. Use the actual status evidence. Also remember that robots.txt controls crawling, not indexing. A disallowed URL is not automatically noindex.

Other features that use stored robots.txt

Custom Checks can search stored robots.txt text for a value that should be found or not found. On a failed fetch, they can evaluate the previous stored text rather than an empty replacement.

Benign cases include intentional comments, formatting edits, or documented AI crawler restrictions. Review the diff and crawler-specific evidence before resolving them. For unexpected changes, restore direct public access or correct the relevant user-agent rules.

After a fix, select Check now in Site Resources. Confirm the robots.txt row has a usable status, then verify the related result in Sitewide Checks. A usable fetch is required before a skipped policy check can be judged again.

Related articles