Respond to Robots.txt changes and blocked URLs

Investigate availability, content, monitored-path, and AI crawler incidents.

Last checked 21 August 2026

Start with the exact incident title, severity, and evidence. One robots.txt event can affect several checks, but each answers a different question:

  • Robots.txt unavailable is Warning. The current request did not return usable 2xx or 304. Redirects are not followed.
  • Robots.txt content changed is Warning. A successful body differs from the previous successful body.
  • Robots.txt blocks monitored URLs is Critical. Enabled monitored paths changed from allowed to blocked for Googlebot.
  • AI crawler access removed is Warning. One or more supported AI crawlers changed from allowed to blocked.

Investigate an availability incident

  1. Request the exact origin path, such as https://www.example.com/robots.txt.
  2. Compare its status with Expected and Current. 0 (Failed) means a timeout or network failure.
  3. Publish redirects directly at the monitored origin.
  4. For 401, 403, 429, or 5xx, review authentication, CDN, firewall, rate limits, and origin health.
  5. If another device succeeds, test from the TechDash computer and network.

The failed attempt updates the visible robots.txt status but does not overwrite the last successful body. This protects the comparison baseline.

Availability and policy are separate. During an unavailable fetch, Robots.txt availability fails. Robots.txt blocks monitored URLs and AI crawler access cannot evaluate the current policy, so their current evaluation is skipped. A skipped result would appear as Advisory, not Good. If either policy check was already failing, TechDash preserves that failure. It does not report a false recovery or close the incident merely because robots.txt became unreadable.

Review a content change

Read the entire highlighted comparison. Harmless edits such as comments, line ordering, or formatting still change the body hash and can open Robots.txt content changed.

Check:

  • Which User-agent group changed
  • Whether Disallow became broader
  • Whether an Allow exception disappeared
  • Whether sitemap declarations moved or changed
  • Whether a CMS, CDN rule, staging template, or security tool generated the file

If the edit was intentional and benign, document it and resolve the incident after review. If it was accidental, restore or regenerate the intended file.

Investigate blocked monitored URLs

The incident lists newly blocked monitored paths. TechDash uses explicit googlebot groups when present, otherwise User-agent: *. It supports * wildcards and trailing $ end anchors. The longest matching rule wins, with Allow winning an equal-length tie.

For each URL, compare its path and query with every applicable rule. For example, Disallow: /*.pdf$ blocks a path ending in .pdf, but not the same path followed by a query string. A longer Allow: /products/public/ can override Disallow: /products/.

Remember that a crawl block and noindex are different controls. If a page should disappear from search, blocking it can prevent a crawler from seeing a page-level noindex. If the page should remain searchable, restore crawl access.

An incident may offer an in-app Remove Page action. Use it only when monitoring should stop. It does not repair robots.txt or alter search-engine access.

Investigate AI crawler access

TechDash evaluates GPTBot, ClaudeBot, PerplexityBot, and Google-Extended separately. The first successful observation stores a per-crawler baseline. A later incident lists only crawlers that changed from allowed to blocked.

Review each named crawler's explicit group and User-agent: * fallback. Blocking can be intentional but may reduce relevant AI visibility. Document the decision and resolve only when the evidence matches it.

If the restriction was accidental, update the relevant rule or CDN bot setting. Do not add a broad Allow: / without checking whether it weakens other deliberate protections.

Read pass, fail, and evidence

Good means the successful current policy did not introduce a new monitored-path or crawler block, or that TechDash stored the first baseline. It does not prove every path is allowed. A crawler that was already blocked can remain blocked without creating a new failure.

Availability evidence shows the HTTP result. Content evidence shows previous and current text. Policy evidence lists newly blocked paths or crawler names.

Use that evidence to avoid fixing the wrong layer. An HTTP failure needs server, CDN, or network work. A policy failure needs a robots.txt or bot-policy decision.

Verify the result

After correcting the server or file, use Check & Verify when available, or select Check now in Site Resources. The action validates the license and connectivity, then queues a resource run.

Confirm the Site resources job completed, the robots.txt row is usable, and Sitewide Checks shows recovery.

If the incident remains open, check that monitoring is not paused and re-read the latest evidence. Do not repeatedly queue runs while the origin still returns the same response.

Related articles