Robots.txt monitoring for SEO teams
robots.txt is a small file with outsized power. A single Disallow can hide a directory. A missing Sitemap line can slow discovery. Watch the file itself, not just page HTML.
Updated 2026-09-01
What to watch
- The file is reachable and not a soft 404 or a login page.
- Content changes: new Disallow/Allow, user-agent blocks, vanished Sitemap lines.
- A custom check for a path you must never block, or a path you must always block.
After an alert
- Diff the stored text against what you expected.
- Check whether a deploy, plugin or CDN injected the rule.
- Restore the intended file. Do not “fix” it only in a cache.
- Verify in TechDash, then fetch as Google if the directory is sensitive.
TechDash treats robots.txt as a site resource with its own schedule. The commercial page is noindex, canonical and robots.txt monitoring.
Questions
Does robots.txt noindex a page?
No. robots.txt asks crawlers not to fetch. Combined with a missing canonical or a sitemap gap it can still hide a URL. noindex belongs in robots meta or headers.
Should staging rules ever reach production?
No. That is the most common emergency. Treat any production Disallow: / as a Critical incident until proven intentional.
Buy the desktop licence. Keep the crawl data local.
One-off licence, 77 shipped checks, Slack/Teams/Discord alerts, and no monthly monitoring subscription. Current launch price: $89.50.