Understand indexability and canonical checks

TechDash derives indexability signals from rendered HTML and response headers, then runs target and robots.txt checks separately. A pass can mean unchanged, intentionally non-indexable, absent, baseline stored, or currently reachable. It is not proof that a search engine will index the URL.

Noindex and indexability state

Noindex newly introduced and X-Robots-Tag noindex added are Critical because an accidental directive can remove a page from search. Each uses case-insensitive comma or whitespace tokens. The first checks rendered robots meta; the second checks the response header and can apply to non-HTML resources. After a baseline, either fails only when noindex is present now and was absent before. A pass means no new token appeared, including when noindex was already present. Evidence shows the prior directive or an expected no-noindex label and the current value. It does not prove which crawler honored it.

Indexability state is a separate summary. The current check has four reachable states:

• CONFLICTING when combined robots meta and X-Robots-Tag contain both index and noindex, or when multiple canonicals have distinct values.
• NOINDEX when either directive contains noindex without the above conflict.
• CANONICALISED when any canonical differs from the requested URL after trailing-slash removal.
• INDEXABLE otherwise.

After its baseline, Became CONFLICTING, Became NOINDEX, or Became CANONICALISED fails when the derived state changes to that value. Became NOINDEX is Critical; the other two are Warning. Returning to INDEXABLE, remaining in the same state, or having no prior state passes. CONFLICTING is reachable and evaluated before NOINDEX. HTTP availability and robots.txt blocking are separate families.

These incidents can legitimately reflect a deliberate duplicate, retirement, or access policy. Compare meta, headers, and every canonical, then align them with the intended URL. Run Check & Verify after correction. Recovery means the derived state or new-token trigger clears, not that Google selected or indexed the page.

Canonical changes and target health

Canonical removed and Canonical changed are Warning. TechDash compares only the first rendered canonical. Removal fails when a previously non-empty first value becomes empty. Change fails when that prior non-empty string differs from the current value. A first missing canonical establishes a baseline and passes; unchanged or still absent also passes. Evidence shows the first previous and current values. Textual URL variations can therefore fail, while changes to later duplicate tags rely on the duplicate and conflict checks.

Canonical points at an unusable URL is Warning and runs after every enabled Monitored Page has a snapshot. It probes each unique, non-self declared canonical without following redirects. It fails if any target is a concrete error or redirect (for example 3xx, 404, or 500). Rate limits (429), auth or WAF blocks, timeouts, and soft overload (503) are treated as inconclusive and do not open this incident. Evidence lists source path, target, and status. A pass means all probed targets returned successfully, remaining probes were inconclusive, or no non-self canonical was declared. It does not test target noindex, robots.txt policy, content equivalence, or whether the target is the right canonical.

Intentional cross-page canonicals are benign when the source is a true duplicate and all signals agree. Restore or update broken targets, remove redirect hops, and verify target indexability separately. Recovery requires a successful finalization probe after all monitored snapshots are current.

Duplicate and conflicting directives

Duplicate canonical tags and Duplicate robots sources are Warning regression checks. The first fails when canonical count moves from one or fewer to more than one. The second fails when collected robots sources move from one or fewer to more than one, including robots or googlebot meta and X-Robots-Tag. A pass means no new duplication, so an already duplicated page can pass unchanged. Evidence shows canonical markup or robots source lines before and after.

Conflicting canonical values and Conflicting robots directives are Warning. After a baseline, canonical conflict fails whenever the current set has more than one distinct exact href; robots conflict fails whenever combined tokens contain both index and noindex. Unlike duplicate checks, a current conflict can fail even if the prior snapshot also conflicted once the family has an eligible baseline. Canonical normalization is not applied here, so textually different equivalent URLs can be reported. Only the first detected conflict type is returned, with canonical conflicts taking precedence.

Multiple identical tags can be benign in immediate effect but remain fragile. Distinct values or opposing robots tokens can split crawler interpretation and cause deindexing or consolidation to the wrong URL. Remove duplicate emitters in templates, plugins, headers, or CDN rules and retain one unambiguous policy. Check & Verify should show agreement and no newly duplicated source.

Robots.txt and Google evidence

Robots.txt blocks monitored URLs is Critical and compares Googlebot access for monitored paths across successful hourly files. It fails only for newly blocked paths and passes when paths are allowed or the same paths remain blocked. robots.txt unavailability is handled separately and should not be mistaken for an access pass. A block prevents crawling, not indexing by itself.

Connected Search Console can add Google chose a different canonical, a Warning when Google's selected canonical differs from Google's declared canonical. Its evidence is quota-limited and reflects Google's last inspection, not necessarily the latest deployment.

For any failure, establish the intended index URL, fix contradictory directives, publish one direct and usable canonical, align links, sitemap, and hreflang, and review robots.txt. Verify locally with Check & Verify, then use fresh URL Inspection when Google-specific confirmation matters.

Related articles

• Respond to robots.txt changes and blocked URLs
• Interpret Google fetch, crawl-recency, and canonical incidents