Understand crawl access and raw-versus-rendered checks
Learn how TechDash detects bot challenges, rendering divergence, and invalid head markup.
Last checked 21 August 2026
TechDash fetches bounded raw HTML and renders the page in a mobile browser. These families detect access changes, selected raw-to-rendered differences, and invalid head structure. A pass can mean unchanged, absent, baseline stored, or not enough evidence to trigger. It does not prove crawlability or SEO quality.
Bot challenge detection
Crawler newly blocked or challenged is Warning. It matters because challenge pages can prevent TechDash and possibly other bots from receiving the intended content. TechDash classifies recognizable response-header and interstitial signatures, including CDN mitigation signals, into one challenged value.
After the first eligible baseline, the check fails only when challenged changes from false to true. A pass means no challenge was detected or the challenge was already present; the latter is not healthy. When challenged, TechDash suppresses non-availability page checks for that run so challenge HTML does not create false title, content, schema, or link incidents. Non-HTML responses and unsuccessful fetches can also skip relevant browser-derived checks.
The incident records that a challenge was detected, but the check does not preserve the exact matched signature, header, or interstitial snippet in its result. It cannot identify the responsible WAF rule or prove Googlebot is blocked because TechDash does not impersonate Googlebot. A login wall or deliberately protected environment can be benign.
Check the response and CDN or WAF logs for the TechDash request, then allow the configured crawler only where appropriate. Run Check & Verify. Recovery requires a successful render with no detected challenge; use Search Console URL Inspection or verified Googlebot logs for Google-specific confirmation.
Raw versus rendered HTML
Index directives differ raw vs rendered is Critical because crawlers may process initial and rendered signals at different times. On each eligible HTML crawl, it compares robots meta only when both raw and rendered values are non-empty. It compares the first canonical only when both values are non-empty and their normalized destinations differ.
A fail requires either comparison to differ. Evidence renders raw robots or canonical markup as expected and rendered markup as actual. A pass means no eligible difference was found. It can therefore pass when a directive exists in only one version, when neither version has it, or when values agree. It does not compare X-Robots-Tag, later duplicate canonicals, or the whole DOM. Intentional JavaScript normalization can be benign, but conflicting index intent is high risk.
Main content requires JavaScript is Advisory. It fails when raw word count is greater than zero and rendered word count is more than 1.7 times raw count, provided a directive difference did not already return first. Evidence is only the two word counts. Despite product documentation wording, the check does not use internal-link divergence. A pass can mean counts are close, raw count is zero, or a directive result took precedence. It does not judge content quality, meaning, completeness, or whether JavaScript rendering succeeds for search engines.
A deliberately client-rendered application can produce a benign content finding, but it remains dependent on rendering. Compare raw source with the rendered page manually. Emit important canonical and robots instructions consistently in initial HTML. Restore server-side rendering, static generation, or a contentful fallback when the dependency is accidental. Check & Verify recovers when directives agree and the word-count ratio no longer crosses the trigger.
HTML head validity
Head terminated early is Warning. It detects a newly malformed rendered head because parsers may ignore canonical, robots, title, or metadata after the structural break. The underlying signal looks for body-level elements or substantial text inside <head>, <body> before </head>, or important head-only elements after the head ended.
After a baseline, it fails only when headTerminatedEarly changes from false to true. A pass means the signal is false or was already true, so an unchanged malformed head can pass. The result does not include the exact offending tag or parser location. Browser repair behavior can also differ from another crawler's parser. Deliberate but invalid markup is not a safe benign case, although extractor false positives should be checked against raw and rendered HTML.
Inspect the earliest structural break, including tag-manager, consent, template, and CMS injections. Fix the source nesting and keep head-only elements inside one valid head. Run an HTML validator and Check & Verify. Recovery requires the rendered signal to become false; separately confirm that critical metadata is present and correct.