The single most expensive SEO bug is "we accidentally noindexed a section".
A deploy switches production to staging config.
An SEO plugin resets.
A robots.txt experiment never gets rolled back.
Catching these takes minutes; recovering from indexation collapse takes months.
Step 1. Crawl pulling all robots directives
Run a crawl with the parser enabled. The 2-UA crawler extracts three sources simultaneously:
<meta name="robots">tags.X-Robots-TagHTTP response headers.- The live
/robots.txtevaluated against every crawled URL.
Step 2. Cross-reference with your sitemap
Filter to URLs that appear in your sitemap and carry a noindex, nofollow, or robots-blocked flag. These are pages you are telling search engines to both index (sitemap) and not index (meta) — a contradiction Google flags as low-quality.
Step 3. Inspect the X-Robots-Tag header carefully
This is where the most experienced teams get burned.
The HTML looks fine; the noindex hides in the HTTP response header.
The 2-UA parser captures both — filter x_robots_tag CONTAINS noindex to surface every header-based block.
Step 4. Continuous robots.txt monitoring
Use Site → Robots to track robots.txt diffs over time.
Every change is recorded with a timestamp; you see exactly when Disallow: / got committed
and which engineer the commit traces back to.
Three patterns that cause silent deindex incidents
- Staging meta tags shipped to prod —
<meta name="robots" content="noindex">conditioned on<html data-env="staging">that ships incorrectly. - CDN injecting X-Robots-Tag — check Cloudflare or Fastly rules; many teams forget the CDN can rewrite headers.
- Plugin "noindex archive pages" set too aggressively — disables ranking on tag and category pages that actually have traffic.
Noindex vs robots.txt: do not confuse them
robots.txt blocks crawling. Already-indexed URLs stay in the index without content snippets. noindex meta tag blocks indexing. Google must be able to crawl the page to read the noindex, which means it must not be blocked in robots.txt. Using both for the same URL is a logic error: Google never crawls the page, never reads the noindex, and the URL stays indexed.
Use the free robots.txt validator to test directives against your top URLs, then add a project for continuous robots, meta, and header monitoring.