The robots.txt file is a 200-byte deployment capable of removing your entire site from Google.
It is also one of the least-monitored deployments in most teams.
Most "we lost our rankings overnight" stories trace back to an accidental Disallow: / line that no one reviewed.
Step 1. Validate once, end-to-end
Open the free robots.txt validator, paste your site URL, and check:
- Syntax errors and typos.
- Conflicting rules — allow and disallow for the same path with different user-agents.
- Sitemap reference must be an absolute URL.
- Crawl-delay — ignored by Google but read by Bing.
Step 2. Test against your most important URLs
Submit a list of your top 20 important URLs and confirm each is allowed for Googlebot. False positives here are how SEO migrations die — every important URL must be testable in seconds, not in days of debugging.
Step 3. Enable continuous robots monitoring
Add the site to a 2-UA project — Site → Robots polls hourly and stores diffs. Any change generates a Slack or Telegram alert with a side-by-side view of the previous and current file. You see exactly which line was added, removed, or modified.
Step 4. Add a canary URL per section
Pick one URL inside each major section of your site and add it to Tracked URLs with Robots check: must be allowed. The moment robots changes break a section, you see a red flag in the dashboard before the section drops from the index.
Three robots.txt mistakes that recur
Disallow:with trailing space — silently ignored, but looks correct in code review.- User-agent specificity backwards — rules for
Googlebotoverride rules for*only when more specific; people get this wrong. - Blocking JS or CSS assets — Google needs to render; blocking
/assets/kills mobile-first indexing.
How robots.txt and noindex differ
robots.txt stops crawling.
noindex stops indexing.
Already-indexed URLs blocked in robots.txt remain in the index without snippets, which is the worst outcome for both you and the user.
Use noindex to deindex, and robots.txt only for genuine crawl-budget control.
The free robots.txt validator requires no signup. Continuous robots.txt monitoring with diff alerts is included in every 2-UA project.