Technical SEO crawler settings that actually matter
A no-fluff guide to user agent, JS rendering, robots rules, headers, and crawl strategy choices.
Practical playbooks
Try a free technical check
Guides in this collection
Crawl report analysis: filters and operators for faster SEO decisions
Turn raw crawl data into action using query operators, segment filters, and practical triage logic.
Mobile vs desktop crawl comparison for SEO: find rendering gaps before Google does
Find parity issues between classic and JS-rendered versions, and between device profiles.
Outgoing links and top words: two fast audits for SEO hygiene
Use crawl-derived outbound links and top word reports to spot quality and relevance issues quickly.
Sitemap monitoring with daily URL trends by sitemap
Track sitemap history and daily page count changes both site-wide and per individual sitemap file.
Free online SEO crawler: evaluate your site before you buy
Run a free crawl, understand your baseline issues, then scale into full monitoring when needed.
2-UAcomBot: how we identify ourselves and respect robots.txt
Every page snapshot 2-UA takes with 2-UAcomBot is gated by the target host's robots.txt. Here is exactly how our bot behaves and how to control it.
How to find broken links (4xx and 5xx) on your website without crashing production
Crawl your site, filter 4xx/5xx, export to XLSX, and set up auto-alerts so broken links never reach users again.
How to detect redirect chains and redirect loops that eat your crawl budget
Find every redirect chain longer than one hop, fix loops, and recover crawl budget Googlebot is wasting on 302/301 ping-pong.
How to find orphan pages on your website (pages with zero internal links)
Combine a full crawl with your sitemap and GSC export to surface every page Google knows about but your site does not link to.