Crawling SEO guides

Deep technical guides on crawl setup, rendering parity, report analysis, and crawl-led troubleshooting.

Start Free Trial View Plans
Crawling SEO guides overview with live crawl status distribution and depth buckets for example.com
Latest crawl report data used as a live visual overview for crawling guides.
Crawling 11 min
Technical SEO crawler settings that actually matter

A no-fluff guide to user agent, JS rendering, robots rules, headers, and crawl strategy choices.

TOFU Read article
Crawling 10 min
Crawl report analysis: filters and operators for faster SEO decisions

Turn raw crawl data into action using query operators, segment filters, and practical triage logic.

MOFU Read article
Crawling 8 min
Mobile vs desktop crawl comparison for SEO: find rendering gaps before Google does

Find parity issues between classic and JS-rendered versions, and between device profiles.

MOFU Read article
Content & Links 7 min
Outgoing links and top words: two fast audits for SEO hygiene

Use crawl-derived outbound links and top word reports to spot quality and relevance issues quickly.

TOFU Read article
Indexation 8 min
Sitemap monitoring with daily URL trends by sitemap

Track sitemap history and daily page count changes both site-wide and per individual sitemap file.

MOFU Read article
Acquisition 7 min
Free online SEO crawler: evaluate your site before you buy

Run a free crawl, understand your baseline issues, then scale into full monitoring when needed.

BOFU Read article
Crawling 6 min
2-UA competitor bot (2-UAcomBot): how we identify ourselves and respect robots.txt

Every competitor snapshot 2-UA takes is identified by the 2-UAcomBot user agent and gated by the target host's robots.txt. Here is exactly how our bot behaves and how to control it.

TOFU Read article