Crawl settings directly affect audit quality. Many SEO teams collect crawl data that is technically correct, but strategically wrong because the crawl profile does not reflect production behavior.
Settings that matter most
- User agent profile (desktop and mobile).
- JavaScript execution on/off to compare rendered vs classic HTML.
- Robots handling (respect or ignore, based on audit goal).
- Concurrency and delay to balance speed and server safety.
- Custom request headers for environment-specific behavior.
- Parseable MIME types to keep crawls focused.
Crawl strategies in practice
- Spider mode: best for broad technical discovery.
- URL list mode: best for controlled validation sets.
- Sitemap mode: best for indexation alignment checks.
Common configuration mistakes
- Running only desktop and assuming mobile parity.
- Mixing diagnostic crawls with production-monitoring crawls in one profile.
- Ignoring robots rules in recurring workflows when your objective is production realism.
- Using no URL exclusion patterns, resulting in noisy crawl scope.
Build at least two reusable crawl profiles: one for technical diagnostics and one for recurring production checks. That separation keeps data cleaner and recommendations more trustworthy.