Crawl budget is not a score: what Googlebot demand, capacity, and server health actually mean

Imagine an ecommerce team watching Googlebot requests fall sharply after a platform migration. The launch calendar includes tens of thousands of seasonal product URLs, so the team buys more origin capacity, removes a rate limit, and asks engineering to make every response faster. Two weeks later the request count is higher, the infrastructure bill is higher, and most new products still have not been crawled.

The intervention failed because the team treated crawl budget as a gauge that needed to move upward. The real bottleneck was URL inventory: navigation exposed millions of sortable and faceted combinations, old redirects remained in internal links, and the sitemap's modification dates changed on every build. Googlebot could crawl more. It did not have a clean reason to spend that capacity on the URLs the business cared about.

The opposite failure is equally expensive. A publisher may spend a quarter consolidating parameters while its origin returns bursts of 503 responses whenever Googlebot and real traffic overlap. Demand exists; capacity collapses under it. More canonicals cannot repair an unavailable host. In both cases, the visible symptom—late crawling—is the same. The correct intervention depends on which side of the system is binding.

Crawl budget is an intersection, not a score

Google's current documentation defines a site's crawl budget as the set of URLs Google can crawl and wants to crawl. “Can” is the crawl capacity limit, also called hostload: the time a server spends holding connections open, affected by parallel connections, their duration, response stability, and overload signals. “Wants” is crawl demand: the crawler's need to discover or refresh URLs based on the perceived inventory, site size, update frequency, content quality and relevance, popularity, and staleness.¹

That definition has no public 0–100 value. It does not expose a quota you can request, a target number of daily hits, or a formula that an SEO tool can reproduce. Capacity and demand are dynamic, and a hostname can sit below its capacity simply because demand is low. Google's guide is explicit: even when the capacity limit is not reached, low demand means less crawling. It also says the capacity limit is shared across Google crawlers while each crawler has its own demand.¹

This is why “crawl budget score” dashboards are dangerous. They usually combine proxies such as response time, status codes, duplicate counts, click depth, sitemap coverage, and observed bot hits. Those inputs can be useful. Their weighted sum is not Google's allocation. A score can improve when you remove pages Google never wanted to crawl, or worsen during a healthy migration that temporarily raises discovery demand. Keep the evidence as separate measures and connect each measure to a decision.

Scroll horizontally to read the full graphic.

First prove that crawl budget is the constraint

Most sites do not need an advanced crawl-budget project. Google's July 2026 guide is aimed primarily at sites with roughly one million or more unique pages that change moderately often, sites with roughly 10,000 or more unique pages that change daily, or sites with a large share of URLs marked “Discovered – currently not indexed.” Google labels those numbers rough estimates, not exact thresholds. Its Crawl Stats documentation goes further for small sites: if a site has fewer than about one thousand pages, owners generally should not need that report or this level of crawling detail. ¹³

Size is only a screening question. A small site can still have a crawl failure caused by broken discovery, accidental blocking, or a sick server. A large site can have no material constraint when its important URLs are found and refreshed within the business's required window. The outcome that matters is not requests per day. It is whether index-worthy URLs are discovered or refreshed too late for a real decision: a product launch, news cycle, price change, availability update, migration, or removal.

Write that requirement before looking at charts. “Googlebot should first request 95% of new in-stock product pages within 48 hours” is testable. “Improve crawl budget” is not. For a weekly knowledge base, a seven-day discovery lag may be harmless. For limited inventory that sells out in two days, it may make the search opportunity irrelevant. The same observed crawl rate can be adequate for one business and costly for another.

Do not confuse discovery, crawling, indexing, and serving

Crawl budget only belongs to one part of the pipeline. A URL can fail before crawling because Google has not discovered it or robots.txt disallows the request. It can be crawled successfully and still not be indexed because Google consolidates it with another URL, sees a noindex directive, classifies it as a soft 404, or decides it does not merit indexing. An indexed URL can fail to appear for a query because ranking and relevance are separate systems. Google's documentation says directly that not every crawled page is indexed, and its historical crawl-budget explanation states that a higher crawl rate is not a ranking signal.¹¹¹

Stop a crawl-budget investigation as soon as the evidence crosses into another stage. If a representative URL was fetched after the latest change but remains excluded as a duplicate, work on canonical consistency and content differentiation. If Google does not know the URL, fix discoverable links and the sitemap. If requests arrive but time out or receive 5xx responses, investigate capacity. Calling every exclusion a “crawl-budget issue” hides ownership and makes the chosen fix unfalsifiable.

Scroll horizontally to read the full graphic.

Use three evidence layers, with logs first

A reliable diagnosis joins three layers rather than asking one report to answer everything. First, verified CDN or origin access logs show individual requests: time, path, status, bytes, latency if logged, and user agent. Google calls raw access logs the best source for identifying what resources its crawlers request. Verification matters because a user-agent string can be spoofed; classify Google crawler traffic with Google's published IP verification method or ranges before calculating URL-family shares.⁵

Second, Search Console's Crawl Stats report shows Google's property-scoped aggregates: total requests, total download size, average response time, host status, response codes, file types, crawl purpose, and Googlebot type. The report distinguishes discovery from refresh and covers a 90-day host status window. It is useful for direction and correlation, but its example URLs are representative rather than comprehensive, and Google documents a current coverage gap in which some requests may not be counted. Redirect hops are counted as separate requests; external-host resources are outside the property's scope.³

Third, a site crawl and URL inventory describe what the site makes discoverable: index-worthy pages, canonical clusters, parameters, redirect chains, response codes, orphan candidates, sitemaps, robots rules, and internal links. This is not Googlebot evidence. It is a map you can compare with the requests. The absence of a URL in Crawl Stats examples proves nothing; the absence of verified requests in complete logs over a defined period is stronger, though it still must be interpreted against whether Google discovered and wanted the URL.

Label conclusions at four levels. A platform fact comes from current documentation, such as the dimensions in Crawl Stats. A site observation comes from your verified logs or inventory. An inference connects coincident evidence to a candidate constraint and must name alternatives. An editorial decision sets the business window, internal service objective, and stop rule. Keeping the labels visible prevents a Google statement from being stretched into a site-specific diagnosis.

Diagnose a capacity limit from server evidence

Google's capacity limit is designed to avoid overwhelming a host. The current guide says stable or improving latency and Time to First Byte can let the limit rise; longer response times, 5xx errors, and 429 rate-limit signals can make it fall. Crawl troubleshooting recommends correlating Crawl Stats availability warnings with failing URLs and treating a URL Inspection “Hostload exceeded” result as direct evidence that Google cannot crawl as many known URLs as it discovered.¹²

Look for coincidence, not a single average. Plot verified Googlebot request concurrency, response-time percentiles, connection failures, 5xx and 429 rates, origin saturation, and Crawl Stats request volume on the same timeline. Break them down by hostname and resource type. A daily average can hide a 20-minute overload window that repeatedly interrupts discovery. A response-time rise without delayed important URLs is an infrastructure warning, but it has not yet proved a crawl-budget loss.

Do not invent a universal TTFB target for Googlebot. Google publishes direction—stable and faster is better for capacity—but not a threshold that guarantees more crawling. Set an internal service objective from your own baseline and user workload. For example, require the p95 for cacheable HTML under normal bot concurrency to remain within 20% of the previous four-week median, with an error budget for 5xx and 429 responses. That is your operating contract, not Google's score.

Use overload responses as an emergency brake, not an optimization tactic

Google documents 5xx and 429 responses as signals that temporarily reduce crawling. Its emergency guidance recommends them only for a short period— a couple of hours or one to two days—and warns that persistent errors can cause URLs to be dropped from Google products. The effect applies across the hostname, including URLs that are still healthy. Google also says not to use 401 or 403 as crawl-rate controls. ²⁷

HTTP semantics give 503 a precise meaning: temporary inability to handle a request because of overload or maintenance. A server may send Retry-After as an HTTP date or delay in seconds. That is the correct protocol expression for a temporary outage, but RFC 9110 does not promise that Googlebot will retry at that exact time. Pair the response with recovery monitoring and remove it promptly; never leave a broad bot-only 503 rule in place after the origin is healthy.⁹

Treat robots.txt availability as production availability

A broken robots.txt endpoint can stop an otherwise healthy crawl. RFC 9309 says that when robots.txt is unreachable because of server or network errors, a crawler must initially assume complete disallow; a cached copy may be used under its caching rules. Google's Crawl Stats help says frequent robots.txt failures can slow or stop crawling until Google gets an acceptable response. Monitor the file from the same edge and hostname as users, and test the status separately from its rule content.⁸³

Robots.txt is not access control. The standard explicitly says its rules are not authorization and warns that listed paths are public. Use it to guide compliant crawlers away from unwanted crawl spaces, not to protect accounts, staging systems, customer data, or administrative URLs. Those require authentication and network controls.⁸

Make repeated fetches cheaper

Google supports conditional requests using If-Modified-Since and If-None-Match in some crawl contexts. When content has not changed, a correct 304 response lets Google reuse its copy without a response body, saving origin processing and bandwidth. Google's current guide recommends HTTP caching for crawl efficiency. Test validators end to end through the CDN; a false 304 can preserve stale content, while a missing validator forces a full response.¹²

Rendering resources also matter. Google says same-host JavaScript, CSS, images, and video requests consume that hostname's crawl budget, although WRS attempts to cache JavaScript and CSS for up to 30 days. Stable resource URLs improve reuse. Moving critical resources to a new hostname merely to shift crawl accounting can add connection overhead and harm rendering performance, so treat hostname changes as architecture decisions rather than crawl-budget tricks.⁵

A separate 2026 retrieval boundary is easy to mislabel as budget: Googlebot currently fetches only the first 2 MB of a non-PDF URL, including response headers, and sends that prefix to indexing and rendering as if it were complete. Referenced resources receive their own per-URL limit. If essential canonicals, structured data, or content sit after the cutoff, they are unseen—but reducing an ordinary 300 KB page to 200 KB does not by itself create more demand. This is a byte-placement and retrieval problem first.⁴

Diagnose demand through the inventory Google sees

A healthy server can serve far more than Google wants to request. Google's current demand model includes site size, change frequency, page quality and relevance, popularity, and staleness. Site moves can create temporary demand as systems reprocess URLs. The part owners can influence most directly is perceived inventory: the URLs exposed through links, sitemaps, redirects, canonicals, parameters, and historical discovery.¹

Do not interpret low demand as a technical throttle until you establish that index-worthy URLs are known and changing. If old pages are stable and new pages are rare, fewer refresh requests may be efficient. If important URLs were crawled but not indexed, more repeated fetches may add cost without changing Google's assessment. Improving server speed is worthwhile for users and resilience, but Google's troubleshooting guidance explicitly warns that making low-quality pages faster will not persuade Googlebot to crawl more of them.²

Create an index-worthy inventory with one row per canonical URL and the business reason it deserves discovery or refresh. Record template, state, last meaningful change, required discovery window, sitemap membership, internal path, canonical target, robots state, status, and expected device profile. Then group every observed Googlebot request into that inventory or a named waste family. “Unknown” must remain a visible category; it often reveals new parameter spaces, alternate hosts, or generated paths that a conventional crawler did not anticipate.

Choose the URL control by the outcome you want

Inventory fixes fail when teams treat robots.txt, noindex, canonicalization, redirects, and deletion as interchangeable. They communicate different outcomes and act at different stages.

• Consolidate true duplicates: remove duplicate routes where possible, link to the preferred URL, and use a consistent canonical signal when variants must remain reachable.
• Prevent crawling of a space that should never be searched: use robots.txt for compliant crawlers, but remember that blocked URLs can remain known and the rule cannot remove sensitive data or guarantee deindexing.
• Remove a URL permanently: return 404 or 410. Google's guide says a 404 is a strong signal not to crawl again, while a blocked known URL can remain in the crawl queue longer.
• Exclude an accessible page from Google Search: use noindex only when Google can crawl the response and read the directive. It still requires a request, so it is not a way to conserve crawling for a large unwanted space.
• Move content: redirect directly to the final relevant destination and update internal links. Every server-side hop is a separate request in Crawl Stats.
• Signal a real update: include the canonical URL in a sitemap and set lastmod to the page's actual meaningful modification time, not the time the sitemap was regenerated.

Sitemaps make URLs available to supporting crawlers; they do not guarantee crawling or indexing. The protocol requires loc, makes lastmod optional, and defines that timestamp as the linked page's modification date. A deployment that stamps every URL “today” destroys the field's ability to distinguish real changes. Use a content or product update timestamp, and validate that timezone and serialization do not rewrite unchanged rows.¹⁰

Facets are an inventory design problem

Faceted navigation is the clearest example of demand being diluted by perceived inventory. Each color, size, price, sort, pagination, and session parameter can create another URL. Google says crawlers often must fetch many combinations before determining that they are useless, which can consume compute and slow discovery of new useful URLs. Its December 2025 guidance recommends preventing crawling when facet URLs do not need search exposure; when they do, use standard parameter separators, stable ordering, no duplicate filters, and real 404 responses for empty combinations. ⁶

This is not a blanket instruction to block all filters. Some category combinations satisfy distinct demand and deserve indexable landing pages. Make that a finite product decision: define the allowed dimensions and combinations, give them stable URLs and unique value, link to them intentionally, and prevent the rest from becoming crawlable paths. A rule like “index every combination with products” can still expose millions of near-duplicates; a rule like “index combinations with approved demand and differentiated content” creates an auditable inventory.

Canonical tags can help consolidate signals, but Google's faceted-navigation documentation says they may only decrease crawling of non-canonical variants over time and are generally less effective than preventing unwanted crawling. A canonical is not a scheduler command. Likewise, nofollow is fragile as an inventory control because every discoverable link to the URL would need consistent treatment. ⁶

Scroll horizontally to read the full graphic.

Worked example: a request count that hides the bottleneck

Consider a hypothetical retailer with 85,000 index-worthy product and category URLs. Its navigation can generate 4.2 million parameter combinations. During a 14-day baseline, verified access logs contain 1,200,000 Googlebot requests. The total looks healthy until the team assigns every request to a mutually exclusive URL family:

• 660,000 requests, or 55%, reached faceted, sort, and session URLs;
• 168,000, or 14%, were redirect hops left in internal navigation and feeds;
• 72,000, or 6%, reached empty-result pages returning 200 or other soft-error patterns;
• 180,000, or 15%, fetched same-host page resources; and
• 120,000, or 10%, fetched index-worthy HTML URLs.

The first three avoidable classes account for 900,000 requests, or 75% of the observed ledger. That does not mean fixing them produces 900,000 requests for products. Google's current guide warns that blocking or hiding URLs does not transfer the newly available capacity elsewhere unless the site was already hitting its serving limit. The business case must therefore rely on improved discovery or refresh latency for important URLs, not a promise to “recover” every request.¹

The team makes three bounded changes. It allows only approved search-facing facets, updates navigation and feeds to final destinations, and returns 404 for impossible combinations. It leaves useful facets indexable and keeps their canonical and sitemap signals consistent. It does not change origin capacity because p95 response time, 5xx, 429, DNS, robots.txt, and Hostload evidence were healthy.

Thirty days later, the success question is not “Did total crawling rise?” The test is whether the share of requests for known unwanted families fell, whether first-request lag for new in-stock products improved, and whether refresh lag for price and availability changes met the predefined window. If waste falls but important lag does not move, capacity was not the binding constraint or demand remains low. The team should then investigate discovery, quality, relevance, internal prominence, and index selection rather than purchasing servers.

The strongest counterposition is often correct: do nothing

Google automatically adjusts crawling, most sites are small, and engineering time spent on crawler micro-optimization can be better used on content, product, reliability, or conversion. That is the strongest reasonable objection to a crawl-budget program, and the evidence supports it. Google's own advanced guide says sites without large, rapidly changing inventories—or sites whose pages are crawled the same day—do not need the guide. ¹

The conclusion survives because the proposed framework begins with a stop rule. If important URLs meet their required discovery and refresh windows, host health is stable, and the indexable inventory is finite, there is no diagnosed crawl-budget constraint. Monitor after major changes and work on a more valuable problem. A lower request count may simply mean that unchanged pages do not need refresh or that duplicate inventory was removed.

Large platforms have a different boundary. Marketplaces, publishers, ecommerce catalogs, classifieds, and sites with calendar or search pages can create new URLs faster than crawlers can evaluate them. There, even a small improvement in URL-family policy can remove millions of unnecessary responses and reduce origin cost. The method still needs outcome evidence: request-family share, server resource use, and important-URL latency. Scale makes the measurement worthwhile; it does not turn a score into truth.

Build a 30-day measurement plan

Use a before/after design with a stable URL taxonomy. It will not prove a ranking or traffic effect, but it can determine whether the intervention changed the diagnosed crawl constraint. Avoid changing server capacity, navigation, sitemaps, canonicals, and content quality at the same time unless the business needs an incident response; otherwise you will not know which mechanism moved.

Scroll horizontally to read the full graphic.

Days 1–7: establish the baseline

1. Define index-worthy URL cohorts and the required first-crawl or refresh window for each.
2. Verify Googlebot traffic, retain complete access logs, and classify each request by URL family, hostname, status, resource type, and purpose where inferable.
3. Export Crawl Stats totals, download bytes, average response time, host status, response mix, Googlebot type, and discovery-versus-refresh split.
4. Record p50 and p95 origin latency, connection failures, DNS incidents, robots.txt status, 5xx and 429 rates, and saturation windows.
5. Sample important URLs in URL Inspection and record discovery, last crawl, canonical, and indexing outcomes without treating the sample as a census.

Days 8–14: change one diagnosed constraint

For a capacity diagnosis, choose one operational change such as caching an expensive template, separating a failing upstream dependency, or adding capacity for the measured saturation window. For an inventory diagnosis, choose one URL family and one intended outcome: remove generation, update links, disallow permanent unwanted space, consolidate a duplicate, return 404/410, or correct sitemap membership and truthful lastmod. Record the release time and affected patterns.

Days 15–30: observe outcomes, not activity

Compare like-for-like cohorts and weekday patterns. Track median and p90 time from publication or meaningful update to the first verified Googlebot request; the share of index-worthy URLs meeting their required window; request share by URL family; status and latency distributions; Crawl Stats host warnings; and discovery versus refresh movement. Keep deploys, promotions, migrations, and outages as annotations because they can change demand.

Change course after 30 days only when the result matches the mechanism. A capacity change earns continuation when overload evidence improves and the late important cohort is crawled sooner. An inventory change earns continuation when unwanted-family share falls and important discovery or refresh improves without creating indexability regressions. If only total requests change, the result is inconclusive. If URLs are fetched on time but remain excluded, close the crawl-budget project and open an indexing, canonicalization, or content-quality investigation.

What 2-UA can verify—and what still requires other evidence

What the evidence does not show

What to do this week

1. Write one measurable crawl requirement for a valuable cohort, including the event, denominator, target percentage, and time window.
2. Export 90 days of Crawl Stats and annotate migrations, releases, outages, campaigns, and large inventory changes.
3. Obtain seven days of verified Googlebot access logs from the CDN or origin; do not substitute a crawler simulation.
4. Classify requests into index-worthy HTML, redirects, errors, resources, approved alternates, unwanted parameters, and unknown URLs.
5. Check whether late URLs were undiscovered, blocked, unfetched, unsuccessfully fetched, or fetched and excluded from indexing.
6. Choose one capacity or inventory intervention, write the expected mechanism, and set the 30-day stop or continue rule before release.

Continue observing for 30 days unless the change is an incident fix. Change course when the evidence crosses stages: healthy, timely fetches followed by exclusion are no longer a crawl-budget problem; falling waste without faster important discovery means saved capacity was not the bottleneck; stable demand alongside recurring overload means inventory work alone is insufficient. The purpose of the measurement is to stop the wrong project as confidently as it supports the right one.

Optimize the constraint, not the chart

Crawl budget is a useful systems model precisely because it resists a single score. Googlebot must want a URL, the host must be able to serve it, and the result must still pass separate indexing and ranking systems. Each boundary leaves different evidence and belongs to a different owner.

Start with late, valuable URLs and a business deadline. Use verified logs to see requests, Crawl Stats to see Google's aggregate history and host signals, and a crawl inventory to explain what the site exposes. Repair server health when capacity binds. Reduce or clarify URL inventory when unwanted spaces consume attention. Improve discovery and truthful freshness signals when Google does not know what changed. Stop calling it crawl budget when the page was fetched and the problem moved downstream.

The winning outcome may be more crawling, less crawling, or the same number of requests aimed at a better inventory. A score cannot tell those stories apart. A claim-to-source ledger, a URL-family request ledger, and a predeclared 30-day decision can.

Sources

1. Google Crawling Infrastructure, “Optimize your crawl budget”, updated July 22, 2026. First-party platform documentation.
2. Google Search Central, “Troubleshoot Google Search crawling errors”, updated December 18, 2025. First-party platform documentation.
3. Google Search Console Help, “Crawl Stats report”, reviewed September 7, 2026. First-party product documentation.
4. Gary Illyes, Google Search Central, “Inside Googlebot: demystifying crawling, fetching, and the bytes we process”, March 31, 2026. First-party engineering explanation.
5. Martin Splitt and Gary Illyes, Google Search Central, “Crawling December: The how and why of Googlebot crawling”, December 2024. First-party engineering explanation.
6. Google Crawling Infrastructure, “Managing crawling of faceted navigation URLs”, updated December 18, 2025. First-party platform documentation.
7. Google Crawling Infrastructure, “How HTTP status codes affect Google's crawlers”, updated February 4, 2026. First-party platform documentation.
8. Martijn Koster, Gary Illyes, Henner Zeller, and Lizzi Sassman, “Robots Exclusion Protocol”, RFC 9309, September 2022. IETF standards-track RFC.
9. Roy Fielding, Mark Nottingham, and Julian Reschke, “HTTP Semantics”, RFC 9110 / STD 97, June 2022. IETF Internet Standard.
10. Sitemaps.org, “Sitemaps XML format”, reviewed September 7, 2026. Industry protocol maintained by participating search engines.
11. Gary Illyes, Google Search Central, “What crawl budget means for Googlebot”, January 16, 2017. Historical first-party explanation; Google now flags parts as potentially outdated.