How fast does search learn that a page changed? Sitemaps, lastmod, IndexNow, and the limits of “instant indexing”

Search engines can learn about a changed page quickly, but “learn” hides several independent events. A system may receive a notification in seconds, crawl the URL hours later, process a new version later still, select a different canonical, and show different versions across search surfaces while caches converge. If a team records only the submission receipt, it cannot tell which boundary failed.

The primary sources do not support a universal indexing-speed promise. Google says a requested crawl can take from a few days to a few weeks and does not guarantee inclusion; its troubleshooting guidance says most sites should expect three days or more rather than same-day processing.^{3, 10} IndexNow makes the notification event immediate for participating engines, but its protocol says an HTTP 200 means only that the engine received the URLs.⁴ Bing recommends combining IndexNow with sitemaps and says no tool can guarantee when or how content will appear.⁷

The practical answer is not one number. It is a measurement design: choose the right notification path for each engine and change type, publish truthful modification evidence, then time notification, crawl, index state, and serving separately. That gives you a diagnosis instead of an “instant indexing” story.

The answer in one minute

The four clocks hidden inside “indexed”

Google describes crawling, indexing, and serving as separate stages, and not every page reaches every stage.¹³ Notification happens before those stages. Treating the four events as one produces false confidence when the early event succeeds and a later one does not.

Clock 1: notification

This is the interval from your publication event to a successful signal: a sitemap containing the correct URL and timestamp, a valid IndexNow response, or a manual URL Inspection request. It answers, “Did we tell the intended system?” It does not answer whether the engine fetched the page. Keep the request body, endpoint, response code, attempt count, and time. For IndexNow, distinguish 200 from 202: the latter means the URL was received while key validation is pending.⁴

Clock 2: crawl

This is the first verified crawler request that retrieves the changed URL after publication. Exact URL-level evidence normally lives in origin, CDN, or edge logs. Google’s crawling guidance explicitly sends site owners to logs for this question because Search Console does not provide a complete per-URL crawl history.¹⁰ Verify Googlebot rather than trusting a user-agent string; store the response status, bytes, cache outcome, response time, and content version or hash.

A request is still not proof that a new representation was usable. The crawler may receive a 304 based on a bad validator, a cached old body, a soft 404, a blocked resource, an error page, or a redirect to another URL. Crawl success should mean that the expected changed representation was delivered—not merely that a bot appeared in a log.

Clock 3: indexed state

This is the first observation that a search engine’s indexed version reflects the expected URL and version. Google’s URL Inspection API can report verdict, coverage state, indexing state, page-fetch state, last crawl time, declared and selected canonicals, crawler type, known sitemaps, and referring URLs.¹¹ Those fields help distinguish “not fetched,” “fetched but excluded,” and “indexed under a different canonical.” They do not provide a perfect processing-event stream.

Clock 4: serving

This is the first controlled observation that a relevant search surface presents the new information. It is the noisiest event. Search results can vary by query, locale, device, user context, data center, feature, and canonical. A changed title in one query does not prove universal propagation. Define the query, country, device, surface, and evidence captured. Treat serving checks as samples, not an index census.

These clocks turn a vague complaint into a diagnosis. If notification succeeded but no verified crawler requested the URL, inspect discovery and crawl constraints. If the new body was fetched but URL Inspection reports an exclusion or a different canonical, work on indexability and consolidation. If the indexed state is current but one result looks stale, investigate the serving surface and snippet selection instead of resubmitting the URL.

Choose a mechanism by engine and change type

The wrong question is “Which indexing tool is fastest?” The useful question is “Which system must learn which fact, at what scale, with what urgency?” A complete sitemap, an event-level notification, and a manual inspection do different jobs.

The default design for an ordinary public website is additive: stable crawlable links, a complete canonical sitemap, truthful lastmod, and engine-specific event notification where supported. Manual inspection is an escalation and diagnostic tool, not the publishing pipeline. A protocol should remove ambiguity about delivery; it cannot remove a search engine’s quality, canonicalization, rendering, and serving decisions.

What sitemaps and truthful lastmod can do

A sitemap helps a search engine discover the URL inventory you want considered. Google tells publishers to include absolute, canonical URLs they want shown in Search. One sitemap is limited to 50,000 URLs or 50 MB uncompressed; larger inventories require multiple files and optionally a sitemap index.^{1, 8} These are file limits, not a crawl promise.

The <lastmod> field can make that inventory more useful. Google says it uses the value when it is consistently and verifiably accurate, and defines a significant update as a change to main content, structured data, or links—not a copyright-year tick.¹ Its 2023 explanation is unusually direct: if a page changed years ago but the sitemap says yesterday, Google may eventually stop believing that page’s modification dates. If your CMS cannot determine a truthful date for a composite page, omitting the field is acceptable.²

Define “significant” in page semantics

A useful implementation starts with facts a search result or user decision could depend on. For a product page, those may be price, availability, primary description, variants, structured offer data, and canonical destination. For an article, they may be the substantive text, author correction, evidence, title, or internal links. For a location page, opening hours, address, services, and closure state matter. A rotating recommendation widget or global footer change usually should not reset every page’s timestamp.

Compute a semantic content version from owned source fields or a normalized subset of the rendered representation. Store the time that version changed. Do not derive every URL’s lastmod from deployment time, database backup time, cache purge time, or sitemap generation time. Those events describe infrastructure, not necessarily the page.

Do not confuse two “last modified” contracts

Sitemap <lastmod> is discovery metadata about a URL. The HTTP Last-Modified response header is a validator timestamp for the selected representation. RFC 9110 says its derivation is implementation-defined, it must not be in the future relative to the response, and an ETag can be more reliable when timestamp resolution or maintenance is inadequate.⁹ The values may share an underlying content version, but copying one blindly into the other can be wrong when a URL has negotiated representations, assembled fragments, or multiple updates within a second.

The deprecated Google ping is not a shortcut

Google completed deprecation of the unauthenticated sitemap ping endpoint. Requests to it return 404 and do nothing useful. Sitemaps remain discoverable through robots.txt, Search Console, and the Search Console API.² If an old plugin still “pings Google,” inspect the endpoint and response rather than assuming a green plugin badge means Google was notified.

Sitemap index timestamps need their own truth

The Sitemaps protocol says a sitemap-index <lastmod> describes the child sitemap file, not the pages listed inside it. Accurate values enable crawlers to retrieve only changed sitemap files.⁸ On a large site, that is useful: shard by stable page family, update only the child file whose inventory or metadata changed, and publish the matching index timestamp. Rewriting every shard on every build defeats the incremental signal.

What IndexNow actually acknowledges

IndexNow changes the first clock. Instead of waiting for a crawler to revisit a sitemap or link, a publisher can notify a participating endpoint immediately when a URL is added, updated, or deleted. A POST can contain up to 10,000 URLs. Ownership is verified through a key file on the submitted host or within an authorized path.⁴

The response contract is the guardrail. HTTP 200 means the engine received the URLs. HTTP 202 means it received them while key validation is pending. The protocol also defines errors for malformed requests, invalid keys, host or schema mismatch, and excessive requests. None of the success responses means “the new content is in the index.” The IndexNow FAQ says the engine still evaluates whether to crawl, and each participant decides independently whether to index.⁵

Participating engines are a real boundary

Current IndexNow guidance lists the global IndexNow endpoint and endpoints for Amazon, Bing, Naver, and Seznam.cz. A submission to one is shared across IndexNow-enabled participants.⁵ Google is not in that documented endpoint list. Do not sell IndexNow as a Google indexing method; implement Google’s sitemap and URL Inspection guidance separately.

Bing’s “instant” language needs protocol-level translation

Bing’s 2025 adoption update uses phrases such as “instant indexing,” “real-time indexing,” and “near real time” while describing Shopify, Amazon, and Milestone integrations.⁶ The page publishes no sample, denominator, before/after distribution, crawl completion rate, or independent measurement. It is valid evidence of Bing’s product direction and named integrations, not a latency benchmark.

Bing’s more operational sitemap guidance is narrower. It recommends sitemaps for comprehensive coverage and IndexNow for URL-level changes, says a submitted or robots.txt-declared sitemap is typically revisited at least daily, and explicitly states that no tool can guarantee when or how content appears.⁷ “Sitemap fetched daily” is also not “each listed URL crawled daily.” Keep the clocks separate.

Event-driven does not mean event-spam

Notify after a meaningful, successfully deployed change—not when an editor opens a draft, a job enters a queue, or a cache purge starts. Deduplicate retries by URL and content version. Respect 429 responses and their retry guidance. Repeated unchanged submissions add noise, consume crawl attention, and weaken the audit trail. For high-frequency user-generated content, batch or threshold notifications according to the protocol’s guidance rather than firing on every reaction or counter increment.

Google’s documented submission paths

For ordinary pages, Google documents two recrawl paths: URL Inspection for a few URLs and sitemaps for many. It says crawling can take from a few days to a few weeks, a request does not guarantee instant inclusion or inclusion at all, and repeated requests for the same URL do not accelerate crawling.³ That makes manual submission appropriate for diagnosis and a small number of urgent pages—not a release webhook for an entire catalog.

URL Inspection: useful, quota-limited, and not a completion receipt

The tool can test whether a live URL appears indexable and can place an eligible URL into an indexing queue. Its indexed-view report is more valuable after submission: it can expose last crawl time, fetch state, indexability, and Google-selected canonical. A team should record what it is checking. “Live test passed” means the current page can be fetched and evaluated at test time; it does not mean the indexed version has changed. “Request indexing submitted” means a queue request was accepted; it does not reveal completion time.

The Indexing API is not a loophole for ordinary content

Google’s Indexing API is restricted to pages with JobPosting or a livestream BroadcastEvent embedded in VideoObject. The default onboarding quota is 200 publish requests per project per day, and additional use requires approval. A batch can combine up to 100 calls, but quota is counted for every URL.¹²

Most importantly, the API’s metadata endpoint reports the last notification Google received. Its documentation says it does not tell you when Google indexed or removed the URL.¹² An SEO tool that displays that metadata as an “indexed at” timestamp is labelling the wrong event. For product pages, articles, category pages, and ordinary landing pages, do not add fake job or livestream markup to gain API access; use the supported paths.

Build a trustworthy freshness pipeline

A reliable implementation starts at the content transaction and ends in evidence. The goal is not to call every endpoint. It is to preserve a causal chain from “version changed” to “new version observed,” with enough detail to locate a failure.

1. Create a semantic version event

When publishing succeeds, calculate or retrieve a content-version identifier. It can be a database revision, immutable deployment identifier, or hash of normalized fields that matter to the page. Store the canonical URL, change type, changed fields, publication time, old version, new version, and urgency class. If the public response has not changed yet, do not emit a notification.

2. Route by engine, scale, and urgency

• All canonical public pages: preserve crawlable internal links and sitemap coverage.
• Significant page changes: update that URL’s truthful sitemap lastmod.
• Participating engines: send an IndexNow notification after the new public version is available.
• Google, a few high-risk URLs: use URL Inspection manually after verifying the live response.
• Google, many URLs: rely on sitemap coverage and architecture; do not automate repeated manual requests.
• Eligible job or livestream pages: consider Google’s Indexing API under its actual markup and approval rules.

3. Preserve delivery evidence

Store the endpoint, request time, content version, URL count, response status, retry count, and final outcome. Do not log the IndexNow key. For sitemap publication, store the generated file hash, URL count, child-shard identity, last successful fetch observation, and validation result. A queue job marked “done” is not enough if it does not preserve the external response.

4. Verify the public representation

Fetch the canonical URL as a normal user and the relevant crawler profile. Confirm status, canonical, robots directives, structured data, changed fact, and rendered main content. Record a version marker or normalized hash. This catches the embarrassing case where the database changed but the CDN, rendered application, API hydration, or alternate device route remained stale.

5. Observe downstream events without inventing them

Join verified crawler logs to the change event by canonical URL and time. Query URL Inspection at a measured cadence for sampled Google URLs, respecting quotas. For serving observations, define a limited, reproducible set of queries and regions. Never backfill a missing event with the timestamp of the previous one. If you did not observe index processing, keep the field null.

Worked example: one update, five timestamps

Consider a fictional ecommerce page whose price changes from $89 to $79 at 09:00. The update is material because visible HTML, Product structured data, and the purchasing decision change. The team publishes the page, changes the sitemap timestamp, and notifies IndexNow. The following times are illustrative; they are not platform benchmarks.

At 09:00:08 the IndexNow integration was healthy. It would have been incorrect to close the freshness incident because the business requirement concerned an obsolete offer in search, not API delivery. At 09:14 the team could say Bingbot fetched the new version, but not yet that Bing indexed or served it. Googlebot’s later request did not demonstrate a failure in IndexNow because IndexNow was never the Google notification path.

The ledger also prevents a misleading engine comparison. The team did not capture a Bing index-state timestamp or equivalent serving observation, so it cannot claim Bing indexed faster. It can report two crawler-request lags and one end-to-end Google serving sample. Missing evidence remains missing.

A 30-day measurement plan

A useful freshness program measures your site rather than importing a vendor adjective. Run the following plan on low-risk, representative changes. Do not delay product recalls, legal corrections, security notices, or other urgent updates to create an experiment.

This week: instrument the pipeline

1. Define three change classes: urgent fact, material content, and non-material presentation.
2. Choose a latency budget for each class and engine. For example, an urgent availability removal may require immediate notification and human escalation; an evergreen article correction may tolerate a longer observation window.
3. Implement semantic version IDs and truthful sitemap timestamps for the page families in scope.
4. Log sitemap publication and IndexNow delivery without secrets.
5. Verify crawler identities and retain URL-level origin or CDN logs for the measurement window.
6. Create an inspection sample stratified by template, importance, update frequency, canonical complexity, and device-rendering risk.
7. Write the serving-observation protocol before looking at results: query, country, device, surface, cadence, screenshot or result capture, and stop rule.

Observe for 30 days

For each eligible change, capture publication-to-notification, publication-to-first verified crawl of the new version, crawl-to-current index-state observation, and current index state-to-first defined serving observation. Report the median and slower tail—such as the 90th percentile—only when the sample is large enough to make that summary meaningful. Always show the count of eligible changes and the count with each event observed.

Segment before aggregating. New URLs and updates to known URLs have different discovery problems. Deletions have different evidence from additions. Product pages can differ from editorial pages. A large site should split by sitemap shard, template, crawl demand, canonical pattern, and response performance. A small site may have too few changes for stable percentiles; use an event log and review each miss.

Record failure states, not only completed rows

• Notification failed, rate-limited, or key validation pending.
• Sitemap file was unreachable, invalid, stale, or missing the canonical URL.
• Crawler requested the page but received an old version, redirect, error, or 304 with incorrect validators.
• The new page was crawlable but carried noindex, a conflicting canonical, or materially different rendered content.
• The page was crawled but remained excluded or canonicalized elsewhere.
• Index state looked current but the defined serving observation stayed stale.
• No downstream evidence was collected; do not misclassify this as a failure or success.

Use change-course rules

Change the system when the evidence points to a controllable boundary. If more than a trivial share of notifications fail, repair retries, key placement, URL ownership, or release ordering. If accepted notifications frequently precede stale public content, move the event after cache convergence. If crawl lag breaches the business budget for one template, inspect links, sitemap accuracy, duplication, server health, rendering cost, and crawl demand. If pages are crawled promptly but excluded, stop tuning the notifier and investigate quality, indexability, and canonical selection.

Do not change course because a single query shows an old snippet once. Require repeated observations under the written protocol, or stronger indexed-state evidence. Conversely, do not dismiss a high-risk stale fact because the site-wide median looks healthy. Latency budgets attach to change classes and page value, not only portfolio averages.

The strongest counterposition

The strongest reasonable counterposition is that this framework is over-engineered. IndexNow is designed to notify engines immediately, modern CMS platforms implement it automatically, and most teams should enable the integration and move on. Adding ledgers, verified logs, inspection samples, and serving checks can cost more than the occasional delay.

That argument is correct for low-risk sites if the decision is merely “should we enable a supported, low-cost notification?” Enable it. A small brochure site does not need a search-freshness observability platform. A truthful sitemap, crawlable links, working IndexNow integration for participants, and occasional Search Console inspection may be sufficient.

The argument fails when stale search information has material cost or when a vendor promises an outcome it cannot observe. A marketplace can misstate availability across thousands of URLs. A publisher can leave a corrected claim in old snippets. A job board can show expired roles. In those contexts, seconds-to-receipt is not the business outcome. The additional instrumentation is justified because it separates publisher-controlled failures from search-engine processing and makes the escalation specific.

The conclusion survives the counterposition in narrower form: implement the simplest valid notification stack by default; add downstream measurement in proportion to the cost and frequency of stale information. Never label a delivery receipt as indexing, even when you choose not to measure the later stages.

Small-site, large-platform, and product boundaries

Small sites

A small site should prioritize correctness over automation. Keep one valid sitemap, truthful dates for pages whose changes can be known, and clear internal links. Use URL Inspection for a handful of important Google updates. If IndexNow is built into the CMS or CDN, enable it and test the key and response once. Maintain a simple sheet with change time, notification result, crawler observation if available, and URL Inspection result for important releases. Do not build percentiles from five events.

Large platforms

A large site needs event idempotency, stable sitemap sharding, bounded retries, secrets management, per-host ownership, crawler verification, and sampling. Avoid regenerating all sitemap shards for one URL. Preserve content versions so a bot fetch can be matched to the intended change. Measure by template and urgency, because a healthy median can hide a broken locale, inventory shard, or JavaScript route. Keep deletion events, redirects, and canonical changes as distinct classes.

Where 2-UA fits—and where it does not

2-UA can monitor sitemap availability, hierarchy, declared URL counts, daily count changes, page responses, and public page snapshots. Those checks can reveal a missing URL cohort, a stale or failed sitemap, an unexpected status, or a page version that did not reach the public web. Its read-only Search Console integration imports search-performance data; it does not submit indexing requests, expose Google’s URL Inspection index-state fields, or replace origin/CDN access logs. Exact crawler-request timing requires your logs, Google index-state diagnosis requires Search Console URL Inspection or its API, and serving evidence requires a defined search-observation process.

What the evidence does not show

Implementation checklist

Publishing contract

• Define significant changes per page type.
• Generate an immutable content-version identifier.
• Emit freshness events only after the public version is available.
• Store canonical URL, change type, urgency, and changed fields.
• Keep visible HTML, structured data, feeds, and transactional truth aligned where applicable.

Sitemap contract

• Include absolute canonical URLs intended for search.
• Respect the 50,000-URL and 50 MB uncompressed limits.
• Set lastmod from the last significant page change.
• Omit the timestamp when it cannot be derived reliably.
• Do not use changefreq or priority as Google crawl controls.
• Timestamp child sitemap files independently in the sitemap index.
• Reference the sitemap in robots.txt and/or submit it through the relevant webmaster tools.
• Remove calls to Google’s deprecated sitemap ping endpoint.

Notification contract

• Use IndexNow only for engines and services that participate.
• Host and test the ownership key without writing it to application logs.
• Batch no more than 10,000 URLs in one IndexNow POST.
• Label 200 as received and 202 as received with key validation pending.
• Deduplicate by URL and content version.
• Back off on 429 and do not resubmit unchanged URLs.
• Use Google URL Inspection for a few important URLs and sitemaps for many.
• Use Google’s Indexing API only for eligible job and livestream pages.

Evidence contract

• Verify the changed fact in the public response before notification.
• Retain notification request and response metadata.
• Verify crawler identity and record the served content version.
• Record indexed state, last crawl, fetch result, and selected canonical separately.
• Define query, country, device, and surface for serving observations.
• Leave unobserved events null.
• Review misses by boundary before changing the notification mechanism.

The durable operating rule is simple: publish truthful change evidence, notify the engines that accept the mechanism, and never let an early success claim answer a later question. “200 received” is useful. It just is not “indexed.”

References

1. Google Search Central, “Build and submit a sitemap”, updated July 8, 2026.
2. Gary Illyes, Google Search Central, “Sitemaps ping endpoint is going away”, June 26, 2023; deprecation complete.
3. Google Search Central, “Ask Google to recrawl your URLs”, updated December 10, 2025.
4. IndexNow, protocol documentation, accessed September 28, 2026.
5. IndexNow, implementation FAQ, accessed September 28, 2026.
6. Fabrice Canel and Krishna Madhavan, Microsoft Bing, “IndexNow drives smarter and faster content discovery”, May 19, 2025.
7. Fabrice Canel and Krishna Madhavan, Microsoft Bing, “Keeping content discoverable with sitemaps in AI-powered search”, July 31, 2025.
8. Sitemaps.org, Sitemaps protocol, accessed September 28, 2026.
9. R. Fielding, M. Nottingham, and J. Reschke, RFC 9110: HTTP Semantics, June 2022, Sections 8.8.2–8.8.3.
10. Google Search Central, “Troubleshoot Google Search crawling errors”, accessed September 28, 2026.
11. Google Search Console API, “UrlInspectionResult”, accessed September 28, 2026.
12. Google Search Central, Indexing API documentation, including the quickstart, notification status, and quota pages, updated July 16, 2026.
13. Google Search Central, “In-depth guide to how Google Search works”, accessed September 28, 2026.