Soft 404s: how a successful HTTP response can erase a page from search

A retailer retires 18,000 discontinued product URLs. The application still returns 200 OK for every route because its single-page app needs an HTML shell. After JavaScript calls the product API, visitors see “This product no longer exists.” Engineering’s uptime monitor stays green, the SEO crawler reports 18,000 successful URLs, and the XML sitemap has already removed them. A week later, Search Console groups many of those URLs under Soft 404.

No system in that chain is necessarily lying. The uptime monitor observed transport success. The crawler recorded the response code it received. The browser rendered the application’s final message. Google evaluated the page content and concluded that the successful response did not represent a live resource. The failure is the contract between those layers: status said “success” while the representation said “gone.”

That conflict can make a page disappear from Google Search. Google’s current troubleshooting documentation says detected soft-404 pages are excluded from Search, and Search Console reports the classification as a non-indexed reason. It does not publish a word-count threshold, classifier score, detection delay, false-positive rate, or recovery service level. The practical task is therefore not to reverse-engineer a magic amount of text. It is to make status, content, routing, and resource state agree.⁴⁸

The useful model: two channels, one resource truth

Every HTTP response communicates through at least two channels. The status code describes the result of handling the request. The response content carries a representation or an explanation. RFC 9110 defines 200 OK as a successful request; for a GET request, its content represents the target resource. A polished “not found” page sent as 200 is therefore not merely an SEO quirk. It is contradictory protocol semantics. ¹

Google adds a third layer: its indexing systems evaluate what they received. Google’s crawler reference states that a 2xx response allows content to move to the next processing step but does not guarantee indexing. When successful content looks empty or error-like, Search Console may show a soft 404. Conversely, Google says it ignores content returned with most 4xx responses and removes previously indexed URLs over time. The protocol status is an input; Google’s classification is a later platform decision.⁵

Five states that teams routinely collapse into one

The empty-but-valid row needs judgment. A search for an obscure phrase can be a legitimate interaction even when it finds zero products. That does not automatically make the URL a missing resource. But if arbitrary query parameters create unlimited, indexable, nearly identical zero-result pages, the site has an inventory problem as well as a content problem. Preserve a useful 200 experience for visitors, then decide whether the page belongs in search with a crawlable information architecture and an explicit indexing policy. Do not turn every no-result state into 404 just to avoid a label.

What Google calls a soft 404

Google’s current definition is broader than an error template with the wrong code. Its troubleshooting page describes a URL that tells the user the page does not exist while returning 200, and adds that a page with no main content or an empty page can also be classified. Documented causes include a missing server-side include, a broken database connection, an empty internal-search result, or JavaScript that failed to load. The underlying pattern is missing page meaning behind a success response.⁴

This explains a common false diagnosis: “Google thinks the copy is too short.” Short pages can be valid. Long error pages can still be errors. The sources provide no minimum word count or uniqueness score. A better question is: does the rendered main content fulfill the identity and purpose of this URL? A product page needs the actual product state and facts. A location page needs that location. An article needs its promised subject. A shell, global navigation, cookie banner, and recommendations do not prove that the target resource exists.

Google also documents the reverse case: a good page can be labelled soft 404 because it did not load properly for Googlebot, critical resources were unavailable, or a prominent error appeared during rendering. If the rendered result is blank, nearly blank, or dominated by an error message, the fix is to restore the page’s evidence rather than manufacture filler. Inspect the returned status, rendered HTML, screenshot, resource failures, and server behavior at the crawl time.⁴

Do not choose a status from the SEO outcome you want

A status code should report the resource state that exists now. It is not an instruction to rank, preserve traffic, or transfer unspecified authority. That discipline prevents four popular but damaging shortcuts.

200 is not a preservation switch

Returning 200 for a removed page does not preserve it as a valid search result. Google explicitly says a successful response does not guarantee indexing. If the page communicates absence, the mismatch creates the condition Google calls a soft 404. The server may also encourage browsers and intermediaries to cache the successful representation: both 200 and common negative responses can be heuristically cacheable under HTTP rules. Correct the status, purge or revalidate affected caches, and test the delivered response at the edge rather than assuming the origin fix is visible everywhere.¹²⁵

A redirect needs a real successor

When the same resource has a new permanent URL, a server-side 301 or 308 is the cleanest contract. Google describes permanent redirects as signals that the target should become canonical and recommends server-side redirects when possible. The word same matters. A retired blue-widget page can redirect to a direct replacement model or a genuinely consolidated successor. Redirecting it to the homepage because the homepage is important does not preserve the original intent; it converts a missing resource into a confusing navigation outcome.⁷

Temporary redirects solve a different problem. If a resource will return at its existing URL but visitors must temporarily see an incident page, a 302 or 307 can retain the source URL as the intended long-term location. Do not make that the default response for permanent deletion. Google distinguishes temporary and permanent redirects in its canonical processing, even though both are valid HTTP mechanisms.

404 and 410 are not quality failures

RFC 9110 says 404 Not Found means the origin has no current representation or is unwilling to disclose one; it does not say whether that condition is temporary or permanent. Use 410 Gone when the server knows the unavailability is likely permanent. That is a semantic distinction, not a promise of faster Google removal. Google’s current crawler reference groups all 4xx responses except 429 together for indexing behavior. Choose 410 because your resource lifecycle knows “intentionally retired,” not because an SEO myth says it wins a race. ¹⁵

A correct 404 can still be friendly. The response body may explain what happened, preserve site navigation, offer search, suggest relevant destinations, and let a visitor report a broken link. Google explicitly recommends useful custom error pages while requiring the server to keep the 404 status. The body serves the human; the status preserves machine-readable truth.⁴

Temporary failure is not absence

If the database is down, the application does not know that every product vanished. A global 404 would assert the wrong state. RFC 9110 defines 503 Service Unavailable for temporary overload or maintenance and allows Retry-After. RFC 6585 defines 429 Too Many Requests for rate limiting and also permits Retry-After. Those signals tell clients to treat the condition as retriable rather than concluding that the target resource does not exist.¹³

Temporary does not mean harmless forever. Google says 5xx and 429 responses reduce crawling, preserve indexed URLs initially, and can lead to removal when they persist. Its crawling-error guidance warns that returning those statuses for more than a few days can cause URLs to be dropped. Treat them as incident signals with paging, recovery verification, and an owner; never as a long-term indexing strategy. ⁴⁵

Why single-page applications create the perfect mismatch

Traditional server rendering can resolve a route and record lookup before sending headers. A client-rendered application often sends the same 200 app shell for every syntactically valid path. Only after the browser downloads JavaScript, starts the application, and calls an API does it learn whether the entity exists. By then, the original document status is immutable. JavaScript can change the DOM and URL, but it cannot rewrite the status of the response that delivered the document.

Google describes crawling, rendering, and indexing as separate phases. It says 200 pages are sent to the rendering queue unless indexing is blocked, while a non-200 error page may skip rendering. Its SPA guidance offers two client-side fallbacks for missing entities: redirect to a server URL that returns 404, or inject a robots noindex directive into the error view. These can prevent an error view from being indexed, but they are not equal architecture choices.⁶

The stronger cross-client design resolves public resource state at the server, edge, or framework route boundary and returns the final status with the first HTML response. Server-side rendering, static generation with deletion handling, or an edge lookup can do this. If that is not currently possible, Google’s redirect-to-a-real-404 workaround is clearer than leaving every missing entity at 200. Dynamic noindex is a bounded Google workaround, not a general substitute for correct HTTP semantics: other clients may never execute it, and the response still claims success.

Framework failure modes to test

• Wildcard fallback: the web server rewrites every unknown path to /index.html with 200.
• API/document split: the API correctly returns 404, but the document remains 200 and renders the API error.
• Hydration replacement: valid server HTML is replaced by a generic error because client state, locale, or an experiment assignment differs.
• Resource failure: blocked, timed-out, or version-mismatched JavaScript leaves only an app shell, spinner, or error boundary.
• Authentication leak: public crawlers receive a sign-in shell with 200 for URLs that should be public, or a private route masquerades as missing.
• Edge cache drift: one region or device profile serves a stale error shell while the origin and developer browser show the live resource.

Test at least the raw response, a browser-rendered response with JavaScript, and the production edge. Add desktop and mobile profiles when routing, rendering, or personalization can differ. A synthetic request with a Googlebot user-agent is useful for exposing user-agent branches, but it does not prove that verified Googlebot made the request or that Google’s Web Rendering Service produced the same DOM.

A worked example: the retired product that was not really replaced

Consider /products/atlas-9-blue. The item is permanently discontinued. The catalog has an Atlas 10, but it is a different size, price, and compatibility class. The current implementation sends a 200 app shell, calls /api/products/atlas-9-blue, receives an API 404, and renders “Product unavailable” plus four popular items. The HTML title remains “Shop products” and the canonical points to the missing URL.

The first decision is whether Atlas 10 fulfills substantially the same intent. Marketing would like the redirect because the new model converts. Support says customers use the old page to verify compatibility and download a manual. Those facts argue against a blind product redirect. A better product decision may be to keep a durable 200 support page for Atlas 9 with discontinued status, specifications, documentation, and a clearly labelled successor comparison. The resource still exists as useful historical information even though it cannot be purchased.

If the business will not maintain that support page and there is no equivalent destination, return 410 because the catalog lifecycle knows the URL is permanently retired. Keep a human-friendly body with search and compatible alternatives. Remove the URL from sitemaps and internal merchandise links, but do not block it in robots.txt: Google needs to crawl the response to observe the removal. External links may continue to send useful referral traffic to the custom error page.

If product identity truly transferred one-to-one—for example, only the slug changed while SKU, specifications, and purpose remained the same— issue a 301 or 308 directly to the new URL. Update internal links, canonical references, structured data, and sitemap entries to the destination. Avoid redirect chains. The result is a consistent contract: the old resource moved, the new resource is live, and every layer says so.

Use the redirect-or-remove decision in this order

1. Can the system determine state? If a dependency prevents the lookup, this is an availability incident. Return 503, or 429 when the client is rate-limited, and recover quickly.
2. Does a meaningful resource still exist at this URL? If yes, return 200 with the promised main content. Out-of-stock, archived, or zero-result does not automatically mean nonexistent.
3. Is there a clear successor that fulfills substantially the same intent? If yes and the move is permanent, use 301 or 308. Update every internal signal to the target.
4. Is the absence known to be permanent? If yes, 410 is semantically precise. Otherwise use 404.
5. Does the body help a person without contradicting the status? Provide navigation, search, and relevant options, but keep the chosen status intact.

This sequence also handles accidental URLs. A typo or invented path normally has no current representation, so 404 is correct. You do not need to create a redirect for every malformed URL Google or a visitor discovers. Search Console says correct 404s are not necessarily problems; fix the ones created by your internal links or sitemaps, and investigate high-value external demand, but do not treat every reported 404 as a defect. ⁸¹⁰

Build a response contract before writing monitoring rules

Monitoring a URL for “up” is insufficient because both the intended page and the error shell may return 200. Define a small contract for each representative URL family. The contract should include expected status, allowed redirect destination, required identity evidence, forbidden error evidence, index policy, and device profiles. Then choose examples that exercise every state transition, not only the happy path.

Minimum automated cases

What to collect when a contract fails

• Final status, redirect hops, Location, Retry-After, content type, cache headers, and the edge cache result.
• Raw HTML and rendered HTML, plus a screenshot and browser console or resource errors.
• Required identity markers and forbidden error markers, evaluated after rendering where JavaScript owns content.
• Canonical and robots directives from both HTTP headers and rendered document head.
• Template, locale, device profile, authentication state, experiment assignment, and release identifier.
• For Google-specific investigation, the Search Console classification, last crawl date, live inspection output, and indexed-versus-live distinction.

Access logs answer a different question: which verified requests reached the CDN or origin, when, and what that boundary returned. Search Console is not a complete request log, and a third-party crawl is not Google. Keep those evidence layers separate. If the incident involves a particular Googlebot fetch, obtain CDN or origin logs and verify crawler identity; neither a user-agent simulation nor a rendered screenshot can reconstruct that event.

What to do this week

1. Write the state table. For each high-value template, define live, empty-valid, moved, gone, and temporary-failure behavior with expected status and body evidence.
2. Find status/content contradictions. Crawl representative live and retired identifiers, random impossible paths, empty searches, and known redirects. Render JavaScript routes as well as fetching source HTML.
3. Fix the state decision at the earliest capable layer. Prefer server, framework, or edge resolution before headers are sent. Keep client-side noindex or redirect workarounds narrow and documented.
4. Repair discovery signals. Remove gone URLs from sitemaps and internal links; point links, canonicals, and structured data at permanent successors; do not block missing URLs before crawlers can observe the response.
5. Purge and retest caches. Confirm the production edge, not only the origin, returns the new contract in all relevant regions and device variants.
6. Add release tests. Make the six cases above part of deployment QA and monitor representative URLs continuously.

A 30-day measurement plan

The goal is not to force Search Console’s soft-404 count to zero. Some discovered junk URLs can be correctly absent, report data can lag, and Google may revisit known URLs for a while. The goal is to prove that valuable live URLs remain meaningful, retired URL families return the intended contract, and new contradictions are caught at release time.

Days 0–3: establish the denominator

• Export or record affected Search Console examples with report date and last-crawl date; do not treat the examples as a complete inventory.
• Group URLs by template, route pattern, lifecycle state, rendering mode, and expected outcome.
• Create a fixed test cohort containing live controls, corrected URLs, known-good redirects, and intentional errors.
• Record source status, rendered meaning, edge cache state, sitemap membership, internal-link count, and device profile.

Days 4–14: validate the implementation

• Run daily contract tests against the fixed cohort after deployments and cache changes.
• Inspect representative corrected URLs in Search Console, keeping the indexed view separate from the live test.
• Watch Crawl Stats response-code and host-availability trends, but use access logs for URL-level production request proof.
• Track recurrence by release and template. One returning wildcard fallback is more actionable than a fluctuating site-wide label count.

Days 15–30: decide whether to change course

Continue the rollout when the fixed cohort consistently returns the expected status and meaning, corrected live pages render their identity without critical resource failure, and no new template-scale contradictions appear. Investigate further when Search Console’s latest crawl still labels a valuable live page soft 404: compare the crawl date with the release, inspect the rendered screenshot and resources, verify user-agent and locale paths, and check the production edge.

Change course when a planned redirect sends users to a materially different resource, a 410 decision destroys useful support demand, valid zero-result pages are being removed even though users need them, or a temporary incident is lasting long enough to threaten index retention. The correct response is the one that matches user intent and resource truth, not the one that makes one dashboard line move fastest.

The strongest counterposition: let Google classify the content

The strongest reasonable objection is that soft-404 detection is useful. If Google can recognize an error page despite a broken CMS, why spend engineering time producing exact statuses? On a small site with a handful of accidental URLs, the classifier may indeed prevent useless pages from appearing. Correct 404s are not ranking improvements, and a team should not delay more valuable work to polish every typo URL.

The conclusion survives because classification is a safety net, not a resource contract. It occurs after crawling and sometimes rendering, it is Google-specific, it is not exposed as a deterministic API, and it can also flag a valid page whose content failed to load. Other clients, caches, analytics systems, monitoring tools, link checkers, and search engines still receive the contradictory 200. At inventory scale, Google’s historical crawl-budget explanation also lists soft error pages among low-value URL classes that can consume resources, while explicitly warning that crawl rate is not a ranking signal.⁹

The bounded recommendation is proportional. A small brochure site should fix its shared error handler and add a few route tests. A large marketplace or UGC platform needs lifecycle-aware states, edge-cache controls, template cohorts, log evidence, mobile and desktop rendering checks, and deployment gates. Neither needs a project to imitate Google’s classifier.

What the evidence does not show

• It does not reveal Google’s soft-404 classifier, content threshold, confidence score, false-positive rate, detection delay, or recovery SLA.
• It does not show that every thin, empty, out-of-stock, or zero-result page is a soft 404. Resource purpose and rendered evidence matter.
• It does not show that 410 is removed faster than 404. Google’s current crawler reference treats non-429 4xx responses alike for indexing behavior.
• It does not show that correct status codes guarantee crawling, indexing, ranking, impressions, or traffic.
• It does not quantify the crawl cost of soft errors. Google’s 2017 crawl-budget post supplies no denominator or effect size and warns that some details may be outdated.
• It does not make Search Console a real-time or complete URL-level log. Report examples, live tests, and indexed information answer different questions.
• It does not prove that a user-agent simulation reproduces verified production Googlebot or Google’s rendering and indexing systems.

Where 2-UA fits—and where it stops

2-UA can support the contract layer: crawl links, report response codes, audit redirects, monitor an expected status, and compare source with a bounded browser-rendered result for public pages. That can reveal a 200 error shell, a redirect chain, a broken internal link, or a rendered page that lost its identity. It does not determine Google’s soft-404 classification, provide verified Googlebot access logs, reproduce Google’s Web Rendering Service, or establish indexed state. Those questions require Search Console and, for request history, CDN or origin logs.

Implementation checklist

• Map resource lifecycle states before selecting response codes.
• Return the final status from the server or edge when feasible.
• Keep a friendly error body, but never let it overwrite the error status.
• Redirect only to a clear successor; avoid homepage and unrelated-category catch-alls.
• Use 404 when no current representation exists and 410 only when permanence is known.
• Use 503 for temporary service failure and 429 for rate limiting; monitor recovery immediately.
• Test raw and rendered content, desktop and mobile variants, JavaScript failure, and production edge caches.
• Remove gone URLs from sitemaps and internal links without blocking crawlers from seeing the removal status.
• Record Search Console crawl dates before judging whether a fix failed.
• Keep access-log, crawler, rendered, indexed, and search-performance evidence in separate columns.

References

1. IETF, RFC 9110: HTTP semantics. Sections 15.3.1, 15.5.5, 15.5.11, and 15.6.4. Internet Standard, June 2022.
2. IETF, RFC 9111: HTTP caching. Sections 2 and 4.2.2. Internet Standard, June 2022.
3. IETF, RFC 6585: additional HTTP status codes. Section 4, 429 Too Many Requests. Proposed Standard, April 2012.
4. Google Search Central, Troubleshoot Google Search crawling errors. Soft-404 and temporary-overload guidance; updated December 18, 2025.
5. Google Crawling Infrastructure, How HTTP status codes affect Google’s crawlers. Updated February 4, 2026.
6. Google Search Central, Understand the JavaScript SEO basics. Crawl-render-index sequence and SPA soft-404 guidance.
7. Google Search Central, Redirects and Google Search. Updated April 14, 2026.
8. Google Search Console Help, Page indexing report. Soft 404 and Not found report reasons.
9. Google Search Central, What crawl budget means for Googlebot. Historical platform post, January 16, 2017; page warns that some information may be outdated.
10. Google Search Console Help, Crawl Stats report. Aggregate crawling evidence and correct-404 guidance.