What is a crawl error?

A crawl error is a problem encountered when a crawler tries to obtain a web resource. It can involve resolving the hostname, connecting to the server, following a redirect or receiving the intended response. A report label needs context: a missing old URL may correctly return an error status, while a current service page should be available.

Google’s HTTP guidance distinguishes response types and their processing consequences. A successful response allows further consideration; it does not guarantee indexing. A robots.txt restriction is a crawl-access rule, rather than proof that your server failed.

Seven common problems and the checks they need

These are selected diagnostic categories, not an exhaustive classification for every crawler. The same visible symptom can originate in the application, hosting platform or delivery layer.

Inspect the response before choosing a repair
ObservationWhat to checkImportant distinction
DNS failureHostname resolution and domain configuration.A connection cannot begin if the hostname cannot be resolved.
Server errorApplication, hosting and gateway responses.Persistent failures differ from a recovered temporary incident.
Unavailable robots.txtThe actual file response and Google’s last fetched rules.A missing file differs from a server or network failure.
404 Not FoundWhether the resource exists or has a suitable replacement.A genuinely removed page can correctly return this response.
Soft 404The content served with a successful status.Google may classify an apparent error or empty response as missing content.
403 ForbiddenAuthentication, firewall and delivery rules.Confirm the intended public access before changing a security rule.
Redirect failureThe actual sequence and final destination.A terminating move differs from a loop or unavailable target.

The Crawl Stats documentation explains DNS, connectivity, robots.txt and response observations. Google’s robots.txt specification describes separate handling for file-fetch failures, including cached rules. An unavailable file does not establish an indefinite site-wide stop under every condition.

Prioritize by affected purpose and scope

An unexpected error on your public booking page needs a different response from a correct not-found status on a retired article. Start with what the address should do and which templates or visitors it affects. A shared server failure deserves investigation at that shared layer.

Google’s HTTP guidance says server errors and overload responses can reduce crawling, and persistent server errors can eventually lead to affected URLs leaving the index. It does not establish an immediate ranking loss for every failed request. Keep the observed failure, its duration and its search consequences separate.

Combine reported observations with current output

Search Console can show what Google encountered. Crawl Stats provides aggregate requests and representative examples; it is not a complete URL-level request log. The Page indexing report describes indexing reasons, with limited example lists.

For an important address, compare URL Inspection’s stored information and live test with the response you serve now. A local crawler tests its own request conditions and settings. It cannot reproduce every Googlebot request or certify Google’s indexing decision.

Available server or CDN records can help identify what was requested and returned. Check logging coverage, intermediary responses and crawler identity before calling them a complete account of Google traffic. Google provides request-verification methods; a claimed user-agent name alone is not adequate verification.

Illustrative responses to different errors

A service page starts returning a server error after a deployment. The team checks the application change and public response, restores the intended page, and records what changed. It then compares the repair with later crawl observations. This scenario has no measured traffic loss or recovery deadline.

A discontinued item has no replacement and correctly returns not found. The owner removes misleading internal links to it rather than redirecting every old product to the homepage. If an equivalent replacement exists, review that destination and the move separately.

A crawl investigation asks whether Google could fetch the intended resource. An indexing investigation also asks what was processed and whether it was included. A page marked noindex may have been fetched successfully, while a successful response can still contain an unintended error message.

Use the indexing entry for the later processing stage and soft 404 for apparent missing content served with a success response. Neither a short page nor an unindexed page automatically establishes a crawl error.

Work from the intended response

  1. Record the exact URL, reported reason and observation time.
  2. State whether the page should exist, redirect, be restricted or be gone.
  3. Inspect the current response, redirect destination and relevant delivery rules.
  4. Check other pages using the same template or infrastructure.
  5. Fix the confirmed cause and update misleading internal links.
  6. Save the deployed observation and compare it with subsequent reporting.

For removed content without a replacement, Google documents both 404 and 410 responses. Its current HTTP reference treats these client-error responses alike for indexing purposes. Choose a truthful status, rather than promise that one code completes removal faster.

Avoid changing the wrong layer

A broad firewall exemption or redirect rule can affect more than the reported address. Verify the requester and the intended public access before relaxing protection. Test nearby routes after changing a shared rule.

Another mistake is treating every non-success label as damage that must be hidden. Returning an empty success page to remove a not-found warning can create a soft 404. Repair the visitor’s actual destination or retain a truthful missing-resource response.

Must every 404 be redirected?

No. Google’s Crawl Stats guidance says a 404 can be correct when a page is genuinely gone without a replacement. Redirect when there is an appropriate destination, and repair links that incorrectly promise an available page. A clean report count is not a reason to send unrelated old URLs to the homepage.

What to bring to a technical discussion

Bring affected URLs, their intended responses, recent deployment changes and any authorized crawl or hosting evidence. If you discuss help with theStacc, confirm who can inspect and change the relevant system. A software signup does not establish that server access, log analysis or a specific audit deliverable is included.

Robots.txt explains permitted fetching. Redirect chains concern the route to a destination. Index coverage concerns reported inclusion, while crawl budget covers capacity and demand.

Sources and editorial scope

Official Google references are linked beside the diagnostic distinctions and were read on October 12, 2026. AI assisted this revision. The scenarios are illustrative; no private logs, Search Console property, customer result or named human approval was reviewed. The original publication field is retained with unresolved historical provenance.