A crawl error is any issue that prevents Googlebot or another search-engine crawler from successfully fetching a page. Errors fall into two families: site-level (DNS failure, server 5xx, robots.txt unreachable) and URL-level (404 not found, 403 forbidden, soft 404, redirect loop). Both drain crawl budget and delay indexing.
Every crawl error is a page Googlebot tried, failed, and may not try again for weeks. A handful of them is normal on any site. A pile of them - especially the site-level kind - is one of the fastest ways to lose organic traffic without ever changing your content.
What is a crawl error?
A crawl error is a fetch failure. Every time a crawler makes a request to your server, the server returns an HTTP response. If the response is a 2xx (success) or a properly signalled 3xx (redirect), the crawl succeeded. Anything else - a 4xx client error, a 5xx server error, a DNS timeout, a robots.txt block - is a crawl error.
Google discontinued the old "Crawl Errors" report in 2019 and rebuilt reporting inside the Google Search Console Pages and Crawl Stats reports. The underlying concept did not change - just the interface.
Google treats site-level errors as urgent - if your DNS fails, no page can rank. Individual URL errors are diagnostic - they signal broken structure but usually do not hurt overall visibility unless the affected URLs matter.
7 types of crawl errors
| Type | Cause | Impact | Fix |
|---|---|---|---|
| DNS failure | Domain does not resolve | Site-wide, urgent | Check DNS records, TTL |
| Server error (5xx) | App crash, overload, 500/502/503 | Blocks whole path | Fix code / scale server |
| Robots.txt unreachable | robots.txt returns 5xx | Google pauses crawling | Make robots.txt return 200 or 404 |
| 404 Not Found | URL does not exist | Only harmful if page matters | Redirect or accept |
| Soft 404 | Empty page returning 200 OK | Wastes crawl budget | Return real 404 or fill content |
| 403 Forbidden | Auth block or firewall | Prevents indexing | Whitelist Googlebot, remove block |
| Redirect loop / chain | A → B → A or 5+ hops | Wastes crawl, breaks indexing | Point 301 directly to final URL |
Why crawl errors matter
- Blocked indexing. A URL Googlebot cannot fetch cannot appear in search results.
- Wasted crawl budget. Every failed fetch is a request Googlebot spent on nothing - budget it will not spend on important pages.
- Ranking signal loss. When a page becomes unreachable, the links pointing to it lose their downstream value.
- User experience damage. Errors that Googlebot sees are usually errors real users hit too - broken UX and lost revenue.
How to detect crawl errors
Every serious crawl-error workflow uses three sources together.
1. Google Search Console
The Pages report lists every URL Google could not index and the reason. The Crawl Stats report (Settings → Crawl stats) shows response codes over time - a spike in 5xx errors is a hard signal something broke.
2. Site crawlers (Screaming Frog, Sitebulb)
These tools crawl your site the way Googlebot does and surface every error, including ones GSC has not caught yet. Run monthly at minimum, weekly on large sites.
3. Server log files
The ground truth. Logs show every Googlebot hit and the status code returned. Nothing else gives you the actual crawler view.
URL: /products/discontinued-item/
Last crawled: 3 days ago
Response: 404 Not Found
Referring pages: 4 internal links
# Fix: either restore, 301, or remove links
Real crawl error examples
1. E-commerce site during a deploy
A retailer's checkout template throws a 500 error during a deploy. Googlebot hits 2,400 URLs over 30 minutes, all returning 5xx. Rankings for those product pages drop within 48 hours until the deploy is rolled back and re-crawled.
2. Blog with legacy 404s
A publisher migrated CMS in 2019 without redirects. 780 old blog URLs still appear in Search Console as 404. Because none receive traffic or backlinks, the errors are harmless - Google is simply cleaning up. No action needed.
3. Soft 404 on a category page
A category page returns HTTP 200 but shows "No products found." Google flags it as a soft 404. Fix: either populate the page with default recommendations or return a real 404 when the category is truly empty.
Crawl error vs indexing error - what is the difference
Crawl error
- Fetch failed - Googlebot never saw content
- Cause: DNS, server, URL structure
- Fixed at the server or code level
- Impact: page never enters the index
- Detected in: Crawl Stats + Pages report
Indexing error
- Fetch succeeded - Google decided not to index
- Cause: noindex, canonical, thin content
- Fixed at the content or directive level
- Impact: page fetched but excluded
- Detected in: Pages report exclusions
7 best practices for crawl errors
- Monitor Crawl Stats weekly. A 5xx spike is a five-minute check that catches serious problems fast.
- Set alerts on 5xx errors. Uptime monitoring plus a Search Console alert should ping you the moment errors cross a threshold.
- Fix errors on pages with links or traffic first. A 404 on a URL with 30 backlinks is urgent. A 404 on an orphaned URL is not.
- Do not mass-redirect 404s to the homepage. Google treats it as a soft 404 and gives it no equity.
- Return 410 Gone for permanently retired URLs. Stronger signal than 404, removes them from the crawl queue faster.
- Never let robots.txt return 5xx. Google interprets an unreachable robots.txt as "stop crawling entirely" and pauses your whole site.
- Fix internal links to broken URLs. Every internal link to a 404 wastes budget and confuses signals.
Not every crawl error needs a fix. 404s on genuinely retired URLs are healthy - Google expects them. Focus on 4xx and 5xx errors affecting URLs that have traffic, backlinks, or internal links pointing to them. Everything else is noise.
Common crawl error mistakes to avoid
- Redirecting bulk 404s to the homepage. Treated as soft 404, gains no equity.
- Ignoring 5xx spikes. Server errors compound - fix within hours, not days.
- Blocking Googlebot in error handlers. Firewall rules that treat crawlers as bots often return 403 by mistake.
- Fixing errors on unimportant URLs first. Sort by traffic and inbound links, not alphabetically.
- Assuming Search Console is real-time. Crawl Stats lag 2-3 days. Use log files for live diagnostics.
Frequently asked questions
A crawl error is any issue that prevents Googlebot or another search engine crawler from successfully fetching a page. Crawl errors fall into two families: site-level (DNS, server 5xx, robots.txt failures) and URL-level (404 not found, 403 forbidden, soft 404, redirect loops).
Google Search Console's Pages report and Crawl Stats report list every URL Google could not fetch, along with the error type. Screaming Frog, Sitebulb, and server log files add deeper diagnostics.
Yes - but only certain types. 5xx server errors and DNS failures actively hurt rankings by making pages unreachable. 404s on unimportant URLs are harmless. Soft 404s hurt because they waste crawl budget on valueless pages.
A 404 returns an HTTP 404 status code, telling Google the page does not exist. A soft 404 returns HTTP 200 OK on a page that looks empty or missing to Google. Soft 404s waste crawl budget because Google keeps trying them.
Site-wide errors (DNS, 5xx, robots.txt) need immediate action - they block the whole site. Individual 404s on retired URLs can be triaged over 1-2 weeks. Prioritize by traffic and inbound links to the affected URL.
How theStacc helps
theStacc's technical audit runs a full crawl-error sweep across your site and log files. Every error is scored by traffic, backlinks, and internal-link exposure so you fix the ones that actually matter first. No noise, no orphan-URL panic.
Related glossary terms
Sources
