Crawl budget is the total number of URLs Googlebot will fetch from a site during a given period. It equals crawl rate limit (server capacity Googlebot can safely use) multiplied by crawl demand (how much Google wants to recrawl based on freshness and popularity). Optimization matters most for sites with 10,000+ URLs or heavy publishing cadence.
Below 1,000 URLs, crawl budget is not a problem. Above 10,000 - especially on e-commerce and news sites - it becomes the single biggest reason new content sits unindexed for weeks. Fix crawl budget and indexing speed drops from weeks to days.
What is crawl budget?
Crawl budget is a technical SEO concept that describes how much of your site Googlebot will crawl in a given time window. Google introduced the term officially in a 2017 blog post to clarify years of confusion.
Two factors combine to set the budget:
- Crawl rate limit. The maximum number of parallel connections Googlebot can open to your site without degrading performance for real users. Fast servers get a higher limit; slow servers get throttled.
- Crawl demand. How badly Google wants to crawl your URLs. Popular, frequently updated pages have high demand. Stale, orphaned pages have low demand.
Multiply the two and you get the number of URLs Googlebot will fetch per day. Waste those fetches on parameter URLs or broken links and important pages sit unindexed.
Google states that most sites do not need to worry about crawl budget. It becomes a concern above roughly 10,000 URLs, or on sites publishing 30+ pages per month that show indexing delays in Search Console.
Why crawl budget matters
- Faster indexing. More budget spent on important URLs means new content ranks in days instead of weeks.
- Prioritization. Well-budgeted sites let Google discover the pages that matter first, not filter permutations.
- Site health signal. Google's ranking systems pay attention to crawl efficiency. A site full of soft 404s and redirect chains looks unmaintained.
- Server cost. Every wasted crawl is a wasted request on your infrastructure - real money at e-commerce scale.
How Googlebot allocates crawl budget
The allocation runs as a two-part negotiation between your server and Google's schedulers.
Crawl rate limit (server-driven)
Googlebot samples your response times. Fast, healthy responses (under 200ms) let it open more parallel connections. Slow responses (above 500ms) or 5xx errors cause it to back off. The limit exists to prevent Google from taking your site down.
Crawl demand (Google-driven)
Google decides which URLs it wants to crawl and how often based on:
- Popularity. URLs with more inbound links and impressions get recrawled more often.
- Staleness. URLs that change frequently get revisited on a shorter interval.
- Type. Home pages, category pages, and high-authority pages sit at the top of the demand list.
Crawl budget = Crawl rate limit × Crawl demand
# Example: mid-sized e-commerce site
Rate limit: 50 requests/sec
Effective demand: ~40% utilisation
Daily crawl: ~1.7M requests
What wastes crawl budget the most
| Waster | How it burns budget | Fix |
|---|---|---|
| Faceted navigation | Millions of filter combinations | Robots.txt or canonical to core category |
| Session-ID / tracking params | Same page, infinite URL variants | Canonical, URL parameters tool |
| Broken links (4xx) | Googlebot re-checks failed URLs | Return 410 Gone or fix the link |
| Redirect chains | Multiple hops per crawl | Point 301s directly to final URL |
| Infinite calendars | Every future date = new URL | Nofollow future links, robots block |
| Duplicate content | Multiple URLs same content | Canonical to primary URL |
| Soft 404s | Empty pages returning 200 OK | Return real 404 or add content |
| Low-value tag / archive pages | Thin content Googlebot ignores | Noindex and consolidate |
Crawl budget examples
The scenarios below are illustrative composites with worked numbers, not client data.
1. E-commerce site with faceted navigation
A retailer has 12,000 products but exposes 1.2M URLs due to color, size, and brand filters. Googlebot spends 80% of its crawl budget on filter permutations. New product pages take 3-4 weeks to index. Blocking the top 6 filter parameters in robots.txt drops crawlable URLs to 40,000 - indexing time drops to 2-3 days.
2. News publisher with archive drift
A news site publishes 40 articles daily. Over 8 years it accumulates 115,000 archived pages. Googlebot spends significant budget re-crawling stale archives. Consolidating tag pages and noindexing 60,000 low-traffic articles frees up budget for the daily fresh content that actually drives traffic.
3. Small site optimization
A B2B SaaS with only 800 URLs will never hit a crawl budget limit. Time is better spent on internal linking and content freshness than technical crawl-budget optimisation.
Crawl budget vs crawl rate - what is the difference
Crawl rate
- Requests per second Googlebot makes
- Set by server performance
- Measured in real time
- Adjustable per-domain in Search Console (legacy)
- The "how fast" dimension
Crawl budget
- Total URLs Googlebot fetches per period
- Rate limit multiplied by demand
- Aggregated across a day / week
- Cannot be manually raised
- The "how many" dimension
8 best practices for crawl budget
- Improve server response time. Sub-200ms TTFB lifts crawl rate limits automatically. Slow servers get throttled.
- Return 410 Gone for permanently removed pages. 410 is a stronger signal than 404 and gets removed from Google's crawl queue faster.
- Keep your XML sitemap clean. Only include canonical, indexable URLs. Every dead or redirected URL in a sitemap is a wasted crawl.
- Block low-value URLs in robots.txt. Faceted filters, internal search results, and admin URLs should never see Googlebot.
- Eliminate redirect chains. Chains multiply crawl cost. Point 301s directly to the final destination.
- Fix internal links pointing to redirects or 404s. Broken internal links waste every future crawl of that page.
- Consolidate thin content. Merge low-traffic tag pages, thin category pages, and outdated posts.
- Use the Crawl Stats report. Google Search Console's Crawl Stats shows requests per day, response times, and file-type breakdowns.
Adding noindex does not stop crawling. Googlebot still has to fetch the page to see the tag. Only after repeated crawls confirm noindex does the URL move to a low-priority queue. To stop crawling entirely, block the URL in robots.txt.
Common crawl budget mistakes to avoid
- Blocking already-indexed URLs in robots.txt. Google cannot see the noindex tag on a blocked URL and may keep it indexed.
- Ignoring log files. Search Console shows summaries. Log files show exactly which URLs Googlebot hits and at what frequency.
- Assuming small sites need optimisation. Below 1,000 URLs it does not matter.
- Blocking CSS or JS. Googlebot needs these to render pages. Block only what is truly non-indexable.
- Publishing endlessly without pruning. An 8-year archive with 60% zero-traffic pages drags crawl efficiency down.
Frequently asked questions
Crawl budget is the total number of URLs Googlebot will fetch from your site during a given period. It equals crawl rate limit (server capacity Googlebot can safely use) multiplied by crawl demand (how much Google wants to recrawl based on freshness and popularity).
Google says most sites under 1,000 URLs never need to worry about crawl budget. Optimization starts to matter at 10,000+ indexable URLs, or if a smaller site publishes 30+ new pages per month and is seeing indexing delays.
Google Search Console's Crawl Stats report shows requests per day, response times, and file-type breakdowns. Combine that with server log-file analysis to see exactly which URLs Googlebot is spending budget on.
Faceted navigation, session-ID URLs, infinite calendars, redirect chains, broken links, and duplicate parameter variants. Each wastes fetches Googlebot could have spent on important pages.
Eventually. Googlebot still has to fetch a page to see the noindex tag, so the first crawls consume budget. Once Google confirms noindex, it visits the URL less often. Block in robots.txt if you never want it crawled.
How theStacc helps
theStacc's technical audit runs a full crawl-budget analysis: which URL patterns are wasting fetches, where redirect chains leak equity, and which parameter families should be blocked. Every finding is ranked by projected indexing lift so you know what to fix first - and what to leave alone.
Related glossary terms
Sources
