What is crawl budget?
Crawl budget describes the URLs Google can and wants to crawl on a site. Its two main elements are crawl capacity and crawl demand. Google’s crawl-budget guidance describes their interaction; it does not provide a multiplication formula that predicts daily requests.
A fetch is also different from indexing. An accessible page can be crawled without becoming a search result. Before diagnosing a missing page as a budget problem, establish whether Google has already fetched it and whether an indexing rule or content-selection decision is involved.
When to investigate crawl allocation
Google frames its advanced guidance around large or frequently changing sites and substantial discovery backlogs. Its size examples are approximate, not a boundary that makes every smaller site immune to crawl problems. If important pages are discovered promptly, start with the page’s actual indexing state rather than assuming a larger crawl allowance is needed.
Write down the symptom. “New product pages have not been fetched” and “fetched product pages are not indexed” need different investigations. The Page indexing report distinguishes discovered, crawled and exclusion states. It also explains that an indexed observation and a live inspection can differ.
Capacity and demand explain different constraints
Crawl capacity
Capacity concerns what Google can fetch without overwhelming the serving system. Slow responses, server errors and overload signals can reduce crawling. Investigate the actual request and response pattern rather than treating a single response-time target as a switch that raises a quota.
Crawl demand
Demand concerns which URLs Google wants to revisit or discover. Its documentation discusses known URL inventory, popularity and staleness. Unnecessary duplicate variations can add requests without adding useful content.
A healthier server does not make unlimited crawling desirable. Likewise, blocking a URL family does not guarantee that the remaining requests will immediately shift to another family. Google’s guidance specifically cautions against using temporary robots.txt blocks as a budget-reallocation trick.
Investigate URL families before choosing a fix
| Observed pattern | What to inspect | Decision to make |
|---|---|---|
| Many filter combinations | Actual URLs, parameter ordering and useful filtered landing pages. | Which collections need discovery, and which combinations need no crawling. |
| Repeated redirects | The complete chain and links to its old addresses. | Whether links and redirects can point directly to the intended destination. |
| Missing content served as success | The response status and body for representative URLs. | Whether a real missing-page status or restored content is appropriate. |
| Recurring serving errors | Response types, affected templates and host availability. | Which infrastructure or application fault needs repair. |
For filter URLs, Google’s faceted-navigation guidance distinguishes preventing unwanted crawling from supporting useful filtered pages. Review the visitor’s task before applying one rule to the whole family.
For missing pages, Google’s HTTP status documentation treats both 404 and 410 within its non-overload client-error handling. It does not support promising that 410 clears every URL faster. A success response also does not guarantee indexing.
Illustrative diagnostic scenarios
A retailer with filter variations
The retailer sees repeated requests for combinations of colour and size. First, identify whether any combination is a useful search landing page. Then compare the URL inventory with the navigation and serving behavior. Test proposed restrictions against representative useful pages as well as unwanted variants.
Record what changed and observe the resulting requests. This scenario does not assign an invented percentage of wasted crawling or promise a shorter indexing time. The evidence has to come from the retailer’s own authorised reports and logs.
A publisher with an old archive
Older pages are not automatically unwanted. Review whether an archive contains useful reference material, genuine duplicates or removed content. Decide on maintenance, consolidation or an appropriate missing-page response from that review. Low recent traffic alone does not prove that deleting an article improves crawl allocation.
A small service website
If a service page is absent from search, inspect that URL first. Check that its response, crawl access and indexing instructions match the intended behavior. A site-wide crawl-budget project is a larger intervention than correcting one accidental noindex header.
Crawl budget vs crawl rate
Crawl rate describes request activity over time. Crawl budget describes the allocation of capacity and demand across URLs. Neither is a count of indexed pages or a score for the quality of a website.
| Observation | What it describes | What it does not establish |
|---|---|---|
| Crawl requests over a period | Google’s observed request activity. | The number of distinct pages indexed. |
| Average response time | The response-time summary for fetched resources. | A universal threshold for increasing crawl capacity. |
| Discovered or crawled indexing state | A particular stage or outcome for a URL. | A diagnosis of the site’s entire crawl budget. |
The Crawl Stats report includes repeat requests, resources and separate redirect requests. Its URL examples are samples, not a complete log. Differences from server logs can therefore require a scope check before either set is treated as wrong.
Build a useful crawl investigation
- Define the symptom and origin. Identify the important URLs and the host affected. Keep protocol, time period and deployment changes with the notes.
- Collect the available observations. Compare Crawl Stats, indexing details and authorised request logs. Missing data stays unknown.
- Group requests by a meaningful URL pattern. Separate product pages, filters, assets, redirects and failures. Do not classify every parameter as waste.
- Check serving behavior. Investigate repeated errors or slow responses before adding crawl restrictions.
- Choose a control for the actual goal. Duplicate selection, crawl prevention and search exclusion are separate tasks.
- Review after the change. Compare the same scope and keep other migrations or publication changes in view before attributing an outcome.
Search Console’s Crawl Stats documentation states that the report is available for eligible root-level properties. Access to one path-specific property does not establish that you have this report or complete host-level evidence.
Do not treat noindex as crawl prevention
Google must fetch a resource to read noindex. Its crawl-budget guidance therefore discourages noindex as the method for avoiding unwanted fetches. If the goal is removal of an indexed page, keep the exclusion instruction accessible instead. The noindex entry explains that separate task.
A new robots.txt block can stop Google reading a page’s exclusion rule. Do not record “removed from search” merely because a local crawl test says access is blocked.
Do not manufacture overload responses to improve a crawl chart. Google treats 429 and server errors as overload or failure signals. Address a real serving problem with the infrastructure owner and inspect its effects; persistently broken responses are not an optimization result.
Questions about crawl allocation
Can I ask Google to increase my crawl rate?
The Crawl Stats documentation says you cannot tell Google to raise its crawl rate. Investigate unnecessary URL inventory and serving problems, and use current guidance for genuinely large or frequently changing sites. A requested recrawl is not a guaranteed budget increase.
Do more crawl requests mean the change worked?
Not necessarily. More requests could involve useful discovery, repeated redirects, resource fetches or failures. Compare the request types with the original problem and the relevant URL outcomes before calling the change successful.
What to bring to an implementation discussion
If you want help choosing a next step, bring representative URLs, the observed indexing states, crawl-response details and any recent migration or template changes. Confirm the work to be investigated and who can change the site. Do not assume a software signup includes log analysis, server changes or a predicted indexing gain.
You can sign up for free to explore theStacc software or book a demo to discuss your setup. Confirm any managed technical work separately.
Related glossary terms
- Robots.txt: choose and test crawl restrictions.
- Noindex: handle search exclusion without blocking the instruction.
- Canonical URL: distinguish duplicate preferences from crawling controls.
- Soft 404: investigate missing content served with a success response.
Sources and editorial note
Official Google documentation is linked beside the relevant explanations and was checked on October 12, 2026. AI assisted this revision. Scenarios are illustrative; no private Search Console data, production logs, customer improvement benchmark or named expert approval were tested. The existing publication date is preserved, with historical provenance still unresolved.
Explore this topic: Crawl Budget Optimization (2026): Index More Pages, Faster.