Crawl rate is the number of requests per second that Googlebot makes to a site during crawling. Google adjusts it automatically based on two signals: crawl capacity (your server's ability to respond quickly - sub-200ms lifts the ceiling, 500ms+ triggers throttling) and crawl demand (how fresh and valuable Google considers your content).
Crawl rate is the speed dimension of Google's relationship with your site. It is not the same as crawl budget - budget is the total, rate is the throughput. Lift crawl rate and every downstream metric (indexing speed, freshness detection, rankings on new content) improves.
What is crawl rate?
Crawl rate is a technical SEO metric that describes how rapidly Googlebot fetches URLs from your site. Google measures it in parallel connections per second. A crawl rate of 5 requests/second means Googlebot is fetching five URLs in parallel at that moment.
Crawl rate is dynamic. It changes minute-to-minute based on how your server responds. When your responses are fast, Googlebot opens more connections. When responses slow or return errors, it backs off to avoid hurting real-user performance.
In January 2024 Google removed the manual crawl-rate limit setting from Search Console. Google now sets crawl rate entirely automatically. Your only levers are server performance, content freshness, and site authority.
How Google determines crawl rate
Two signals combine to set the rate:
1. Crawl capacity limit (server-driven)
Googlebot constantly measures how your server responds:
- Under 200ms average response: Google opens more parallel connections. Crawl rate rises.
- 200-500ms: Steady state. Rate remains where it is.
- Above 500ms or spike of 5xx errors: Google throttles down to protect your infrastructure. Rate falls.
2. Crawl demand (Google-driven)
Even if your server is instant, Googlebot only crawls if it wants to. Demand rises with:
- New content published frequently
- Pages with strong inbound-link profiles
- URLs that change often (blog, news, product pricing)
- High organic-traffic pages
Server responds 120ms avg → Googlebot opens more connections
Rate climbs 5/sec → 18/sec
# If server slows to 700ms
Googlebot detects strain → Throttles back
Rate drops 18/sec → 3/sec
Why crawl rate matters
- Indexing speed. Higher crawl rate means new URLs get discovered and indexed faster - hours instead of days on well-optimized sites.
- Freshness detection. Updates to existing pages (prices, publish dates, content revisions) get recognized sooner.
- Crawl budget utilization. Rate is the throughput inside your crawl budget. High rate lets Google spend its budget faster and see more pages.
- Ranking recovery. After site changes (migrations, redesigns), a high crawl rate speeds up re-evaluation.
Factors that affect crawl rate
| Factor | Impact on crawl rate | Optimization |
|---|---|---|
| Server response time (TTFB) | Direct - fastest lever | CDN, caching, faster hosting |
| Site size and depth | Deeper sites need higher rates | Flat architecture, internal linking |
| Content freshness | Fresh content raises demand | Regular publishing cadence |
| Site authority / backlinks | High-authority sites earn more demand | Link-building, PR |
| Redirect chains | Slows effective rate | Point 301s directly to final URL |
| Crawl errors (5xx) | Triggers throttling | Monitor Crawl Stats weekly |
| Robots.txt directives | Blocks reduce wasted requests | Block low-value URL patterns |
Real crawl rate examples
1. E-commerce site optimizing TTFB
An online retailer's server response time was 1,200ms - Googlebot averaged 200 requests/day. New product pages took 2 weeks to index. After moving to a CDN and enabling edge caching, TTFB dropped to 180ms. Crawl rate climbed to 2,000+ requests/day. Product indexing time fell to 48 hours.
2. Content publisher with steady cadence
A blog publishes 30 articles per month. Average server response is 150ms. Google runs a steady 500 requests/day and indexes new posts within 24-48 hours. No optimization needed - the pattern is healthy.
3. Site suffering 5xx spike after deploy
A SaaS deploys a broken template that returns 502 on 15% of pages. Googlebot immediately throttles crawl rate 60% and does not recover for 5 days after the fix. The team missed a full week of indexing on new content.
Crawl rate vs crawl budget vs crawl demand
Crawl rate
- Requests per second (throughput)
- Set by server performance
- Changes minute-to-minute
- The "how fast" dimension
- Lever: TTFB
Crawl budget
- Total URLs fetched per period
- Rate limit multiplied by demand
- Aggregated across a day / week
- The "how many" dimension
- Lever: clean site architecture
7 best practices to improve crawl rate
- Optimize server response time. Sub-200ms TTFB is the single biggest lever for crawl rate. CDN + edge caching + database indexes.
- Fix 5xx errors immediately. Server errors trigger throttling. Set alerts to catch spikes within minutes.
- Publish on a regular cadence. A predictable publishing rhythm signals Google to visit more often.
- Submit an updated XML sitemap. Include lastmod timestamps so Google prioritizes updated URLs.
- Eliminate redirect chains. Every hop slows effective crawl rate.
- Use HTTP/2 or HTTP/3. Multiplexed connections let Googlebot pull more resources per connection.
- Monitor Crawl Stats weekly. A downward trend often precedes a bigger technical problem.
Cloudflare, AWS WAF, and similar security layers often flag Googlebot as a suspicious bot and throttle or block it. Verify Googlebot's IP using reverse DNS and whitelist it. A CDN block can silently halve your crawl rate for weeks.
Common crawl rate mistakes to avoid
- Assuming higher crawl rate is always better. If your server cannot serve it, Google will throttle - the lift is not free.
- Ignoring TTFB. Time-to-first-byte is the metric Googlebot actually measures - not page load time.
- Confusing crawl rate with crawl budget. Rate is speed, budget is volume.
- Blocking Googlebot in security layers. Verify with reverse DNS lookup before adding rules.
- Reacting to daily noise. Crawl rate fluctuates naturally. Judge trends on 4-week rolling averages.
Frequently asked questions
Crawl rate is the number of requests per second that Googlebot makes to your site during crawling. It measures how quickly Googlebot fetches pages. Google adjusts crawl rate automatically based on server response time and crawl demand.
Google uses two signals. Crawl capacity - your server's ability to serve requests without slowing down (sub-200ms responses raise the ceiling, 500ms+ trigger throttling). Crawl demand - how fresh and important Google considers your content.
Google removed the manual crawl-rate control from Search Console in 2024. You now influence crawl rate indirectly by improving server response time, fixing errors, publishing frequently, and earning inbound links to signal higher demand.
Under 200ms is ideal - Googlebot will open more parallel connections. Responses between 200-500ms are acceptable. Above 500ms Google begins throttling to protect your server, which slows indexing.
Open Google Search Console, go to Settings and click Crawl Stats. The report shows total requests, average response time, and file-type breakdown for the last 90 days. Server log files give real-time detail.
How theStacc helps
theStacc's technical audit measures your live TTFB against the sub-200ms threshold Googlebot uses, benchmarks your crawl rate against your category, and ranks fixes by projected indexing lift. Faster crawl means faster ranking - and faster ranking on fresh content compounds.
Related glossary terms
Sources
