A crawl trap is a structural issue on a website that causes crawlers to get stuck fetching infinite or near-infinite URL variations. Common types include faceted navigation with millions of filter combinations, calendar widgets with unlimited future dates, session-ID URLs, and redirect loops. Traps waste crawl budget on valueless URLs and delay indexing of pages that actually matter.
Every technical SEO who has run a 12,000-URL site through Screaming Frog and watched it climb past 3 million knows the feeling. Crawl traps are the reason. Fix them and indexing accelerates, crawl budget frees up, and Google finally spends time on pages that rank.
What is a crawl trap?
A crawl trap - sometimes called a "spider trap" or "infinite space" (Google's original term) - is any structural pattern on a site that generates an effectively unlimited number of URLs. Every URL variation is a separate fetch for Googlebot. When the pattern has no natural stop, crawlers get "trapped" fetching pages that have no unique value.
Traps rarely block a site from indexing entirely. They quietly eat crawl budget, delay indexing of new content, and sometimes flood Google's index with near-duplicates that dilute topical authority.
Google's Search Central docs formally warn against creating "infinite spaces". Site owners are advised to use robots.txt, canonical tags, or nofollow to prevent crawlers from following URL patterns that have no natural termination.
6 common types of crawl traps
| Trap type | How it forms | Fix |
|---|---|---|
| Faceted navigation | Filter combos: color, size, brand, price | Robots block + canonical to core category |
| Infinite calendars | Widget generates every future date URL | Nofollow future links, robots block |
| Session IDs in URLs | Every visitor spawns a new URL | Store session in cookie, canonical tag |
| Sorting parameters | ?sort=asc, ?sort=price, ?sort=name... | Canonical to base URL |
| Redirect loops | A → B → A | Fix redirect rules to terminate |
| Orphan pagination | /page=999 returns empty page 200 OK | Return 404 past the last real page |
Why crawl traps matter
- Crawl budget drain. A trap with 500,000 variants easily consumes 80% of Googlebot's fetches, starving important pages.
- Indexing delays. New content sits unindexed for weeks because Googlebot never reaches it.
- Index bloat. Even without ranking, indexed variants dilute topical authority. Google sees hundreds of near-duplicates instead of one canonical page.
- Server load. Every trapped fetch is a real database query and page render - real infrastructure cost.
- Analytics distortion. Traffic reports show hundreds of "pages" that are actually the same content under different URLs.
How to detect a crawl trap
Every trap detection uses three tools together.
1. Site crawler
Run Screaming Frog or Sitebulb with a 500,000-URL limit. If the crawl count balloons past your real indexable page count, you have a trap. Look at URL patterns clustered by parameter or path segment.
2. Google Search Console Crawl Stats
The Crawl Stats report groups requests by URL patterns. A pattern responsible for 40% of daily crawls that does not contain unique content is almost always a trap.
3. Server log files
The definitive view. Log files show every Googlebot hit and the URL it fetched. Group by URL template - if one template accounts for 50%+ of Googlebot's daily requests, investigate it.
/shoes/?color=red
/shoes/?color=red&size=10
/shoes/?color=red&size=10&brand=nike
/shoes/?color=red&size=10&brand=nike&price=100-200
# 8 filters × 6 values each = 262,144 combinations
# Fix: block the parameter set in robots.txt
Disallow: /shoes/?*color=
Real crawl trap examples
1. E-commerce with facet explosion
A fashion retailer has 18,000 products but exposes 2.4M crawlable URLs due to 12 filter dimensions. Google spends 91% of its crawl budget on filter combinations. Blocking six filters (color, size, brand, material, price, availability) in robots.txt drops crawlable URLs to 42,000. Product indexing time falls from 3 weeks to 2 days.
2. Calendar widget on a booking site
A venue booking site includes a "next month" link on its calendar that has no stopping condition. Googlebot follows it into the year 2087. The site logs show 340,000 date-URL fetches in 90 days - all with identical content. Adding rel="nofollow" to the "next" link stops the loop.
3. Session IDs in URLs
An older PHP app appends ?PHPSESSID=abc123... to every URL for anonymous visitors. Googlebot sees a new URL on every crawl. Moving session state into cookies and adding a canonical tag stops the trap - crawl volume drops 78% within two weeks.
Crawl trap vs redirect loop - what is the difference
Crawl trap
- Infinite unique URLs generated by structure
- Every URL returns HTTP 200
- Wastes crawl budget on duplicates
- Fixed with robots, canonical, nofollow
- Example: facet filters, calendars
Redirect loop
- Two or more URLs that redirect back to each other
- Never returns a 200 - endless 3xx cycles
- Googlebot gives up after ~5 hops
- Fixed with redirect rule cleanup
- Example: A → B → A
7 best practices to prevent crawl traps
- Audit URLs quarterly. Run a full crawler on a fresh 500K limit at least every quarter to catch new traps before they compound.
- Block filter parameters in robots.txt. Faceted URLs almost never need to be indexed - blocking is safer than canonicals for traps.
- Cap pagination. Return 404 (not 200) once a paginated series exceeds the actual number of items.
- Nofollow calendar widgets and "load more" links. These are user features, not indexing targets.
- Move session IDs into cookies. Never let a session token appear in a crawlable URL.
- Canonical every filter and sort variant to the base URL. Even blocked URLs benefit from a canonical fallback.
- Monitor crawl volume in Search Console. A sudden 3x spike in daily crawls with no publishing spike usually signals a new trap.
Canonical tags are a hint, not a directive. Google can and does ignore canonicals when it disagrees with the choice. For a genuine crawl trap, block the pattern in robots.txt first - then add canonical as a safety net for URLs already indexed.
Common crawl trap mistakes to avoid
- Assuming canonicals are enough. They save index bloat but not crawl budget - Googlebot still fetches every variant.
- Blocking a trap after millions of variants are indexed. Once indexed, blocking in robots.txt prevents Google from seeing the noindex tag needed to remove them.
- Nofollowing internal links to fix a trap. Google largely ignores nofollow for crawl direction. Use robots.txt.
- Ignoring log files. Search Console shows aggregates - logs show the pattern in real time.
- Fixing structure without validating. After a fix, re-run the crawler and confirm the URL count drops. Trust but verify.
Frequently asked questions
A crawl trap is a structural issue on a website that causes crawlers to get stuck fetching infinite or near-infinite URL variations. Common examples include faceted navigation with millions of filter combinations, calendar widgets with unlimited future dates, session-ID URLs, and redirect loops.
Run a Screaming Frog or Sitebulb crawl - if the URL count balloons past your indexable page count, you have a trap. Server log files show the exact pattern Googlebot is chasing. Google Search Console's Crawl Stats also flags parameter explosions.
The top six are: faceted navigation (filters), infinite calendars (future dates), session ID URLs, redirect loops, sorting parameter combinations, and orphaned pagination that generates empty pages beyond the last real one.
Block trap patterns in robots.txt for whole URL families, add rel canonical to point variants at a canonical URL, use nofollow on links that create the trap, and return 410 Gone for URLs that should never have existed.
Indirectly, yes. They drain crawl budget from valuable pages, delay indexing of new content by weeks, and can dilute topical authority when Google indexes hundreds of duplicate variants of the same page.
How theStacc helps
theStacc's technical audit runs a full URL-explosion analysis against your site. Every trap pattern - faceted, calendar, session, pagination - is mapped, quantified, and ranked by projected crawl-budget recovery. Fix the biggest first and watch indexing time collapse.
Related glossary terms
Sources
