Robots.txt is a plain-text file placed at the root of a domain (yoursite.com/robots.txt) that tells search engine crawlers which URLs they are allowed or disallowed from accessing. Google, Bing, and other major crawlers check this file before requesting any page. It controls crawl budget allocation but does not prevent pages from appearing in Google's index if they are linked from other sites.
According to Google's documentation: "Googlebot checks robots.txt before making any request to your server." That single file gates every crawl decision. Get it wrong and you hand Google a list of pages not to crawl — sometimes including your most important content.
What is robots.txt?
Robots.txt implements the Robots Exclusion Protocol, a de facto web standard since 1994. It's a plain-text file — no HTML, no markup — containing a series of rules that cooperative web crawlers read before accessing any page on your domain.
The file uses three core components:
- User-agent: Specifies which crawler the rule targets.
User-agent: *applies to all bots.User-agent: Googlebottargets only Google's crawler. - Disallow: Blocks access to specific paths.
Disallow: /admin/prevents crawlers from requesting any URL starting with /admin/. A blankDisallow:means allow everything. - Allow: Overrides a broader Disallow for specific sub-paths. Useful when you block a directory but need one file within it to be crawlable.
If Google receives a 200 response for robots.txt, it follows the rules. A 404 (file not found) means no restrictions — everything is crawlable. A 5xx server error causes Google to temporarily limit crawling of the whole site to protect a struggling server.
Why robots.txt matters for SEO
On small sites with 50 pages, robots.txt is low-stakes. On sites with thousands of pages, it's a critical traffic allocation tool. Four reasons it matters:
- Crawl budget protection. Google allocates a crawl budget to every site — a rate-limited number of pages it will crawl per day. Blocking low-value pages (admin panels, staging areas, duplicate filter variations) keeps that budget focused on the pages you want indexed.
- Preventing sensitive area indexing. Internal search results, login pages, cart pages, and staging environments don't belong in Google's index. Robots.txt keeps crawlers away from these paths before they even land on the page.
- Faster discovery of new content. When crawlers stop wasting requests on junk pages, they find new blog posts and product pages faster. A site that publishes 30 new articles a month benefits from every crawl being spent on real content.
- Server load management. Blocking unnecessary crawling reduces resource consumption. For shared hosting or small VPS deployments, this can meaningfully reduce server strain during crawl bursts.
How robots.txt directives work
A robots.txt file is read from top to bottom. The first matching rule wins. Here's what each directive does in practice:
User-agent: *
Disallow:
# Block all bots from the entire site (dangerous — avoid in production)
User-agent: *
Disallow: /
# Block Googlebot from admin and staging only
User-agent: Googlebot
Disallow: /admin/
Disallow: /staging/
# Point to XML sitemap
Sitemap: https://example.com/sitemap.xml
Types of robots.txt directives
| Directive | Function | Example |
|---|---|---|
| Disallow | Block crawler from path | Disallow: /wp-admin/ |
| Allow | Override a Disallow for sub-path | Allow: /wp-admin/admin-ajax.php |
| User-agent | Target specific crawler | User-agent: Googlebot |
| Sitemap | Point to XML sitemap | Sitemap: https://site.com/sitemap.xml |
| Crawl-delay | Rate-limit crawler requests | Crawl-delay: 10 |
Real robots.txt examples
1. WordPress site — blocking admin and filter URLs
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /cart/
Disallow: /demo/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap.xml
2. E-commerce site — blocking filter parameter URLs
An online retailer with 50 products but 3,000 filter-generated URL combinations uses one Disallow line to save crawl budget:
Disallow: /products?filter=
Disallow: /products?sort=
Disallow: /products?color=
Sitemap: https://store.example.com/sitemap.xml
3. The production mistake that cost 3 weeks of indexing
A marketing agency left Disallow: / in their production robots.txt after copying from a staging environment. The result: zero new pages indexed for 3 weeks and a steep traffic drop before the error was caught via Search Console. One line, massive consequence.
Robots.txt vs meta robots tag — which to use
| Aspect | Robots.txt | Meta Robots Tag |
|---|---|---|
| Location | Root directory file | HTML <head> of individual pages |
| Timing | Before crawling | After crawling (page must be fetched) |
| Scope | Entire directories or paths | Individual pages |
| Prevents indexing? | No — only prevents crawling | Yes — noindex removes from search |
| Best for | Blocking site sections at scale | Removing specific pages from search |
If you block a page in robots.txt, Google cannot crawl it and therefore cannot read the <meta name="robots" content="noindex"> tag. The page may still appear in results if external sites link to it. To remove a page from Google, you need it to be crawlable so Google can see the noindex directive.
5 robots.txt best practices
- Always include a Sitemap directive. Point crawlers directly to your XML sitemap so they discover all indexable URLs systematically rather than relying on link-following alone.
- Never block CSS or JavaScript files. Google needs to render your pages to understand content. Blocking JS or CSS causes Google to see a stripped-down, partially-broken version of your page — hurting quality assessment.
- Test every change before deploying. Use Google Search Console's robots.txt tester or the URL Inspection tool to verify your rules work as intended. A single typo can break a critical rule.
- Review quarterly as your site grows. New directories, new CMS plugins, and new URL patterns can create unblocked junk paths. Robots.txt should evolve with the site.
- Use Disallow for directories, not individual files. Block entire paths (
/admin/) rather than individual filenames. Directories are stable; filenames change. Path-level rules are more durable.
Common robots.txt mistakes to avoid
- Deploying staging robots.txt to production. The most common catastrophic error. Always check that production never has
Disallow: /. - Blocking CSS and JavaScript. Prevents Google from rendering pages correctly. Causes Googlebot to evaluate unstyled, broken page layouts.
- Blocking pages to remove them from Google. Blocked pages can still be indexed if linked externally. Use
noindexinstead for removal. - Forgetting subdomain isolation. A robots.txt file at example.com does not apply to blog.example.com. Each subdomain needs its own file.
- No sitemap reference. Missing the
Sitemap:directive means crawlers must rely entirely on link discovery to find your pages. - Relying on robots.txt for security. It's a courtesy protocol, not a security mechanism. Malicious bots ignore it. Don't put sensitive endpoints behind robots.txt alone.
Frequently asked questions
Not directly. Robots.txt prevents crawling, not indexing. If other sites link to a blocked page, Google may still show it in results without a description snippet. Use a noindex meta tag to fully remove a page from search results.
Place it at your domain root: https://yoursite.com/robots.txt. Subdirectory placement doesn't work. Each subdomain needs its own robots.txt file — a rule at example.com/robots.txt does not apply to blog.example.com.
Indirectly, yes. Blocking low-value pages preserves crawl budget for important content. On large sites this means faster discovery and indexing of new pages, potentially speeding up ranking improvements for the pages that matter.
Legitimate search engine bots (Googlebot, Bingbot) respect robots.txt. Malicious bots and scrapers typically ignore it. Don't rely on robots.txt for security — it's a guideline for cooperative crawlers, not a firewall.
If Google gets a 5xx server error when fetching robots.txt, it temporarily limits crawling of the entire site to avoid overloading a struggling server. A 404 (file not found) is treated as no restrictions — all pages are crawlable.
Related glossary terms
Sources
