What is robots.txt?

Robots.txt is a plain-text file that gives cooperating crawlers rules about URLs they may request. It implements the Robots Exclusion Protocol, standardised in RFC 9309. Those rules are crawl instructions, not permissions that protect private data.

For a website, put the file at /robots.txt on the origin you want to control. Google’s robots.txt specification limits its scope to that protocol, host and port. A file on https://example.com/ does not govern a different subdomain or the HTTP version.

Decide whether you want to prevent a fetch or prevent a search result. Robots.txt addresses the former. A URL blocked from crawling can still appear in Google Search without its fetched content. Use a readable noindex instruction when search exclusion is the goal.

Why crawl rules deserve careful review

A broad rule can stop Googlebot fetching important pages or resources. That matters on a small site as well as a large one. Review an important service page’s crawl access even if the rest of the site has only a few pages.

Crawl-volume work is a different question. Google’s crawl-budget guidance concerns very large or rapidly changing sites and sites with substantial discovery backlogs. It describes crawl capacity and demand, not a universal daily allowance or a formula that predicts indexing. Changing a rule is not a guarantee of faster discovery or better rankings.

Before blocking a URL family, establish what visitors and searchers need from it. Review the content, navigation and existing indexing state. Removing access to a useful collection just because its address contains a parameter is a poor substitute for that decision.

How Google chooses the applicable rules

Google selects the most specific matching user-agent group. Multiple groups for that same specific agent are combined; the wildcard group is not added to a matching specific group. Within the selected group, the most specific matching path wins. An equally specific allow/disallow conflict is resolved in favour of allowing the fetch. File order does not create a “first matching rule wins” policy.

Group merging and the longest matching rule are also described in the protocol’s syntax and matching rules. The distinction matters when you add a Googlebot-specific group to a file that already has general rules.

Fields and patterns you will encounter

Check support for the crawler you actually want to control
FieldPurposeImportant limit
User-agentIdentify the crawler group.A wildcard group is a fallback, not an extra layer on every specific group.
DisallowSpecify a path the selected crawler must not fetch.An empty value creates no restriction.
AllowSpecify an allowed path.Compare specificity with matching disallow rules.
SitemapProvide an absolute sitemap URL.It does not override crawl restrictions.
Crawl-delayA crawler-specific extension found in some files.Google does not support this field.

Google supports * for a sequence of characters and $ for the end of the URL. Path matching is case-sensitive. Check complete URLs, including relevant query strings, rather than assuming two visually similar addresses match the same rule.

Illustrative robots.txt examples

A narrow restriction with a public exception

The example below blocks an illustrative directory while allowing a particular public path within it. It is not a recommended production configuration for every website:

User-agent: *
Disallow: /internal/
Allow: /internal/public-guide/

Sitemap: https://example.com/sitemap.xml
Expected matching outcomes for the example’s wildcard group
URL pathFetch allowed?Reason
/internal/report/No.Matches the disallow path without the exception.
/internal/public-guide/start/Yes.The longer allow path is more specific.
/Internal/report/Yes, under these rules.The differently capitalised path does not match.

These outcomes concern rule matching only. They do not predict whether the URL exists, which response it returns, or whether it will be indexed. The public exception is not a way to make the blocked directory private.

Filter URLs require a page-purpose decision

Google’s faceted-navigation guidance recommends preventing unnecessary crawling when filter combinations need not be indexed. Its example patterns account for parameters appearing after other query values. A narrow prefix such as /products?color= does not cover every possible ordering of a colour parameter.

Take representative URLs from your own navigation and logs, including reordered parameters. Check any useful filtered landing pages separately. If an indexed URL must be removed, a new robots.txt block will not let Google discover a noindex instruction on it.

The meta robots tag and X-Robots-Tag header are read with the fetched resource. Their rules can address indexing and search presentation. Robots.txt is consulted for crawl access. They operate at different stages and cannot substitute for each other.

A blocked fetch hides page-level instructions

Google’s noindex guidance requires the instruction to be accessible. Do not block a page in robots.txt and expect Google to read a new meta tag or response header through a sitemap or backlink.

For genuinely private content, use authentication or another effective access restriction. RFC 9309 explicitly warns that listing paths in robots.txt makes them discoverable and that the protocol is not an access-authorisation mechanism.

Review a change against real URLs

  1. State the intended restriction. Identify the URL family and crawler. Explain why fetching those URLs is unnecessary.
  2. Read the whole applicable group. Check existing exceptions and whether a specific user agent replaces the wildcard fallback.
  3. Test representative paths. Include a blocked example, a permitted exception, and a page that must stay crawlable. Add case and query-order variants where relevant.
  4. Protect rendering requirements. Check that useful page resources are accessible rather than blocking an entire asset directory without inspection.
  5. Inspect the deployed file. Confirm its origin, status, content and intended rules. A local configuration file is not proof of the public response.
  6. Record the observation. Save the change, its owner and the URLs tested. Review it again when templates, parameter handling or origins change.

The current Search Console robots.txt report shows fetched versions, times and parsing issues for eligible properties. It can request a recrawl after an important correction. Such a request does not guarantee that affected pages are immediately recrawled.

Fetch errors and misleading assumptions

A missing robots.txt and an unreachable robots.txt are different conditions. Google treats most client errors, including a missing-file response, as having no crawl restrictions, but excludes 429 from that rule. Server and network errors can suspend crawling and later involve cached rules. See the documented error handling when diagnosing an incident.

Do not turn that sequence into a fixed recovery promise. Restore the actual file response and inspect what Google last fetched. A successful change in your hosting dashboard does not establish that Google has already seen it.

Other mistakes include assuming Crawl-delay limits Googlebot, treating a sitemap as a permission override, and keeping a development-wide Disallow: / on the production origin. Review the emitted file before launch rather than relying on the name of a setting.

Questions about robots.txt

Can I block one file instead of a directory?

Yes. Rules match URL paths, so a rule can target a file as well as a directory. Check whether the pattern also matches extensions, suffixes or query variants you want to allow. There is no general requirement to block only directories.

Is a sitemap line required?

A sitemap line is a supported discovery aid, not a prerequisite for a valid robots.txt file. Include the correct absolute address when it helps your setup. Missing that line does not prove that crawlers cannot discover any pages through other means.

  • Noindex: exclude a resource after the crawler can read its instruction.
  • Meta robots tag: understand page-level instructions separately from crawl rules.
  • Canonical URL: distinguish a duplicate preference from a crawl restriction.

Sources and editorial note

Google’s implementation documentation and RFC 9309 are linked beside the relevant explanations. Sources were checked on October 12, 2026; the rules and examples were reviewed with AI assistance. Examples are illustrative, not client incidents. No production robots.txt, private Search Console property, crawl-rate improvement or named expert approval was tested. The retained original publication date has unresolved historical provenance.

Explore this topic: Robots.txt Optimization: 8 Steps to a Cleaner Crawl.