Optimize robots.txt by stating which crawler and URL family a rule should restrict, preserving useful page resources, and testing both blocked paths and public exceptions. Inspect the deployed file and Google’s last fetched version after a change. Use separate controls for private access and search exclusion; robots.txt cannot guarantee either.
A broad Disallow rule can affect a useful service page, a rendering resource or an entire origin. The practical work is to understand the current policy, make a narrow change and verify its output. Reducing a crawl-request total is not, by itself, evidence of better indexing.
This eight-step workflow uses Google’s implementation specification, RFC 9309 and current provider documentation. The examples are illustrative configurations, not files to paste into every production site.
Choose the outcome before writing a rule
Robots.txt gives cooperating crawlers rules about URLs they may fetch. It does not authenticate visitors or hide the existence of a path. RFC 9309 explicitly distinguishes the protocol from access authorization; protect private content with effective access controls.
Search exclusion is also a separate task. Google can show a blocked URL without its fetched content, and it cannot read a new noindex instruction behind a crawl block. The noindex entry explains exclusion after fetching; the robots.txt entry explains the protocol’s basic mechanics.
Step 1: Identify the live origin and controlling system
Inspect /robots.txt on the origin you want to control. Google scopes the rules to the protocol, hostname and port. A file on the public HTTPS hostname does not automatically govern another subdomain or the HTTP version.
Save the current text, response status and the CMS, server or deployment rule that supplies it. A virtual CMS file and a static file can be controlled differently. Use your platform’s current documentation to locate the correct setting rather than assume a particular plugin or theme menu.
If no file exists, do not treat that absence as a verified SEO fault. Google’s robots.txt report says a missing file can be fine. A missing-file response differs from a network or server failure that prevents Google obtaining rules. Diagnose the actual response before creating another layer.
Step 2: Read groups and matching rules together
User-agent chooses the applicable group
Google’s group-selection guidance uses the most specific matching user agent. Repeated groups for the same specific agent are combined; a matching specific group does not inherit the wildcard group. Before adding a special bot group, decide which general restrictions need repeating there.
The word “Google” is not one universal crawler token. Check the relevant product and documented token, rather than assume a Googlebot rule controls every Google fetcher. Other providers can have different supported fields or request behavior.
Disallow matches a URL path
A rule such as Disallow: /internal/ restricts the matching path for the selected group. An empty Disallow value creates no restriction. Disallow: / matches the origin’s path space, so inspect production output for development-wide blocks before launch.
Allow is resolved by specificity
Google uses the most specific matching path, rather than the first rule in file order. At equal specificity, the least restrictive rule wins. This example restricts an illustrative directory while permitting a narrower public section:
User-agent: *
Disallow: /internal/
Allow: /internal/public-guide/
Sitemap supplies a discovery reference
A Sitemap line uses an absolute file URL. It does not override crawl rules and is not required to make the file a valid robots.txt policy. Keep references accurate and preserve the surrounding policy when adding one.
Step 3: Review URL families before restricting them
List representative addresses and what each family does for visitors. Google’s crawl-budget guidance focuses on very large or frequently changing sites and substantial discovery backlogs. It cautions against temporary blocks as a way to reallocate crawling to other pages.
Protect administrative access separately
An administrative area or private development environment needs effective access protection. A robots restriction may express an additional crawl preference, but the public file is not a password. Test the production origin independently so a development-wide policy is not carried into customer-facing routes.
Trace filters, search and pagination
Google’s faceted-navigation guidance distinguishes unnecessary filter combinations from filtered collections you want potentially indexed. Check parameter ordering and useful exceptions before restricting the whole family. A parameter is not proof of duplicate content.
Likewise, do not block every pagination URL. Google’s pagination guidance recommends distinct page URLs and crawlable links through the sequence. Determine where actual content ends and avoid generating nonexistent subsequent pages.
A public confirmation page may need an indexing exclusion rather than a fetch restriction. Choose that outcome deliberately; placing its route in robots.txt does not establish that Google has removed it from results.
Step 4: Preserve content and rendering requirements
Identify the resources the public page needs
Check intended public pages and the CSS, JavaScript or other resources needed to understand their content. Google’s JavaScript guidance explains fetching, rendering and indexing. Blocking a necessary resource can change what Google can render; it is not a measured Core Web Vitals penalty for every blocked file.
Compare actual output with expected access
Inspect the current response and relevant rendered content for representative pages. Review the selected robots group and matching rule for their resources. A successful test of one page does not establish that every template or asset remains available.
A Sitemap reference cannot grant permission to a disallowed path. Keep useful public destinations and resources in your test set so a cleanup rule does not unintentionally hide them.
Step 5: Keep sitemap references accurate
Use the actual published sitemap address, which can be a sitemap index. Google supports multiple Sitemap lines and does not tie them to the preceding user-agent group. The line can appear anywhere in the file; putting it last is an organizational choice.
Sitemap: https://example.com/sitemap.xml
Fetch that file and inspect its children rather than assume the reference is correct because it looks conventional. The XML sitemap implementation guide covers format, inventory and submission checks. Neither a reference nor direct submission guarantees that every listed page is fetched or indexed.
Step 6: Choose AI access by provider and purpose
Search crawling, model development and user-triggered retrieval can use separate controls. Review current provider documentation and record the intended use you want to allow or restrict. A token does not prove that every request from the same company has the same purpose.
| Provider control | Documented purpose | Important limit |
|---|---|---|
| OAI-SearchBot | OpenAI search crawling. | It is separate from the training and user-triggered agents. |
| GPTBot | OpenAI model-training crawling preference. | A training preference does not select the search policy. |
| ChatGPT-User | Some user-triggered retrieval. | OpenAI says robots rules may not apply to these user-initiated actions. |
| ClaudeBot, Claude-SearchBot, Claude-User | Anthropic distinguishes training, search and user-directed retrieval. | Review the relevant token instead of treating them as one bot. |
| Google-Extended | Controls documented Gemini training and grounding uses. | No separate HTTP request user agent; it does not control Google Search inclusion or ranking. |
The table follows OpenAI’s crawler documentation, Anthropic’s crawler policy and Google’s product-token documentation. Allowing access does not guarantee citation, traffic or appearance for a query.
Restrict documented training uses
If your policy is to restrict a provider’s supported training crawl, use its documented token and scope. This is a forward-looking provider preference, not proof that you have erased existing model data or blocked every crawler on the internet. Keep the file readable by the provider so it can obtain the policy.
Separate a search policy from a training policy
OpenAI documents independent OAI-SearchBot and GPTBot settings. The example below retains an illustrative internal-directory restriction in both the wildcard and specific search group, while expressing a GPTBot restriction:
User-agent: *
Disallow: /internal/
User-agent: OAI-SearchBot
Disallow: /internal/
User-agent: GPTBot
Disallow: /
The repeated directory rule matters: a specific group does not automatically add the wildcard group’s restrictions under Google’s documented selection rules. Verify each target provider’s behavior and any firewall restrictions separately; the example does not promise effective access across your deployment.
Choose the fallback policy deliberately
A wildcard group is the fallback for applicable crawlers without a more specific group. A permissive fallback is a policy choice, not a way to force indexing or AI visibility. Keep genuine private access protected regardless of the crawl preference.
Step 7: Test representative paths and the deployed file
Read the current robots.txt report
Google’s robots.txt report shows fetched files, versions and parsing issues for eligible domain-level or root URL-prefix properties. It can request a recrawl after a critical correction. That request does not guarantee immediate crawling of newly unblocked pages.
The report’s file history is not a complete simulator for every page and bot. Compare the last fetched version with the public file, and use relevant URL-level evidence for an affected page. Do not rely on an old instruction to paste policy text into the former Search Console tester.
Use a matching tool for local rules
Google publishes a robots.txt parser and matcher for local URL and user-agent tests. Matching a local file does not certify the deployed response, caching, authentication or every product-specific crawler behavior. Keep those checks in the test record separately.
Include allowed, restricted and near-match cases
For the directory example in Step 2, the table shows expected matches for the wildcard group. These results concern matching only; they do not assert that the illustrative pages exist or are indexed.
| Path | Fetch permitted by these rules? | Reason |
|---|---|---|
| /internal/report/ | No. | Matches the restricted directory. |
| /internal/public-guide/start/ | Yes. | The longer Allow path is more specific. |
| /Internal/report/ | Yes. | The differently capitalized path does not match. |
| /internal | Yes. | It lacks the trailing slash in the restricted prefix. |
Add your public homepage, intended service page, relevant assets and a genuine restricted example. For query-pattern rules, include reordered parameters and an unrelated value that must stay allowed. Test the complete address and correct crawler group, not just a path that seems similar.
Check group and path conflicts separately
First select the applicable group, then compare matching path rules. A more specific group can change which rules exist for that crawler. Within that group, longest-path matching and an Allow tie differ from “last line wins.” Save the expected outcome and reason for each test.
Step 8: Recheck after relevant site changes
Review the policy when origins, URL patterns, templates, asset paths or provider choices change. Assign an owner who can explain and reverse a rule. A universal quarterly schedule cannot replace a check after a migration or accidental block.
Google can use cached rules, with cache behavior affected by availability and response conditions. Its specification distinguishes a missing file from server and network failures. Compare the deployed file with Google’s observed version rather than promise that every change is applied within a fixed period.
Compare fetch evidence with the original goal
The Crawl Stats report includes repeated requests, resources and representative samples, not a count of unique indexed pages. Review the resource and response mix alongside relevant URL outcomes. An apparent decrease in requests is not proof that Google reassigned every saved request to important pages.
Avoid security and indexing substitutions
Do not use the public policy as access protection
Listing a confidential path makes that path discoverable. Apply effective authentication or another access control to private resources. Changing a robots rule does not establish that unauthorized visitors can no longer retrieve the content.
Keep an exclusion instruction readable
If a public resource should be excluded from search, Google must be able to fetch its noindex instruction. Do not add a new crawl block and assume Google read the tag behind it. A broad rule may also prevent fetching public pages or resources you intended to preserve.
For urgent result hiding, authorized owners can review the Removals tool. Its result is temporary and does not remove the underlying content from the internet. Record the lasting exclusion or access-control task separately.
Record what you actually verified
A useful completion record includes the intended crawler and scope, the deployed file, representative matching outcomes and the observed file version. Later page-fetch or indexing observations answer additional questions. No step here establishes a percentage of saved crawl budget, a ranking lift or guaranteed zero requests to unwanted pages.
If you discuss help with theStacc, bring the rule, affected URLs and available observations, and confirm implementation responsibilities separately. A content software account does not establish a server-level robots audit or guaranteed technical outcome.
Robots.txt FAQ
Does every website need a robots.txt file?
No. Google’s robots.txt report says a missing file can be fine. Create rules when you have a defined crawl policy to express, and diagnose a fetch error separately from a correct missing-file response. Absence alone does not show that your public pages cannot be discovered.
What is the maximum file size for robots.txt?
Google’s specification sets a 500 KiB limit and ignores content beyond it. Keep the intended rules within that limit and check the actual fetched file. Other providers can differ, so the Google limit does not establish one universal size rule for every crawler.
Should I block AI bots from crawling my site?
Choose a policy for the documented provider and request purpose, then test it. Training preferences and search access need not be the same. OpenAI says robots rules may not apply to some user-triggered fetches; Anthropic separately documents training, search and user agents. Allowing a bot does not guarantee citation.
AI assisted this revision for theStacc. Primary references were read on October 12, 2026; policy examples are illustrative. No live production rule, private account, provider request experiment, crawl saving or named human approval was tested. Original publication fields are retained with unresolved historical provenance.
Sources & references
- [1] Google: Crawl budget guidance
- [2] Google: Common crawlers and product tokens
- [3] Google: Faceted navigation guidance
- [4] Google: robots.txt specification
- [5] Google: Block indexing with noindex
- [6] Google: JavaScript SEO basics
- [7] Google: Pagination and incremental loading
- [8] OpenAI: Crawler purposes and controls
- [9] Google: Robots parser and matcher
- [10] Anthropic: Crawler policy
- [11] Google Search Console: robots.txt report
- [12] Google Search Console: Crawl Stats report
- [13] Google Search Console: Removals tool
- [14] RFC 9309: Robots Exclusion Protocol
Plan topics, review article drafts, and publish to a connected CMS.
Weekly local SEO teardowns
One practical email a week. Map Pack, GBP, AI Overviews. No fluff. Unsubscribe anytime.