What is the meta robots tag?

A meta robots tag gives supported search crawlers instructions about an HTML page’s indexing, link following and search presentation. The usual placement is the document’s head. Google can read those instructions only when it can fetch the page, as its robots meta specification explains.

Choose the instruction for the outcome you need. Excluding a public confirmation page from search is different from limiting an article’s snippet. Neither choice makes the page private.

Match the instruction to the publishing decision

Before changing a CMS setting, write down the desired result. Should the page remain a search destination? Should Google use its links for discovery? Do you want to restrict a preview of the content? Treat those as separate decisions.

Check the actual page rather than applying the same combination to every archive or filter. A useful category page may deserve to be indexed. If duplicate selection is the problem, consider the canonical URL instead. Google’s canonicalization guidance discourages using noindex to choose a canonical within a site.

How page-level instructions are supplied

In HTML, the tag uses a crawler name and a content value. For non-HTML resources such as PDFs, use an X-Robots-Tag response header instead. A response header can also carry the rules for HTML.

<!-- In the head of an HTML page -->
<meta name="robots" content="noindex">

# An alternative response-header implementation
X-Robots-Tag: noindex

These examples show syntax; they do not set an exclusion instruction on this glossary entry. Google also supports the meta name googlebot for its text search results. Other crawlers may interpret supported rules differently.

Resolve conflicts at their source

For Google, restrictive instructions combine. A permissive tag does not cancel a noindex header. Inspect the template, CMS plugin, server and CDN before adding another tag, then verify the response a visitor or crawler actually receives.

Common directives and their limits

Google’s documented behavior, not a ranking-value table
InstructionPurposeLimit to remember
noindexExclude the fetched page from search results.Google must be able to read it.
nofollowGive a page-wide instruction not to follow links.It is not a measured PageRank-transfer percentage.
nosnippetPrevent text snippets and video previews; restrict direct input to AI Overviews and AI Mode.A static image thumbnail may still be shown.
max-snippet:50Limit a text snippet to fifty characters.Separate content permissions can affect this limit; it does not cover every preview format.
indexifembeddedPermit indexing of content embedded in another page despite noindex.It works only alongside noindex.

Google no longer uses noarchive to control a cached search link, because that feature no longer exists. See the historical rules. Do not present it as a current Google privacy or cache-removal control.

When no nofollow rule is supplied, Google may use the page’s links for discovery. That is distinct from guaranteeing that every linked page is crawled, indexed or receives a particular ranking benefit.

Illustrative decisions for two page types

A public confirmation page

A business wants an enquiry-confirmation page available after submission but absent from search. The implementer checks that the correct template emits noindex and that ordinary service pages remain unaffected. They also confirm that crawl access allows Google to read the instruction. This is an illustrative workflow, not a customer result.

An article with a limited search snippet

A publisher wants a shorter text preview while keeping the article eligible for search. A snippet control addresses that requirement. It does not establish that shorter previews increase clicks. Compare actual outcomes with authorised data before making a commercial claim about the change.

Read the intended output before deploying a combination
CombinationResult to evaluate
noindex, nofollowSearch exclusion plus a page-wide link-following instruction.
noindex, indexifembeddedExclusion of the standalone page with permission to index embedded content.
max-snippet:50, max-image-preview:largeA text-preview limit plus permission for a large image preview.

Robots.txt controls fetch access for cooperating crawlers. A page-level instruction is discovered with the resource. Blocking the URL can prevent Google seeing its noindex tag, as the noindex documentation explains.

For private information, use effective access controls. RFC 9309 makes clear that robots.txt is not authorisation. Listing a private path there also exposes its name publicly.

Inspect a representative response

  1. Identify the page and outcome. Record why a directive is needed and which template or setting supplies it.
  2. Check both delivery methods. Inspect the HTML and response headers so a hidden header rule is not overlooked.
  3. Check crawl access. Confirm that robots.txt and serving failures do not prevent the instruction being read.
  4. Check a neighbouring template. Inspect a normal page that should remain indexable after the same deployment.
  5. Compare the observations. Record the deployed output and change date alongside the last crawl details in Search Console.

The Page indexing report explains how to investigate an observed noindex exclusion. A live response and the response Google previously fetched can differ; use the timestamps to understand that difference.

Avoid a JavaScript-only repair for initial noindex

If the initial HTML contains noindex, removing it later with JavaScript may not work as expected. Google’s JavaScript SEO guidance says it may skip rendering and script execution when it encounters that tag. Fix the initial output when the page should be indexable.

Check the emitted rule, not only the setting name

A staging toggle can affect a whole template. A CDN can add a response header separately from the CMS. Inspect representative production output after a change rather than assuming one dashboard field controls every instruction.

Questions about page-level robots instructions

Is index, follow required on every page?

No. You do not need to add permissive defaults to make a page eligible for Google Search. Check for unintended restrictive instructions instead. Removing a restriction still does not guarantee that Google will index the page.

Can I hide one passage from a snippet instead of the whole page?

Google supports data-nosnippet on span, div and section for passage-level snippet exclusion. It is a boolean attribute: setting its value to “false” does not disable it. Remove the attribute when exclusion is not wanted. This differs from a page-wide nosnippet rule.

Sources and editorial note

Official Google documentation and RFC 9309 are linked beside the relevant explanations, checked on October 12, 2026. AI assisted this revision. Examples are illustrative; no production settings, private Search Console property, click-through experiment or named expert approval were tested. The existing publication date is preserved, with its historical provenance still unresolved.