Meta Robots Tag

An ecommerce store with 500 filter combination pages (color + size + material) risks having Google index hundreds of near-duplicate thin pages, splitting ranking signals and diluting the authority of the main category page. Adding noindex, follow to filter pages solves this cleanly — keeping the filter pages crawlable for internal link equity, but removing them from search results where they'd compete against (and cannibalize) the real category pages.

What is the meta robots tag?

The meta robots tag is a standard HTML element that communicates indexing and crawling preferences to search engine bots directly at the page level. Its basic syntax:

Key difference from robots.txt: robots.txt blocks Google from fetching a URL entirely. The meta robots tag is read after Google fetches the page — which means Google must be allowed to crawl the page for the noindex tag to take effect. Pages blocked in robots.txt will never be deindexed by noindex, because Google can't read the noindex tag if it can't crawl the page.

Why the meta robots tag matters for SEO

Indexing control is one of the highest-leverage technical SEO tools available. Four reasons it deserves attention:

1. Crawl budget efficiency. Google allocates a finite crawl budget to each site. Pages with noindex that are also blocked from indexing free up that budget for pages that actually matter. Large sites with thin generated pages (faceted navigation, session-ID URLs, parameter variations) can waste significant crawl budget on content that should never rank.
2. Duplicate content prevention. Faceted navigation, filter pages, tag pages, and URL parameter variants often create near-identical content at different URLs. Without noindex, these compete against each other in Google's index, diluting ranking signals for all variants.
3. Snippet control. The nosnippet and max-snippet directives let publishers control how much of their content Google displays in search results — relevant for news publishers and paywalled content who want to encourage clicks rather than giving content away in snippets.
4. Granular privacy and access control. Login pages, admin pages, internal search results, and staging environments can be excluded from Google's index precisely — without removing them from the live site.

How meta robots tag directives work

The tag accepts multiple directives in a single content="" attribute, comma-separated. Google supports the following:

Index / Noindex

Controls whether the page appears in Google's search results index. index is the default (you don't need to add it explicitly). noindex tells Google to exclude the page from its index — it will crawl it but won't show it in search results.

Follow / Nofollow

Controls whether Google follows the links on the page. follow is the default. nofollow tells Google not to follow or pass PageRank through any links on the page. Note: nofollow on the meta tag differs from rel="nofollow" on individual link elements — the meta tag applies to all links on the page.

Nosnippet

Prevents Google from showing a text snippet or video preview for this page in search results. The title and URL still appear, but no description text is shown. Used by publishers who want to encourage clicks rather than give content away in the result preview.

Max-snippet:[n]

Limits the number of characters Google can use for the text snippet. max-snippet:50 restricts Google to 50 characters. A news publisher used this to limit snippets to 50 characters, which increased their click-through rate from search results by removing the ability to read the article summary without visiting the page.

Noarchive

Prevents Google from caching a copy of the page. Relevant for frequently updated content, time-sensitive information, or content with legal or confidentiality considerations around archived versions.

Common meta robots combinations and when to use them

Real meta robots tag examples

1. Ecommerce filter pages (noindex, follow)

An online clothing store has filter combinations at URLs like /dresses/?color=red&size=M. There are potentially 500+ such combinations, most with only 2-3 products — thin content by any measure. Adding <meta name="robots" content="noindex, follow"> to all filter pages removes them from Google's index, preventing thin content dilution, while keeping follow so internal link equity from filtered pages still flows to the main category pages.

2. News publisher limiting snippets (max-snippet)

A news site used max-snippet:50 to limit snippet display to 50 characters across article pages. The constraint encourages searchers to click through to read the full article rather than consuming the summary in the SERP. The publisher saw a measurable click-through rate increase on articles with limited snippets compared to full-snippet articles at equivalent ranking positions.

Meta robots tag vs. robots.txt — critical difference

7 best practices for meta robots tags

1. Never block a noindexed page in robots.txt. If Google can't crawl the page, it can't see the noindex tag. Pages that need to be deindexed must remain crawlable. This is the most common and most damaging meta robots mistake.
2. Use noindex, follow (not noindex, nofollow) for filter pages. The follow directive allows link equity from filter pages to pass to the category page they filter. Using nofollow isolates those pages and blocks the equity flow.
3. Audit staging environments before launch. Staging sites often have noindex set globally. When you launch or migrate to production, verify the noindex is removed — a forgotten global noindex on a production site will deindex the entire website.
4. Request recrawling after adding noindex. Submit the URL in Google Search Console's URL Inspection tool to trigger faster recrawling and removal from the index. Don't wait weeks for Google's normal crawl cycle.
5. Use X-Robots-Tag for non-HTML files. PDFs and image files don't have HTML head sections, so meta robots tags can't be placed in them. Use the X-Robots-Tag HTTP response header instead for the same directives on non-HTML files.
6. Verify with Google's URL Inspection tool. After adding a meta robots tag, inspect the URL in Search Console to confirm Google is reading the tag as intended and the indexing status reflects your directive.
7. Document every noindex decision. Large sites accumulate noindex tags across dozens of page types. Without documentation, future team members can't distinguish deliberate noindex decisions from accidental ones — leading to confusion and potentially incorrect tag removal.

Common meta robots tag mistakes to avoid

• Blocking crawling in robots.txt for pages you want to noindex — Google can't read the noindex tag on a crawl-blocked page and won't deindex it.
• Using nofollow when follow is appropriate — noindex, nofollow on filter pages wastes internal link equity that noindex, follow would pass.
• Forgetting staging noindex on production launch — always check the site's global template for meta robots on every production deployment.
• Using noindex to handle duplicate content — canonical tags are usually a better solution for duplicate content management than noindex, which removes the page from Google entirely.
• Not verifying the tag is being read — JavaScript-rendered meta robots tags may not be processed correctly by Googlebot. Verify with URL Inspection that the tag is reflected in Google's rendered view of the page.

Frequently asked questions

Related glossary terms

Sources

Akshay VR

Marketing Head · theStacc · ex-Sr Marketing Specialist, ARKA 360 · Malappuram, Kerala

Akshay leads editorial and content operations at theStacc. He writes about technical SEO, crawl architecture, and the indexing decisions that determine which pages Google treats as authoritative and which it filters out.

LinkedIn Author page More terms by Akshay