Robots meta tag and X-Robots-Tag

Two ways to tell search engines what to do with a page once they have fetched it: a meta tag in the page and an HTTP header.

The tag and the header

In a page:

<meta name="robots" content="noindex, nofollow">

For any file type, in the HTTP response:

X-Robots-Tag: noindex, nofollow

Google's documentation says that rules without a user agent apply to all crawlers, and that the header name, the user agent name and the values are not case sensitive. To address one crawler, use its name in place of robots, for example googlebot.

The common values

Google says that when rules conflict, the more restrictive one applies, and that a nosnippet rule wins over a max-snippet value. Our generator therefore leaves out max-snippet when nosnippet is on.

An important trap

These rules are found only when the page is crawled. Google's page says that if a page is disallowed from crawling in robots.txt, then the indexing and serving rules on it will not be found and will be ignored. To remove a page from results with noindex, the page must stay crawlable.

Using the header

The header is the way to apply a rule to PDFs, images and other files. Examples for servers are in Google's page: in Apache, a Header set X-Robots-Tag "noindex, nofollow" inside a <Files> block, and in nginx an add_header X-Robots-Tag "noindex, nofollow"; line inside a location.

This is an independent guide. It is not connected with Google, Meta, X, schema.org or any company named here. Facts were read on 14 November 2026 from the pages named on each page. The generators run in your browser and send nothing anywhere.