Meta Robots Tag

01What it is and how it works

The meta robots tag is a piece of code placed inside the section of an HTML document. It uses the element with the name attribute set to "robots" and a content attribute that lists one or more directives. Common directives include "index" (allow search engines to index the page), "noindex" (prevent indexing), "follow" (follow links on the page), and "nofollow" (do not follow links). Multiple directives can be combined, for example, content="index, follow". The tag is interpreted by most search engine crawlers, including Google, Bing, and others, as a way to communicate preferences on a per-page basis. It operates at the page level, unlike robots.txt which controls access at the directory or file level. When a crawler encounters the tag, it respects the directives when deciding whether to include the page in its search index and whether to pass link equity through the page's outgoing links. The tag is part of the broader Robots Exclusion Protocol but is specifically for meta tags. For more details, see the Google Search Central documentation.

02What to do about it

To implement the meta robots tag, first identify the pages where you want to control indexing or link following. Edit the HTML of those pages by adding a line like within the area. If you use a content management system, look for a setting that inserts this tag automatically. After adding the tag, verify it by viewing the page source and checking for the meta element. You can also use SEO tools or Google Search Console's URL Inspection tool to see how Google treats the page. If you want to allow indexing but discourage link passing, use content="index, nofollow". For temporary blocking of indexing, consider using the tag and remove it later. Always test changes on a small set of pages before rolling out sitewide. Regular audits ensure the tag is applied correctly across your site.

03How it is measured or noticed

The most direct way to check the meta robots tag is to view the page source in a browser (right-click and select "View page source") and search for the word "robots". SEO software such as Screaming Frog or Ahrefs can crawl your site and report which pages have noindex or nofollow directives. In Google Search Console, the "Pages" report shows indexing status; if a page is excluded, you can inspect it to see if a meta robots tag is the cause. Additionally, the "Coverage" report may list pages with issues that could be related to the tag. Monitoring your site's indexation over time helps notice the impact of these tags. You can also use the "site:" search operator in Google to see if your pages appear in search results; missing pages might indicate a noindex directive.

04Common mistakes

• Using noindex on pages that should be in the search index, which removes them from search results.
• Combining noindex with nofollow when only one directive is needed, causing unintended restrictions.
• Relying on the meta tag to block crawling; it only prevents indexing, not access by crawlers.
• Forgetting that the tag applies only to the current page and does not affect subpages or directories.
• Using non-standard directive names that search engines ignore, leading to unexpected behavior.

05Limits

The meta robots tag does not prevent a search engine from crawling a page; it only controls whether the page is indexed and whether links are followed. For crawl control, robots.txt is the appropriate tool. The tag is ignored by some crawlers or may not be supported in all contexts, such as within frames or certain dynamic rendering setups. It also cannot override a site's overall robots.txt settings if they are more restrictive. Additionally, the tag is specific to HTML documents; it has no effect on non-HTML resources like PDFs or images unless the server provides HTML wrappers. Some search engines may not honor all directives, so relying solely on this tag for critical restrictions is risky. It is often confused with the robots.txt file, which is a separate mechanism for crawl budget management.

06Worked example

Suppose you have a staging site at https://example.com/staging/ that you do not want indexed. You would add the tag above to every HTML page in that directory. After deployment, you can verify by checking the page source or using Google Search Console to see that the pages are not indexed. If you later want them indexed, remove the tag or change the content to "index, follow". This example shows how the tag can be used to exclude specific sections of a site from search results while allowing other parts to be crawled and indexed normally.

<meta name="robots" content="noindex, nofollow">

Frequently asked questions

Related terms

More in SEO