Meta Tag Robots

The robots meta tag is an HTML element that provides instructions to search engines on how they should deal with a specific web page for crawling and indexing purposes. This tag can be used to prevent search engines from indexing a page (with the “noindex” attribute), from following links on it (with the “nofollow” attribute), or from storing a cached copy of the page in search results (with the “noarchive” attribute).

The robots meta tag is an important tool for controlling the access of search engine crawlers and for managing the visibility of web pages in search results. Some of the most commonly used attributes for refining control over search engines include:

• noindex. This is the command to request not to index the page.
• nofollow. It is the command to request not to follow the link.
• noarchive. Blocks the appearance of the “Cached copy” link in search results.
• nosnippet. Asks to block the presence of a preview snippet in SERPs.
• noimageindex. This attribute prevents search engines from indexing the images on the page. It is useful if you want images to not appear in image search results.
• none. Equivalent to using “noindex, nofollow” and tells search engines not to index the page and not to follow links on it.
• all. In contrast to “none,” “all” indicates that you want both indexing of the page and following of links. However, this attribute is usually redundant since search engines operate this way by default.
• unavailable_after. Specifies a date and time after which the page should no longer be indexed. This can be useful for pages with promotional content or events with an expiration date.

The uses of robots meta tags for SEO and site management are many, ranging from controlling indexing to managing crawl budgets (by limiting crawling of less important pages, you can direct search engines’ attention to more relevant pages) to protecting content and developing a new site or a new section of an existing site.

It is important to note that while the robots meta tag provides guidance to search engines, it is not an absolute guarantee that the guidelines will be followed in every case. Search engines may choose to ignore these directions under certain circumstances, especially if they believe that blocked content is of particular value to users.