Meta Robots Tag
What is a Meta Robots Tag?
A meta robots tag is an HTML tag that provides instructions to search engine crawlers regarding how to index, crawl, and display a page in search results. It is placed in the <head> section of the webpage.
This is how it looks in the code:
<meta name="robots" content="noindex, nofollow">
The meta robots tag is primarily used to control page indexing and crawling and is visible in the HTML code of the page. For pages that lack an HTML head section or for resources such as images, PDFs, etc., the X-Robots tag can be used instead.
Unlike the meta robots tag, the X-Robots tag is an HTTP header that is sent with the response of a web page.
The easiest method to verify the Meta Robots Tag for a page is by using Ahrefs SEO Toolbar browser extension. The indexability section provides directives for crawlers based on both the Meta tags and HTTP headers.
It is important to note that search engine crawlers can only follow the instructions specified in the Meta Robots Tag for pages that are allowed to be crawled according to the rules defined in the robots.txt file.
If you don’t want a page to appear in the search results, it is crucial to ensure that the crawling of that specific page is permitted so that search engines can read the noindex directive.
Apart from controlling page indexing and crawling, the meta robots tag also has a few lesser-known applications. These include instructing search engines not to display a text snippet or video preview for the page in search results or to refrain from storing a cached version of the page.
Why are meta robots tags important?
Webmasters may want to prevent certain pages from showing up in the search engines, such as pages with thin content, PPC landing pages, or admin and “thank you” pages. The meta robots tag provides webmasters with the most reliable way to do that using the “noindex” directive.
Additionally, the “nofollow” attribute in the meta robots tag will mane all links on the page “nofollow” for search engine crawlers such as Googlebot.
It can also be used to control the way an individual page is indexed and served to users. For example, webmasters can use a “nosnippet” directive to prevent search engines from displaying a snippet of text from the page in the search results.
What rules can you use in the meta robots tag?
There are several rules or directives that can be used with the meta robots tag:
- noindex - Instructs the search engines to not index the page. This prevents the page from showing in search results.
- nofollow - Instructs the search engines to not follow any links on the page.
- none - It is the same as using noindex or nofollow.
- all - Equivalent to index, follow. There’s no need to use this directive.
- noarchive - Prevents the search engines from showing a cached version of the page.
- nositelinkssearchbox - Prevents Google from showing a sitelinks search box in the search results.
- nosnippet - Instructs the search engines to not display a text snippet from the web page in the search results.
- indexifembedded - This instructs Google to index content embedded through iframes or similar HTML tags on a page with a noindex directive.
- max-snippet - Specifies the maximum number of characters that Google can display in their text snippets.
- max-image-preview - Tells the search engines if and how big of an image they can use for the image snippets.
- max-video-preview - Sets the maximum number of seconds for video snippets in search results.
- notranslate - This directive instructs Google to not provide translation for the web page.
- noimageindex - Tells Google not to index the images on a web page.
- unavailable_after - Instructs Google not to show a web page in search results after a specified date/time.
For more information on the different rules, visit Google’s documentation on the robots meta tag.
How to set up the meta robots tag?
Implementing the meta robots tag in WordPress is quite easy with popular SEO plugins like Yoast and RankMath. Here’s how to do it:
Using Yoast SEO plugin:
Go to the Advanced Section below the editing blocks for your post or page.
The dropdown “Allow search engines to show this Post in search results?” controls the “noindex” attribute on the page.
Using RankMath plugin:
Noindexing pages with RankMath plugin is similar. Open the Advanced tab and select the No Index option.
