{"id":30778,"date":"2019-10-08T14:22:25","date_gmt":"2019-10-08T22:22:25","guid":{"rendered":"https:\/\/ahrefs.com\/blog\/?p=30778"},"modified":"2024-03-18T08:06:38","modified_gmt":"2024-03-18T13:06:38","slug":"meta-robots","status":"publish","type":"post","link":"https:\/\/ahrefs.com\/blog\/meta-robots\/","title":{"rendered":"Robots Meta Tag &amp; X-Robots-Tag: Everything You Need to&nbsp;Know"},"content":{"rendered":"<div class=\"intro-txt\">Guiding&nbsp;search&nbsp;engines to crawl and index your website the way you want can be a challenging task. While robots.txt manages the accessibility of your content to crawlers, it doesn\u2019t tell them whether they should index the content or&nbsp;not.<\/div>\n<p>That\u2019s what robots meta tags and the x-robots-tag HTTP header are&nbsp;for.<\/p>\n<p>Let\u2019s get one thing straight from the beginning. You can\u2019t control indexation with robots.txt. That\u2019s a common misconception.<\/p>\n<p>The noindex rule in robots.txt was never officially supported by Google. And in July 2019, it was officially <a href=\"https:\/\/webmasters.googleblog.com\/2019\/07\/a-note-on-unsupported-rules-in-robotstxt.html\" target=\"_blank\" rel=\"noopener noreferrer\">deprecated<\/a>.<\/p>\n<div class=\"intro-tok\" id=\"intro_tok\" style=\"display:none;\"><div class=\"intro-title\">Contents<\/div><a href=\"#\" class=\"expand-dots\"><span><\/span><span><\/span><span><\/span><\/a><\/div>\n<div class=\"hub-link\"><img decoding=\"async\" alt=\"Beginner's guide to technical SEO\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/svg\/4.svg\"><div class=\"hl-title\">New to technical SEO? Check out&nbsp;our<\/div><div class=\"hl-content\"><a href=\"https:\/\/ahrefs.com\/blog\/technical-seo\/\" target=\"_blank\">Beginner\u2019s guide to technical SEO<\/a><\/div><\/div>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"What is a robots meta tag?\" data-section=\"1-what-is-robots-meta-tag\">\n<h2 id=\"what-is-a-robots-meta-tag\">What is a robots meta&nbsp;tag?<\/h2>\n<\/div><\/div>\n<p><span style=\"font-weight: 400;\">A robots meta tag is an HTML snippet that tells search engine robots what they can and cannot do on a certain page. It lets you control crawling, indexing and how information from that page is displayed in the search results. It\u2019s placed into the &lt;head&gt; section of a webpage.<\/span><\/p>\n<h3>Example<\/h3>\n<p><code>&lt;meta name=\"robots\" content=<span style=\"font-weight: 400;\">\"noindex, nofollow\"<\/span>&gt;<\/code><\/p>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"Why is the robots meta tag important for SEO?\" data-section=\"2-why-is-robots-meta-tag-important-for-seo\">\n<h2 id=\"why-the-robots-meta-tag-is-important\">Why is the robots meta tag important for&nbsp;SEO?<\/h2>\n<\/div><\/div>\n<p>The meta robots tag is commonly used to prevent pages showing up in search results, although it does have other uses (more on those&nbsp;later).<\/p>\n<p>There are various types of content that you might want to prevent search engines from indexing:<\/p>\n<ul>\n<li>Thin pages with little or no value for the&nbsp;user;<\/li>\n<li>Pages in the staging environment;<\/li>\n<li>Admin and thank-you pages;<\/li>\n<li>Internal search results;<\/li>\n<li>PPC landing pages;<\/li>\n<li>Pages about upcoming promotions, contests or product launches;<\/li>\n<li>Duplicate content (use <a href=\"https:\/\/ahrefs.com\/blog\/canonical-tags\/\">canonical tags<\/a>&nbsp;to suggest the best version for indexing);<\/li>\n<\/ul>\n<p>Generally, the bigger your website is, the more you\u2019ll deal with managing crawlability and indexation. You also want Google and other search engines to crawl and index your pages as efficiently as possible. Correctly combining page-level directives with <a href=\"https:\/\/ahrefs.com\/blog\/robots-txt\/\">robots.txt<\/a> and <a href=\"https:\/\/ahrefs.com\/blog\/how-to-create-a-sitemap\/\">sitemaps<\/a> is crucial for&nbsp;SEO.<\/p>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"What are the values and attributes of a robots meta tag?\" data-section=\"3-what-are-values-attributes-robots-meta-tag\">\n<h2 id=\"meta-robots-values-and-attributes\">What are the values and attributes of a robots meta&nbsp;tag?<\/h2>\n<\/div><\/div>\n<p>Robots meta tags consist of two attributes: <em>name<\/em>&nbsp;and <em>content<\/em>.<\/p>\n<p>You <strong>must<\/strong>&nbsp;specify values for each of these attributes. Let\u2019s explore what these&nbsp;are.<\/p>\n<h3>The <em>name<\/em>&nbsp;attribute and user-agent values<\/h3>\n<p>The <em>name<\/em>&nbsp;attribute specifies which crawlers should follow these instructions. This value is also known as a user-agent (UA) because crawlers need to be identified with their UA to request a page. Your UA reflects the browser you\u2019re using, but Google\u2019s user-agents are, for example, <a href=\"https:\/\/ahrefs.com\/blog\/googlebot\/\">Googlebot<\/a> or Googlebot-image.<\/p>\n<p>The UA value \u201crobots\u201d applies to all crawlers. You can also add as many robots meta tags into the &lt;head&gt; section as you need. For example, if you want to prevent your images from showing up in a Google or Bing image search, add the following meta&nbsp;tags:<\/p>\n<p><code>&lt;meta name=\"googlebot-image\" content=\"noindex\"&gt;<\/code><\/p>\n<p><code>&lt;meta name=\"MSNBot-Media\" content=\"noindex\"&gt;<\/code><\/p>\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div> Both name and content attributes are non-case sensitive. \u201cGooglebot-Image,\u201d \u201cmsnbot-media\u201d and \u201cNoindex\u201d attributes also work for the examples above.&nbsp;<\/div>\n<h3>The <em>content<\/em> attribute and crawling\/indexing directives<\/h3>\n<p>The <em>content<\/em>&nbsp;attribute provides instructions on how to crawl and index information on the page. If there is no robots meta tag available, crawlers interpret it as index and follow. That gives them permission to show the page in search results and crawl all links on the page (unless stated otherwise with the <a href=\"https:\/\/ahrefs.com\/blog\/nofollow-links\/\">rel=\u201dnofollow\u201d tag<\/a>).<\/p>\n<p>The following&nbsp;are the <a href=\"https:\/\/developers.google.com\/search\/reference\/robots_meta_tag\" target=\"_blank\" rel=\"noopener noreferrer\">supported values for the <\/a><em><a href=\"https:\/\/developers.google.com\/search\/reference\/robots_meta_tag\" target=\"_blank\" rel=\"noopener noreferrer\">content<\/a><\/em><a href=\"https:\/\/developers.google.com\/search\/reference\/robots_meta_tag\" target=\"_blank\" rel=\"noopener noreferrer\">&nbsp;attribute by Google<\/a>:<\/p>\n<h4>all<\/h4>\n<p>The default value of \u201cindex, follow\u201d, no need to ever use this directive.<\/p>\n<p><code>&lt;meta name=\"robots\" content=\"all\"&gt;<\/code><\/p>\n<h4>noindex<\/h4>\n<p>Instructs search engines not to index the page. That prevents it from showing in search results.<\/p>\n<p><code>&lt;meta name=\"robots\" content=\"noindex\"&gt;<\/code><\/p>\n<h4>nofollow<\/h4>\n<p>Stops robots from crawling all links on the page.&nbsp;Please note that those URLs still may be indexable, especially if they have backlinks pointing to&nbsp;them.<\/p>\n<p><code>&lt;meta name=\"robots\" content=\"nofollow\"&gt;<\/code><\/p>\n<h4>none<\/h4>\n<p>The combination of <em>noindex, nofollow<\/em>. Avoid using this as other search engines (e.g., <a href=\"https:\/\/www.bing.com\/webmaster\/help\/which-robots-metatags-does-bing-support-5198d240\" target=\"_blank\" rel=\"noopener noreferrer\">Bing<\/a>) don\u2019t support this.<\/p>\n<p><code>&lt;meta name=\"robots\" content=\"none\"&gt;<\/code><\/p>\n<h4>noarchive<\/h4>\n<p>Prevents Google from showing a cached copy of the page in the&nbsp;SERP.<\/p>\n<p><code>&lt;meta name=\"robots\" content=\"noarchive\"&gt;<\/code><\/p>\n<h4>notranslate<\/h4>\n<p>Prevents Google from offering a translation of the page in the&nbsp;SERP.<\/p>\n<p><code>&lt;meta name=\"robots\" content=\"notranslate\"&gt;<\/code><\/p>\n<h4>noimageindex<\/h4>\n<p>Prevents Google from indexing images embedded on the&nbsp;page.<\/p>\n<p><code>&lt;meta name=\"robots\" content=\"noimageindex\"&gt;<\/code><\/p>\n<h4>unavailable_after:<\/h4>\n<p>Tells Google not to show a page in search&nbsp;results after a specified date\/time. Basically a <em>noindex<\/em>&nbsp;directive with a timer. The date\/time must be specified using the <a href=\"https:\/\/tools.ietf.org\/html\/rfc850\" target=\"_blank\" rel=\"noopener noreferrer\">RFC 850 format<\/a>.<\/p>\n<p><code>&lt;meta name=\"robots\" content=\"unavailable_after: Sunday, 01-Sep-19 12:34:56 GMT\"&gt;<\/code><\/p>\n<h4>nosnippet<\/h4>\n<p>Opts out of all text and video snippets within the SERP. It also works as <em>noarchive<\/em>&nbsp;at the same&nbsp;time.<\/p>\n<p><code>&lt;meta name=\"robots\" content=\"nosnippet\"&gt;<\/code><\/p>\n<div class=\"recommendation\"><div class=\"recommendation-title\">IMPORTANT NOTE<\/div><div class=\"recommendation-content\">\n<p>Since October 2019, Google offers <a href=\"https:\/\/webmasters.googleblog.com\/2019\/09\/more-controls-on-search.html\" target=\"_blank\" rel=\"noopener noreferrer\">more granular options<\/a>&nbsp;to control if and how you want to display your snippets in the search results. This is in part due to the <a href=\"https:\/\/ec.europa.eu\/digital-single-market\/en\/modernisation-eu-copyright-rules\" target=\"_blank\" rel=\"noopener noreferrer\">European Copyright Directive<\/a>,&nbsp;which was first implemented by France with its <a href=\"https:\/\/france.googleblog.com\/2019\/09\/comment-nous-respectons-le-droit-dauteur.html\" target=\"_blank\" rel=\"noopener noreferrer\">new copyright law<\/a>.<\/p>\n<p>Crucially, this legislation already affects <strong>all<\/strong>&nbsp;website owners. How? Because Google no longer displays snippets (text, image or video) from your site to users in France unless you opt-in using their new robots meta&nbsp;tags.<\/p>\n<p>We discuss how each of these new tags works below. That said, if this concerns your business and you\u2019re looking for a quick solution, add the following HTML snippet to every page on your site to tell Google that you want no restrictions on your snippets:<\/p>\n<p><code>&lt;meta name=\"robots\" content=\u201dmax-snippet:-1, max-image-preview:large, max-video-preview:-1\"&gt;<\/code><\/p>\n<p>Note that if you use Yoast SEO, <a href=\"https:\/\/yoast.com\/robots-meta-changes-for-google\/\" target=\"_blank\" rel=\"noopener noreferrer\">this piece of code is added automatically<\/a>&nbsp;on every page unless you added <em>noindex<\/em>&nbsp;or <em>nosnippet <\/em>directives.<\/p>\n<\/div><\/div>\n<h4>max-snippet:<\/h4>\n<p>Specifies a maximum number of characters Google can show in their text snippets. Using 0 will opt out of the text snippets, -1 declares no limit on the text preview.<\/p>\n<p>The following tag will set up the limit to 160 characters (similar to standard meta description length):<\/p>\n<p><code>&lt;meta name=\"robots\" content=\"max-snippet:160\"&gt;<\/code><\/p>\n<h4>max-image-preview:<\/h4>\n<p>Tells Google if and how big an image it can use for image snippets. This directive has three possible values:<\/p>\n<ul>\n<li>none no image snippet will be&nbsp;shown<\/li>\n<li>standard - a default image preview may be&nbsp;shown<\/li>\n<li>large - the largest possible image preview may&nbsp; be shown<\/li>\n<\/ul>\n<p><code>&lt;meta name=\"robots\" content=\"max-image-preview:large\"&gt;<\/code><\/p>\n<p><span style=\"font-weight: 400;\">Using the <\/span><i><span style=\"font-weight: 400;\">large<\/span><\/i><span style=\"font-weight: 400;\"> value along with images that are at least 1200px wide is recommended because it increases the likelihood of getting featured in <\/span><a href=\"https:\/\/ahrefs.com\/blog\/google-discover\/\"><span style=\"font-weight: 400;\">Google Discover<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n<h4>max-video-preview:<\/h4>\n<p>Sets up a maximum number of seconds for a video snippet. As with the text snippet, 0 will opt out completely, -1 places no limits.<\/p>\n<p>The following tag would allow Google to show maximum of 15 seconds:<\/p>\n<p><code>&lt;meta name=\"robots\" content=\"max-video-preview:15\"&gt;<\/code><\/p>\n<div class=\"recommendation\"><div class=\"recommendation-title\">a quick note on using a <em>data-nosnippet <\/em>HTML attribute<\/div><div class=\"recommendation-content\">\n<p>Alongside the new robots directives introduced in October 2019, Google also introduced the data-nosnippet HTML attribute. You can use this to tag parts of text that you don\u2019t want Google to use as a snippet.<\/p>\n<p>This can be done in HTML on div, span, and section elements. The data-nosnippet is considered a boolean attribute, meaning that it\u2019s valid with or without a&nbsp;value.<\/p>\n<pre>&lt;p&gt;This is some text in a paragraph that can be shown as a snippet&lt;span data-nosnippet&gt;excluding this part&lt;\/span&gt;&lt;\/p&gt;\n<\/pre>\n<pre>&lt;div data-nosnippet&gt;This will not appear in a snippet&lt;\/div&gt;&lt;div data-nosnippet=\"true\"&gt;And neither will this&lt;\/div&gt;\n<\/pre>\n<\/div><\/div>\n<h4>nositelinkssearchbox<\/h4>\n<p><span style=\"font-weight: 400;\">Prevents Google from showing a search box as a part of your <\/span><a href=\"https:\/\/ahrefs.com\/blog\/sitelinks\/\"><span style=\"font-weight: 400;\">sitelinks<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><code>&lt;meta name=\"robots\" content=<span style=\"font-weight: 400;\">\"nositelinkssearchbox\"<\/span>&gt;<\/code><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-149589\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/1-zillow-sitelinks-searchbox.jpg\" alt=\"Sitelinks searchbox for Zillow\" width=\"2286\" height=\"1828\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/1-zillow-sitelinks-searchbox.jpg 2286w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/1-zillow-sitelinks-searchbox-531x425.jpg 531w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/1-zillow-sitelinks-searchbox-768x614.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/1-zillow-sitelinks-searchbox-1536x1228.jpg 1536w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/1-zillow-sitelinks-searchbox-2048x1638.jpg 2048w\" sizes=\"auto, (max-width: 2286px) 100vw, 2286px\"><\/p>\n<h4>indexifembedded<\/h4>\n<p><span style=\"font-weight: 400;\">Allows Google to index content embedded through iframes or similar HTML tags on a page with a noindex directive. It only works when both directives are present, like&nbsp;this:<\/span><\/p>\n<p><code>&lt;meta name=\"robots\" content=<span style=\"font-weight: 400;\">\"noindex, indexifembedded\"<\/span>&gt;<\/code><\/p>\n<p><span style=\"font-weight: 400;\">Google\u2019s documentation nicely <\/span><a href=\"https:\/\/developers.google.com\/search\/blog\/2022\/01\/robots-meta-tag-indexifembedded\"><span style=\"font-weight: 400;\">explains the use case<\/span><\/a><span style=\"font-weight: 400;\"> here.<\/span><\/p>\n<h3>Using these directives<\/h3>\n<p>Most SEOs don\u2019t need to go beyond the noindex and nofollow directives, but it\u2019s good to know that there are other options. <span style=\"font-weight: 400;\">Keep in mind that all directives we talk about here are based on what Google supports. There are a few directives unique to other search engines, but they aren\u2019t worth diving into.<\/span><\/p>\n<p>Let\u2019s check the comparison <a href=\"https:\/\/www.bing.com\/webmasters\/help\/which-robots-metatags-does-bing-support-5198d240\"><span style=\"font-weight: 400;\">with Bing<\/span><\/a>:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-149593\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/meta_robots.png\" alt=\"Robots meta tags supported by Bing\" width=\"1800\" height=\"2770\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/meta_robots.png 1800w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/meta_robots-276x425.png 276w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/meta_robots-768x1182.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/meta_robots-998x1536.png 998w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/meta_robots-1331x2048.png 1331w\" sizes=\"auto, (max-width: 1800px) 100vw, 1800px\"><\/p>\n<p>You can use multiple directives at once and combine them. But if they conflict (e.g., \u201cnoindex, index\u201d) or one is a subset of another (e.g., \u201cnoindex, noarchive\u201d), <a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots-meta-tag\" target=\"_blank\" rel=\"noopener noreferrer\">Google will use the most restrictive one<\/a>. In these cases, it would be just \u201cnoindex\u201d.<\/p>\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div>&nbsp;Snippet directives may be overridden by <a href=\"https:\/\/ahrefs.com\/blog\/structured-data\/\" data-ahr=\"https:\/\/ahrefs.com\/blog\/what-is-structured-data\/\">structured data<\/a>&nbsp;that allows Google to use any information within the annotation. If you want to prevent Google from showing snippets, adjust the annotation accordingly and make sure that you don\u2019t have any license agreement with Google.<\/div>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"How to set up the robots meta tag\" data-section=\"4-how-to-set-up-robots-meta-tag\">\n<h2 id=\"how-to-set-up-meta-robots\">How to set up the robots meta&nbsp;tag<\/h2>\n<\/div><\/div>\n<p>Now that you know what all these directives do and look like, it\u2019s time to get to the actual implementation on your website.<\/p>\n<p>Robots meta tags belong to the <code>&lt;head&gt;<\/code> &nbsp;section of a page. It\u2019s pretty straightforward if you edit the code using HTML editors such as Notepad++ or Brackets. But what if you\u2019re using a CMS with SEO plugins?<\/p>\n<p>Let\u2019s focus on the most popular option out&nbsp;there.<\/p>\n<h3>Implementing robots meta tags in WordPress using Yoast&nbsp;SEO<\/h3>\n<p>Go to the \u201cAdvanced\u201d section below the editing block of each post or page. Set up the robots meta tag according to your needs. The following settings would implement \u201cnoindex, nofollow\u201d directives.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1300\" height=\"690\" class=\"wp-image-30775\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-02-at-21.51.32-1.png\" alt=\"Screenshot 2019 09 02 at 21.51.32 1\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-02-at-21.51.32-1.png 1300w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-02-at-21.51.32-1-768x408.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-02-at-21.51.32-1-680x361.png 680w\" sizes=\"auto, (max-width: 1300px) 100vw, 1300px\"><\/p>\n<p>The \u201cMeta robots advanced\u201d row gives you the option to implement directives other than <em>noindex<\/em>&nbsp;and <em>nofollow<\/em>,&nbsp;such as <em>noimageindex<\/em>.<\/p>\n<p>You also have the option to apply these directives sitewide. Go to \u201cSearch Appearance\u201d in the Yoast menu. There you can set up meta robots tags on all posts, pages, or just on specific taxonomies or archives.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1298\" height=\"988\" class=\"wp-image-30773\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-02-at-21.45.25-1.png\" alt=\"Screenshot 2019 09 02 at 21.45.25 1\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-02-at-21.45.25-1.png 1298w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-02-at-21.45.25-1-768x585.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-02-at-21.45.25-1-558x425.png 558w\" sizes=\"auto, (max-width: 1298px) 100vw, 1298px\"><\/p>\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div> Yoast isn\u2019t the only way to control meta robots tags in WordPress. There are plenty of <a href=\"https:\/\/ahrefs.com\/blog\/best-seo-plugins-for-wordpress\/\">other WordPress SEO plugins<\/a>&nbsp;with similar functionality.&nbsp;<\/div>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"What is an X-Robots-Tag?\" data-section=\"5-what-is-x-robots-tag\">\n<h2 id=\"what-is-x-robots-tag\">What is an X-Robots-Tag?<\/h2>\n<\/div><\/div>\n<p>The robots meta tag is fine for implementing <em>noindex<\/em>&nbsp;directives on HTML pages here and there. But what if you want to prevent search engines from indexing files such as images or PDFs? This is when x-robots-tags come into&nbsp;play.<\/p>\n<p>The X-Robots-Tag is an HTTP header sent from a web server. Unlike the meta robots tag, it isn\u2019t placed in the HTML of the page. Here\u2019s what it can look&nbsp;like:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-149592\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/2-http-response-headers.png\" alt=\"X-Robots-Tag example\" width=\"878\" height=\"524\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/2-http-response-headers.png 878w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/2-http-response-headers-680x406.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/2-http-response-headers-768x458.png 768w\" sizes=\"auto, (max-width: 878px) 100vw, 878px\"><\/p>\n<p><span style=\"font-weight: 400;\">The easiest way to check HTTP headers is with the free <\/span><a href=\"https:\/\/ahrefs.com\/seo-toolbar\"><span style=\"font-weight: 400;\">Ahrefs SEO toolbar browser extension<\/span><\/a><span style=\"font-weight: 400;\">. Just head over to the HTTP header tab and see if the X-Robots-Tag is present:<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-149591\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/3-ahrefs-toolsbar-http-headers.jpg\" alt=\"X-Robots-Tag example from Ahrefs SEO toolbar\" width=\"1372\" height=\"730\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/3-ahrefs-toolsbar-http-headers.jpg 1372w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/3-ahrefs-toolsbar-http-headers-680x362.jpg 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/3-ahrefs-toolsbar-http-headers-768x409.jpg 768w\" sizes=\"auto, (max-width: 1372px) 100vw, 1372px\"><\/p>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"How to set up the X-Robots-Tag\" data-section=\"6-how-to-set-up-x-robots-tag\">\n<h2 id=\"how-to-set-up-x-robots-tag\">How to set up the X-Robots-Tag<\/h2>\n<\/div><\/div>\n<p>The configuration depends on the type of web server you\u2019re using and which pages or files you want to keep out of the&nbsp;index.<\/p>\n<p>The line of code look like&nbsp;this:<\/p>\n<p><code>Header set X-Robots-Tag \u201cnoindex, nofollow\u201d<\/code><\/p>\n<p>This example takes into account the most widespread server type\u2014Apache. The most practical way of adding the HTTP header is by modifying the main configuration file (usually <em>httpd.conf<\/em>)&nbsp;or <em>.htaccess<\/em>&nbsp;files. Sounds familiar? This is the place where <a href=\"https:\/\/ahrefs.com\/blog\/301-redirects\/\">redirects<\/a>&nbsp;also happen.<\/p>\n<p>You use the same values and directives for the x-robots-tag as a meta robots tag. That said, implementing these changes should be left for the experienced. Backups are your friends because even a small syntax error can break the whole website.<\/p>\n<div class=\"recommendation\"><div class=\"recommendation-title\">PRO TIP<\/div><div class=\"recommendation-content\">\n<p>If you use a CDN that supports <a href=\"https:\/\/www.searchenginejournal.com\/edge-seo\/273983\/\" target=\"_blank\" rel=\"noopener noreferrer\">serverless applications for Edge SEO<\/a>, you can modify both robots meta tags and X-Robots-Tags on the edge server without making changes to the underlying codebase.<\/p>\n<\/div><\/div>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"When to use the robots meta tag vs. x-robots-tag?\" data-section=\"7-when-to-use-robots-meta-tag-vs-x-robots-tag\">\n<h2 id=\"meta-robots-vs-x-robots-tag\">When to use the robots meta tag vs. x-robots-tag?<\/h2>\n<\/div><\/div>\n<p>While adding an HTML snippet looks like the easiest and most straightforward option, it falls short in some&nbsp;cases.<\/p>\n<h3>Non-HTML files<\/h3>\n<p>You can\u2019t place the HTML snippet into non-HTML files such as PDFs or images. X-Robots-Tag is the only&nbsp;way.<\/p>\n<p>The following snippet (on an Apache server) would configure <em>noindex<\/em>&nbsp;HTTP headers on all PDF files on the&nbsp;site.<\/p>\n<pre>&lt;Files ~ \"\\.pdf$\"&gt;\nHeader set X-Robots-Tag \"noindex\"\n&lt;\/Files&gt;\n<\/pre>\n<h3>Applying directives at&nbsp;scale<\/h3>\n<p>If you need to noindex a whole (sub)domain, subdirectory, pages with certain <a href=\"https:\/\/ahrefs.com\/blog\/url-parameters\/\">parameters<\/a> or anything else that requires bulk editing, use x-robots-tags. It\u2019s easier.<\/p>\n<p>HTTP header modifications can be matched against URLs and file names using regular expressions. Complex bulk editing in HTML using the search and replace function would usually require more time and computing power.<\/p>\n<h3>Traffic from search engines other than Google<\/h3>\n<p>Google supports both meta robots tags and x-robots-tags, but this isn\u2019t the case for all search engines.<\/p>\n<p>For example, Seznam, a Czech search engine only supports robots meta tags. If you want to control how this search engine crawls and indexes your pages, using x-robots-tags won\u2019t work. You need to use the HTML snippets.<\/p>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"How to avoid crawlability and (de)indexation mistakes\" data-section=\"8-how-to-avoid-crawlability-deindexation-mistakes\">\n<h2 id=\"avoiding-mistakes\">How to avoid crawlability and (de)indexation mistakes<\/h2>\n<\/div><\/div>\n<p>You want to show all valuable pages, avoid <a href=\"https:\/\/ahrefs.com\/blog\/duplicate-content\/\">duplicate content<\/a>,&nbsp;issues and keep specific pages out of the index. If you manage a huge website then <a href=\"https:\/\/ahrefs.com\/blog\/crawl-budget\/\" target=\"_blank\" rel=\"noopener noreferrer\">crawl budget<\/a>&nbsp;management is another thing to pay attention to.<\/p>\n<p>Let\u2019s have a look at the most common mistakes people make regarding robots directives.<\/p>\n<h3>Mistake #1: Adding noindex directives to&nbsp;pages disallowed in robots.txt<\/h3>\n<p>Never disallow crawling of content that you\u2019re trying to get deindexed in robots.txt. Doing so prevents search engines from recrawling the page and discovering the noindex directive.<\/p>\n<p>If you feel you may have made that mistake in the past, crawl your site with <a href=\"https:\/\/ahrefs.com\/webmaster-tools\"><span style=\"font-weight: 400;\">Ahrefs Webmaster Tools<\/span><\/a><span style=\"font-weight: 400;\"> for free<\/span>.&nbsp;Look for pages with \u201cNoindex page receives organic traffic\u201d errors.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1600\" height=\"682\" class=\"wp-image-30774\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-22-at-23.54.19-1.png\" alt=\"Screenshot 2019 09 22 at 23.54.19 1\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-22-at-23.54.19-1.png 1600w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-22-at-23.54.19-1-768x327.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-22-at-23.54.19-1-680x290.png 680w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\"><\/p>\n<p>Noindexed pages that receive organic traffic are clearly still indexed. If you didn\u2019t add the noindex tag recently, chances are this is due to a crawl block in your robots.txt file. Check for issues and fix them as appropriate.<\/p>\n<h3>Mistake #2: Bad sitemaps management<\/h3>\n<p>If you\u2019re trying to get content deindexed using a meta robots tag or x-robots-tag, don\u2019t remove it from your <a href=\"https:\/\/ahrefs.com\/blog\/how-to-create-a-sitemap\/\">sitemap<\/a>&nbsp;until it\u2019s been successfully deindexed. Otherwise, Google may be slower to recrawl the&nbsp;page.<\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en-gb\">\n<p dir=\"ltr\" lang=\"en\"><a href=\"https:\/\/twitter.com\/nishanthstephen?ref_src=twsrc%5Etfw\">@nishanthstephen<\/a> generally anything you put in a sitemap will be picked up sooner\u2014 Gary \u201c\u9be8\u7406\u201d Illyes (@methode) <a href=\"https:\/\/twitter.com\/methode\/status\/653980524264878080?ref_src=twsrc%5Etfw\">13 October 2015<\/a><\/p>\n<\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<p>To potentially speed up the deindexing process further, set the lastmod date in your sitemap to the date you added the noindex tag. This encourages recrawling and reprocessing.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-172999\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2024-03-18-at-13.06.26.png\" alt=\"John Mueller explaining how to use lastmod to potentially speed up recrawling\" width=\"1192\" height=\"482\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2024-03-18-at-13.06.26.png 1192w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2024-03-18-at-13.06.26-680x275.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2024-03-18-at-13.06.26-768x311.png 768w\" sizes=\"auto, (max-width: 1192px) 100vw, 1192px\"><\/p>\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div>&nbsp;John is talking about 404 pages here. That said, we\u2019re assuming that this also makes sense for other changes like when you add or remove a noindex directive.&nbsp;<\/div>\n<div class=\"recommendation\"><div class=\"recommendation-title\">IMPORTANT NOTE<\/div><div class=\"recommendation-content\">\n<p>Don\u2019t include noindexed pages in your sitemap in the long-term. Once content has been deindexed, remove it from your sitemap.<\/p>\n<p>If you\u2019re worried that old, successfully deindexed content may still exist in your sitemap, check the \u201cNoindex page in sitemap\u201d error in <a href=\"https:\/\/ahrefs.com\/webmaster-tools\"><span style=\"font-weight: 400;\">Ahrefs Webmaster Tools<\/span><\/a>.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1600\" height=\"566\" class=\"wp-image-30771\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-23-at-00.23.07-1.png\" alt=\"Screenshot 2019 09 23 at 00.23.07 1\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-23-at-00.23.07-1.png 1600w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-23-at-00.23.07-1-768x272.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/Screenshot-2019-09-23-at-00.23.07-1-680x241.png 680w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\"><\/p>\n<\/div><\/div>\n<h3>Mistake #3: Not removing noindex directives from the production environment<\/h3>\n<p>Preventing robots from crawling and indexing anything in the staging environment is a good practice. However, it sometimes gets pushed into production, forgotten, and your organic traffic plunges.<\/p>\n<p>Even worse, the organic traffic drop might not be that noticeable if you\u2019re involved in a <a href=\"https:\/\/ahrefs.com\/blog\/website-migration\/\">site migration<\/a> using 301 redirects. If the new URLs contain the noindex directive or are disallowed in robots.txt, you\u2019ll still receive organic traffic from the old ones for some time. It can take Google up to a few weeks to deindex the old&nbsp;URLs.<\/p>\n<p><span style=\"font-weight: 400;\">Whenever there are such changes on your website, keep an eye on the noindex warnings in the <\/span><b>Indexability<\/b><span style=\"font-weight: 400;\"> report:<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-149590\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/4-indexability-issues-site-audit.png\" alt=\"Indexability report in Ahrefs Site Audit\" width=\"2520\" height=\"2180\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/4-indexability-issues-site-audit.png 2520w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/4-indexability-issues-site-audit-491x425.png 491w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/4-indexability-issues-site-audit-768x664.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/4-indexability-issues-site-audit-1536x1329.png 1536w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/4-indexability-issues-site-audit-2048x1772.png 2048w\" sizes=\"auto, (max-width: 2520px) 100vw, 2520px\"><\/p>\n<p>To help prevent similar issues in the future, enrich the dev team\u2019s checklist with instructions for removing disallow rules from robots.txt and noindex directives&nbsp;before pushing to production.<\/p>\n<h3>Mistake #4: Adding \u201csecret\u201d URLs to robots.txt instead of noindexing them<\/h3>\n<p>Developers often try to hide pages about upcoming promotions, discounts, or product launches by disallowing access to them in the site\u2019s robots.txt file. This is bad practice because humans can still view a robots.txt file. As such, these pages are easily leaked.<\/p>\n<p>Fix this by keeping \u201csecret\u201d pages out of robots.txt and noindexing them instead.<\/p>\n<h2>Final thoughts<\/h2>\n<p><span style=\"font-weight: 400;\">Properly understanding and managing the crawling and indexing of your website is the cornerstone of SEO. It\u2019s nothing really complicated. Well, at least in comparison to other tricky stuff within the field of <\/span><a href=\"https:\/\/ahrefs.com\/seo\/technical-seo\" data-ahr=\"https:\/\/ahrefs.com\/blog\/technical-seo\/\"><span style=\"font-weight: 400;\">technical SEO<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">I hope that you\u2019re now prepared to apply the best practices for long-term solutions at&nbsp;scale.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Got any questions? Let me know on <\/span><a href=\"https:\/\/twitter.com\/michalpecanek\"><span style=\"font-weight: 400;\">Twitter<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>That\u2019s what robots meta tags and the x-robots-tag HTTP header are&nbsp;for. Let\u2019s get one thing straight from the beginning. You can\u2019t control indexation with robots.txt. That\u2019s a common misconception. The noindex rule in robots.txt was never officially supported by Google.<span class=\"ellipsis\">\u2026<\/span><\/p>\n<div class=\"read-more\">Read more \u203a<\/div>\n<p><!-- end of .read-more --><\/p>\n","protected":false},"author":149,"featured_media":30794,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"wp_typography_post_enhancements_disabled":false,"footnotes":""},"categories":[329],"tags":[],"coauthors":[375],"class_list":["post-30778","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technical-seo","odd"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.3 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Robots Meta Tag &amp; X-Robots-Tag: Everything You Need to Know<\/title>\n<meta name=\"description\" content=\"A robots meta tag is an HTML snippet that tells search engine robots what they can and cannot do on a certain page. Understanding this is the cornerstone of SEO.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ahrefs.com\/blog\/meta-robots\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Robots Meta Tag &amp; X-Robots-Tag: Everything You Need to Know\" \/>\n<meta property=\"og:description\" content=\"Tell search engines where they can and can&#039;t go on your site.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ahrefs.com\/blog\/meta-robots\/\" \/>\n<meta property=\"og:site_name\" content=\"SEO Blog by Ahrefs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Ahrefs\/\" \/>\n<meta property=\"article:published_time\" content=\"2019-10-08T22:22:25+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-03-18T13:06:38+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/fb-meta-robots-tag-2.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"990\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Michal Pec\u00e1nek\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:description\" content=\"Tell search engines where they can and can&#039;t go on your site.\" \/>\n<meta name=\"twitter:creator\" content=\"@michalpecanek\" \/>\n<meta name=\"twitter:site\" content=\"@ahrefs\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/meta-robots\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/meta-robots\\\/\"},\"author\":{\"name\":\"Michal Pec\u00e1nek\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/b66ba944c48e71d0abac1af2f11265ec\"},\"headline\":\"Robots Meta Tag &amp; X-Robots-Tag: Everything You Need to&nbsp;Know\",\"datePublished\":\"2019-10-08T22:22:25+00:00\",\"dateModified\":\"2024-03-18T13:06:38+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/meta-robots\\\/\"},\"wordCount\":2730,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/meta-robots\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/fb-meta-robots-tag-2.jpg\",\"articleSection\":[\"Technical SEO\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/ahrefs.com\\\/blog\\\/meta-robots\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/meta-robots\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/meta-robots\\\/\",\"name\":\"Robots Meta Tag & X-Robots-Tag: Everything You Need to Know\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/meta-robots\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/meta-robots\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/fb-meta-robots-tag-2.jpg\",\"datePublished\":\"2019-10-08T22:22:25+00:00\",\"dateModified\":\"2024-03-18T13:06:38+00:00\",\"description\":\"A robots meta tag is an HTML snippet that tells search engine robots what they can and cannot do on a certain page. Understanding this is the cornerstone of SEO.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ahrefs.com\\\/blog\\\/meta-robots\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/meta-robots\\\/#primaryimage\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/fb-meta-robots-tag-2.jpg\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/fb-meta-robots-tag-2.jpg\",\"width\":1920,\"height\":990},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"name\":\"SEO Blog by Ahrefs\",\"description\":\"Link Building Strategies &amp; SEO Tips\",\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\",\"name\":\"Ahrefs\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"width\":2048,\"height\":768,\"caption\":\"Ahrefs\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/Ahrefs\\\/\",\"https:\\\/\\\/x.com\\\/ahrefs\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/ahrefs\\\/\",\"https:\\\/\\\/www.youtube.com\\\/c\\\/ahrefscom\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/b66ba944c48e71d0abac1af2f11265ec\",\"name\":\"Michal Pec\u00e1nek\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/08c8f97481700ebb7fa875d1c0ce2a09bacba7ec4e37d19883377184bc111f03?s=96&d=mm&r=gd907208e0da9767ada4256cffbd6f311\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/08c8f97481700ebb7fa875d1c0ce2a09bacba7ec4e37d19883377184bc111f03?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/08c8f97481700ebb7fa875d1c0ce2a09bacba7ec4e37d19883377184bc111f03?s=96&d=mm&r=g\",\"caption\":\"Michal Pec\u00e1nek\"},\"description\":\"SaaS SEO consultant with over 6 years of experience. Before going freelance, Michal was an SEO &amp; Marketing Educator at Ahrefs\u2013creating content for the blog and managing a team of guest writers.\",\"sameAs\":[\"https:\\\/\\\/www.michalpecanek.com\\\/\",\"michalpecanek\",\"https:\\\/\\\/x.com\\\/michalpecanek\",\"joshua.hardwick@ahrefs.com\"],\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/author\\\/michal-pecanek\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Robots Meta Tag & X-Robots-Tag: Everything You Need to Know","description":"A robots meta tag is an HTML snippet that tells search engine robots what they can and cannot do on a certain page. Understanding this is the cornerstone of SEO.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ahrefs.com\/blog\/meta-robots\/","og_locale":"en_US","og_type":"article","og_title":"Robots Meta Tag & X-Robots-Tag: Everything You Need to Know","og_description":"Tell search engines where they can and can't go on your site.","og_url":"https:\/\/ahrefs.com\/blog\/meta-robots\/","og_site_name":"SEO Blog by Ahrefs","article_publisher":"https:\/\/www.facebook.com\/Ahrefs\/","article_published_time":"2019-10-08T22:22:25+00:00","article_modified_time":"2024-03-18T13:06:38+00:00","og_image":[{"width":1920,"height":990,"url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/fb-meta-robots-tag-2.jpg","type":"image\/jpeg"}],"author":"Michal Pec\u00e1nek","twitter_card":"summary_large_image","twitter_description":"Tell search engines where they can and can't go on your site.","twitter_creator":"@michalpecanek","twitter_site":"@ahrefs","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ahrefs.com\/blog\/meta-robots\/#article","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/meta-robots\/"},"author":{"name":"Michal Pec\u00e1nek","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/b66ba944c48e71d0abac1af2f11265ec"},"headline":"Robots Meta Tag &amp; X-Robots-Tag: Everything You Need to&nbsp;Know","datePublished":"2019-10-08T22:22:25+00:00","dateModified":"2024-03-18T13:06:38+00:00","mainEntityOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/meta-robots\/"},"wordCount":2730,"commentCount":0,"publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/meta-robots\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/fb-meta-robots-tag-2.jpg","articleSection":["Technical SEO"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/ahrefs.com\/blog\/meta-robots\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/ahrefs.com\/blog\/meta-robots\/","url":"https:\/\/ahrefs.com\/blog\/meta-robots\/","name":"Robots Meta Tag & X-Robots-Tag: Everything You Need to Know","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/meta-robots\/#primaryimage"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/meta-robots\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/fb-meta-robots-tag-2.jpg","datePublished":"2019-10-08T22:22:25+00:00","dateModified":"2024-03-18T13:06:38+00:00","description":"A robots meta tag is an HTML snippet that tells search engine robots what they can and cannot do on a certain page. Understanding this is the cornerstone of SEO.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ahrefs.com\/blog\/meta-robots\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/meta-robots\/#primaryimage","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/fb-meta-robots-tag-2.jpg","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/fb-meta-robots-tag-2.jpg","width":1920,"height":990},{"@type":"WebSite","@id":"https:\/\/ahrefs.com\/blog\/#website","url":"https:\/\/ahrefs.com\/blog\/","name":"SEO Blog by Ahrefs","description":"Link Building Strategies &amp; SEO Tips","publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ahrefs.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ahrefs.com\/blog\/#organization","name":"Ahrefs","url":"https:\/\/ahrefs.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","width":2048,"height":768,"caption":"Ahrefs"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Ahrefs\/","https:\/\/x.com\/ahrefs","https:\/\/www.linkedin.com\/company\/ahrefs\/","https:\/\/www.youtube.com\/c\/ahrefscom"]},{"@type":"Person","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/b66ba944c48e71d0abac1af2f11265ec","name":"Michal Pec\u00e1nek","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/08c8f97481700ebb7fa875d1c0ce2a09bacba7ec4e37d19883377184bc111f03?s=96&d=mm&r=gd907208e0da9767ada4256cffbd6f311","url":"https:\/\/secure.gravatar.com\/avatar\/08c8f97481700ebb7fa875d1c0ce2a09bacba7ec4e37d19883377184bc111f03?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/08c8f97481700ebb7fa875d1c0ce2a09bacba7ec4e37d19883377184bc111f03?s=96&d=mm&r=g","caption":"Michal Pec\u00e1nek"},"description":"SaaS SEO consultant with over 6 years of experience. Before going freelance, Michal was an SEO &amp; Marketing Educator at Ahrefs\u2013creating content for the blog and managing a team of guest writers.","sameAs":["https:\/\/www.michalpecanek.com\/","michalpecanek","https:\/\/x.com\/michalpecanek","joshua.hardwick@ahrefs.com"],"url":"https:\/\/ahrefs.com\/blog\/author\/michal-pecanek\/"}]}},"_links":{"self":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/30778","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/users\/149"}],"replies":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/comments?post=30778"}],"version-history":[{"count":0,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/30778\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media\/30794"}],"wp:attachment":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media?parent=30778"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/categories?post=30778"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/tags?post=30778"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/coauthors?post=30778"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}