{"id":30996,"date":"2019-10-15T15:51:14","date_gmt":"2019-10-15T23:51:14","guid":{"rendered":"https:\/\/ahrefs.com\/blog\/?p=30996"},"modified":"2025-12-18T08:37:53","modified_gmt":"2025-12-18T13:37:53","slug":"robots-txt","status":"publish","type":"post","link":"https:\/\/ahrefs.com\/blog\/robots-txt\/","title":{"rendered":"Robots.txt and SEO: Everything You Need to&nbsp;Know"},"content":{"rendered":"<div class=\"intro-txt\">Robots.txt is one of the simplest files on a website, but it\u2019s also one of the easiest to mess up. Just one character out of place can wreak havoc on your SEO and prevent search engines from accessing important content on your&nbsp;site.<\/div>\n<p>This is why robots.txt misconfigurations are extremely common\u2014even amongst experienced SEO professionals.<\/p>\n<p>In this guide, you\u2019ll learn:<\/p>\n<ul>\n<li><a href=\"#what-is-robots-txt\">What a robots.txt file&nbsp;is<\/a><\/li>\n<li><a href=\"#what-does-robots-txt-look-like\">What robots.txt looks&nbsp;like<\/a><\/li>\n<li><a href=\"#user-agents-directives\">Robots.txt user-agents and directives<\/a><\/li>\n<li><a href=\"#do-you-need-robots-txt\">Whether you need a robots.txt file<\/a><\/li>\n<li><a href=\"#how-to-find-robots-txt\">How to find your robots.txt file<\/a><\/li>\n<li><a href=\"#how-to-create-robots-txt\">How to create a robots.txt file<\/a><\/li>\n<li><a href=\"#robots-txt-best-practices\">Robots.txt best practices<\/a><\/li>\n<li><a href=\"#example-robots-txt\">Example robots.txt files<\/a><\/li>\n<li><a href=\"#robots-txt-errors\">How to audit your robots.txt file for issues<\/a><\/li>\n<\/ul>\n<div class=\"hub-link\"><img decoding=\"async\" alt=\"Beginner's guide to technical SEO\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/svg\/4.svg\"><div class=\"hl-title\">New to technical SEO? Check out&nbsp;our<\/div><div class=\"hl-content\"><a href=\"https:\/\/ahrefs.com\/blog\/technical-seo\/\" target=\"_blank\">Beginner\u2019s guide to technical SEO<\/a><\/div><\/div>\n<h2 id=\"what-is-robots-txt\">What is a robots.txt file?<\/h2>\n<p>A robots.txt file tells search engines where they can and can\u2019t go on your&nbsp;site.<\/p>\n<p>Primarily, it lists all the content you want to lock away from search engines like Google. You can also tell some search engines (not Google) <em>how<\/em>&nbsp;they can crawl allowed content.<\/p>\n<div class=\"recommendation\"><div class=\"recommendation-title\">important note<\/div><div class=\"recommendation-content\">\n<p>Most search engines are obedient. They aren\u2019t in the habit of breaking an entry. That said, some aren\u2019t shy about picking a few metaphorical locks.<\/p>\n<p>Google isn\u2019t one of those search engines. They obey the instructions in a robots.txt file.<\/p>\n<p>Just know that some search engines ignore it completely.<\/p>\n<\/div><\/div>\n<h2 id=\"what-does-robots-txt-look-like\">What does a robots.txt file look&nbsp;like?<\/h2>\n<p>Here\u2019s the basic format of a robots.txt file:<\/p>\n<pre>Sitemap: [URL location of sitemap]\n\nUser-agent: [bot identifier]\n[directive 1]\n[directive 2]\n[directive ...]\n\nUser-agent: [another bot identifier]\n[directive 1]\n[directive 2]\n[directive ...]\n<\/pre>\n<p>If you\u2019ve never seen one of these files before, that might seem daunting. However, the syntax is quite simple. In short, you assign rules to bots by stating their <em>user-agent<\/em>&nbsp;followed by <em>directives<\/em>.<\/p>\n<p>Let\u2019s explore these two components in more detail.<\/p>\n<h2 id=\"user-agents-directives\">User-agents<\/h2>\n<p>Each search engine identifies itself with a different user-agent. You can set custom instructions for each of these in your robots.txt file. There are <a href=\"http:\/\/www.user-agents.org\/\" target=\"_blank\" rel=\"noopener noreferrer\">hundreds of user-agents<\/a>, but here are some useful ones for&nbsp;SEO:<\/p>\n<ul>\n<li><strong>Google: <\/strong><a href=\"https:\/\/ahrefs.com\/blog\/googlebot\/\">Googlebot<\/a><\/li>\n<li><strong>Google Images:<\/strong>&nbsp;Googlebot-Image<\/li>\n<li><strong>Bing: <\/strong>Bingbot<\/li>\n<li><strong>Yahoo: <\/strong>Slurp<\/li>\n<li><strong>Baidu<\/strong>: Baiduspider<\/li>\n<li><strong>DuckDuckGo:<\/strong>&nbsp;DuckDuckBot<\/li>\n<\/ul>\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div>&nbsp;All user-agents are case sensitive in robots.txt.&nbsp;<\/div>\n<p>You can also use the star (*) wildcard to assign directives to all user-agents.<\/p>\n<p>For example, let\u2019s&nbsp;say that you wanted to block all bots except Googlebot from crawling your site. Here\u2019s how you\u2019d do&nbsp;it:<\/p>\n<pre>User-agent: *\nDisallow: \/\n\nUser-agent: Googlebot\nAllow: \/\n<\/pre>\n<p>Know that your robots.txt file can include&nbsp;directives for as many user-agents as you like. That said, every time you declare a new user-agent, it acts as a clean slate. In other words, if you add directives for multiple user-agents, the directives declared for the first user-agent don\u2019t apply to the second, or third, or fourth, and so&nbsp;on.<\/p>\n<p>The exception to that rule is when you declare the same user-agent more than once. In that case, all relevant directives are combined and followed.<\/p>\n<div class=\"recommendation\"><div class=\"recommendation-title\">IMPORTANT NOTE<\/div><div class=\"recommendation-content\">\n<p>Crawlers only follow the rules declared under the user-agent(s) that <strong>most accurately applies to them<\/strong>. That\u2019s why the robots.txt file above blocks all bots except Googlebot&nbsp;(and other Google bots) from crawling the site. Googlebot ignores the less specific user-agent declaration.<\/p>\n<\/div><\/div>\n<h2>Directives<\/h2>\n<p>Directives are rules that you want the declared user-agents to follow.<\/p>\n<h3>Supported directives<\/h3>\n<p>Here are directives that Google currently supports, along with their&nbsp;uses.<\/p>\n<h4 id=\"h.xeq6vut3hh39\">Disallow<\/h4>\n<p>Use this directive to instruct search engines not to access files and pages that fall under a specific path. For example, if you wanted to block all search engines from accessing your blog and all its posts, your robots.txt file might look like&nbsp;this:<\/p>\n<pre>User-agent: *\nDisallow: \/blog\n<\/pre>\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div>&nbsp;If you fail to define a path after the disallow directive, search engines will ignore it.&nbsp;<\/div>\n<h4 id=\"h.u53y90qh3toz\">Allow<\/h4>\n<p>Use this directive to allow search engines to crawl a subdirectory or page\u2014even in an otherwise disallowed directory. For example, if you wanted to prevent search engines from accessing every post on your blog except for one, then your robots.txt file might look like&nbsp;this:<\/p>\n<pre>User-agent: *\nDisallow: \/blog\nAllow: \/blog\/allowed-post\n<\/pre>\n<p>In this example, search engines can access <code>\/blog\/allowed-post<\/code>. But they can\u2019t access:<\/p>\n<p><code>\/blog\/another-post<\/code><br>\n<code>\/blog\/yet-another-post<\/code><br>\n<code>\/blog\/download-me.pdf<\/code><\/p>\n<p>Both Google and Bing&nbsp;support this directive.<\/p>\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div>&nbsp;As with the disallow directive, if you fail to define a path after the allow directive, search engines will ignore it.&nbsp;<\/div>\n<div class=\"recommendation\"><div class=\"recommendation-title\">a note about conflicting rules<\/div><div class=\"recommendation-content\">\n<p>Unless you\u2019re careful, disallow and allow directives can easily conflict with one another. In the example below, we\u2019re disallowing access to <code>\/blog\/<\/code>&nbsp;and allowing access to <code>\/blog<\/code>.<\/p>\n<pre>User-agent: *\nDisallow: \/blog\/\nAllow: \/blog\n<\/pre>\n<p>In this case, the URL <code>\/blog\/post-title\/<\/code>&nbsp;seems to be both disallowed and allowed. So which&nbsp;wins?<\/p>\n<p>For Google and Bing, the rule is that the directive with the most characters wins.&nbsp;Here, that\u2019s the disallow directive.<\/p>\n<p><code>Disallow: \/blog\/<\/code> <em>(6 characters)<\/em><br>\n<code>Allow: \/blog<\/code>&nbsp;<em>(5 characters)<\/em><\/p>\n<p>If the allow and disallow directives are equal in length, then the least restrictive directive wins. In this case, that would be the allow directive.<\/p>\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div>&nbsp;Here, <code>\/blog<\/code>&nbsp;(without the <a href=\"https:\/\/ahrefs.com\/blog\/trailing-slash\/\">trailing slash<\/a>) is still accessible and crawlable.&nbsp;<\/div>\n<p>Crucially, <strong>this is only the case for Google and Bing<\/strong>. Other search engines listen to the first matching directive. In this case, that\u2019s disallow.<\/p>\n<\/div><\/div>\n<h4 id=\"h.bcoscz7v58m1\">Sitemap<\/h4>\n<p>Use this directive to specify the location of your sitemap(s) to search engines. If you\u2019re unfamiliar with sitemaps, they generally include the pages that you want search engines to crawl and&nbsp;index.<\/p>\n<p>Here\u2019s an example of a robots.txt file using the sitemap directive:<\/p>\n<pre>Sitemap: https:\/\/www.domain.com\/sitemap.xml\n\nUser-agent: *\nDisallow: \/blog\/\nAllow: \/blog\/post-title\/\n<\/pre>\n<p>How important is including your sitemap(s) in your robots.txt file? If you\u2019ve already submitted through Search Console, then it\u2019s somewhat redundant for Google. However, it does tell other search engines like Bing where to find your sitemap, so it\u2019s still good practice.<\/p>\n<p>Note that you don\u2019t need to repeat the sitemap directive multiple times for each user-agent. It doesn\u2019t apply to only one. &nbsp;So you\u2019re best to include sitemap directives at the beginning or end of your robots.txt file. For example:<\/p>\n<pre>Sitemap: https:\/\/www.domain.com\/sitemap.xml\n\nUser-agent: Googlebot\nDisallow: \/blog\/\nAllow: \/blog\/post-title\/\n\nUser-agent: Bingbot\nDisallow: \/services\/\n<\/pre>\n<p>Google <a href=\"https:\/\/developers.google.com\/search\/reference\/robots_txt\" target=\"_blank\" rel=\"noopener noreferrer\">supports<\/a>&nbsp;the sitemap directive, as do Ask, Bing, and&nbsp;Yahoo.<\/p>\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div>&nbsp;You can include as many sitemaps as you like in your robots.txt file.&nbsp;<\/div>\n<h3>Unsupported directives<\/h3>\n<p>Here are the directives that are <a href=\"https:\/\/webmasters.googleblog.com\/2019\/07\/a-note-on-unsupported-rules-in-robotstxt.html\" target=\"_blank\" rel=\"noopener noreferrer\">no longer supported by Google<\/a>\u2014some of which technically never&nbsp;were.<\/p>\n<h4 id=\"h.nh0j3m4oqdg5\">Crawl-delay<\/h4>\n<p>Previously, you could use this directive to specify a crawl delay in seconds. For example, if you wanted Googlebot to wait 5 seconds after each crawl action, you\u2019d set the crawl-delay to 5 like&nbsp;so:<\/p>\n<pre>User-agent: Googlebot\nCrawl-delay: 5\n<\/pre>\n<p>Google no longer supports this directive, but <a href=\"https:\/\/blogs.bing.com\/webmaster\/2012\/05\/03\/to-crawl-or-not-to-crawl-that-is-bingbots-question\/\" target=\"_blank\" rel=\"noopener noreferrer\">Bing<\/a>&nbsp;and <a href=\"https:\/\/yandex.com\/support\/webmaster\/controlling-robot\/robots-txt.html#crawl-delay\" target=\"_blank\" rel=\"noopener noreferrer\">Yandex<\/a>&nbsp;do.<\/p>\n<p>That said, be careful when setting this directive, especially if you have a big site. If you set a crawl-delay of 5 seconds, then you\u2019re limiting bots to crawl a maximum of 17,280 URLs a day. That\u2019s not very helpful if you have millions of pages, but it could save bandwidth if you have a small website.<\/p>\n<h4 id=\"h.adsbtb5ez6n8\">Noindex<\/h4>\n<p>This directive was never officially supported by Google. However, until recently, it\u2019s thought that Google had some \u201ccode that handles unsupported and unpublished rules (such as noindex).\u201d So if&nbsp;you wanted to prevent Google from indexing all posts on your blog, you could use the following directive:<\/p>\n<pre>User-agent: Googlebot\nNoindex: \/blog\/\n<\/pre>\n<p>However, on September 1st, 2019, Google made it clear that <strong>this directive is not supported<\/strong>. If you want to exclude a page or file from search engines, use the meta robots tag or x-robots HTTP header&nbsp;instead.<\/p>\n<h4 id=\"h.8cnsyktzcwz2\">Nofollow<\/h4>\n<p>This is another directive that Google never officially supported, and was used to instruct search engines not to follow links on pages and files under a specific path. For example, if you wanted to stop Google from following all links on your blog, you could use the following directive:<\/p>\n<pre>User-agent: Googlebot\nNofollow: \/blog\/\n<\/pre>\n<p>Google announced that this directive is officially unsupported on September 1st, 2019. If you want to nofollow all links on a page now, you should use the robots meta tag or x-robots header. If you want to tell Google not to follow specific links on a page, use the rel=\u201cnofollow\u201d link attribute.<\/p>\n<h2 id=\"do-you-need-robots-txt\">Do you need a robots.txt file?<\/h2>\n<p>Having a robots.txt file isn\u2019t crucial for a lot of websites, especially small&nbsp;ones.<\/p>\n<p>That said, there\u2019s no good reason not to have one. It gives you more control over where search engines can and can\u2019t go on your website, and that can help&nbsp;with things like:<\/p>\n<ul>\n<li>Preventing the crawling of <a href=\"https:\/\/ahrefs.com\/blog\/duplicate-content\/\" target=\"_blank\" rel=\"noopener noreferrer\">duplicate content<\/a>;<\/li>\n<li>Keeping sections of a website private (e.g., your staging site);<\/li>\n<li>Preventing the crawling of internal search results pages;<\/li>\n<li>Preventing server overload;<\/li>\n<li>Preventing Google from wasting \u201c<a href=\"https:\/\/ahrefs.com\/blog\/crawl-budget\/\" target=\"_blank\" rel=\"noopener noreferrer\">crawl budget<\/a>.\u201d<\/li>\n<li>Preventing <a href=\"https:\/\/support.google.com\/webmasters\/answer\/35308\" target=\"_blank\" rel=\"noopener noreferrer\">images<\/a>, <a href=\"https:\/\/support.google.com\/webmasters\/answer\/156442#block_video\" target=\"_blank\" rel=\"noopener noreferrer\">videos<\/a>,&nbsp;and resources files from appearing in Google search results.<\/li>\n<\/ul>\n<p>Note that while Google doesn\u2019t typically index web pages that are blocked in robots.txt, <strong>there\u2019s no way to guarantee exclusion from search results using the robots.txt file<\/strong>.<\/p>\n<p>Like <a href=\"https:\/\/support.google.com\/webmasters\/answer\/6062608?hl=en\" target=\"_blank\" rel=\"noopener noreferrer\">Google says<\/a>, if content is linked to from other places on the web, it may still appear in Google search results.<\/p>\n<h2 id=\"how-to-find-robots-txt\">How to find your robots.txt file<\/h2>\n<p>If you already have a robots.txt file on your website, it\u2019ll be accessible at <em>domain.com\/robots.txt<\/em>. Navigate to the URL in your browser. If you see something like this, then you have a robots.txt file:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"249\" height=\"131\" class=\"wp-image-30989\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/ahrefs-robots-5.png\" alt=\"ahrefs robots 5\"><\/p>\n<h2 id=\"how-to-create-robots-txt\">How to create a robots.txt file<\/h2>\n<p>If you don\u2019t already have a robots.txt file, creating one is easy. Just open a blank .txt document and begin typing directives. For example, if you wanted to disallow all search engines from crawling your <code>\/admin\/<\/code> directory, it would look something like&nbsp;this:<\/p>\n<pre>User-agent: *\nDisallow: \/admin\/\n<\/pre>\n<p>Continue to build up the directives until you\u2019re happy with what you have. Save your file as \u201crobots.txt.\u201d<\/p>\n<p>Alternatively, you can also use a robots.txt generator like <a href=\"http:\/\/tools.seobook.com\/robots-txt\/generator\/\" target=\"_blank\" rel=\"noopener noreferrer\">this one<\/a>.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"722\" height=\"326\" class=\"wp-image-30991\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/robots-creator-3.png\" alt=\"robots creator 3\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/robots-creator-3.png 722w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/robots-creator-3-680x307.png 680w\" sizes=\"auto, (max-width: 722px) 100vw, 722px\"><\/p>\n<p>The advantage of using a tool like this is that it minimizes syntax errors. That\u2019s good because one mistake could result in an SEO catastrophe for your site\u2014so it pays to err on the side of caution.<\/p>\n<p>The disadvantage is that they\u2019re somewhat limited in terms of customizability.<\/p>\n<h2>Where to put your robots.txt file<\/h2>\n<p>Place your robots.txt file in the root directory of the subdomain to which it applies. For example, to control crawling behavior on <em>domain.com<\/em>, the robots.txt file should be accessible at <em>domain.com\/robots.txt<\/em>.<\/p>\n<p>If you want to control crawling on a subdomain like<em>&nbsp;blog.domain.com<\/em>, then the robots.txt file should be accessible at <em>blog.domain.com\/robots.txt<\/em>.<\/p>\n<h2 id=\"robots-txt-best-practices\">Robots.txt file best practices<\/h2>\n<p>Keep these in mind to avoid common mistakes.<\/p>\n<h3>Use a new line for each directive<\/h3>\n<p>Each directive should sit on a new line. Otherwise, it\u2019ll confuse search engines.<\/p>\n<p><strong>Bad: <\/strong><\/p>\n<pre>User-agent: * Disallow: \/directory\/ Disallow: \/another-directory\/\n<\/pre>\n<p><strong>Good: <\/strong><\/p>\n<pre>User-agent: * \nDisallow: \/directory\/ \nDisallow: \/another-directory\/\n<\/pre>\n<h3>Use wildcards to simplify instructions<\/h3>\n<p>Not only can you use wildcards (*) to apply directives to all user-agents, but also to match URL patterns when declaring directives. For example, if you wanted to prevent search engines from accessing <a href=\"https:\/\/ahrefs.com\/blog\/url-parameters\/\">parameterized<\/a> product category URLs on your site, you could list them out like&nbsp;this:<\/p>\n<pre>User-agent: * \nDisallow: \/products\/t-shirts?\nDisallow: \/products\/hoodies?\nDisallow: \/products\/jackets?\n\u2026\n<\/pre>\n<p>But that\u2019s not very efficient. It would be better to simplify things with a wildcard like&nbsp;this:<\/p>\n<pre>User-agent: * \nDisallow: \/products\/*?\n<\/pre>\n<p>This example blocks search engines from crawling all URLs under the \/product\/ subfolder that contain a question mark. In other words, any parameterized product category URLs.<\/p>\n<h3>Use \u201c$\u201d to specify the end of a&nbsp;URL<\/h3>\n<p>Include the \u201c$\u201d symbol to mark the end of a URL. For example, if you wanted to prevent search engines accessing all .pdf files on your site, your robots.txt file might look like&nbsp;this:<\/p>\n<pre>User-agent: * \nDisallow: \/*.pdf$\n<\/pre>\n<p>In this example, search engines can\u2019t access any URLs ending with .pdf. That means they can\u2019t access \/file.pdf, but they can access \/file.pdf?id=68937586 because that doesn\u2019t end with \u201c.pdf\u201d.<\/p>\n<h3>Use&nbsp;each user-agent only&nbsp;once<\/h3>\n<p>If you specify the same user-agent multiple times, Google doesn\u2019t mind. It will merely combine all rules from the various declarations into one and follow them all. For example, if you had the following user-agents and directives in your robots.txt file\u2026<\/p>\n<pre>User-agent: Googlebot\nDisallow: \/a\/\n\nUser-agent: Googlebot \nDisallow: \/b\/\n<\/pre>\n<p>\u2026 Googlebot <strong>wouldn\u2019t<\/strong>&nbsp;crawl either of those subfolders.<\/p>\n<p>That said, it makes sense to declare each user-agent only once because it\u2019s less confusing. In other words, you\u2019re less likely to make critical mistakes by keeping things neat and simple.<\/p>\n<h3>Use specificity to avoid unintentional errors<\/h3>\n<p>Failure to provide specific instructions when setting directives can result in easily-missed mistakes that can have a catastrophic impact on your SEO. For example, let\u2019s assume that you have a multilingual site, and you\u2019re working on a German version that will be available under the \/de\/ subdirectory.<\/p>\n<p>Because it isn\u2019t quite ready to go, you want to prevent search engines from accessing it.<\/p>\n<p>The robots.txt file below will&nbsp;prevent search engines from accessing that subfolder and everything in&nbsp;it:<\/p>\n<pre>User-agent: *\nDisallow: \/de\n<\/pre>\n<p>But it will also prevent search engines from crawling of any pages or files beginning with&nbsp;<code>\/de<\/code>.<\/p>\n<p>For example:<\/p>\n<p><code>\/designer-dresses\/<\/code><br>\n<code>\/delivery-information.html<\/code><br>\n<code>\/depeche-mode\/t-shirts\/<\/code><br>\n<code>\/definitely-not-for-public-viewing.pdf<\/code><\/p>\n<p>In this instance, the solution is simple: add a trailing slash.<\/p>\n<pre>User-agent: *\nDisallow: \/de\/\n<\/pre>\n<h3>Use comments to explain your robots.txt file to humans<\/h3>\n<p>Comments help explain your robots.txt file to developers\u2014and potentially even your future self. To include a comment, begin the line with a&nbsp;hash&nbsp;(#).<\/p>\n<pre># This instructs Bing not to crawl our site.\nUser-agent: Bingbot\nDisallow: \/\n<\/pre>\n<p>Crawlers will ignore everything on lines that start with a&nbsp;hash.<\/p>\n<h3>Use a separate robots.txt file for each subdomain<\/h3>\n<p>Robots.txt only controls crawling behavior on the subdomain where it\u2019s hosted. If you want to control crawling on a different subdomain, you\u2019ll need a separate robots.txt file.<\/p>\n<p>For example, if your main site sits on <em>domain.com<\/em>&nbsp;and your blog sits on <em>blog.domain.com<\/em>, then you would need two robots.txt files. One should go in the root directory of the main domain, and the other in the root directory of the&nbsp;blog.<\/p>\n<h2 id=\"example-robots-txt\">Example robots.txt files<\/h2>\n<p>Below are a few examples of robots.txt files. These are mainly for inspiration but if one happens to match your requirements, copy-paste it into a text document, save it as \u201crobots.txt\u201d and upload it to the appropriate directory.<\/p>\n<h3>All-Access for all&nbsp;bots<\/h3>\n<pre>User-agent: *\nDisallow:\n<\/pre>\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div> Failing to declare a URL after a directive renders that directive redundant. In other words, search engines ignore it. That\u2019s why this disallow directive has no effect on the site. Search engines can still crawl all pages and&nbsp;files.&nbsp;<\/div>\n<h3>No access for all&nbsp;bots<\/h3>\n<pre>User-agent: *\nDisallow: \/\n<\/pre>\n<h3>Block one subdirectory for all&nbsp;bots<\/h3>\n<pre>User-agent: *\nDisallow: \/folder\/\n<\/pre>\n<h3>Block one subdirectory for all bots (with one file within allowed)<\/h3>\n<pre>User-agent: *\nDisallow: \/folder\/\nAllow: \/folder\/page.html\n<\/pre>\n<h3>Block one file for all&nbsp;bots<\/h3>\n<pre>User-agent: *\nDisallow: \/this-is-a-file.pdf\n<\/pre>\n<h3>Block one filetype (PDF) for all&nbsp;bots<\/h3>\n<pre>User-agent: *\nDisallow: \/*.pdf$\n<\/pre>\n<h3>Block all parameterized URLs for Googlebot only<\/h3>\n<pre>User-agent: Googlebot\nDisallow: \/*?\n<\/pre>\n<h2 id=\"robots-txt-errors\">How to audit your robots.txt file for errors<\/h2>\n<p>Robots.txt mistakes can slip through the net fairly easily, so it pays to keep an eye out for issues.<\/p>\n<p>To do this, regularly check for issues related to robots.txt in the \u201cCoverage\u201d report in <a href=\"https:\/\/search.google.com\/search-console\/about\" target=\"_blank\" rel=\"noopener noreferrer\">Search Console<\/a>. Below are some of the errors you might see, what they mean, and how you might fix&nbsp;them.<\/p>\n<div class=\"recommendation\"><div class=\"recommendation-title\">Need to check for errors related to a certain page?<\/div><div class=\"recommendation-content\">\n<p>Paste a URL into Google\u2019s URL Inspection tool in Search Console. If it\u2019s blocked by robots.txt, you should see something like&nbsp;this:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"752\" height=\"370\" class=\"wp-image-30995\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/blocked-by-robots-txt-7.png\" alt=\"blocked by robots txt 7\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/blocked-by-robots-txt-7.png 752w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/blocked-by-robots-txt-7-680x335.png 680w\" sizes=\"auto, (max-width: 752px) 100vw, 752px\"><\/p>\n<\/div><\/div>\n<h3>Submitted URL blocked by robots.txt<\/h3>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"365\" height=\"101\" class=\"wp-image-30993\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/submitted-url-blocked-by-robots-3.png\" alt=\"submitted url blocked by robots 3\"><\/p>\n<p>This means that at least one of the URLs in your submitted sitemap(s) are blocked by robots.txt.<\/p>\n<p>If you <a href=\"https:\/\/ahrefs.com\/blog\/how-to-create-a-sitemap\/\">created your sitemap correctly<\/a>&nbsp;and excluded <a href=\"https:\/\/ahrefs.com\/blog\/canonical-tags\/\">canonicalized<\/a>, noindexed, and <a href=\"https:\/\/ahrefs.com\/blog\/301-redirects\/\">redirected<\/a>&nbsp;pages, then <strong>no submitted pages should be blocked by robots.txt<\/strong>. If they are, investigate which pages are affected, then adjust your robots.txt file accordingly to remove the block for that&nbsp;page.<\/p>\n<p>You can use <a href=\"https:\/\/support.google.com\/webmasters\/answer\/6062598?hl=en\" target=\"_blank\" rel=\"noopener noreferrer\">Google\u2019s robots.txt tester<\/a>&nbsp;to see which directive is blocking the content. Just be careful when doing this. It\u2019s easy to make mistakes that affect other pages and&nbsp;files.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"734\" height=\"558\" class=\"wp-image-30994\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/robots-txt-tester-3.png\" alt=\"robots txt tester 3\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/robots-txt-tester-3.png 734w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/robots-txt-tester-3-559x425.png 559w\" sizes=\"auto, (max-width: 734px) 100vw, 734px\"><\/p>\n<h3>Blocked by robots.txt<\/h3>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"355\" height=\"97\" class=\"wp-image-30990\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/blocked-by-robots-txt-6.png\" alt=\"blocked by robots txt 6\"><\/p>\n<p>This means you have content blocked by robots.txt that isn\u2019t currently indexed in Google.<\/p>\n<p>If this content is important and should be indexed, remove the crawl block in robots.txt. (It\u2019s also worth making sure that the content isn\u2019t noindexed). If you\u2019ve blocked content in robots.txt with the intention of excluding it from Google\u2019s index, remove the crawl block and use a robots meta tag or x-robots-header instead. That\u2019s the only way to guarantee the exclusion of content from Google\u2019s index.<\/p>\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div>&nbsp;Removing the crawl block when attempting to exclude a page from the search results is crucial. Fail to do this, and Google won\u2019t see the noindex tag or HTTP header\u2014so it will stay indexed.&nbsp;<\/div>\n<h3>Indexed, though blocked by robots.txt<\/h3>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"379\" height=\"99\" class=\"wp-image-30992\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/indexed-though-blocked-3.png\" alt=\"indexed though blocked 3\"><\/p>\n<p>This means that some of the content blocked by robots.txt is still indexed in Google.<\/p>\n<p>Once again, if you\u2019re trying to exclude this content from Google\u2019s search results, robots.txt isn\u2019t the correct solution. Remove the crawl block and instead use a <a href=\"https:\/\/ahrefs.com\/blog\/meta-robots\/\">meta robots tag or x-robots-tag HTTP header<\/a>&nbsp;to prevent indexing.<\/p>\n<p>If you blocked this content by accident and want to keep it in Google\u2019s index, remove the crawl block in robots.txt. This may help to improve the visibility of the content in Google search.<\/p>\n<p><strong>Recommended reading:<\/strong> <em><a href=\"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/\">How to Fix \u201cindexed, though blocked by robots.txt\u201d in&nbsp;GSC<\/a><\/em><\/p>\n<h2>FAQs<\/h2>\n<p>Here are a few frequently asked questions that didn\u2019t fit naturally elsewhere in our guide. Let us know in the comments if anything is missing, and we\u2019ll update the section accordingly.<\/p>\n<h3>What\u2019s the maximum size of a robots.txt file?<\/h3>\n<p><a href=\"https:\/\/developers.google.com\/search\/reference\/robots_txt\" target=\"_blank\" rel=\"noopener noreferrer\">500 kilobytes<\/a>&nbsp;(roughly).<\/p>\n<h3>Where is robots.txt in WordPress?<\/h3>\n<p>Same place: <em>domain.com\/robots.txt<\/em>.<\/p>\n<h3>How do I edit robots.txt in WordPress?<\/h3>\n<p>Either manually, or using one of <a href=\"https:\/\/ahrefs.com\/blog\/best-seo-plugins-for-wordpress\/\">the many WordPress SEO plugins<\/a>&nbsp;like Yoast that let you edit robots.txt from the WordPress backend.<\/p>\n<h3>What happens if I disallow access to noindexed content in robots.txt?<\/h3>\n<p>Google will never see the noindex directive because it can\u2019t crawl the&nbsp;page.<\/p>\n<blockquote class=\"twitter-tweet\">\n<p dir=\"ltr\" lang=\"en\">DYK blocking a page with both a robots.txt disallow &amp; a noindex in the page doesn\u2019t make much sense cos Googlebot can\u2019t \u201csee\u201d the noindex? <a href=\"https:\/\/t.co\/N4639rCCWt\">pic.twitter.com\/N4639rCCWt<\/a>\u2014 Gary \u201c\u9be8\u7406\u201d Illyes (@methode) <a href=\"https:\/\/twitter.com\/methode\/status\/829952819373494272?ref_src=twsrc%5Etfw\">February 10,&nbsp;2017<\/a><\/p>\n<\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<h2>Final thoughts<\/h2>\n<p>Robots.txt is a simple but powerful file. Use it wisely, and it can have a positive impact on SEO. Use it haphazardly and, well, you\u2019ll live to regret it.<\/p>\n<p>Got more questions? Leave a comment or <a href=\"https:\/\/twitter.com\/joshuachardwick?lang=en\" target=\"_blank\" rel=\"noopener noreferrer\">ping me on Twitter<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is why robots.txt misconfigurations are extremely common\u2014even amongst experienced SEO professionals. In this guide, you\u2019ll learn: What a robots.txt file&nbsp;is What robots.txt looks&nbsp;like Robots.txt user-agents and directives Whether you need a robots.txt file How to find your robots.txt file<span class=\"ellipsis\">\u2026<\/span><\/p>\n<div class=\"read-more\">Read more \u203a<\/div>\n<p><!-- end of .read-more --><\/p>\n","protected":false},"author":114,"featured_media":31008,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"wp_typography_post_enhancements_disabled":false,"footnotes":""},"categories":[329],"tags":[],"coauthors":[336],"class_list":["post-30996","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technical-seo","odd"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.3 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Robots.txt and SEO: Everything You Need to Know<\/title>\n<meta name=\"description\" content=\"Robots.txt file tells search engines where they can and can\u2019t go on your site. It also controls how they can crawl allowed content.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ahrefs.com\/blog\/robots-txt\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Robots.txt and SEO: Everything You Need to Know\" \/>\n<meta property=\"og:description\" content=\"Learn how to avoid common robots.txt misconfigurations that can wreak SEO havoc.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ahrefs.com\/blog\/robots-txt\/\" \/>\n<meta property=\"og:site_name\" content=\"SEO Blog by Ahrefs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Ahrefs\/\" \/>\n<meta property=\"article:published_time\" content=\"2019-10-15T23:51:14+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-12-18T13:37:53+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/fb-robots-txt-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"990\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Joshua Hardwick\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Robots.txt and SEO: Everything You Need to Know\" \/>\n<meta name=\"twitter:description\" content=\"Learn how to avoid common robots.txt misconfigurations that can wreak SEO havoc.\" \/>\n<meta name=\"twitter:creator\" content=\"@JoshuaCHardwick\" \/>\n<meta name=\"twitter:site\" content=\"@ahrefs\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/robots-txt\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/robots-txt\\\/\"},\"author\":{\"name\":\"Joshua Hardwick\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/e6a89cbde8e750d22996aa26e213e712\"},\"headline\":\"Robots.txt and SEO: Everything You Need to&nbsp;Know\",\"datePublished\":\"2019-10-15T23:51:14+00:00\",\"dateModified\":\"2025-12-18T13:37:53+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/robots-txt\\\/\"},\"wordCount\":3018,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/robots-txt\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/fb-robots-txt-1.png\",\"articleSection\":[\"Technical SEO\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/ahrefs.com\\\/blog\\\/robots-txt\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/robots-txt\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/robots-txt\\\/\",\"name\":\"Robots.txt and SEO: Everything You Need to Know\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/robots-txt\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/robots-txt\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/fb-robots-txt-1.png\",\"datePublished\":\"2019-10-15T23:51:14+00:00\",\"dateModified\":\"2025-12-18T13:37:53+00:00\",\"description\":\"Robots.txt file tells search engines where they can and can\u2019t go on your site. It also controls how they can crawl allowed content.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ahrefs.com\\\/blog\\\/robots-txt\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/robots-txt\\\/#primaryimage\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/fb-robots-txt-1.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/fb-robots-txt-1.png\",\"width\":1920,\"height\":990},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"name\":\"SEO Blog by Ahrefs\",\"description\":\"Link Building Strategies &amp; SEO Tips\",\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\",\"name\":\"Ahrefs\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"width\":2048,\"height\":768,\"caption\":\"Ahrefs\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/Ahrefs\\\/\",\"https:\\\/\\\/x.com\\\/ahrefs\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/ahrefs\\\/\",\"https:\\\/\\\/www.youtube.com\\\/c\\\/ahrefscom\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/e6a89cbde8e750d22996aa26e213e712\",\"name\":\"Joshua Hardwick\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/meme.jpg109e89523fcea81015d3cc08c79f9036\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/meme.jpg\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/meme.jpg\",\"caption\":\"Joshua Hardwick\"},\"description\":\"Head of Content @ Ahrefs (or, in plain English, I'm the guy responsible for ensuring that every blog post we publish is EPIC).\",\"sameAs\":[\"https:\\\/\\\/x.com\\\/JoshuaCHardwick\"],\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/author\\\/joshua-hardwick\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Robots.txt and SEO: Everything You Need to Know","description":"Robots.txt file tells search engines where they can and can\u2019t go on your site. It also controls how they can crawl allowed content.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ahrefs.com\/blog\/robots-txt\/","og_locale":"en_US","og_type":"article","og_title":"Robots.txt and SEO: Everything You Need to Know","og_description":"Learn how to avoid common robots.txt misconfigurations that can wreak SEO havoc.","og_url":"https:\/\/ahrefs.com\/blog\/robots-txt\/","og_site_name":"SEO Blog by Ahrefs","article_publisher":"https:\/\/www.facebook.com\/Ahrefs\/","article_published_time":"2019-10-15T23:51:14+00:00","article_modified_time":"2025-12-18T13:37:53+00:00","og_image":[{"width":1920,"height":990,"url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/fb-robots-txt-1.png","type":"image\/png"}],"author":"Joshua Hardwick","twitter_card":"summary_large_image","twitter_title":"Robots.txt and SEO: Everything You Need to Know","twitter_description":"Learn how to avoid common robots.txt misconfigurations that can wreak SEO havoc.","twitter_creator":"@JoshuaCHardwick","twitter_site":"@ahrefs","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ahrefs.com\/blog\/robots-txt\/#article","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/robots-txt\/"},"author":{"name":"Joshua Hardwick","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/e6a89cbde8e750d22996aa26e213e712"},"headline":"Robots.txt and SEO: Everything You Need to&nbsp;Know","datePublished":"2019-10-15T23:51:14+00:00","dateModified":"2025-12-18T13:37:53+00:00","mainEntityOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/robots-txt\/"},"wordCount":3018,"commentCount":0,"publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/robots-txt\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/fb-robots-txt-1.png","articleSection":["Technical SEO"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/ahrefs.com\/blog\/robots-txt\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/ahrefs.com\/blog\/robots-txt\/","url":"https:\/\/ahrefs.com\/blog\/robots-txt\/","name":"Robots.txt and SEO: Everything You Need to Know","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/robots-txt\/#primaryimage"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/robots-txt\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/fb-robots-txt-1.png","datePublished":"2019-10-15T23:51:14+00:00","dateModified":"2025-12-18T13:37:53+00:00","description":"Robots.txt file tells search engines where they can and can\u2019t go on your site. It also controls how they can crawl allowed content.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ahrefs.com\/blog\/robots-txt\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/robots-txt\/#primaryimage","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/fb-robots-txt-1.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/fb-robots-txt-1.png","width":1920,"height":990},{"@type":"WebSite","@id":"https:\/\/ahrefs.com\/blog\/#website","url":"https:\/\/ahrefs.com\/blog\/","name":"SEO Blog by Ahrefs","description":"Link Building Strategies &amp; SEO Tips","publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ahrefs.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ahrefs.com\/blog\/#organization","name":"Ahrefs","url":"https:\/\/ahrefs.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","width":2048,"height":768,"caption":"Ahrefs"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Ahrefs\/","https:\/\/x.com\/ahrefs","https:\/\/www.linkedin.com\/company\/ahrefs\/","https:\/\/www.youtube.com\/c\/ahrefscom"]},{"@type":"Person","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/e6a89cbde8e750d22996aa26e213e712","name":"Joshua Hardwick","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/meme.jpg109e89523fcea81015d3cc08c79f9036","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/meme.jpg","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/10\/meme.jpg","caption":"Joshua Hardwick"},"description":"Head of Content @ Ahrefs (or, in plain English, I'm the guy responsible for ensuring that every blog post we publish is EPIC).","sameAs":["https:\/\/x.com\/JoshuaCHardwick"],"url":"https:\/\/ahrefs.com\/blog\/author\/joshua-hardwick\/"}]}},"as_json":null,"json_reviewers":[149],"_links":{"self":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/30996","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/users\/114"}],"replies":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/comments?post=30996"}],"version-history":[{"count":0,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/30996\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media\/31008"}],"wp:attachment":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media?parent=30996"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/categories?post=30996"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/tags?post=30996"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/coauthors?post=30996"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}