{"id":131285,"date":"2021-01-28T11:39:20","date_gmt":"2021-01-28T16:39:20","guid":{"rendered":"https:\/\/ahrefs.com\/blog\/?p=131285"},"modified":"2025-01-02T15:57:11","modified_gmt":"2025-01-02T20:57:11","slug":"indexed-though-blocked-by-robots-txt","status":"publish","type":"post","link":"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/","title":{"rendered":"\u201cIndexed, though blocked by robots.txt\u201d Can Be More Than A Robots.txt Block"},"content":{"rendered":"
\u201cIndexed, though blocked by robots.txt\u201d tells you that Google has indexed URLs that you blocked them from crawling using the robots.txt file on your website.<\/div>\n\n\n\n

In most cases, this will be a straightforward issue where you blocked crawling in your robots.txt file. But there are a few additional conditions that can trigger the problem, so let\u2019s go through the following troubleshooting process to diagnose and fix things as efficiently as possible.<\/p>\n

How to fix \u2018Indexed, though blocked by robots.txt\u2019<\/h2>\n\n\n\n
\"How<\/figure>\n\n\n\n

You can see that the first step is to ask yourself whether you want Google to index the URL.<\/p>\n\n\n\n

If you don\u2019t want the URL indexed\u2026<\/h2>\n\n\n\n

Just add a noindex meta robots tag<\/a> and make sure to allow crawling\u2014assuming it\u2019s canonical.<\/p>\n\n\n\n

If you block a page from being crawled, Google may still index it because crawling and indexing are two different things. Unless Google can crawl a page, they won\u2019t see the noindex meta tag and may still index it because it has links.<\/p>\n\n\n\n

If the URL canonicalizes to another page, don\u2019t add a noindex meta robots tag. Just make sure proper canonicalization signals<\/a> are in place, including a canonical tag<\/a> on the canonical page, and allow crawling so signals pass and consolidate correctly.<\/p>\n\n\n\n

If you do want the URL indexed\u2026<\/h2>\n\n\n\n

You need to figure out why Google can\u2019t crawl the URL and remove the block.<\/p>\n\n\n\n

The most likely cause is a crawl block in robots.txt<\/a>. But there are a few other scenarios where you may see messages saying that you\u2019re blocked. Let\u2019s go through these in the order you should probably be looking for them.<\/p>\n\n\n\n

    \n
  1. Check for a crawl block in robots.txt<\/a><\/li>\n
  2. Check for intermittent blocks<\/a><\/li>\n
  3. Check for a user-agent block<\/a><\/li>\n
  4. Check for an IP block<\/a><\/li>\n<\/ol>\n\n\n\n

    Check for a crawl block in robots.txt<\/h3>\n\n\n\n

    The easiest way to see the issue is with the robots.txt tester in GSC<\/a>, which will flag the blocking rule.

    \"Fixing<\/p>\n\n\n\n

    If you know what you\u2019re looking for or you don\u2019t have access to GSC, you can navigate to domain.com\/robots.txt to find the file. We have more information in our robots.txt article<\/a>, but you\u2019re likely looking for a disallow statement like:<\/p>\n\n\n\n

    Disallow: \/<\/pre>\n\n\n\n
    There may be a specific user-agent mentioned, or it may block everyone. If your site is new or has recently launched, you may want to look for:<\/p>\n\n\n\n
    User-agent: *
    Disallow: \/<\/pre>\n\n\n
    Can\u2019t find an issue?<\/div>
    \n
    It\u2019s possible that someone already fixed the robots.txt block and resolved the issue before you\u2019re looking into the issue. That\u2019s the best-case scenario. However, if the problem appears to be resolved but appears again shortly after, you may have an intermittent block<\/a>.<\/p>\n<\/div><\/div>\n\n\n\n
    How to fix \u201cindexed, though blocked by robots.txt\u201d in WordPress<\/h4>\n\n\n\n
    You\u2019ll want to remove the disallow statement causing the block. How you do this varies depending on the technology you\u2019re using.<\/p>\n\n\n\n\n\n
    If the issue impacts your entire website, the most likely cause is that you checked a setting in WordPress to disallow indexing. This mistake is common on new websites and following website migrations<\/a>. Follow these steps to check for it:<\/p>\n\n\n\n
      \n
    1. Click \u2018Settings\u2019<\/li>\n
    2. Click \u2018Reading\u2019<\/li>\n
    3. Make sure \u2018Search Engine Visibility\u2019 is unchecked.<\/li>\n<\/ol>\n\n\n\n
      \"3-wordpress-search-engine-block.png\"<\/figure>\n\n\n\n
      WordPress with Yoast<\/h5>\n\n\n\n
      If you\u2019re using the Yoast SEO plugin<\/a>, you can directly edit the robots.txt file to remove the blocking statement.<\/p>\n\n\n\n
        \n
      1. Click \u2018Yoast SEO\u2019<\/li>\n
      2. Click \u2018Tools\u2019<\/li>\n
      3. Click \u2018File editor\u2019<\/li>\n<\/ol>\n\n\n\n
        WordPress with Rank Math<\/h5>\n\n\n\n
        Similar to Yoast, Rank Math<\/a> allows you to edit the robots.txt file directly.<\/p>\n\n\n\n
          \n
        1. Click \u2018Rank Math\u2019<\/li>\n
        2. Click \u2018General Settings\u2019<\/li>\n
        3. Click \u2018Edit robots.txt\u2019<\/li>\n<\/ol>\n\n\n\n
          FTP or hosting<\/h5>\n\n\n\n
          If you have FTP access to the site, you can directly edit the robots.txt file to remove the disallow statement causing the issue. Your hosting provider may also give you access to a File Manager that allows you to access the robots.txt file directly.<\/p>\n\n\n\n
          Check for intermittent blocks<\/h3>\n\n\n\n
          Intermittent issues can be more difficult to troubleshoot because the conditions causing the block may not always be present.<\/p>\n\n\n\n
          What I\u2019d recommend is checking the history of your robots.txt file. For instance, in the GSC robots.txt tester<\/a>, if you click the dropdown, you\u2019ll see past versions of the file that you can click and see what they contained.<\/p>\n\n\n\n
          \"4-historic-robots-txt.gif\"<\/figure>\n\n\n\n
          The Wayback Machine<\/a> on archive.org also has a history of the robots.txt files for the websites they crawl. You can click on any of the dates they have data for and see what the file included on that particular day.<\/p>\n\n\n\n
          \"5-wayback-machine.png\"<\/figure>\n\n\n\n
          Or use the beta version of the Changes report, which lets you easily see content changes between two different versions.<\/p>\n\n\n\n
          \"6-wayback-machine.gif\"<\/figure>\n\n\n\n
          How to fix intermittent blocks<\/h4>\n\n\n\n
          The process for fixing intermittent blocks will depend on what is causing the issue. For example, one possible cause would be a shared cache between a test environment and a live environment. When the cache from the test environment is active, the robots.txt file may include a blocking directive. And when the cache from the live environment is active, the site may be crawlable. In this case, you would want to split the cache or maybe exclude .txt files from the cache in the test environment.<\/p>\n\n\n\n
          Check for user-agent blocks<\/h3>\n\n\n\n
          User-agent blocks are when a site blocks a specific user-agent like Googlebot<\/a> or AhrefsBot. In other words, the site is detecting a specific bot and blocking the corresponding user-agent.<\/p>\n\n\n\n
          If you can view a page fine in your regular browser but get blocked after changing your user-agent, it means that the specific user-agent you entered is blocked.<\/p>\n\n\n\n
          You can specify a particular user agent using Chrome devtools<\/a>. Another option is to use a browser extension to change user agents like this one<\/a>.<\/p>\n\n\n\n
          Alternatively, you can check for user-agent blocks with a cURL command. Here\u2019s how to do this on Windows:<\/p>\n\n\n\n
            \n
          1. Press Windows+R to open a \u201cRun\u201d box.<\/li>\n
          2. Type \u201ccmd\u201d and then click \u201cOK.\u201d<\/li>\n
          3. Enter a cURL command like this:<\/li>\n<\/ol>\n\n\n\n
            curl -A \u201cuser-agent-name-here\u201d -Lv [URL]curl -A \u201cMozilla\/5.0 (compatible; AhrefsBot\/7.0; +http:\/\/ahrefs.com\/robot\/)\u201d -Lv https:\/\/ahrefs.com<\/pre>\n\n\n\n
            How to fix user-agent blocks<\/h4>\n\n\n\n
            Unfortunately, this is another one where knowing how to fix it will depend on where you find the block. Many different systems may block a bot, including .htaccess, server config, firewalls, CDN, or even something you may not be able to see that your hosting provider controls. Your best bet may be to contact your hosting provider or CDN and ask them where the block is coming from and how you can resolve it.<\/p>\n\n\n\n
            For example, here are two different ways to block a user agent in .htaccess that you might need to look for.<\/p>\n\n\n\n
            RewriteEngine On
            RewriteCond %{HTTP_USER_AGENT} Googlebot [NC]RewriteRule .* - [F,L]<\/pre>\n\n\n\n
            Or\u2026<\/p>\n\n\n\n
            BrowserMatchNoCase \"Googlebot\" bots
            Order Allow,Deny
            Allow from ALL
            Deny from env=bots<\/pre>\n\n\n\n
            Check for IP blocks<\/h3>\n\n\n\n
            If you\u2019ve confirmed you\u2019re not blocked by robots.txt and ruled out user-agent blocks, then it\u2019s likely an IP block.<\/p>\n\n\n\n
            How to fix IP blocks<\/h4>\n\n\n\n
            IP blocks are difficult issues to track down. As with user-agent blocks, your best bet may be to contact your hosting provider or CDN and ask them where the block is coming from and how you can resolve it.<\/p>\n\n\n\n
            Here\u2019s one example of something you may be looking for in .htaccess:<\/p>\n\n\n\n
            deny from 123.123.123.123<\/code><\/p>\n\n\n\n
            Final thoughts<\/h2>\n\n\n\n
            Most of the time, the \u201cindexed, though blocked by robots.txt\u201d warning results from a robots.txt block. Hopefully, this guide helped you find and fix the issue if that wasn\u2019t the case for you.<\/p>\n\n\n\n
            Have questions? Let me know on Twitter<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
            In most cases, this will be a straightforward issue where you blocked crawling in your robots.txt file. But there are a few additional conditions that can trigger the problem, so let\u2019s go through the following troubleshooting process to diagnose and\u2026<\/span><\/p>\n
            Read more \u203a<\/div>\n
            <\/p>\n","protected":false},"author":150,"featured_media":131391,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"wp_typography_post_enhancements_disabled":false,"footnotes":""},"categories":[329],"tags":[],"coauthors":[377],"class_list":["post-131285","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technical-seo","odd"],"yoast_head":"\n"Indexed, though blocked by robots.txt" Can Be More Than A Robots.txt Block<\/title>\n<meta name=\"description\" content=\"Follow this troubleshooting process to fix the \u201cIndexed, though blocked by robots.txt\u201d warning in Google Search Console.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\""Indexed, though blocked by robots.txt" Can Be More Than A Robots.txt Block\" \/>\n<meta property=\"og:description\" content=\"Follow this troubleshooting process.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/\" \/>\n<meta property=\"og:site_name\" content=\"SEO Blog by Ahrefs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Ahrefs\/\" \/>\n<meta property=\"article:author\" content=\"patrickstox\" \/>\n<meta property=\"article:published_time\" content=\"2021-01-28T16:39:20+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-01-02T20:57:11+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/01\/fb-indexed-though-blocked-by-robots-txt-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"990\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Patrick Stox\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:description\" content=\"Follow this troubleshooting process.\" \/>\n<meta name=\"twitter:creator\" content=\"@patrickstox\" \/>\n<meta name=\"twitter:site\" content=\"@ahrefs\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/indexed-though-blocked-by-robots-txt\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/indexed-though-blocked-by-robots-txt\\\/\"},\"author\":{\"name\":\"Patrick Stox\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/14bf754248f3c561786477e4e5fd2067\"},\"headline\":\"\u201cIndexed, though blocked by robots.txt\u201d Can Be More Than A Robots.txt Block\",\"datePublished\":\"2021-01-28T16:39:20+00:00\",\"dateModified\":\"2025-01-02T20:57:11+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/indexed-though-blocked-by-robots-txt\\\/\"},\"wordCount\":1232,\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/indexed-though-blocked-by-robots-txt\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/01\\\/fb-indexed-though-blocked-by-robots-txt-1.png\",\"articleSection\":[\"Technical SEO\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/indexed-though-blocked-by-robots-txt\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/indexed-though-blocked-by-robots-txt\\\/\",\"name\":\"\\\"Indexed, though blocked by robots.txt\\\" Can Be More Than A Robots.txt Block\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/indexed-though-blocked-by-robots-txt\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/indexed-though-blocked-by-robots-txt\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/01\\\/fb-indexed-though-blocked-by-robots-txt-1.png\",\"datePublished\":\"2021-01-28T16:39:20+00:00\",\"dateModified\":\"2025-01-02T20:57:11+00:00\",\"description\":\"Follow this troubleshooting process to fix the \u201cIndexed, though blocked by robots.txt\u201d warning in Google Search Console.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ahrefs.com\\\/blog\\\/indexed-though-blocked-by-robots-txt\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/indexed-though-blocked-by-robots-txt\\\/#primaryimage\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/01\\\/fb-indexed-though-blocked-by-robots-txt-1.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/01\\\/fb-indexed-though-blocked-by-robots-txt-1.png\",\"width\":1920,\"height\":990},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"name\":\"SEO Blog by Ahrefs\",\"description\":\"Link Building Strategies & SEO Tips\",\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\",\"name\":\"Ahrefs\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"width\":2048,\"height\":768,\"caption\":\"Ahrefs\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/Ahrefs\\\/\",\"https:\\\/\\\/x.com\\\/ahrefs\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/ahrefs\\\/\",\"https:\\\/\\\/www.youtube.com\\\/c\\\/ahrefscom\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/14bf754248f3c561786477e4e5fd2067\",\"name\":\"Patrick Stox\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/11\\\/Screenshot-2019-11-06-at-00.57.29.pngbade1fd182f70b6825c334271c12533e\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/11\\\/Screenshot-2019-11-06-at-00.57.29.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/11\\\/Screenshot-2019-11-06-at-00.57.29.png\",\"caption\":\"Patrick Stox\"},\"description\":\"Patrick Stox is a Product Advisor, Technical SEO, & Brand Ambassador at Ahrefs. He was the lead author for the SEO chapter of the 2021 Web Almanac and a reviewer for the 2022 SEO chapter. He also co-wrote the SEO Book For Beginners by Ahrefs and was the Technical Review Editor for The Art of SEO 4th Edition. He\u2019s an organizer for the Triangle SEO Meetup, the Tech SEO Connect conference, he runs a Technical SEO Slack group, and is a moderator for \\\/r\\\/TechSEO on Reddit.\",\"sameAs\":[\"https:\\\/\\\/patrickstox.com\\\/\",\"patrickstox\",\"https:\\\/\\\/x.com\\\/patrickstox\"],\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/author\\\/patrick-stox\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"\"Indexed, though blocked by robots.txt\" Can Be More Than A Robots.txt Block","description":"Follow this troubleshooting process to fix the \u201cIndexed, though blocked by robots.txt\u201d warning in Google Search Console.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/","og_locale":"en_US","og_type":"article","og_title":"\"Indexed, though blocked by robots.txt\" Can Be More Than A Robots.txt Block","og_description":"Follow this troubleshooting process.","og_url":"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/","og_site_name":"SEO Blog by Ahrefs","article_publisher":"https:\/\/www.facebook.com\/Ahrefs\/","article_author":"patrickstox","article_published_time":"2021-01-28T16:39:20+00:00","article_modified_time":"2025-01-02T20:57:11+00:00","og_image":[{"width":1920,"height":990,"url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/01\/fb-indexed-though-blocked-by-robots-txt-1.png","type":"image\/png"}],"author":"Patrick Stox","twitter_card":"summary_large_image","twitter_description":"Follow this troubleshooting process.","twitter_creator":"@patrickstox","twitter_site":"@ahrefs","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/#article","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/"},"author":{"name":"Patrick Stox","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/14bf754248f3c561786477e4e5fd2067"},"headline":"\u201cIndexed, though blocked by robots.txt\u201d Can Be More Than A Robots.txt Block","datePublished":"2021-01-28T16:39:20+00:00","dateModified":"2025-01-02T20:57:11+00:00","mainEntityOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/"},"wordCount":1232,"publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/01\/fb-indexed-though-blocked-by-robots-txt-1.png","articleSection":["Technical SEO"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/","url":"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/","name":"\"Indexed, though blocked by robots.txt\" Can Be More Than A Robots.txt Block","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/#primaryimage"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/01\/fb-indexed-though-blocked-by-robots-txt-1.png","datePublished":"2021-01-28T16:39:20+00:00","dateModified":"2025-01-02T20:57:11+00:00","description":"Follow this troubleshooting process to fix the \u201cIndexed, though blocked by robots.txt\u201d warning in Google Search Console.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/indexed-though-blocked-by-robots-txt\/#primaryimage","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/01\/fb-indexed-though-blocked-by-robots-txt-1.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/01\/fb-indexed-though-blocked-by-robots-txt-1.png","width":1920,"height":990},{"@type":"WebSite","@id":"https:\/\/ahrefs.com\/blog\/#website","url":"https:\/\/ahrefs.com\/blog\/","name":"SEO Blog by Ahrefs","description":"Link Building Strategies & SEO Tips","publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ahrefs.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ahrefs.com\/blog\/#organization","name":"Ahrefs","url":"https:\/\/ahrefs.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","width":2048,"height":768,"caption":"Ahrefs"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Ahrefs\/","https:\/\/x.com\/ahrefs","https:\/\/www.linkedin.com\/company\/ahrefs\/","https:\/\/www.youtube.com\/c\/ahrefscom"]},{"@type":"Person","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/14bf754248f3c561786477e4e5fd2067","name":"Patrick Stox","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.pngbade1fd182f70b6825c334271c12533e","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.png","caption":"Patrick Stox"},"description":"Patrick Stox is a Product Advisor, Technical SEO, & Brand Ambassador at Ahrefs. He was the lead author for the SEO chapter of the 2021 Web Almanac and a reviewer for the 2022 SEO chapter. He also co-wrote the SEO Book For Beginners by Ahrefs and was the Technical Review Editor for The Art of SEO 4th Edition. He\u2019s an organizer for the Triangle SEO Meetup, the Tech SEO Connect conference, he runs a Technical SEO Slack group, and is a moderator for \/r\/TechSEO on Reddit.","sameAs":["https:\/\/patrickstox.com\/","patrickstox","https:\/\/x.com\/patrickstox"],"url":"https:\/\/ahrefs.com\/blog\/author\/patrick-stox\/"}]}},"as_json":null,"json_reviewers":[114],"_links":{"self":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/131285","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/users\/150"}],"replies":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/comments?post=131285"}],"version-history":[{"count":0,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/131285\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media\/131391"}],"wp:attachment":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media?parent=131285"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/categories?post=131285"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/tags?post=131285"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/coauthors?post=131285"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}