{"id":187899,"date":"2025-05-22T04:30:00","date_gmt":"2025-05-22T09:30:00","guid":{"rendered":"https:\/\/ahrefs.com\/blog\/?p=187899"},"modified":"2025-05-25T12:33:19","modified_gmt":"2025-05-25T17:33:19","slug":"seo-bot-block-rates","status":"publish","type":"post","link":"https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/","title":{"rendered":"The SEO Bots That ~140 Million Websites Block the&nbsp;Most"},"content":{"rendered":"\n<div class=\"intro-txt\">Ever wonder which SEO bots are the most blocked? This can impact the quality of the data the tools provide.<\/div>\n\n\n\n<p>Blocking these bots will mostly impact the link index of the tools. They won\u2019t be able to crawl the pages, so they can\u2019t check where those pages are linking. It doesn\u2019t matter for traffic estimates, keyword rankings, top pages, etc. Those are built from different data sources.<\/p>\n\n\n\n<p>For Ahrefs, it would also impact the internal links we show and the page history feature that shows changes to your pages over time, which you might need at some point. Ahrefsbot also powers the index for our search engine, <a href=\"https:\/\/yep.com\/\">Yep.com<\/a>, so blocking Ahrefsbot means you wouldn\u2019t show in Yep\u2019s search results.<\/p>\n\n\n\n<p>We looked at ~140 million websites to see how often SEO bots were blocked. I want to give a huge thanks to our data scientist <a href=\"https:\/\/sg.linkedin.com\/in\/xibeijia-guan\">Xibeijia Guan<\/a> for pulling this&nbsp;data.<\/p>\n\n\n\n<div class=\"intro-tok\" id=\"intro_tok\" style=\"display:none;\"><div class=\"intro-title\">Contents<\/div><a href=\"#\" class=\"expand-dots\"><span><\/span><span><\/span><span><\/span><\/a><\/div>\n\n\n\n<p><a id=\"post-187899-_gjdgxs\"><\/a><\/p><div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"Key takeaways\" data-section=\"key-takeaways\">\n\n\n\n<h2 class=\"wp-block-heading\">Key takeaways<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>Here are the top 3 most blocked SEO&nbsp;bots:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>MJ12bot (Majestic)<\/strong>. Blocked by 6.49% of all websites.<\/li>\n\n\n\n<li><strong>SemrushBot<\/strong>. Blocked by 6.34% of all websites.<\/li>\n\n\n\n<li><strong>AhrefsBot<\/strong>. Blocked by 6.31% of all websites.<\/li>\n<\/ol>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"How often are SEO bots blocked?\" data-section=\"how-often\">\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-187899-_30j0zll\"><\/a>How often are SEO bots blocked?<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>We looked at the total number of websites blocking the bots. There are many ways to block bots with robots.txt, and this accounts for all of them including:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Explicit blocks<\/strong>, where the bot is mentioned and disallowed<\/li>\n\n\n\n<li><strong>General blocks<\/strong>, where all bots may be blocked<\/li>\n\n\n\n<li>Any instances where a directive <strong>allowed the bot<\/strong>, after blocking all&nbsp;bots<\/li>\n<\/ul>\n\n\n\n<p>Caveats: this doesn\u2019t include any other block types such as firewalls or IP blocks. It is also Ahrefs view of the web, which means if we\u2019re blocked by one of these methods, we may not see the robots.txt files at&nbsp;all.<\/p>\n\n\n\n<p>As I mentioned earlier, the most blocked bot is MJ12bot from Majestic. I suspect there are a few reasons for&nbsp;this.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>They\u2019re a distributed crawler, meaning you can\u2019t look up or block them by IPs, which makes them less trusted and also more likely to be blocked by user&nbsp;agent.<\/li>\n\n\n\n<li>They\u2019ve been crawling the web for longer.<\/li>\n\n\n\n<li>They have a smaller user base than more popular SEO tools and therefore less leverage to remove any blocks.<\/li>\n<\/ol>\n\n\n\n<p>Here are the most blocked SEO&nbsp;bots:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1293\" height=\"2048\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/seo-bots-block-rate.jpg\" alt=\"SEO bots block rate\" class=\"wp-image-187901\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/seo-bots-block-rate.jpg 1293w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/seo-bots-block-rate-268x425.jpg 268w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/seo-bots-block-rate-768x1216.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/seo-bots-block-rate-970x1536.jpg 970w\" sizes=\"auto, (max-width: 1293px) 100vw, 1293px\"><\/figure>\n\n\n\n<p>And the total websites blocking SEO&nbsp;bots:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1293\" height=\"2048\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/total-blocks-of-seo-bots.jpg\" alt=\"Total blocks of SEO bots\" class=\"wp-image-187902\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/total-blocks-of-seo-bots.jpg 1293w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/total-blocks-of-seo-bots-268x425.jpg 268w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/total-blocks-of-seo-bots-768x1216.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/total-blocks-of-seo-bots-970x1536.jpg 970w\" sizes=\"auto, (max-width: 1293px) 100vw, 1293px\"><\/figure>\n\n\n\n<p>Here\u2019s the&nbsp;data:<\/p>\n\n\n\n<table id=\"tablepress-413\" class=\"tablepress tablepress-id-413 tablepress-responsive tablepress-ahrefs-width-720px\">\n<thead>\n<tr class=\"row-1 odd\">\n\t<th class=\"column-1\">Bot Name<\/th><th class=\"column-2\">Count<\/th><th class=\"column-3\">Percentage %<\/th><th class=\"column-4\">Bot Operator<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-2 even\">\n\t<td class=\"column-1\">MJ12bot<\/td><td class=\"column-2\">9081205<\/td><td class=\"column-3\">6.49<\/td><td class=\"column-4\">Majestic<\/td>\n<\/tr>\n<tr class=\"row-3 odd\">\n\t<td class=\"column-1\">SemrushBot<\/td><td class=\"column-2\">8868486<\/td><td class=\"column-3\">6.34<\/td><td class=\"column-4\">Semrush<\/td>\n<\/tr>\n<tr class=\"row-4 even\">\n\t<td class=\"column-1\">AhrefsBot<\/td><td class=\"column-2\">8831316<\/td><td class=\"column-3\">6.31<\/td><td class=\"column-4\">Ahrefs<\/td>\n<\/tr>\n<tr class=\"row-5 odd\">\n\t<td class=\"column-1\">dotbot<\/td><td class=\"column-2\">8569766<\/td><td class=\"column-3\">6.13<\/td><td class=\"column-4\">Moz<\/td>\n<\/tr>\n<tr class=\"row-6 even\">\n\t<td class=\"column-1\">BLEXBot<\/td><td class=\"column-2\">8374216<\/td><td class=\"column-3\">5.99<\/td><td class=\"column-4\">SEO PowerSuite<\/td>\n<\/tr>\n<tr class=\"row-7 odd\">\n\t<td class=\"column-1\">serpstatbot<\/td><td class=\"column-2\">7878935<\/td><td class=\"column-3\">5.63<\/td><td class=\"column-4\">Serpstat<\/td>\n<\/tr>\n<tr class=\"row-8 even\">\n\t<td class=\"column-1\">DataForSeoBot<\/td><td class=\"column-2\">7872939<\/td><td class=\"column-3\">5.63<\/td><td class=\"column-4\">DataForSEO<\/td>\n<\/tr>\n<tr class=\"row-9 odd\">\n\t<td class=\"column-1\">SemrushBot-CT<\/td><td class=\"column-2\">7855400<\/td><td class=\"column-3\">5.62<\/td><td class=\"column-4\">Semrush<\/td>\n<\/tr>\n<tr class=\"row-10 even\">\n\t<td class=\"column-1\">Barkrowler<\/td><td class=\"column-2\">7804425<\/td><td class=\"column-3\">5.58<\/td><td class=\"column-4\">Babbar<\/td>\n<\/tr>\n<tr class=\"row-11 odd\">\n\t<td class=\"column-1\">SemrushBot-BA<\/td><td class=\"column-2\">7796785<\/td><td class=\"column-3\">5.57<\/td><td class=\"column-4\">Semrush<\/td>\n<\/tr>\n<tr class=\"row-12 even\">\n\t<td class=\"column-1\">SemrushBot-SWA<\/td><td class=\"column-2\">7789812<\/td><td class=\"column-3\">5.57<\/td><td class=\"column-4\">Semrush<\/td>\n<\/tr>\n<tr class=\"row-13 odd\">\n\t<td class=\"column-1\">SemrushBot-SI<\/td><td class=\"column-2\">7789062<\/td><td class=\"column-3\">5.57<\/td><td class=\"column-4\">Semrush<\/td>\n<\/tr>\n<tr class=\"row-14 even\">\n\t<td class=\"column-1\">SEOkicks<\/td><td class=\"column-2\">7758904<\/td><td class=\"column-3\">5.55<\/td><td class=\"column-4\">SEOkicks<\/td>\n<\/tr>\n<tr class=\"row-15 odd\">\n\t<td class=\"column-1\">Screaming Frog SEO Spider<\/td><td class=\"column-2\">7711108<\/td><td class=\"column-3\">5.51<\/td><td class=\"column-4\">Screaming Frog<\/td>\n<\/tr>\n<tr class=\"row-16 even\">\n\t<td class=\"column-1\">linkdexbot<\/td><td class=\"column-2\">7704425<\/td><td class=\"column-3\">5.51<\/td><td class=\"column-4\">LinkDex<\/td>\n<\/tr>\n<tr class=\"row-17 odd\">\n\t<td class=\"column-1\">DomainStatsBot<\/td><td class=\"column-2\">7696944<\/td><td class=\"column-3\">5.5<\/td><td class=\"column-4\">Domainstats<\/td>\n<\/tr>\n<tr class=\"row-18 even\">\n\t<td class=\"column-1\">ZoomBot<\/td><td class=\"column-2\">7669495<\/td><td class=\"column-3\">5.48<\/td><td class=\"column-4\">SEOZoom<\/td>\n<\/tr>\n<tr class=\"row-19 odd\">\n\t<td class=\"column-1\">SiteCheckerBotCrawler<\/td><td class=\"column-2\">7666545<\/td><td class=\"column-3\">5.48<\/td><td class=\"column-4\">Sitechecker<\/td>\n<\/tr>\n<tr class=\"row-20 even\">\n\t<td class=\"column-1\">Cocolyzebot<\/td><td class=\"column-2\">7666233<\/td><td class=\"column-3\">5.48<\/td><td class=\"column-4\">Cocolyze<\/td>\n<\/tr>\n<tr class=\"row-21 odd\">\n\t<td class=\"column-1\">SeobilityBot<\/td><td class=\"column-2\">7664228<\/td><td class=\"column-3\">5.48<\/td><td class=\"column-4\">Seobility<\/td>\n<\/tr>\n<tr class=\"row-22 even\">\n\t<td class=\"column-1\">SenutoBot<\/td><td class=\"column-2\">7655145<\/td><td class=\"column-3\">5.47<\/td><td class=\"column-4\">Senuto<\/td>\n<\/tr>\n<tr class=\"row-23 odd\">\n\t<td class=\"column-1\">hypestat<\/td><td class=\"column-2\">7648671<\/td><td class=\"column-3\">5.47<\/td><td class=\"column-4\">HypeStat<\/td>\n<\/tr>\n<tr class=\"row-24 even\">\n\t<td class=\"column-1\">online-webceo-bot<\/td><td class=\"column-2\">7648444<\/td><td class=\"column-3\">5.47<\/td><td class=\"column-4\">WebCEO<\/td>\n<\/tr>\n<tr class=\"row-25 odd\">\n\t<td class=\"column-1\">BrightEdge Crawler<\/td><td class=\"column-2\">7648139<\/td><td class=\"column-3\">5.47<\/td><td class=\"column-4\">BrightEdge<\/td>\n<\/tr>\n<tr class=\"row-26 even\">\n\t<td class=\"column-1\">SEOlizer<\/td><td class=\"column-2\">7648112<\/td><td class=\"column-3\">5.47<\/td><td class=\"column-4\">SEOLizer<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n<p>It gets a little more complicated to analyze. For the above, we looked at the main robots.txt file for a website, but every subdomain can have their own set of instructions. If we look at the ~461M robots.txt in total, then the most blocked SEO bot is SemrushBot at 5.76%. Here are the top&nbsp;5:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>SemrushBot<\/strong>: 5.76%<\/li>\n\n\n\n<li><strong>Dotbot (Moz)<\/strong>: 5.34%<\/li>\n\n\n\n<li><strong>MJ12bot (Majestic)<\/strong>: 4.96%<\/li>\n\n\n\n<li><strong>BLEXBot<\/strong>: 4.88%<\/li>\n\n\n\n<li><strong>Ahrefsbot<\/strong>: 4.67%<\/li>\n<\/ol>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"How often are SEO bots specifically targeted\" data-section=\"targeted\">\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-187899-_1fob9te\"><\/a>How often are SEO bots specifically targeted<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>For this measure, we\u2019re looking only at cases where a particular bot is disallowed. It does not include any overall disallow statements or cases where only certain bots may be allowed. In these cases, website owners went out of their way to specifically block certain bots.<\/p>\n\n\n\n<p>Majestic\u2019s bot is the most targeted, followed by Moz\u2019s&nbsp;bot.<\/p>\n\n\n\n<p>Here are the most blocked SEO bots by explicit mentions:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1293\" height=\"2048\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/explicit-block-rate-of-seo-bots-1.jpg\" alt=\"Explicit block rate of SEO bots\" class=\"wp-image-187918\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/explicit-block-rate-of-seo-bots-1.jpg 1293w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/explicit-block-rate-of-seo-bots-1-268x425.jpg 268w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/explicit-block-rate-of-seo-bots-1-768x1216.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/explicit-block-rate-of-seo-bots-1-970x1536.jpg 970w\" sizes=\"auto, (max-width: 1293px) 100vw, 1293px\"><\/figure>\n\n\n\n<p>Here are the number of websites explicitly blocking SEO&nbsp;bots:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1256\" height=\"2048\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/number-of-websites-explicitly-blocking-seo-bots-1.png\" alt=\"Number of websites explicitly blocking SEO bots\" class=\"wp-image-187921\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/number-of-websites-explicitly-blocking-seo-bots-1.png 1256w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/number-of-websites-explicitly-blocking-seo-bots-1-261x425.png 261w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/number-of-websites-explicitly-blocking-seo-bots-1-768x1252.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/number-of-websites-explicitly-blocking-seo-bots-1-942x1536.png 942w\" sizes=\"auto, (max-width: 1256px) 100vw, 1256px\"><\/figure>\n\n\n\n<p>Here\u2019s the&nbsp;data:<\/p>\n\n\n\n<table id=\"tablepress-414\" class=\"tablepress tablepress-id-414 tablepress-responsive tablepress-ahrefs-width-720px\">\n<thead>\n<tr class=\"row-1 odd\">\n\t<th class=\"column-1\">Bot Name<\/th><th class=\"column-2\">Count<\/th><th class=\"column-3\">Percentage %<\/th><th class=\"column-4\">Bot Operator<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-2 even\">\n\t<td class=\"column-1\">MJ12bot<\/td><td class=\"column-2\">2000372<\/td><td class=\"column-3\">1.43<\/td><td class=\"column-4\">Majestic<\/td>\n<\/tr>\n<tr class=\"row-3 odd\">\n\t<td class=\"column-1\">dotbot<\/td><td class=\"column-2\">1402305<\/td><td class=\"column-3\">1<\/td><td class=\"column-4\">Moz<\/td>\n<\/tr>\n<tr class=\"row-4 even\">\n\t<td class=\"column-1\">AhrefsBot<\/td><td class=\"column-2\">1350771<\/td><td class=\"column-3\">0.97<\/td><td class=\"column-4\">Ahrefs<\/td>\n<\/tr>\n<tr class=\"row-5 odd\">\n\t<td class=\"column-1\">SemrushBot<\/td><td class=\"column-2\">1285857<\/td><td class=\"column-3\">0.92<\/td><td class=\"column-4\">Semrush<\/td>\n<\/tr>\n<tr class=\"row-6 even\">\n\t<td class=\"column-1\">BLEXBot<\/td><td class=\"column-2\">861184<\/td><td class=\"column-3\">0.62<\/td><td class=\"column-4\">SEO PowerSuite<\/td>\n<\/tr>\n<tr class=\"row-7 odd\">\n\t<td class=\"column-1\">serpstatbot<\/td><td class=\"column-2\">354683<\/td><td class=\"column-3\">0.25<\/td><td class=\"column-4\">Serpstat<\/td>\n<\/tr>\n<tr class=\"row-8 even\">\n\t<td class=\"column-1\">DataForSeoBot<\/td><td class=\"column-2\">284694<\/td><td class=\"column-3\">0.2<\/td><td class=\"column-4\">DataForSEO<\/td>\n<\/tr>\n<tr class=\"row-9 odd\">\n\t<td class=\"column-1\">Barkrowler<\/td><td class=\"column-2\">276332<\/td><td class=\"column-3\">0.2<\/td><td class=\"column-4\">Babbar<\/td>\n<\/tr>\n<tr class=\"row-10 even\">\n\t<td class=\"column-1\">SEOkicks<\/td><td class=\"column-2\">219961<\/td><td class=\"column-3\">0.16<\/td><td class=\"column-4\">SEOkicks<\/td>\n<\/tr>\n<tr class=\"row-11 odd\">\n\t<td class=\"column-1\">SemrushBot-CT<\/td><td class=\"column-2\">211895<\/td><td class=\"column-3\">0.15<\/td><td class=\"column-4\">Semrush<\/td>\n<\/tr>\n<tr class=\"row-12 even\">\n\t<td class=\"column-1\">linkdexbot<\/td><td class=\"column-2\">166405<\/td><td class=\"column-3\">0.12<\/td><td class=\"column-4\">Linkdex<\/td>\n<\/tr>\n<tr class=\"row-13 odd\">\n\t<td class=\"column-1\">DomainStatsBot<\/td><td class=\"column-2\">157053<\/td><td class=\"column-3\">0.11<\/td><td class=\"column-4\">Domainstats<\/td>\n<\/tr>\n<tr class=\"row-14 even\">\n\t<td class=\"column-1\">SemrushBot-BA<\/td><td class=\"column-2\">154349<\/td><td class=\"column-3\">0.11<\/td><td class=\"column-4\">Semrush<\/td>\n<\/tr>\n<tr class=\"row-15 odd\">\n\t<td class=\"column-1\">SemrushBot-SI<\/td><td class=\"column-2\">147999<\/td><td class=\"column-3\">0.11<\/td><td class=\"column-4\">Semrush<\/td>\n<\/tr>\n<tr class=\"row-16 even\">\n\t<td class=\"column-1\">SemrushBot-SWA<\/td><td class=\"column-2\">146261<\/td><td class=\"column-3\">0.1<\/td><td class=\"column-4\">Semrush<\/td>\n<\/tr>\n<tr class=\"row-17 odd\">\n\t<td class=\"column-1\">ZoomBot<\/td><td class=\"column-2\">125310<\/td><td class=\"column-3\">0.09<\/td><td class=\"column-4\">SEOZoom<\/td>\n<\/tr>\n<tr class=\"row-18 even\">\n\t<td class=\"column-1\">SiteCheckerBotCrawler<\/td><td class=\"column-2\">122574<\/td><td class=\"column-3\">0.09<\/td><td class=\"column-4\">Sitechecker<\/td>\n<\/tr>\n<tr class=\"row-19 odd\">\n\t<td class=\"column-1\">Cocolyzebot<\/td><td class=\"column-2\">121737<\/td><td class=\"column-3\">0.09<\/td><td class=\"column-4\">Cocolyze<\/td>\n<\/tr>\n<tr class=\"row-20 even\">\n\t<td class=\"column-1\">SeobilityBot<\/td><td class=\"column-2\">117558<\/td><td class=\"column-3\">0.08<\/td><td class=\"column-4\">Seobility<\/td>\n<\/tr>\n<tr class=\"row-21 odd\">\n\t<td class=\"column-1\">Screaming Frog SEO Spider<\/td><td class=\"column-2\">87673<\/td><td class=\"column-3\">0.06<\/td><td class=\"column-4\">Screaming Frog<\/td>\n<\/tr>\n<tr class=\"row-22 even\">\n\t<td class=\"column-1\">SenutoBot<\/td><td class=\"column-2\">54978<\/td><td class=\"column-3\">0.04<\/td><td class=\"column-4\">Senuto<\/td>\n<\/tr>\n<tr class=\"row-23 odd\">\n\t<td class=\"column-1\">hypestat<\/td><td class=\"column-2\">861<\/td><td class=\"column-3\">0<\/td><td class=\"column-4\">HypeStat<\/td>\n<\/tr>\n<tr class=\"row-24 even\">\n\t<td class=\"column-1\">SenutoBot<\/td><td class=\"column-2\">54978<\/td><td class=\"column-3\">0.04<\/td><td class=\"column-4\">Senuto<\/td>\n<\/tr>\n<tr class=\"row-25 odd\">\n\t<td class=\"column-1\">hypestat<\/td><td class=\"column-2\">861<\/td><td class=\"column-3\">0<\/td><td class=\"column-4\">HypeStat<\/td>\n<\/tr>\n<tr class=\"row-26 even\">\n\t<td class=\"column-1\">online-webceo-bot<\/td><td class=\"column-2\">659<\/td><td class=\"column-3\">0<\/td><td class=\"column-4\">WebCEO<\/td>\n<\/tr>\n<tr class=\"row-27 odd\">\n\t<td class=\"column-1\">BrightEdge Crawler<\/td><td class=\"column-2\">289<\/td><td class=\"column-3\">0<\/td><td class=\"column-4\">BrightEdge<\/td>\n<\/tr>\n<tr class=\"row-28 even\">\n\t<td class=\"column-1\">SEOlizer<\/td><td class=\"column-2\">253<\/td><td class=\"column-3\">0<\/td><td class=\"column-4\">SEOLizer<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"SEO bot blocks over time by DR\" data-section=\"DR\"><a id=\"post-187899-_3znysh7\"><\/a>\n\n\n\n<h2 class=\"wp-block-heading\">SEO bot blocks over time by&nbsp;DR<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>We looked at the top 1M sites by DR, which aligns to sites with a DR &gt;45. Semrush is the most blocked followed by Majestic and&nbsp;Moz.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1765\" height=\"2048\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/total-blocks-of-seo-bots-on-the-top-1-million-webs-1.png\" alt=\"Total blocks of SEO bots on the top 1 million websites, >45 DR\" class=\"wp-image-187924\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/total-blocks-of-seo-bots-on-the-top-1-million-webs-1.png 1765w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/total-blocks-of-seo-bots-on-the-top-1-million-webs-1-366x425.png 366w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/total-blocks-of-seo-bots-on-the-top-1-million-webs-1-768x891.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/total-blocks-of-seo-bots-on-the-top-1-million-webs-1-1324x1536.png 1324w\" sizes=\"auto, (max-width: 1765px) 100vw, 1765px\"><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<p><a id=\"post-187899-_2et92p0\"><\/a><\/p><div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"SEO bot blocks over time by DR\" data-section=\"DR\"><a id=\"post-187899-_3znysh7\"><\/a>\n\n\n\n<h2 class=\"wp-block-heading\">SEO bot blocks over time by domain category<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>Here\u2019s how it breaks down for each individual bot in different categories of websites. The top 3&nbsp;are:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Autos_and_Vehicles: <\/strong>39%<\/li>\n\n\n\n<li><strong>Books_and_Literature: <\/strong>27%<\/li>\n\n\n\n<li><strong>Real_Estate: <\/strong>17%<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2048\" height=\"1980\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/block-rate-of-seo-bots-by-domain-category-1.jpg\" alt=\"Block rate of SEO bots by domain category\" class=\"wp-image-187927\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/block-rate-of-seo-bots-by-domain-category-1.jpg 2048w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/block-rate-of-seo-bots-by-domain-category-1-440x425.jpg 440w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/block-rate-of-seo-bots-by-domain-category-1-768x743.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/block-rate-of-seo-bots-by-domain-category-1-1536x1485.jpg 1536w\" sizes=\"auto, (max-width: 2048px) 100vw, 2048px\"><\/figure>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"Which bots crawl the fastest?\" data-section=\"fastest\">\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-187899-_tyjcwt\"><\/a>Which bots crawl the fastest?<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>Going by the bot requests in <a href=\"https:\/\/radar.cloudflare.com\/\">Cloudflare Radar<\/a>, Ahrefs is by far the fastest crawler in the SEO space. ~4.6x faster than Moz and ~6.7x faster than Semrush.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1993\" height=\"739\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/bots-that-crawl-the-most-according-to-cloudflare-r-3.png\" alt=\"Bots that crawl the most according to Cloudflare Radar\" class=\"wp-image-187929\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/bots-that-crawl-the-most-according-to-cloudflare-r-3.png 1993w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/bots-that-crawl-the-most-according-to-cloudflare-r-3-680x252.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/bots-that-crawl-the-most-according-to-cloudflare-r-3-768x285.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/bots-that-crawl-the-most-according-to-cloudflare-r-3-1536x570.png 1536w\" sizes=\"auto, (max-width: 1993px) 100vw, 1993px\"><\/figure>\n\n\n\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Blocking these bots will mostly impact the link index of the tools. They won\u2019t be able to crawl the pages, so they can\u2019t check where those pages are linking. It doesn\u2019t matter for traffic estimates, keyword rankings, top pages, etc.<span class=\"ellipsis\">\u2026<\/span><\/p>\n<div class=\"read-more\">Read more \u203a<\/div>\n<p><!-- end of .read-more --><\/p>\n","protected":false},"author":150,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"wp_typography_post_enhancements_disabled":false,"footnotes":""},"categories":[414],"tags":[],"coauthors":[377,467],"class_list":["post-187899","post","type-post","status-publish","format-standard","hentry","category-data-studies","odd"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>The SEO Bots That ~140 Million Websites Block the Most<\/title>\n<meta name=\"description\" content=\"We looked at ~140 million websites to see how often SEO bots were blocked.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The SEO Bots That ~140 Million Websites Block the Most\" \/>\n<meta property=\"og:description\" content=\"We looked at ~140 million websites to see how often SEO bots were blocked.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/\" \/>\n<meta property=\"og:site_name\" content=\"SEO Blog by Ahrefs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Ahrefs\/\" \/>\n<meta property=\"article:author\" content=\"patrickstox\" \/>\n<meta property=\"article:published_time\" content=\"2025-05-22T09:30:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-25T17:33:19+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/seo-bots-block-rate.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1293\" \/>\n\t<meta property=\"og:image:height\" content=\"2048\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Patrick Stox, Xibeijia Guan\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@patrickstox\" \/>\n<meta name=\"twitter:site\" content=\"@ahrefs\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/seo-bot-block-rates\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/seo-bot-block-rates\\\/\"},\"author\":{\"name\":\"Patrick Stox\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/14bf754248f3c561786477e4e5fd2067\"},\"headline\":\"The SEO Bots That ~140 Million Websites Block the&nbsp;Most\",\"datePublished\":\"2025-05-22T09:30:00+00:00\",\"dateModified\":\"2025-05-25T17:33:19+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/seo-bot-block-rates\\\/\"},\"wordCount\":755,\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/seo-bot-block-rates\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2025\\\/05\\\/the-seo-bots-that-140-million-by-patrick-stox-data-studies-1.jpg\",\"articleSection\":[\"Data &amp; Studies\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/seo-bot-block-rates\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/seo-bot-block-rates\\\/\",\"name\":\"The SEO Bots That ~140 Million Websites Block the Most\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/seo-bot-block-rates\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/seo-bot-block-rates\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2025\\\/05\\\/seo-bots-block-rate.jpg\",\"datePublished\":\"2025-05-22T09:30:00+00:00\",\"dateModified\":\"2025-05-25T17:33:19+00:00\",\"description\":\"We looked at ~140 million websites to see how often SEO bots were blocked.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ahrefs.com\\\/blog\\\/seo-bot-block-rates\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/seo-bot-block-rates\\\/#primaryimage\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2025\\\/05\\\/seo-bots-block-rate.jpg\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2025\\\/05\\\/seo-bots-block-rate.jpg\",\"width\":1293,\"height\":2048,\"caption\":\"SEO bots block rate\"},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"name\":\"SEO Blog by Ahrefs\",\"description\":\"Link Building Strategies &amp; SEO Tips\",\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\",\"name\":\"Ahrefs\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"width\":2048,\"height\":768,\"caption\":\"Ahrefs\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/Ahrefs\\\/\",\"https:\\\/\\\/x.com\\\/ahrefs\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/ahrefs\\\/\",\"https:\\\/\\\/www.youtube.com\\\/c\\\/ahrefscom\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/14bf754248f3c561786477e4e5fd2067\",\"name\":\"Patrick Stox\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/11\\\/Screenshot-2019-11-06-at-00.57.29.pngbade1fd182f70b6825c334271c12533e\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/11\\\/Screenshot-2019-11-06-at-00.57.29.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/11\\\/Screenshot-2019-11-06-at-00.57.29.png\",\"caption\":\"Patrick Stox\"},\"description\":\"Patrick Stox is a Product Advisor, Technical SEO, &amp; Brand Ambassador at Ahrefs. He was the lead author for the SEO chapter of the 2021 Web Almanac and a reviewer for the 2022 SEO chapter. He also co-wrote the SEO Book For Beginners by Ahrefs and was the Technical Review Editor for The Art of SEO 4th Edition. He\u2019s an organizer for the Triangle SEO Meetup, the Tech SEO Connect conference, he runs a Technical SEO Slack group, and is a moderator for \\\/r\\\/TechSEO on Reddit.\",\"sameAs\":[\"https:\\\/\\\/patrickstox.com\\\/\",\"patrickstox\",\"https:\\\/\\\/x.com\\\/patrickstox\"],\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/author\\\/patrick-stox\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The SEO Bots That ~140 Million Websites Block the Most","description":"We looked at ~140 million websites to see how often SEO bots were blocked.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/","og_locale":"en_US","og_type":"article","og_title":"The SEO Bots That ~140 Million Websites Block the Most","og_description":"We looked at ~140 million websites to see how often SEO bots were blocked.","og_url":"https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/","og_site_name":"SEO Blog by Ahrefs","article_publisher":"https:\/\/www.facebook.com\/Ahrefs\/","article_author":"patrickstox","article_published_time":"2025-05-22T09:30:00+00:00","article_modified_time":"2025-05-25T17:33:19+00:00","og_image":[{"width":1293,"height":2048,"url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/seo-bots-block-rate.jpg","type":"image\/jpeg"}],"author":"Patrick Stox, Xibeijia Guan","twitter_card":"summary_large_image","twitter_creator":"@patrickstox","twitter_site":"@ahrefs","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/#article","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/"},"author":{"name":"Patrick Stox","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/14bf754248f3c561786477e4e5fd2067"},"headline":"The SEO Bots That ~140 Million Websites Block the&nbsp;Most","datePublished":"2025-05-22T09:30:00+00:00","dateModified":"2025-05-25T17:33:19+00:00","mainEntityOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/"},"wordCount":755,"publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/the-seo-bots-that-140-million-by-patrick-stox-data-studies-1.jpg","articleSection":["Data &amp; Studies"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/","url":"https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/","name":"The SEO Bots That ~140 Million Websites Block the Most","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/#primaryimage"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/seo-bots-block-rate.jpg","datePublished":"2025-05-22T09:30:00+00:00","dateModified":"2025-05-25T17:33:19+00:00","description":"We looked at ~140 million websites to see how often SEO bots were blocked.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/seo-bot-block-rates\/#primaryimage","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/seo-bots-block-rate.jpg","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/05\/seo-bots-block-rate.jpg","width":1293,"height":2048,"caption":"SEO bots block rate"},{"@type":"WebSite","@id":"https:\/\/ahrefs.com\/blog\/#website","url":"https:\/\/ahrefs.com\/blog\/","name":"SEO Blog by Ahrefs","description":"Link Building Strategies &amp; SEO Tips","publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ahrefs.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ahrefs.com\/blog\/#organization","name":"Ahrefs","url":"https:\/\/ahrefs.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","width":2048,"height":768,"caption":"Ahrefs"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Ahrefs\/","https:\/\/x.com\/ahrefs","https:\/\/www.linkedin.com\/company\/ahrefs\/","https:\/\/www.youtube.com\/c\/ahrefscom"]},{"@type":"Person","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/14bf754248f3c561786477e4e5fd2067","name":"Patrick Stox","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.pngbade1fd182f70b6825c334271c12533e","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.png","caption":"Patrick Stox"},"description":"Patrick Stox is a Product Advisor, Technical SEO, &amp; Brand Ambassador at Ahrefs. He was the lead author for the SEO chapter of the 2021 Web Almanac and a reviewer for the 2022 SEO chapter. He also co-wrote the SEO Book For Beginners by Ahrefs and was the Technical Review Editor for The Art of SEO 4th Edition. He\u2019s an organizer for the Triangle SEO Meetup, the Tech SEO Connect conference, he runs a Technical SEO Slack group, and is a moderator for \/r\/TechSEO on Reddit.","sameAs":["https:\/\/patrickstox.com\/","patrickstox","https:\/\/x.com\/patrickstox"],"url":"https:\/\/ahrefs.com\/blog\/author\/patrick-stox\/"}]}},"as_json":null,"json_reviewers":[194],"_links":{"self":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/187899","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/users\/150"}],"replies":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/comments?post=187899"}],"version-history":[{"count":0,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/187899\/revisions"}],"wp:attachment":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media?parent=187899"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/categories?post=187899"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/tags?post=187899"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/coauthors?post=187899"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}