Data & Studies

The SEO Bots That ~140 Million Websites Block the Most

Patrick Stox
Patrick Stox is a Product Advisor, Technical SEO, & Brand Ambassador at Ahrefs. He was the lead author for the SEO chapter of the 2021 Web Almanac and a reviewer for the 2022 SEO chapter. He also co-wrote the SEO Book For Beginners by Ahrefs and was the Technical Review Editor for The Art of SEO 4th Edition. He’s an organizer for several groups including the Raleigh SEO Meetup (the most successful SEO Meetup in the US), the Beer and SEO Meetup, the Raleigh SEO Conference, Tech SEO Connect, runs a Technical SEO Slack group, and is a moderator for /r/TechSEO on Reddit.
Ever wonder which SEO bots are the most blocked? This can impact the quality of the data the tools provide.

Blocking these bots will mostly impact the link index of the tools. They won’t be able to crawl the pages, so they can’t check where those pages are linking. It doesn’t matter for traffic estimates, keyword rankings, top pages, etc. Those are built from different data sources.

For Ahrefs, it would also impact the internal links we show and the page history feature that shows changes to your pages over time, which you might need at some point. Ahrefsbot also powers the index for our search engine, Yep.com, so blocking Ahrefsbot means you wouldn’t show in Yep’s search results.

We looked at ~140 million websites to see how often SEO bots were blocked. I want to give a huge thanks to our data scientist Xibeijia Guan for pulling this data.

Here are the top 3 most blocked SEO bots:

  1. MJ12bot (Majestic). Blocked by 6.49% of all websites.
  2. SemrushBot. Blocked by 6.34% of all websites.
  3. AhrefsBot. Blocked by 6.31% of all websites.

We looked at the total number of websites blocking the bots. There are many ways to block bots with robots.txt, and this accounts for all of them including:

  • Explicit blocks, where the bot is mentioned and disallowed
  • General blocks, where all bots may be blocked
  • Any instances where a directive allowed the bot, after blocking all bots

Caveats: this doesn’t include any other block types such as firewalls or IP blocks.

As I mentioned earlier, the most blocked bot is MJ12bot from Majestic. I suspect there are a couple reasons for this.

  1. They’re a distributed crawler, meaning you can’t look up or block them by IPs, which makes them less trusted.
  2. They’ve been crawling the web for longer.
  3. They have a smaller user base than more popular SEO tools and therefore less leverage to remove any blocks.

Here are the most blocked SEO bots:

SEO bots block rate

And the total websites blocking SEO bots:

Total blocks of SEO bots

Here’s the data:

Bot NameCountPercentage %Bot Operator
MJ12bot90812056.49Majestic
SemrushBot88684866.34Semrush
AhrefsBot88313166.31Ahrefs
dotbot85697666.13Moz
BLEXBot83742165.99SEO PowerSuite
serpstatbot78789355.63Serpstat
DataForSeoBot78729395.63DataForSEO
SemrushBot-CT78554005.62Semrush
Barkrowler78044255.58Babbar
SemrushBot-BA77967855.57Semrush
SemrushBot-SWA77898125.57Semrush
SemrushBot-SI77890625.57Semrush
SEOkicks77589045.55SEOkicks
Screaming Frog SEO Spider77111085.51Screaming Frog
linkdexbot77044255.51LinkDex
DomainStatsBot76969445.5Domainstats
ZoomBot76694955.48SEOZoom
SiteCheckerBotCrawler76665455.48Sitechecker
Cocolyzebot76662335.48Cocolyze
SeobilityBot76642285.48Seobility
SenutoBot76551455.47Senuto
hypestat76486715.47HypeStat
online-webceo-bot76484445.47WebCEO
BrightEdge Crawler76481395.47BrightEdge
SEOlizer76481125.47SEOLizer

It gets a little more complicated to analyze. For the above, we looked at the main robots.txt file for a website, but every subdomain can have their own set of instructions. If we look at the ~461M robots.txt in total, then the most blocked SEO bot is SemrushBot at 5.76%. Here are the top 5:

  1. SemrushBot: 5.76%
  2. Dotbot (Moz): 5.34%
  3. MJ12bot (Majestic): 4.96%
  4. BLEXBot: 4.88%
  5. Ahrefsbot: 4.67%

For this measure, we’re looking only at cases where a particular bot is disallowed. It does not include any overall disallow statements or cases where only certain bots may be allowed. In these cases, website owners went out of their way to specifically block certain bots.

Majestic’s bot is the most targeted, followed by Moz’s bot.

Here are the most blocked SEO bots by explicit mentions:

Explicit block rate of SEO bots

Here are the number of websites explicitly blocking SEO bots:

Number of websites explicitly blocking SEO bots

Here’s the data:

Bot NameCountPercentage %Bot Operator
MJ12bot20003721.43Majestic
dotbot14023051Moz
AhrefsBot13507710.97Ahrefs
SemrushBot12858570.92Semrush
BLEXBot8611840.62SEO PowerSuite
serpstatbot3546830.25Serpstat
DataForSeoBot2846940.2DataForSEO
Barkrowler2763320.2Babbar
SEOkicks2199610.16SEOkicks
SemrushBot-CT2118950.15Semrush
linkdexbot1664050.12Linkdex
DomainStatsBot1570530.11Domainstats
SemrushBot-BA1543490.11Semrush
SemrushBot-SI1479990.11Semrush
SemrushBot-SWA1462610.1Semrush
ZoomBot1253100.09SEOZoom
SiteCheckerBotCrawler1225740.09Sitechecker
Cocolyzebot1217370.09Cocolyze
SeobilityBot1175580.08Seobility
Screaming Frog SEO Spider876730.06Screaming Frog
SenutoBot549780.04Senuto
hypestat8610HypeStat
SenutoBot549780.04Senuto
hypestat8610HypeStat
online-webceo-bot6590WebCEO
BrightEdge Crawler2890BrightEdge
SEOlizer2530SEOLizer

We looked at the top 1M sites by DR, which aligns to sites with a DR >45. Semrush is the most blocked followed by Majestic and Moz.

Here’s how it breaks down for each individual bot in different categories of websites. The top 3 are:

  1. Autos_and_Vehicles: 39%
  2. Books_and_Literature: 27%
  3. Real_Estate: 17%
Block rate of SEO bots by domain category

Going by the bot requests in Cloudflare Radar, Ahrefs is by far the fastest crawler in the SEO space. ~4.6x faster than Moz and ~6.7x faster than Semrush.

Bots that crawl the most according to Cloudflare Radar