{"id":187900,"date":"2025-05-21T04:30:00","date_gmt":"2025-05-21T09:30:00","guid":{"rendered":"https:\/\/ahrefs.com\/blog\/?p=187900"},"modified":"2025-08-26T09:53:53","modified_gmt":"2025-08-26T14:53:53","slug":"ai-bot-block-rates","status":"publish","type":"post","link":"https:\/\/ahrefs.com\/blog\/ai-bot-block-rates\/","title":{"rendered":"The AI Bots That ~140 Million Websites Block the Most"},"content":{"rendered":"\n
AI bots power some of the most advanced technologies we use today, from search engines to AI assistants. However, their increasing presence has led to a growing number of websites blocking them.<\/div>\n\n\n\n

There\u2019s a cost to bots crawling your websites and there\u2019s a social contract between search engines and website owners, where search engines add value by sending referral traffic to websites. This is what keeps most websites from blocking search engines like Google, even as Google seems intent on taking more of that traffic for themselves. This social contract extends to generative engines.<\/p>\n\n\n\n

When we looked at the traffic makeup of ~35K websites<\/a> in Ahrefs Analytics, we found that AI sends just 0.1% of total referral traffic\u2014far behind that of search.<\/p>\n\n\n\n

\"Ahrefs<\/figure>\n\n\n\n

I think many site owners want to let these bots learn about their brand, their business, and their products and offerings. But while many people are betting that these systems are the future, they currently run the risk of not adding enough value for website owners. <\/p>\n\n\n\n

The first LLM to add more value to users by showing impressions and clicks to website owners will likely have a big advantage. Companies will report on the metrics from that LLM, which will likely increase adoption and prevent more websites from blocking their bot.<\/p>\n\n\n\n

The bots are using resources, using the data to train their AIs, and creating potential privacy issues. As a result, many websites are choosing to block AI bots.<\/p>\n\n\n\n

We looked at ~140 million websites and our data shows that block rates for AI bots have increased significantly over the past year. I want to give a huge thanks to our data scientist Xibeijia Guan<\/a> for pulling this data.<\/p>\n\n\n\n

Contents<\/div><\/span><\/span><\/span><\/a><\/div>\n\n\n\n

<\/a><\/p>