{"id":188705,"date":"2025-06-18T05:00:00","date_gmt":"2025-06-18T10:00:00","guid":{"rendered":"https:\/\/ahrefs.com\/blog\/?p=188705"},"modified":"2025-06-24T07:53:40","modified_gmt":"2025-06-24T12:53:40","slug":"search-engine-ai-seo-bot-crawling","status":"publish","type":"post","link":"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/","title":{"rendered":"Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots"},"content":{"rendered":"\n<div class=\"intro-txt\">The web is being crawled more than ever, but it\u2019s no longer just search engines doing the crawling. AI bots are soaking up all the data they can to train and power their AI assistants.<\/div>\n\n\n\n<p>The problem is that AI bots are new to web crawling. They\u2019re still making some silly crawl decisions, bumping around, and wasting a lot of resources. We\u2019ve already seen that <a href=\"https:\/\/ahrefs.com\/blog\/ai-bot-block-rates\/\">AI bots are starting to be blocked<\/a> by a lot more of the most trafficked websites.<\/p>\n\n\n\n<p>I have no doubt the AI bots will figure things out, but it will likely be a painful process. I don\u2019t think any of them have even started rendering content yet.<\/p>\n\n\n\n<p>There was a lot of blowback on Bing in the past because of their aggressive crawling. They fixed it, started <a href=\"https:\/\/www.indexnow.org\/\">IndexNow<\/a>, and now they may be the most efficient web crawler.<\/p>\n\n\n\n<div class=\"recommendation\"><div class=\"recommendation-title\">Check out IndexNow<\/div><div class=\"recommendation-content\">\n\n\n\n<p>If you haven\u2019t set up <a href=\"https:\/\/ahrefs.com\/index-now\">IndexNow<\/a> for your website, you should.<\/p>\n\n\n\n<p>If you\u2019re a CDN, CMS, an AI bot, or Google, you should also be getting in on this. Save resources, save money, and save the planet. It\u2019s good for you, it\u2019s good for your users, it\u2019s good for website owners, it\u2019s good for website hosts, and it\u2019s good for the web and the planet.<\/p>\n\n\n\n<p>It\u2019s estimated that <a href=\"https:\/\/web.archive.org\/web\/20250420164754\/https:\/\/blog.cloudflare.com\/crawler-hints-how-cloudflare-is-reducing-the-environmental-impact-of-web-searches\/\">53% of crawler traffic is wasted effort<\/a>, and IndexNow can help cut that number down to almost nothing.<\/p>\n\n\n\n<\/div><\/div>\n\n\n\n<p>By looking at data from <a href=\"https:\/\/radar.cloudflare.com\/\">Cloudflare Radar<\/a>, which reflects a massive slice of internet traffic, we can see how AI bots compare to search engine bots and SEO bots. Cloudflare handles <a href=\"https:\/\/twitter.com\/AxelrodG\/status\/1447938954758705155\">~20% of all internet traffic<\/a>, so this is a large and representative dataset for overall web crawling.<\/p>\n\n\n\n<p>Let\u2019s dive&nbsp;in.<\/p>\n\n\n\n<div class=\"intro-tok\" id=\"intro_tok\" style=\"display:none;\"><div class=\"intro-title\">Contents<\/div><a href=\"#\" class=\"expand-dots\"><span><\/span><span><\/span><span><\/span><\/a><\/div>\n\n\n\n<p><a id=\"post-187899-_gjdgxs\"><\/a><\/p><div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"Requests for AI, Search Engine, and SEO bots\" data-section=\"requests\">\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-188705-_a51v4yz7dkfo\"><\/a>Requests for AI, Search Engine, and SEO&nbsp;bots<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>Search Engine bots crawl the most. AI Bots are relatively new to the web, but they are firmly in 2nd place despite there being fewer of them. If more AI assistants are launched and their bots start crawling the web, they\u2019ll probably pass search engines in crawling. At the rate they\u2019re growing, this could happen in the next couple years.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1600\" height=\"1871\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-search-engine-bots-34-6-ai.jpg\" alt=\"share of bot requests search engine bots 34.6%, ai bots 18.9%, seo bots 7.9%\" class=\"wp-image-188706\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-search-engine-bots-34-6-ai.jpg 1600w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-search-engine-bots-34-6-ai-363x425.jpg 363w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-search-engine-bots-34-6-ai-768x898.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-search-engine-bots-34-6-ai-1314x1536.jpg 1314w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\"><\/figure>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"Individual bot requests\" data-section=\"individual-bots\">\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-188705-_pahu1wpc9x9y\"><\/a>Individual bot requests<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>If we break out some of the individual bots, you can see that Googlebot is eating the internet much more than other&nbsp;bots.<\/p>\n\n\n\n<p>Notice that Google\u2019s AI bot also has a fast crawl rate and so does Google images. There are a lot more <a href=\"https:\/\/ahrefs.com\/blog\/googlebot\/\">Googlebots<\/a> that all perform various tasks.<\/p>\n\n\n\n<p>Amazonbot and GPTBot are both on par with crawling from&nbsp;Bing.<\/p>\n\n\n\n<p>Ahrefsbot is the most active bot by far in the SEO industry. Some other platforms like to claim they crawl faster, but <a href=\"https:\/\/ahrefs.com\/blog\/link-index-comparison\/#crawl-speed\">all the evidence says that\u2019s not true<\/a>. Ahrefs makes nearly as many requests as all 20 other SEO bots combined. We\u2019d beat a combination of 19 of&nbsp;them.<\/p>\n\n\n\n<p>Despite our crawl rate, I\u2019d guarantee that we\u2019re also the most efficient crawler of any SEO bot. We\u2019ve solved a lot of issues that I still see present in these other crawlers. We\u2019re also getting the data from IndexNow via our search engine, <a href=\"https:\/\/yep.com\/\">Yep<\/a>, to help us be even more efficient. At this point, maybe we should be classified as a search engine bot instead.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1394\" height=\"2048\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-for-top-bots-google-and-bin.jpg\" alt=\"share of bot requests for top bots. Google and bing for search engines, amazon and gpt for ai bots, ahrefs for seo bots\" class=\"wp-image-188707\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-for-top-bots-google-and-bin.jpg 1394w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-for-top-bots-google-and-bin-289x425.jpg 289w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-for-top-bots-google-and-bin-768x1128.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-for-top-bots-google-and-bin-1046x1536.jpg 1046w\" sizes=\"auto, (max-width: 1394px) 100vw, 1394px\"><\/figure>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"Requests over time for AI, Search Engine, and SEO bots\" data-section=\"requests-over-time\">\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-188705-_4o0vv76lvrlc\"><\/a>Requests over time for AI, Search Engine, and SEO&nbsp;bots<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>AI bots have ramped up crawling this year. They were nearly 25% of all bot requests in May, just 10% below search bots. That\u2019s amazing to me considering that Googlebot is so dominant overall and there are about half as many AI bots as there are search engine bots.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1946\" height=\"2048\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-over-time-for-search-engine.png\" alt=\"share of bot requests over time for search engine, ai, and seo bots\" class=\"wp-image-188708\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-over-time-for-search-engine.png 1946w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-over-time-for-search-engine-404x425.png 404w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-over-time-for-search-engine-768x808.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-over-time-for-search-engine-1460x1536.png 1460w\" sizes=\"auto, (max-width: 1946px) 100vw, 1946px\"><\/figure>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"Individual bot requests over time\" data-section=\"individual-over-time\">\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-188705-_z8j8f8um3f59\"><\/a>Individual bot requests over&nbsp;time<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>Googlebot is by far the most dominant bot on the web. GPTBot is charging ahead this year. I\u2019m curious if they\u2019ll close the gap more in the future.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1812\" height=\"2048\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-for-individual-bots-top-bot.png\" alt=\"share of bot requests for individual bots. Top bots include googlebot, gptbot, bingbot, GoogleOther\" class=\"wp-image-188709\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-for-individual-bots-top-bot.png 1812w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-for-individual-bots-top-bot-376x425.png 376w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-for-individual-bots-top-bot-768x868.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-for-individual-bots-top-bot-1359x1536.png 1359w\" sizes=\"auto, (max-width: 1812px) 100vw, 1812px\"><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-188705-_5zudbrjq89ha\"><\/a>Final thoughts<\/h2>\n\n\n\n<p>There\u2019s a cost to bots crawling your websites and there\u2019s a social contract between search engines and website owners, where search engines add value by sending referral traffic to websites. Google sends traffic (for now), so they don\u2019t get blocked.<\/p>\n\n\n\n<p>This contract extends to AI assistants. Send traffic to websites, give website owners data on searches and the visibility of their website in your system for reporting, or you will be blocked at the rate you\u2019re crawling. We\u2019re already seeing it happen.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2000\" height=\"1921\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/explicit-blocks-of-ai-bots-on-the-top-1-million-we.png\" alt=\"Explicit blocks of AI bots on the top 1 million websites by traffic\" class=\"wp-image-188710\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/explicit-blocks-of-ai-bots-on-the-top-1-million-we.png 2000w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/explicit-blocks-of-ai-bots-on-the-top-1-million-we-442x425.png 442w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/explicit-blocks-of-ai-bots-on-the-top-1-million-we-768x738.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/explicit-blocks-of-ai-bots-on-the-top-1-million-we-1536x1475.png 1536w\" sizes=\"auto, (max-width: 2000px) 100vw, 2000px\"><\/figure>\n\n\n\n<p>It\u2019s not surprising given that <a href=\"https:\/\/ahrefs.com\/blog\/ai-traffic-research\/\">AI search traffic is only 0.1% of the overall traffic to websites<\/a>. Send more traffic to websites, provide more value to website owners.<\/p>\n\n\n\n<p>If you have any questions, ask me on <a href=\"https:\/\/www.linkedin.com\/in\/patrickstox\/\">LinkedIn<\/a> or <a href=\"https:\/\/x.com\/patrickstox\">X<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The problem is that AI bots are new to web crawling. They\u2019re still making some silly crawl decisions, bumping around, and wasting a lot of resources. We\u2019ve already seen that AI bots are starting to be blocked by a lot<span class=\"ellipsis\">\u2026<\/span><\/p>\n<div class=\"read-more\">Read more \u203a<\/div>\n<p><!-- end of .read-more --><\/p>\n","protected":false},"author":150,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"wp_typography_post_enhancements_disabled":false,"footnotes":""},"categories":[414],"tags":[],"coauthors":[377],"class_list":["post-188705","post","type-post","status-publish","format-standard","hentry","category-data-studies","odd"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots<\/title>\n<meta name=\"description\" content=\"The web is being crawled more than ever, but it\u2019s no longer just search engines doing the crawling. AI bots are soaking up all the data they can to train and power their AI assistants.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots\" \/>\n<meta property=\"og:description\" content=\"The web is being crawled more than ever, but it\u2019s no longer just search engines doing the crawling. AI bots are soaking up all the data they can to train and power their AI assistants.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/\" \/>\n<meta property=\"og:site_name\" content=\"SEO Blog by Ahrefs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Ahrefs\/\" \/>\n<meta property=\"article:author\" content=\"patrickstox\" \/>\n<meta property=\"article:published_time\" content=\"2025-06-18T10:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-06-24T12:53:40+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-search-engine-bots-34-6-ai.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1600\" \/>\n\t<meta property=\"og:image:height\" content=\"1871\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Patrick Stox\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@patrickstox\" \/>\n<meta name=\"twitter:site\" content=\"@ahrefs\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/\"},\"author\":{\"name\":\"Patrick Stox\",\"@id\":\"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/14bf754248f3c561786477e4e5fd2067\"},\"headline\":\"Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots\",\"datePublished\":\"2025-06-18T10:00:00+00:00\",\"dateModified\":\"2025-06-24T12:53:40+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/\"},\"wordCount\":856,\"publisher\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/meet-the-new-web-crawlers-ai-by-patrick-stox-data-studies-1.jpg\",\"articleSection\":[\"Data &amp; Studies\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/\",\"url\":\"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/\",\"name\":\"Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots\",\"isPartOf\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-search-engine-bots-34-6-ai.jpg\",\"datePublished\":\"2025-06-18T10:00:00+00:00\",\"dateModified\":\"2025-06-24T12:53:40+00:00\",\"description\":\"The web is being crawled more than ever, but it\u2019s no longer just search engines doing the crawling. AI bots are soaking up all the data they can to train and power their AI assistants.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/#primaryimage\",\"url\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-search-engine-bots-34-6-ai.jpg\",\"contentUrl\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-search-engine-bots-34-6-ai.jpg\",\"width\":1600,\"height\":1871,\"caption\":\"share of bot requests search engine bots 34.6%, ai bots 18.9%, seo bots 7.9%\"},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ahrefs.com\/blog\/#website\",\"url\":\"https:\/\/ahrefs.com\/blog\/\",\"name\":\"SEO Blog by Ahrefs\",\"description\":\"Link Building Strategies &amp; SEO Tips\",\"publisher\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ahrefs.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/ahrefs.com\/blog\/#organization\",\"name\":\"Ahrefs\",\"url\":\"https:\/\/ahrefs.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png\",\"contentUrl\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png\",\"width\":2048,\"height\":768,\"caption\":\"Ahrefs\"},\"image\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/Ahrefs\/\",\"https:\/\/x.com\/ahrefs\",\"https:\/\/www.linkedin.com\/company\/ahrefs\/\",\"https:\/\/www.youtube.com\/c\/ahrefscom\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/14bf754248f3c561786477e4e5fd2067\",\"name\":\"Patrick Stox\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.pngbade1fd182f70b6825c334271c12533e\",\"url\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.png\",\"contentUrl\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.png\",\"caption\":\"Patrick Stox\"},\"description\":\"Patrick Stox is a Product Advisor, Technical SEO, &amp; Brand Ambassador at Ahrefs. He was the lead author for the SEO chapter of the 2021 Web Almanac and a reviewer for the 2022 SEO chapter. He also co-wrote the SEO Book For Beginners by Ahrefs and was the Technical Review Editor for The Art of SEO 4th Edition. He\u2019s an organizer for the Triangle SEO Meetup, the Tech SEO Connect conference, he runs a Technical SEO Slack group, and is a moderator for \/r\/TechSEO on Reddit.\",\"sameAs\":[\"https:\/\/patrickstox.com\/\",\"patrickstox\",\"https:\/\/x.com\/patrickstox\"],\"url\":\"https:\/\/ahrefs.com\/blog\/author\/patrick-stox\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots","description":"The web is being crawled more than ever, but it\u2019s no longer just search engines doing the crawling. AI bots are soaking up all the data they can to train and power their AI assistants.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/","og_locale":"en_US","og_type":"article","og_title":"Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots","og_description":"The web is being crawled more than ever, but it\u2019s no longer just search engines doing the crawling. AI bots are soaking up all the data they can to train and power their AI assistants.","og_url":"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/","og_site_name":"SEO Blog by Ahrefs","article_publisher":"https:\/\/www.facebook.com\/Ahrefs\/","article_author":"patrickstox","article_published_time":"2025-06-18T10:00:00+00:00","article_modified_time":"2025-06-24T12:53:40+00:00","og_image":[{"width":1600,"height":1871,"url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-search-engine-bots-34-6-ai.jpg","type":"image\/jpeg"}],"author":"Patrick Stox","twitter_card":"summary_large_image","twitter_creator":"@patrickstox","twitter_site":"@ahrefs","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/#article","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/"},"author":{"name":"Patrick Stox","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/14bf754248f3c561786477e4e5fd2067"},"headline":"Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots","datePublished":"2025-06-18T10:00:00+00:00","dateModified":"2025-06-24T12:53:40+00:00","mainEntityOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/"},"wordCount":856,"publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/meet-the-new-web-crawlers-ai-by-patrick-stox-data-studies-1.jpg","articleSection":["Data &amp; Studies"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/","url":"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/","name":"Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/#primaryimage"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-search-engine-bots-34-6-ai.jpg","datePublished":"2025-06-18T10:00:00+00:00","dateModified":"2025-06-24T12:53:40+00:00","description":"The web is being crawled more than ever, but it\u2019s no longer just search engines doing the crawling. AI bots are soaking up all the data they can to train and power their AI assistants.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/#primaryimage","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-search-engine-bots-34-6-ai.jpg","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-search-engine-bots-34-6-ai.jpg","width":1600,"height":1871,"caption":"share of bot requests search engine bots 34.6%, ai bots 18.9%, seo bots 7.9%"},{"@type":"WebSite","@id":"https:\/\/ahrefs.com\/blog\/#website","url":"https:\/\/ahrefs.com\/blog\/","name":"SEO Blog by Ahrefs","description":"Link Building Strategies &amp; SEO Tips","publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ahrefs.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ahrefs.com\/blog\/#organization","name":"Ahrefs","url":"https:\/\/ahrefs.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","width":2048,"height":768,"caption":"Ahrefs"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Ahrefs\/","https:\/\/x.com\/ahrefs","https:\/\/www.linkedin.com\/company\/ahrefs\/","https:\/\/www.youtube.com\/c\/ahrefscom"]},{"@type":"Person","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/14bf754248f3c561786477e4e5fd2067","name":"Patrick Stox","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.pngbade1fd182f70b6825c334271c12533e","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.png","caption":"Patrick Stox"},"description":"Patrick Stox is a Product Advisor, Technical SEO, &amp; Brand Ambassador at Ahrefs. He was the lead author for the SEO chapter of the 2021 Web Almanac and a reviewer for the 2022 SEO chapter. He also co-wrote the SEO Book For Beginners by Ahrefs and was the Technical Review Editor for The Art of SEO 4th Edition. He\u2019s an organizer for the Triangle SEO Meetup, the Tech SEO Connect conference, he runs a Technical SEO Slack group, and is a moderator for \/r\/TechSEO on Reddit.","sameAs":["https:\/\/patrickstox.com\/","patrickstox","https:\/\/x.com\/patrickstox"],"url":"https:\/\/ahrefs.com\/blog\/author\/patrick-stox\/"}]}},"_links":{"self":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/188705","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/users\/150"}],"replies":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/comments?post=188705"}],"version-history":[{"count":0,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/188705\/revisions"}],"wp:attachment":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media?parent=188705"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/categories?post=188705"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/tags?post=188705"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/coauthors?post=188705"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}