{"version":"1.0","provider_name":"SEO Blog by Ahrefs","provider_url":"https:\/\/ahrefs.com\/blog","author_name":"Patrick Stox","author_url":"https:\/\/ahrefs.com\/blog\/author\/patrick-stox\/","title":"Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots","type":"rich","width":600,"height":338,"html":"<blockquote class=\"wp-embedded-content\" data-secret=\"yJT137je0h\"><a href=\"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/\">Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots<\/a><\/blockquote><iframe sandbox=\"allow-scripts\" security=\"restricted\" src=\"https:\/\/ahrefs.com\/blog\/search-engine-ai-seo-bot-crawling\/embed\/#?secret=yJT137je0h\" width=\"600\" height=\"338\" title=\"&#8220;Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots&#8221; &#8212; SEO Blog by Ahrefs\" data-secret=\"yJT137je0h\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\" class=\"wp-embedded-content\"><\/iframe><script type=\"text\/javascript\">\n\/* <![CDATA[ *\/\n\/*! This file is auto-generated *\/\n!function(d,l){\"use strict\";l.querySelector&&d.addEventListener&&\"undefined\"!=typeof URL&&(d.wp=d.wp||{},d.wp.receiveEmbedMessage||(d.wp.receiveEmbedMessage=function(e){var t=e.data;if((t||t.secret||t.message||t.value)&&!\/[^a-zA-Z0-9]\/.test(t.secret)){for(var s,r,n,a=l.querySelectorAll('iframe[data-secret=\"'+t.secret+'\"]'),o=l.querySelectorAll('blockquote[data-secret=\"'+t.secret+'\"]'),c=new RegExp(\"^https?:$\",\"i\"),i=0;i<o.length;i++)o[i].style.display=\"none\";for(i=0;i<a.length;i++)s=a[i],e.source===s.contentWindow&&(s.removeAttribute(\"style\"),\"height\"===t.message?(1e3<(r=parseInt(t.value,10))?r=1e3:~~r<200&&(r=200),s.height=r):\"link\"===t.message&&(r=new URL(s.getAttribute(\"src\")),n=new URL(t.value),c.test(n.protocol))&&n.host===r.host&&l.activeElement===s&&(d.top.location.href=t.value))}},d.addEventListener(\"message\",d.wp.receiveEmbedMessage,!1),l.addEventListener(\"DOMContentLoaded\",function(){for(var e,t,s=l.querySelectorAll(\"iframe.wp-embedded-content\"),r=0;r<s.length;r++)(t=(e=s[r]).getAttribute(\"data-secret\"))||(t=Math.random().toString(36).substring(2,12),e.src+=\"#?secret=\"+t,e.setAttribute(\"data-secret\",t)),e.contentWindow.postMessage({message:\"ready\",secret:t},\"*\")},!1)))}(window,document);\n\/\/# sourceURL=https:\/\/ahrefs.com\/blog\/wp-includes\/js\/wp-embed.min.js\n\/* ]]> *\/\n<\/script>\n","description":"The web is being crawled more than ever, but it\u2019s no longer just search engines doing the crawling. AI bots are soaking up all the data they can to train and power their AI assistants.","thumbnail_url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2025\/06\/share-of-bot-requests-search-engine-bots-34-6-ai.jpg","thumbnail_width":1600,"thumbnail_height":1871}