{"id":178325,"date":"2024-08-19T06:15:42","date_gmt":"2024-08-19T11:15:42","guid":{"rendered":"https:\/\/ahrefs.com\/blog\/?p=178325"},"modified":"2025-01-30T07:15:09","modified_gmt":"2025-01-30T12:15:09","slug":"website-crawlers","status":"publish","type":"post","link":"https:\/\/ahrefs.com\/blog\/website-crawlers\/","title":{"rendered":"Crawl Me Maybe? How Website Crawlers Work"},"content":{"rendered":"<div class=\"intro-txt\"> You might have heard of website crawling before \u2014 you may even have a vague idea of what it\u2019s about \u2014 but do you know why it\u2019s important, or what differentiates it from web crawling? (yes, there is a difference!)&nbsp;<\/div>\n\n\n\n<p>Search engines are increasingly ruthless when it comes to the quality of the sites they allow into the search results.<\/p>\n\n\n\n<p>If you don\u2019t grasp the basics of optimizing for web crawlers (and eventual users), your organic traffic may well pay the&nbsp;price.<\/p>\n\n\n\n<p>A good web<span style=\"text-decoration: underline;\">site<\/span> crawler can show you how to protect and even enhance your site\u2019s visibility.<\/p>\n\n\n\n<p>Here\u2019s what you need to know about both web crawlers and site crawlers.<\/p><div class=\"intro-tok\" id=\"intro_tok\" style=\"display:none;\"><div class=\"intro-title\">Contents<\/div><a href=\"#\" class=\"expand-dots\"><span><\/span><span><\/span><span><\/span><\/a><\/div>\n<p>\n\n<\/p>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"What is a web crawler?\" data-section=\"web-crawler-definition\">\n<p>\n\n<\/p>\n<h2 id=\"web-crawler-definition\" class=\"wp-block-heading\"><a id=\"post-178325-_3q76iu7vw9er\"><\/a>What is a web crawler?<\/h2>\n<p>\n\n<\/p>\n<\/div><\/div>\n<p>\n\n<\/p>\n<p>A web crawler is a software program or script that automatically scours the internet, analyzing and indexing web&nbsp;pages.<\/p>\n<p>\n\n<\/p>\n<p>Also known as a web spider or spiderbot, web crawlers assess a page\u2019s content to decide how to prioritize it in their indexes.<\/p>\n<p>\n\n<\/p>\n<p><a href=\"https:\/\/ahrefs.com\/blog\/googlebot\/\">Googlebot<\/a>, Google\u2019s web crawler, meticulously browses the web, following links from page to page, gathering data, and processing content for inclusion in Google\u2019s search engine.<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_hp80z6yb58oh\"><\/a>How do web crawlers impact SEO?<\/h3>\n<p>\n\n<\/p>\n<p>Web crawlers analyze your page and decide how indexable or rankable it is, which ultimately determines your ability to drive organic traffic.<\/p>\n<p>\n\n<\/p>\n<p>If you want to be discovered in search results, then it\u2019s important you ready your content for crawling and indexing.<\/p>\n<p>\n\n<\/p>\n<div class=\"recommendation\"><div class=\"recommendation-title\">Did you&nbsp;know?<\/div><div class=\"recommendation-content\"> <a href=\"https:\/\/ahrefs.com\/robot\">AhrefsBot<\/a> is a web crawler that:\n<p>\n\n<\/p>\n<ul class=\"wp-block-list\">\n<li>Visits over 8 billion web pages every 24&nbsp;hours<\/li>\n\n\n\n<li>Updates every 15\u201330 minutes<\/li>\n\n\n\n<li>Is the #1 most active SEO crawler (and 4th most active crawler worldwide)<\/li>\n<\/ul>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2048\" height=\"1226\" class=\"wp-image-178361\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-ahrefsbot-crawler-as-the-1-most-a-2.jpg\" alt=\"Graphic showing AhrefsBot crawler as the #1 most active SEO crawler and #4 most active web crawler in the world\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-ahrefsbot-crawler-as-the-1-most-a-2.jpg 2048w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-ahrefsbot-crawler-as-the-1-most-a-2-680x407.jpg 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-ahrefsbot-crawler-as-the-1-most-a-2-768x460.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-ahrefsbot-crawler-as-the-1-most-a-2-1536x920.jpg 1536w\" sizes=\"auto, (max-width: 2048px) 100vw, 2048px\"><\/figure>\n<p>\n\n<\/p>\n<\/div><\/div>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"What are the different types of web crawler?\" data-section=\"web-crawler-types\">\n<h2>What are the different types of web crawler?<\/h2>\n<\/div><\/div>\n<p>Crawlers come in different shapes and sizes. You\u2019ve got web crawlers like Googlebot and Bingbot\u2014they power major search engines, crawling billions of pages 24\/7, to keep search results fresh.<\/p>\n<p>Then you\u2019ve got specialized crawlers that zero-in on select areas\u2014think site crawlers that audit individual sites to spot technical issues, academic crawlers that scour research papers, and\u2014for the code-phobic (like me!)\u2014there\u2019s visual web scrapers that give you a point-and-click interface to grab the data you&nbsp;need.<\/p>\n<p>Below are the main types of web crawlers with examples and real-world applications. In this article, I\u2019m just going to be focusing on web and site crawlers.<\/p>\n\n<table id=\"tablepress-380\" class=\"tablepress tablepress-id-380 tablepress-responsive tablepress-ahrefs-width-720px\">\n<thead>\n<tr class=\"row-1 odd\">\n\t<th class=\"column-1\">Crawler type<\/th><th class=\"column-2\">Example bots<\/th><th class=\"column-3\">What they actually do<\/th>\n<\/tr>\n<\/thead>\n<tbody class=\"row-hover\">\n<tr class=\"row-2 even\">\n\t<td class=\"column-1\">Web crawlers<\/td><td class=\"column-2\">Googlebot, Bingbot<\/td><td class=\"column-3\">Scan billions of web pages&nbsp;to:<br>\n\u2022 Build the search index<br>\n\u2022 Find new\/updated content<br>\n\u2022 Monitor site health<br>\n\u2022 Assess content quality for ranking<\/td>\n<\/tr>\n<tr class=\"row-3 odd\">\n\t<td class=\"column-1\">Site crawlers<\/td><td class=\"column-2\">Ahrefs Site Audit crawler, Majestic<\/td><td class=\"column-3\">Analyze websites for SEO&nbsp;by:<br>\n\u2022 Tracking backlinks<br>\n\u2022 Mapping website structure<br>\n\u2022 Monitoring keyword rankings<br>\n\u2022 Identifying technical SEO issues<\/td>\n<\/tr>\n<tr class=\"row-4 even\">\n\t<td class=\"column-1\">Academic crawlers<\/td><td class=\"column-2\">CiteSeerX, Google Scholar<\/td><td class=\"column-3\">Collect academic research to:<br>\n\u2022 Build citation networks<br>\n\u2022 Identify new publications<br>\n\u2022 Create searchable databases<br>\n\u2022 Monitor research trends<\/td>\n<\/tr>\n<tr class=\"row-5 odd\">\n\t<td class=\"column-1\">Semantic crawlers<\/td><td class=\"column-2\">Apache Nutch, OpenCalais<\/td><td class=\"column-3\">Process web content to:<br>\n\u2022 Understand content meaning and topic relationships<br>\n\u2022 Build knowledge graphs<br>\n\u2022 Categorize content by subject<br>\n\u2022 Identify entities (people, places, organizations)<\/td>\n<\/tr>\n<tr class=\"row-6 even\">\n\t<td class=\"column-1\">Open-source crawlers<\/td><td class=\"column-2\">Scrapy, Heritrix<\/td><td class=\"column-3\">Build custom crawling to:<br>\n\u2022 Monitor competitor prices<br>\n\u2022 Track product inventory<br>\n\u2022 Gather market intelligence<br>\n\u2022 Archive websites<\/td>\n<\/tr>\n<tr class=\"row-7 odd\">\n\t<td class=\"column-1\">Visual web scrapers<\/td><td class=\"column-2\">Octoparse, WebHarvy<\/td><td class=\"column-3\">Help code-averse users extract:<br>\n\u2022 Product information from ecommerce sites<br>\n\u2022 Real estate listings<br>\n\u2022 Contact information from directories<br>\n\u2022 Weather data<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<!-- #tablepress-380 from cache -->\n<p>\n\n<\/p>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"How do web crawlers actually work?\" data-section=\"web-crawlers-work\">\n<p>\n\n<\/p>\n<h2 id=\"web-crawlers-work\" class=\"wp-block-heading\"><a id=\"post-178325-_ipdhfcfygpql\"><\/a>How do web crawlers actually work?<\/h2>\n<p>\n\n<\/p>\n<\/div><\/div>\n<p>\n\n<\/p>\n<p>There are roughly seven stages to web crawling:<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_1khc7cvozvho\"><\/a>1. URL Discovery<\/h3>\n<p>\n\n<\/p>\n<p>When you publish your page (e.g. to your sitemap), the web crawler discovers it and uses it as a \u2018seed\u2019 URL. Just like seeds in the cycle of germination, these starter URLs allow the crawl and subsequent crawling loops to&nbsp;begin.<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_dnt3yksn4b0t\"><\/a>2. Crawling<\/h3>\n<p>\n\n<\/p>\n<p>After URL discovery, your page is scheduled and then crawled. Content like meta tags, images, links, and structured data are <strong>downloaded<\/strong> to the search engine\u2019s servers, where they await parsing and indexing.<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_iym2irt0pa0l\"><\/a>3. Parsing<\/h3>\n<p>\n\n<\/p>\n<p>Parsing essentially means <strong>analysis<\/strong>. The crawler bot extracts the data it\u2019s just crawled to determine how to index and rank the&nbsp;page.<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_x9lgu7hxq0ru\"><\/a>3a. The URL Discovery Loop<\/h3>\n<p>\n\n<\/p>\n<p>Also during the parsing phase, but worthy of its own subsection, is the URL discovery loop. This is when newly discovered links (including links discovered via redirects) are added to a queue of URLs for the crawler to visit. These are effectively new \u2018seed\u2019 URLs, and steps 1\u20133 get repeated as part of the \u2018URL discovery loop\u2019.<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_261szsn2g657\"><\/a>4. Indexing<\/h3>\n<p>\n\n<\/p>\n<p>While new URLs are being discovered, the original URL gets indexed. Indexing is when search engines store the data collected from web pages. It enables them to quickly retrieve relevant results for user queries.<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_d3319lc3mohf\"><\/a>5. Ranking<\/h3>\n<p>\n\n<\/p>\n<p>Indexed pages get ranked in search engines based on quality, relevance to search queries, and ability to meet certain other ranking factors. These pages are then served to users when they perform a search.<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_vkg35vmry3bj\"><\/a>6. Crawl&nbsp;ends<\/h3>\n<p>\n\n<\/p>\n<p>Eventually the entire crawl (including the URL rediscovery loop) ends based on factors like time allocated, number of pages crawled, depth of links followed etc.<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_2w3bglshtafw\"><\/a>7. Revisiting<\/h3>\n<p>\n\n<\/p>\n<p>Crawlers periodically <strong>revisit <\/strong>the page to check for updates, new content, or changes in structure.<\/p>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1600\" height=\"1808\" class=\"wp-image-178362\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-a-7-step-flow-diagram-of-how-web-c-2.jpg\" alt=\"Graphic showing a 7 step flow diagram of how web crawlers work\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-a-7-step-flow-diagram-of-how-web-c-2.jpg 1600w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-a-7-step-flow-diagram-of-how-web-c-2-376x425.jpg 376w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-a-7-step-flow-diagram-of-how-web-c-2-768x868.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-a-7-step-flow-diagram-of-how-web-c-2-1359x1536.jpg 1359w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\"><\/figure>\n<p>\n\n<\/p>\n<p>As you can probably guess, the number of URLs discovered and crawled in this process grows exponentially in just a few&nbsp;hops.<\/p>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1430\" height=\"924\" class=\"wp-image-178363\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/a-graphic-visualizing-website-crawlers-following-l-2.png\" alt=\"A graphic visualizing website crawlers following links exponentially\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/a-graphic-visualizing-website-crawlers-following-l-2.png 1430w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/a-graphic-visualizing-website-crawlers-following-l-2-658x425.png 658w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/a-graphic-visualizing-website-crawlers-following-l-2-768x496.png 768w\" sizes=\"auto, (max-width: 1430px) 100vw, 1430px\"><\/figure>\n<p>\n\n<\/p>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"How do you get search engines to crawl your site in the first place?\" data-section=\"search-engine-crawling\">\n<p>\n\n<\/p>\n<h2 id=\"search-engine-crawling\" class=\"wp-block-heading\"><a id=\"post-178325-_qcawfonuzpb\"><\/a>How do you get search engines to crawl your site in the first&nbsp;place?<\/h2>\n<p>\n\n<\/p>\n<\/div><\/div>\n<p>\n\n<\/p>\n<p>Search engine web crawlers are autonomous, meaning you <a href=\"https:\/\/www.searchenginejournal.com\/how-to-trigger-a-complete-re-indexing\/506211\/\">can\u2019t trigger them to crawl or switch them on\/off<\/a> at&nbsp;will.<\/p>\n<p>\n\n<\/p>\n<p>You can, however, help crawlers out&nbsp;with:<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_hk8v51mu9m1w\"><\/a>XML sitemaps<\/h3>\n<p>\n\n<\/p>\n<p>An <a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/sitemaps\/build-sitemap\">XML sitemap<\/a> is a file that lists all the important pages on your website to help search engines accurately discover and index your content.<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_pyv3avc6xxia\"><\/a>Google\u2019s URL inspection tool<\/h3>\n<p>\n\n<\/p>\n<p>You can ask Google to consider recrawling your site content via its <a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/ask-google-to-recrawl\">URL inspection tool<\/a> in Google Search Console. You may get a message in GSC if Google knows about your URL but hasn\u2019t yet crawled or indexed it. If so, find out <a href=\"https:\/\/ahrefs.com\/blog\/discovered-currently-not-indexed\/\">how to fix \u201cDiscovered \u2014 currently not indexed\u201d<\/a>.<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_b6do1mbkhsl8\"><\/a>IndexNow<\/h3>\n<p>\n\n<\/p>\n<p>Instead of waiting for bots to re-crawl and index your content, you can use <a href=\"https:\/\/ahrefs.com\/index-now\/\" data-ahr=\"https:\/\/ahrefs.com\/blog\/indexnow-yep-ahrefs\/\">IndexNow<\/a> to automatically ping search engines like Bing, Yandex, Naver, Seznam.cz, and <a href=\"https:\/\/yep.com\/\">Yep<\/a>, whenever you:<\/p>\n<p>\n\n<\/p>\n<ul class=\"wp-block-list\">\n<li>Add new&nbsp;pages<\/li>\n\n\n\n<li>Update existing content<\/li>\n\n\n\n<li>Remove outdated pages<\/li>\n\n\n\n<li>Implement redirects<\/li>\n<\/ul>\n<p>\n\n<\/p>\n<p>You can <a href=\"https:\/\/help.ahrefs.com\/en\/articles\/9317209-how-to-submit-pages-to-indexnow-within-site-audit\">set up automatic IndexNow submissions via Ahrefs Site&nbsp;Audit.<\/a><\/p>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2048\" height=\"965\" class=\"wp-image-178364\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-indexnow-api-key-in-ahrefs-site-audi-2.jpg\" alt=\"screenshot of IndexNow API key in Ahrefs Site Audit\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-indexnow-api-key-in-ahrefs-site-audi-2.jpg 2048w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-indexnow-api-key-in-ahrefs-site-audi-2-680x320.jpg 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-indexnow-api-key-in-ahrefs-site-audi-2-768x362.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-indexnow-api-key-in-ahrefs-site-audi-2-1536x724.jpg 1536w\" sizes=\"auto, (max-width: 2048px) 100vw, 2048px\"><\/figure>\n<p>\n\n<\/p>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"How to get Google to crawl more of your pages, more often\" data-section=\"crawling-frequency\">\n<p>\n\n<\/p>\n<h2 id=\"crawling-frequency\" class=\"wp-block-heading\"><a id=\"post-178325-_dfh5ovbl8na8\"><\/a>How to get Google to crawl more of your pages, more&nbsp;often<\/h2>\n<p>\n\n<\/p>\n<\/div><\/div>\n<p>\n\n<\/p>\n<p>Search engine crawling decisions are dynamic and a <em>little<\/em> obscure.<\/p>\n<p>\n\n<\/p>\n<p>Although we don\u2019t know the definitive criteria Google uses to determine when or how often to crawl content, we\u2019ve deduced three of the most important areas.<\/p>\n<p>\n\n<\/p>\n<p>This is based on breadcrumbs dropped by Google in support documentation and rep interviews.<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_ervkgcjvveb\"><\/a>1. Prioritize quality<\/h3>\n<p>\n\n<\/p>\n<p><a href=\"https:\/\/ahrefs.com\/seo\/glossary\/pagerank\">Google PageRank<\/a> evaluates the number and quality of links to a page, considering them as \u201cvotes\u201d of importance.<\/p>\n<p>\n\n<\/p>\n<p>Pages earning quality links are deemed more important and are ranked higher in search results.<\/p>\n<p>\n\n<\/p>\n<p>PageRank is a foundational part of Google\u2019s algorithm. It makes sense then that the quality of your links and content plays a big part in how your site is crawled and indexed.<\/p>\n<p>\n\n<\/p>\n<p>To judge your site\u2019s quality, Google looks at factors such&nbsp;as:<\/p>\n<p>\n\n<\/p>\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/links-crawlable\">Internal links<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/links-crawlable\">External links<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/developers.google.com\/search\/docs\/appearance\/page-experience\">Page experience<\/a><\/li>\n<\/ul>\n<p>\n\n<\/p>\n<p>To assess the pages on your site with the most links, check out the Best by Links report in Ahrefs.<\/p>\n<p>\n\n<\/p>\n<p>Pay attention to the \u201cFirst seen\u201d, \u201cLast check\u201d column, which reveals which pages have been crawled most often, and&nbsp;when.<\/p>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1600\" height=\"907\" class=\"wp-image-178365\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/ahrefs-best-by-links-report-highlighting-first-see-2.jpg\" alt=\"Ahrefs Best by Links report highlighting first seen last check column\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/ahrefs-best-by-links-report-highlighting-first-see-2.jpg 1600w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/ahrefs-best-by-links-report-highlighting-first-see-2-680x385.jpg 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/ahrefs-best-by-links-report-highlighting-first-see-2-768x435.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/ahrefs-best-by-links-report-highlighting-first-see-2-1536x871.jpg 1536w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\"><\/figure>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_fj9jszl40cw9\"><\/a>2. Keep things fresh<\/h3>\n<p>\n\n<\/p>\n<p>According to Google\u2019s Senior Search Analyst, <a href=\"https:\/\/www.linkedin.com\/in\/johnmu\/\">John Mueller<\/a>\u2026<\/p>\n<p>\n\n<\/p>\n<blockquote class=\"small\"><div class=\"quote-content\">Search engines recrawl URLs at different rates, sometimes it\u2019s multiple times a day, sometimes it\u2019s once every few months.<\/div><div class=\"quote-info clearfix\"><div class=\"quote-photo\"><img decoding=\"async\" alt=\"John Mueller\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/02\/john-mueller-google.png\"><\/div><div class=\"extra-box\"><span class=\"quote-author\">John Mueller,<\/span> <span class=\"quote-author-job\">Search Advocate, <a href=\"https:\/\/www.linkedin.com\/in\/johnmu\/\" target=\"_blank\">Google<\/a><\/span><\/div><\/div><\/blockquote>\n<p>\n\n<\/p>\n<p>But if you regularly update your content, you\u2019ll see crawlers dropping by more&nbsp;often.<\/p>\n<p>\n\n<\/p>\n<p>Search engines like Google want to deliver accurate and up-to-date information to remain competitive and relevant, so updating your content is like dangling a carrot on a&nbsp;stick.<\/p>\n<p>\n\n<\/p>\n<p>You can examine just how quickly Google processes your updates by checking your <a href=\"https:\/\/support.google.com\/webmasters\/answer\/9679690?hl=en\">crawl stats in Google Search Console<\/a>.<\/p>\n<p>\n\n<\/p>\n<p>While you\u2019re there, look at the breakdown of crawling \u201cBy purpose\u201d (i.e. percent split of pages refreshed vs pages newly discovered). This will also help you work out just how often you\u2019re encouraging web crawlers to revisit your&nbsp;site.<\/p>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"783\" height=\"307\" class=\"wp-image-178366\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/word-image-178325-6-1.png\" alt srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/word-image-178325-6-1.png 783w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/word-image-178325-6-1-680x267.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/word-image-178325-6-1-768x301.png 768w\" sizes=\"auto, (max-width: 783px) 100vw, 783px\"><\/figure>\n<p>\n\n<\/p>\n<p>To find specific pages that need updating on your site, head to the Top Pages report in Ahrefs Site Explorer, then:<\/p>\n<p>\n\n<\/p>\n<ol class=\"wp-block-list\">\n<li>Set the traffic filter to \u201cDeclined\u201d<\/li>\n\n\n\n<li>Set the comparison date to the last year or&nbsp;two<\/li>\n\n\n\n<li>Look at Content Changes status and update pages with only minor changes<\/li>\n<\/ol>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"976\" height=\"573\" class=\"wp-image-178367\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/3-part-process-of-updating-pages-based-on-content-2.png\" alt=\"3 part process of updating pages based on content changes in Ahrefs\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/3-part-process-of-updating-pages-based-on-content-2.png 976w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/3-part-process-of-updating-pages-based-on-content-2-680x399.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/3-part-process-of-updating-pages-based-on-content-2-768x451.png 768w\" sizes=\"auto, (max-width: 976px) 100vw, 976px\"><\/figure>\n<p>\n\n<\/p>\n<p>Top Pages shows you the content on your site driving the most organic traffic. Pushing updates to these pages will encourage crawlers to visit your best content more often, and (hopefully) boost any declining traffic.<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_jszwkwwfbpvp\"><\/a>3. Refine your site structure<\/h3>\n<p>\n\n<\/p>\n<p>Offering a clear site structure via a logical sitemap, and backing that up with relevant internal links will help crawlers:<\/p>\n<p>\n\n<\/p>\n<ul class=\"wp-block-list\">\n<li>Better navigate your&nbsp;site<\/li>\n\n\n\n<li>Understand its hierarchy<\/li>\n\n\n\n<li>Index and rank your most valuable content<\/li>\n<\/ul>\n<p>\n\n<\/p>\n<p>Combined, these factors will also please users, since they support easy navigation, reduced bounce rates, and increased engagement.<\/p>\n<p>\n\n<\/p>\n<p>Below are some more elements that can potentially influence how your site gets discovered and prioritized in crawling:<\/p>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1850\" height=\"1730\" class=\"wp-image-178368\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-the-factors-that-can-affect-web-cr-2.png\" alt=\"Graphic showing the factors that can affect web crawl discoverability\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-the-factors-that-can-affect-web-cr-2.png 1850w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-the-factors-that-can-affect-web-cr-2-454x425.png 454w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-the-factors-that-can-affect-web-cr-2-768x718.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-the-factors-that-can-affect-web-cr-2-1536x1436.png 1536w\" sizes=\"auto, (max-width: 1850px) 100vw, 1850px\"><\/figure>\n<p>\n\n<\/p>\n<div class=\"recommendation\"><div class=\"recommendation-title\">What is crawl budget?<\/div><div class=\"recommendation-content\"> Crawlers mimic the behavior of human users. Every time they visit a web page, the site\u2019s server gets pinged. Pages or sites that are difficult to crawl will incur errors and slow load times, and if a page is visited too often by a crawler bot, servers and webmasters will block it for overusing resources.\n<p>\n\n<\/p>\n<p>For this reason, each site has a crawl budget, which is the number of URLs a crawler <strong>can<\/strong> and <strong>wants<\/strong> to crawl. Factors like site speed, mobile-friendliness, and a logical site structure impact the efficacy of crawl budget.<\/p>\n<p>\n\n<\/p>\n<p>For a deeper dive into crawl budgets, check out Patrick Stox\u2019s guide: <a href=\"https:\/\/ahrefs.com\/blog\/crawl-budget\/\">When Should You Worry About Crawl Budget?<\/a> <\/p><\/div><\/div>\n<p>\n\n<\/p>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"What is a website crawler?\" data-section=\"website-crawler-definition\">\n<p>\n\n<\/p>\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-ad2f72ca wp-block-group-is-layout-flex\">\n<h2 id=\"website-crawler-definition\" class=\"wp-block-heading\"><a id=\"post-178325-_7873si2f62zu\"><\/a>What is a web<span style=\"text-decoration: underline;\">site<\/span> crawler?<\/h2>\n<\/div>\n<p>\n\n<\/p>\n<\/div><\/div>\n<p>\n\n<\/p>\n<p>Web crawlers like Google crawl the entire internet, and you can\u2019t control which sites they visit, or how&nbsp;often.<\/p>\n<p>\n\n<\/p>\n<p>But what you <em>can <\/em>do is use <a href=\"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/\" target=\"_blank\" rel=\"noopener\">website crawlers<\/a>, which are like your own private bots.<\/p>\n<p>\n\n<\/p>\n<p>Ask them to crawl your website to find and fix important SEO problems, or study your competitor\u2019s site and turn their biggest weakness into your next opportunity.<\/p>\n<p>\n\n<\/p>\n<p>Site crawlers essentially simulate search performance. They help you understand how a search engine\u2019s web crawlers might interpret your pages, based on&nbsp;their:<\/p>\n<p>\n\n<\/p>\n<ul class=\"wp-block-list\">\n<li>Structure<\/li>\n\n\n\n<li>Content<\/li>\n\n\n\n<li>Meta data<\/li>\n\n\n\n<li>Page load&nbsp;speed<\/li>\n\n\n\n<li>Errors<\/li>\n\n\n\n<li>Etc<\/li>\n<\/ul>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_89scbnngpwqg\"><\/a>Example: Ahrefs Site&nbsp;Audit<\/h3>\n<p>\n\n<\/p>\n<p>The <a href=\"https:\/\/ahrefs.com\/robot\/site-audit\">Ahrefs Site Audit<\/a> crawler powers the tools: RankTracker, Projects, and Ahrefs\u2019 main website crawling tool: Site&nbsp;Audit.<\/p>\n<p>\n\n<\/p>\n<p>Site Audit helps SEOs&nbsp;to:<\/p>\n<p>\n\n<\/p>\n<ul class=\"wp-block-list\">\n<li>Analyze 170+ technical SEO issues<\/li>\n\n\n\n<li>Conduct on-demand crawls, with live site performance data<\/li>\n\n\n\n<li>Assess up to 170k URLs a minute<\/li>\n\n\n\n<li>Troubleshoot, maintain, and improve their visibility in search engines<\/li>\n<\/ul>\n<p>\n\n<\/p>\n<p>From URL discovery to revisiting, website crawlers operate very similarly to web crawlers \u2013 only instead of indexing and ranking your page in the SERPs, they store and analyze it in their own database.<\/p>\n<p>\n\n<\/p>\n<p>You can crawl your site either locally or remotely. Desktop crawlers like ScreamingFrog let you download and customize your site crawl, while cloud-based tools like Ahrefs Site Audit perform the crawl without using your computer\u2019s resources \u2013 helping you work collaboratively on fixes and site optimization.<\/p>\n<p>\n\n<\/p>\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"How to crawl your own website\" data-section=\"crawl-site\">\n<p>\n\n<\/p>\n<h2 id=\"crawl-site\" class=\"wp-block-heading\"><a id=\"post-178325-_ftnx4vnoe6j3\"><\/a>How to crawl your own website<\/h2>\n<p>\n\n<\/p>\n<\/div><\/div>\n<p>\n\n<\/p>\n<p>If you want to scan entire websites in real time to detect technical SEO problems, configure a crawl in Site&nbsp;Audit.<\/p>\n<p>\n\n<\/p>\n<p>It will give you visual data breakdowns, site health scores, and detailed fix recommendations to help you understand how a search engine interprets your&nbsp;site.<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_ktfiu6h7yz75\"><\/a>1. Set up your&nbsp;crawl<\/h3>\n<p>\n\n<\/p>\n<p>Navigate to the Site Audit tab and choose an existing project, or <a href=\"https:\/\/help.ahrefs.com\/en\/articles\/4455322-setting-up-your-first-project-in-ahrefs-webmaster-tools-awt\">set one up<\/a>.<\/p>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"949\" height=\"706\" class=\"wp-image-178369\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-import-add-project-page-in-ahrefs-si-2.png\" alt=\"Screenshot of import\/add project page in Ahrefs Site Audit\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-import-add-project-page-in-ahrefs-si-2.png 949w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-import-add-project-page-in-ahrefs-si-2-571x425.png 571w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-import-add-project-page-in-ahrefs-si-2-768x571.png 768w\" sizes=\"auto, (max-width: 949px) 100vw, 949px\"><\/figure>\n<p>\n\n<\/p>\n<p>A project is any domain, subdomain, or URL you want to track over&nbsp;time.<\/p>\n<p>\n\n<\/p>\n<p>Once you\u2019ve <a href=\"https:\/\/help.ahrefs.com\/en\/articles\/9082329-how-should-i-configure-my-site-audit-settings\">configured your crawl settings<\/a> \u2013 including your crawl schedule and URL sources \u2013 you can start your audit and you\u2019ll be notified as soon as it\u2019s complete.<\/p>\n<p>\n\n<\/p>\n<p>Here are some things you can do right&nbsp;away.<\/p>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_ptntag70nrpz\"><\/a>2. Diagnose top errors<\/h3>\n<p>\n\n<\/p>\n<p>The Top Issues overview in Site Audit shows you your most pressing errors, warnings, and notices, based on the number of URLs affected.<\/p>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"823\" height=\"414\" class=\"wp-image-178370\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/word-image-178325-10-1.png\" alt srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/word-image-178325-10-1.png 823w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/word-image-178325-10-1-680x342.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/word-image-178325-10-1-768x386.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/word-image-178325-10-1-400x200.png 400w\" sizes=\"auto, (max-width: 823px) 100vw, 823px\"><\/figure>\n<p>\n\n<\/p>\n<p>Working through these as part of your SEO roadmap will help&nbsp;you:<\/p>\n<p>\n\n<\/p>\n<p>1. Spot <strong>errors (red icons)<\/strong> impacting crawling \u2013 e.g.<\/p>\n<p>\n\n<\/p>\n<ul class=\"wp-block-list\">\n<li>HTTP status code\/client errors<\/li>\n\n\n\n<li>Broken links<\/li>\n\n\n\n<li>Canonical issues<\/li>\n<\/ul>\n<p>\n\n<\/p>\n<p>2. Optimize your content and rankings based on <strong>warnings (yellow) <\/strong>\u2013 e.g.<\/p>\n<p>\n\n<\/p>\n<ul class=\"wp-block-list\">\n<li>Missing alt&nbsp;text<\/li>\n\n\n\n<li>Links to redirects<\/li>\n\n\n\n<li>Overly long meta descriptions<\/li>\n<\/ul>\n<p>\n\n<\/p>\n<p>3. Maintain steady visibility with <strong>notices (blue icon)<\/strong> \u2013 e.g.<\/p>\n<p>\n\n<\/p>\n<ul class=\"wp-block-list\">\n<li>Organic traffic drops<\/li>\n\n\n\n<li>Multiple H1s<\/li>\n\n\n\n<li>Indexable pages not in sitemap<\/li>\n<\/ul>\n<p>\n\n<\/p>\n<h4 class=\"wp-block-heading\"><a id=\"post-178325-_vnztyjm20gsj\"><\/a>Filter issues<\/h4>\n<p>\n\n<\/p>\n<p>You can also prioritize fixes using filters.<\/p>\n<p>\n\n<\/p>\n<p>Say you have thousands of pages with missing meta descriptions. Make the task more manageable and impactful by targeting high traffic pages&nbsp;first.<\/p>\n<p>\n\n<\/p>\n<ol class=\"wp-block-list\">\n<li>Head to the Page Explorer report in Site&nbsp;Audit<\/li>\n\n\n\n<li>Select the advanced filter dropdown<\/li>\n\n\n\n<li>Set an internal pages filter<\/li>\n\n\n\n<li>Select an \u2018And\u2019 operator<\/li>\n\n\n\n<li>Select \u2018Meta description\u2019 and \u2018Not exists\u2019<\/li>\n\n\n\n<li>Select \u2018Organic traffic &gt;&nbsp;100\u2019<\/li>\n<\/ol>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1080\" height=\"332\" class=\"wp-image-178371\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-how-to-find-pages-with-missing-meta-2.png\" alt=\"Screenshot of how to find pages with missing meta descriptions, over 100 organic traffic, in Ahrefs Page Explorer\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-how-to-find-pages-with-missing-meta-2.png 1080w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-how-to-find-pages-with-missing-meta-2-680x209.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-how-to-find-pages-with-missing-meta-2-768x236.png 768w\" sizes=\"auto, (max-width: 1080px) 100vw, 1080px\"><\/figure>\n<p>\n\n<\/p>\n<h4 class=\"wp-block-heading\"><a id=\"post-178325-_x4d7jfac3ecj\"><\/a>Crawl the most important parts of your&nbsp;site<\/h4>\n<p>\n\n<\/p>\n<p>Segment and zero-in on the most important pages on your site (e.g. subfolders or subdomains) using Site Audit\u2019s 200+ filters \u2013 whether that\u2019s your blog, ecommerce store, or even pages that earn over a certain traffic threshold.<\/p>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2048\" height=\"713\" class=\"wp-image-178372\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-ahrefs-site-audit-pointing-out-confi-2.jpg\" alt=\"Screenshot of Ahrefs Site Audit pointing out configure segment option\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-ahrefs-site-audit-pointing-out-confi-2.jpg 2048w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-ahrefs-site-audit-pointing-out-confi-2-680x237.jpg 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-ahrefs-site-audit-pointing-out-confi-2-768x267.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-ahrefs-site-audit-pointing-out-confi-2-1536x535.jpg 1536w\" sizes=\"auto, (max-width: 2048px) 100vw, 2048px\"><\/figure>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_r7gxs9mzjifo\"><\/a>3. Expedite fixes<\/h3>\n<p>\n\n<\/p>\n<p>If you don\u2019t have coding experience, then the prospect of crawling your site and implementing fixes can be intimidating.<\/p>\n<p>\n\n<\/p>\n<p>If you <em>do <\/em>have dev support, issues are easier to remedy, but then it becomes a matter of bargaining for another person\u2019s time.<\/p>\n<p>\n\n<\/p>\n<p>We\u2019ve got a new feature to help you solve for these kinds of headaches. <a href=\"https:\/\/ahrefs.com\/patches\/\" data-ahr=\"https:\/\/ahrefs.com\/blog\/site-audit-patches\/\">Patches<\/a> are fixes you can make autonomously in Site&nbsp;Audit.<\/p>\n<p>\n\n<\/p>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1500\" height=\"650\" class=\"wp-image-178373\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-ahrefs-patches-tool-calling-out-the-2.png\" alt=\"Screenshot of Ahrefs Patches tool calling out the Patch It feature\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-ahrefs-patches-tool-calling-out-the-2.png 1500w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-ahrefs-patches-tool-calling-out-the-2-680x295.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-ahrefs-patches-tool-calling-out-the-2-768x333.png 768w\" sizes=\"auto, (max-width: 1500px) 100vw, 1500px\"><\/figure>\n<p>\n\n<\/p>\n<p>Title changes, missing meta descriptions, site-wide broken links \u2013 when you face these kinds of errors you can hit \u201cPatch it\u201d to publish a fix directly to your website, without having to pester a&nbsp;dev.<\/p>\n<p>\n\n<\/p>\n<p>And if you\u2019re unsure of anything, you can roll-back your patches at any&nbsp;point.<\/p>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1500\" height=\"229\" class=\"wp-image-178374\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-ahrefs-patches-tool-calling-out-draf-2.png\" alt=\"Screenshot of Ahrefs Patches tool calling out drafts, published, and unpublished statuses\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-ahrefs-patches-tool-calling-out-draf-2.png 1500w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-ahrefs-patches-tool-calling-out-draf-2-680x104.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-ahrefs-patches-tool-calling-out-draf-2-768x117.png 768w\" sizes=\"auto, (max-width: 1500px) 100vw, 1500px\"><\/figure>\n<p>\n\n<\/p>\n<h3 class=\"wp-block-heading\"><a id=\"post-178325-_jeroofko2hvh\"><\/a>4. Spot optimization opportunities<\/h3>\n<p>\n\n<\/p>\n<p>Auditing your site with a website crawler is as much about spotting opportunities as it is about fixing bugs.<\/p>\n<p>\n\n<\/p>\n<h4 class=\"wp-block-heading\"><a id=\"post-178325-_gfeuj0p4xihm\"><\/a>Improve internal linking<\/h4>\n<p>\n\n<\/p>\n<p>The Internal Link Opportunities report in Site Audit shows you relevant internal linking suggestions, by taking the top 10 keywords (by traffic) for each crawled page, then looking for mentions of them on your other crawled pages.<\/p>\n<p>\n\n<\/p>\n<p>\u2018Source\u2019 pages are the ones you should link <strong>from<\/strong>, and \u2018Target\u2019 pages are the ones you should link <strong>to<\/strong>.<\/p>\n<p>\n\n<\/p>\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1100\" height=\"435\" class=\"wp-image-178375\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-internal-link-opportunities-report-i-2.png\" alt=\"Screenshot of Internal Link Opportunities report in Ahrefs Site Audit highlighting source page and target page\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-internal-link-opportunities-report-i-2.png 1100w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-internal-link-opportunities-report-i-2-680x269.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/screenshot-of-internal-link-opportunities-report-i-2-768x304.png 768w\" sizes=\"auto, (max-width: 1100px) 100vw, 1100px\"><\/figure>\n<p>\n\n<\/p>\n<p>The more high quality connections you make between your content, the easier it will be for Googlebot to crawl your&nbsp;site.<\/p>\n<p>\n\n<\/p>\n<h2 class=\"wp-block-heading\"><a id=\"post-178325-_4hpkglbpmvkt\"><\/a>Final thoughts<\/h2>\n<p>\n\n<\/p>\n<p>Understanding website crawling is more than just an SEO hack \u2013 it\u2019s foundational knowledge that directly impacts your traffic and&nbsp;ROI.<\/p>\n<p>\n\n<\/p>\n<p>Knowing how crawlers work means knowing how search engines \u201csee\u201d your site, and that\u2019s half the battle when it comes to ranking.<\/p>\n<p><\/p>","protected":false},"excerpt":{"rendered":"<p>Search engines are increasingly ruthless when it comes to the quality of the sites they allow into the search results. If you don\u2019t grasp the basics of optimizing for web crawlers (and eventual users), your organic traffic may well pay<span class=\"ellipsis\">\u2026<\/span><\/p>\n<div class=\"read-more\">Read more \u203a<\/div>\n<p><!-- end of .read-more --><\/p>\n","protected":false},"author":197,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"wp_typography_post_enhancements_disabled":false,"footnotes":""},"categories":[335,329],"tags":[],"coauthors":[464],"class_list":["post-178325","post","type-post","status-publish","format-standard","hentry","category-general-seo","category-technical-seo","odd"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Crawl Me Maybe? How Website Crawlers Work<\/title>\n<meta name=\"description\" content=\"If you don&#039;t grasp the basics of optimizing for web crawlers (and eventual users), your organic traffic may well pay the price.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ahrefs.com\/blog\/website-crawlers\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Crawl Me Maybe? How Website Crawlers Work\" \/>\n<meta property=\"og:description\" content=\"If you don&#039;t grasp the basics of optimizing for web crawlers (and eventual users), your organic traffic may well pay the price.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ahrefs.com\/blog\/website-crawlers\/\" \/>\n<meta property=\"og:site_name\" content=\"SEO Blog by Ahrefs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Ahrefs\/\" \/>\n<meta property=\"article:published_time\" content=\"2024-08-19T11:15:42+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-01-30T12:15:09+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-ahrefsbot-crawler-as-the-1-most-a-2.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2048\" \/>\n\t<meta property=\"og:image:height\" content=\"1226\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Louise Linehan\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@ahrefs\" \/>\n<meta name=\"twitter:site\" content=\"@ahrefs\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/website-crawlers\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/website-crawlers\\\/\"},\"author\":{\"name\":\"Louise Linehan\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/444b3643c35b16b94b763446c5562388\"},\"headline\":\"Crawl Me Maybe? How Website Crawlers Work\",\"datePublished\":\"2024-08-19T11:15:42+00:00\",\"dateModified\":\"2025-01-30T12:15:09+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/website-crawlers\\\/\"},\"wordCount\":2477,\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/website-crawlers\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2024\\\/08\\\/crawl-me-maybe-how-website-crawlers-by-louise-linehan-general-seo.jpg\",\"articleSection\":[\"General SEO\",\"Technical SEO\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/website-crawlers\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/website-crawlers\\\/\",\"name\":\"Crawl Me Maybe? How Website Crawlers Work\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/website-crawlers\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/website-crawlers\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2024\\\/08\\\/graphic-showing-ahrefsbot-crawler-as-the-1-most-a-2.jpg\",\"datePublished\":\"2024-08-19T11:15:42+00:00\",\"dateModified\":\"2025-01-30T12:15:09+00:00\",\"description\":\"If you don't grasp the basics of optimizing for web crawlers (and eventual users), your organic traffic may well pay the price.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ahrefs.com\\\/blog\\\/website-crawlers\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/website-crawlers\\\/#primaryimage\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2024\\\/08\\\/graphic-showing-ahrefsbot-crawler-as-the-1-most-a-2.jpg\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2024\\\/08\\\/graphic-showing-ahrefsbot-crawler-as-the-1-most-a-2.jpg\",\"width\":2048,\"height\":1226,\"caption\":\"Graphic showing AhrefsBot crawler as the #1 most active SEO crawler and #4 most active web crawler in the world\"},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"name\":\"SEO Blog by Ahrefs\",\"description\":\"Link Building Strategies &amp; SEO Tips\",\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\",\"name\":\"Ahrefs\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"width\":2048,\"height\":768,\"caption\":\"Ahrefs\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/Ahrefs\\\/\",\"https:\\\/\\\/x.com\\\/ahrefs\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/ahrefs\\\/\",\"https:\\\/\\\/www.youtube.com\\\/c\\\/ahrefscom\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/444b3643c35b16b94b763446c5562388\",\"name\":\"Louise Linehan\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2024\\\/08\\\/Louise-Linehan.jpg02b05bbed9b25ec9b04e39f0d88f15b0\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2024\\\/08\\\/Louise-Linehan.jpg\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2024\\\/08\\\/Louise-Linehan.jpg\",\"caption\":\"Louise Linehan\"},\"description\":\"Louise is a Content Marketer at Ahrefs. Over the past ten years, she has held senior content positions at SaaS brands: Pi Datametrics, BuzzSumo, and Cision. By day, she writes about content and SEO; by night, you'll find her playing football or screaming down the mic at karaoke.\",\"sameAs\":[\"https:\\\/\\\/www.linkedin.com\\\/in\\\/louise-linehan\\\/\"],\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/author\\\/louise-linehan\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Crawl Me Maybe? How Website Crawlers Work","description":"If you don't grasp the basics of optimizing for web crawlers (and eventual users), your organic traffic may well pay the price.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ahrefs.com\/blog\/website-crawlers\/","og_locale":"en_US","og_type":"article","og_title":"Crawl Me Maybe? How Website Crawlers Work","og_description":"If you don't grasp the basics of optimizing for web crawlers (and eventual users), your organic traffic may well pay the price.","og_url":"https:\/\/ahrefs.com\/blog\/website-crawlers\/","og_site_name":"SEO Blog by Ahrefs","article_publisher":"https:\/\/www.facebook.com\/Ahrefs\/","article_published_time":"2024-08-19T11:15:42+00:00","article_modified_time":"2025-01-30T12:15:09+00:00","og_image":[{"width":2048,"height":1226,"url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-ahrefsbot-crawler-as-the-1-most-a-2.jpg","type":"image\/jpeg"}],"author":"Louise Linehan","twitter_card":"summary_large_image","twitter_creator":"@ahrefs","twitter_site":"@ahrefs","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ahrefs.com\/blog\/website-crawlers\/#article","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/website-crawlers\/"},"author":{"name":"Louise Linehan","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/444b3643c35b16b94b763446c5562388"},"headline":"Crawl Me Maybe? How Website Crawlers Work","datePublished":"2024-08-19T11:15:42+00:00","dateModified":"2025-01-30T12:15:09+00:00","mainEntityOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/website-crawlers\/"},"wordCount":2477,"publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/website-crawlers\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/crawl-me-maybe-how-website-crawlers-by-louise-linehan-general-seo.jpg","articleSection":["General SEO","Technical SEO"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/ahrefs.com\/blog\/website-crawlers\/","url":"https:\/\/ahrefs.com\/blog\/website-crawlers\/","name":"Crawl Me Maybe? How Website Crawlers Work","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/website-crawlers\/#primaryimage"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/website-crawlers\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-ahrefsbot-crawler-as-the-1-most-a-2.jpg","datePublished":"2024-08-19T11:15:42+00:00","dateModified":"2025-01-30T12:15:09+00:00","description":"If you don't grasp the basics of optimizing for web crawlers (and eventual users), your organic traffic may well pay the price.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ahrefs.com\/blog\/website-crawlers\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/website-crawlers\/#primaryimage","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-ahrefsbot-crawler-as-the-1-most-a-2.jpg","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/graphic-showing-ahrefsbot-crawler-as-the-1-most-a-2.jpg","width":2048,"height":1226,"caption":"Graphic showing AhrefsBot crawler as the #1 most active SEO crawler and #4 most active web crawler in the world"},{"@type":"WebSite","@id":"https:\/\/ahrefs.com\/blog\/#website","url":"https:\/\/ahrefs.com\/blog\/","name":"SEO Blog by Ahrefs","description":"Link Building Strategies &amp; SEO Tips","publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ahrefs.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ahrefs.com\/blog\/#organization","name":"Ahrefs","url":"https:\/\/ahrefs.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","width":2048,"height":768,"caption":"Ahrefs"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Ahrefs\/","https:\/\/x.com\/ahrefs","https:\/\/www.linkedin.com\/company\/ahrefs\/","https:\/\/www.youtube.com\/c\/ahrefscom"]},{"@type":"Person","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/444b3643c35b16b94b763446c5562388","name":"Louise Linehan","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/Louise-Linehan.jpg02b05bbed9b25ec9b04e39f0d88f15b0","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/Louise-Linehan.jpg","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/Louise-Linehan.jpg","caption":"Louise Linehan"},"description":"Louise is a Content Marketer at Ahrefs. Over the past ten years, she has held senior content positions at SaaS brands: Pi Datametrics, BuzzSumo, and Cision. By day, she writes about content and SEO; by night, you'll find her playing football or screaming down the mic at karaoke.","sameAs":["https:\/\/www.linkedin.com\/in\/louise-linehan\/"],"url":"https:\/\/ahrefs.com\/blog\/author\/louise-linehan\/"}]}},"as_json":null,"json_reviewers":[194],"_links":{"self":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/178325","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/users\/197"}],"replies":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/comments?post=178325"}],"version-history":[{"count":0,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/178325\/revisions"}],"wp:attachment":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media?parent=178325"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/categories?post=178325"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/tags?post=178325"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/coauthors?post=178325"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}