{"id":133600,"date":"2021-05-28T09:15:20","date_gmt":"2021-05-28T14:15:20","guid":{"rendered":"https:\/\/ahrefs.com\/blog\/?p=133600"},"modified":"2025-03-13T22:37:17","modified_gmt":"2025-03-14T03:37:17","slug":"crawl-budget","status":"publish","type":"post","link":"https:\/\/ahrefs.com\/blog\/crawl-budget\/","title":{"rendered":"When Should You Worry About Crawl Budget?"},"content":{"rendered":"\n<div class=\"intro-txt\">Crawl budget is the amount of time and resources a search engine allows for crawling a website. It is made up crawl demand which is how many pages a search engine wants to crawl on your site and crawl rate which is how fast they can crawl. It\u2019s affected by the amount of resources a crawler wants to use on your site and the amount of crawling your server supports.<\/div>\n\n\n\n<p>More crawling doesn\u2019t mean you\u2019ll rank better, but if your pages aren\u2019t crawled and indexed they aren\u2019t going to rank at&nbsp;all.&nbsp;<\/p>\n\n\n\n<p>Most sites don\u2019t need to worry about crawl budget, but there are few cases where you may want to take a look. Let\u2019s look at some of those&nbsp;cases.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"#when-to-worry\">When should you worry about crawl budget?<\/a><\/li>\n<li><a href=\"#how-to-check-crawl-activity\">How to check crawl activity<\/a><\/li>\n<li><a href=\"#what-counts-against-crawl-budget\">What counts against crawl budget?<\/a><\/li>\n<li><a href=\"#google-adjusts-crawling\">How does Google adjusts its crawling?&nbsp;<\/a><\/li>\n<li><a href=\"#crawl-faster\">How can I make Google crawl faster?<\/a><\/li>\n<li><a href=\"#crawl-slower\">How can I make Google crawl slower?<\/a><\/li>\n<\/ul>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"When should you worry about crawl budget?\" data-section=\"when-to-worry\">\n\n\n\n<h2 class=\"wp-block-heading\">When should you worry about crawl budget?<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>SEOs usually don\u2019t have to worry about crawl budget on popular pages. It\u2019s usually pages that are newer, that aren\u2019t well linked, or don\u2019t change much that are not crawled often.<\/p>\n\n\n\n<p>Crawl budget can be a concern for newer sites, especially those with a lot of pages. Your server may be able to support more crawling, but because your site is new and likely not very popular yet, a <a href=\"https:\/\/ahrefs.com\/seo\/how-do-search-engines-work\" data-ahr=\"https:\/\/ahrefs.com\/blog\/how-do-search-engines-work\/\">search engine<\/a>&nbsp;may not want to crawl your site very&nbsp;much.<\/p>\n<p>\n\n This is mostly a disconnect in expectations. You want your pages crawled and indexed but Google doesn\u2019t know if it\u2019s worth indexing your pages and may not want to crawl as many pages as you want them&nbsp;to.<\/p>\n\n\n\n<p>Crawl budget can also be a concern for larger sites with millions of pages or sites that are frequently updated. In general, if you have lots of pages not being crawled or updated as often as you\u2019d like, then you may want to look into speeding up crawling. We\u2019ll talk about how to do that later in the article.<br><br>If you have a lot of pages in Google Search Console marked as <a href=\"https:\/\/ahrefs.com\/blog\/discovered-currently-not-indexed\/\">Discovered - currently not indexed<\/a>, that means that Google hasn\u2019t crawled them yet and you may have crawl budget issues. Check the Page Indexing report and you\u2019ll see&nbsp;this.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-large wp-image-184486\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/discovered-currently-not-indexed-604x425.png\" alt=\"discovered currently not indexed means not crawled and can indicate crawl budget issues\" width=\"604\" height=\"425\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/discovered-currently-not-indexed-604x425.png 604w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/discovered-currently-not-indexed-768x540.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/discovered-currently-not-indexed-1536x1080.png 1536w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/discovered-currently-not-indexed.png 1812w\" sizes=\"auto, (max-width: 604px) 100vw, 604px\"><\/p>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"How to check crawl activity\" data-section=\"how-to-check-crawl-activity\">\n\n\n\n<h2 class=\"wp-block-heading\">How to check&nbsp;crawl activity<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>If you want to see an overview of Google crawl activity and any issues they identified, the best place to look is the <a href=\"https:\/\/search.google.com\/search-console\/settings\/crawl-stats\">Crawl Stats report<\/a>&nbsp;in Google Search Console.<\/p>\n\n\n\n<figure class=\"wp-block-image is-style-default\"><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/1-crawl-stats-google-search-console.png\" alt=\"crawl stats report in Google Search Console\"><\/figure>\n\n\n\n<p>There are various reports here to help you identify changes in crawling behavior, issues with crawling, and give you more information about how Google is crawling your&nbsp;site.<\/p>\n\n\n\n<p>You definitely want to look into any <a href=\"https:\/\/support.google.com\/webmasters\/answer\/9679690?hl=en\">flagged crawl statuses<\/a>&nbsp;like the ones shown&nbsp;here:<\/p>\n\n\n\n<figure class=\"wp-block-image is-style-default\"><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/2-flagged-crawl-status.png\" alt=\"flagged crawl statuses\"><\/figure>\n\n\n\n<p>There are also timestamps of when pages were last crawled.<\/p>\n\n\n\n<figure class=\"wp-block-image is-style-default\"><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/3-crawl-timestamps.png\" alt=\"crawl timestamps showing last crawled\"><\/figure>\n\n\n\n<p>If you want to see hits from all bots and users, you\u2019ll need access to your log files. Depending on hosting and setup, you may have access to tools like Awstats and Webalizer as is seen here on a shared host with cPanel. These tools show some aggregated data from your log&nbsp;files.<\/p>\n\n\n\n<figure class=\"wp-block-image is-style-default\"><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/4-log-files.png\" alt=\"where to find crawl logs\"><\/figure>\n\n\n\n<p>For more complex setups you\u2019ll have to get access to and store data from the raw log files, possibly from multiple sources. You may also need specialized tools for larger projects such as an&nbsp;ELK (elasticsearch, logstash, kibana) stack which allows for storage, processing, and visualization of log files. There are also log analysis tools such as <a href=\"https:\/\/www.splunk.com\/\">Splunk<\/a>.&nbsp;<\/p>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"What counts against crawl budget?\" data-section=\"what-counts-against-crawl-budget\">\n\n\n\n<h2 class=\"wp-block-heading\">What counts against crawl budget?<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>All URLs and requests count against your crawl budget. This includes alternate URLs like AMP or m-dot pages, hreflang, CSS, embedded content, and JavaScript including XHR requests. XHR requests aren\u2019t cached like other files so you will see significantly more requests on those for sites build on JavaScript frameworks. There are more details about this in our <a href=\"https:\/\/ahrefs.com\/blog\/javascript-seo\/\">JavaScript SEO Guide<\/a>.<\/p>\n\n\n\n<p>These URLs may be found by crawling and parsing pages, or from a variety of other sources including canonical link elements, links on the pages, sitemaps, RSS feeds, submitting URLs for indexing in Google Search Console, or using the <a href=\"https:\/\/developers.google.com\/search\/apis\/indexing-api\/v3\/quickstart\">indexing API<\/a>.<\/p>\n\n\n\n<p>There are also <a href=\"https:\/\/ahrefs.com\/blog\/googlebot\/\">multiple Googlebots<\/a>&nbsp;that share the crawl budget. You can find a list of the various Googlebots crawling your website in the Crawl Stats report in&nbsp;GSC.<\/p>\n\n\n\n<figure class=\"wp-block-image is-style-default\"><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/5-googlebot-type.png\" alt=\"list of the different types of googlebots like desktop, mobile, image\"><\/figure>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"Google adjusts how they crawl\" data-section=\"google-adjusts-crawling\">\n\n\n\n<h2 class=\"wp-block-heading\">Google adjusts how they&nbsp;crawl<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>Each website will have a different crawl budget that\u2019s made up of a few different inputs.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Crawl demand<\/h3>\n\n\n\n<p>Crawl demand is simply how much Google wants to crawl on your website. More popular pages and pages that experience significant changes will be crawled more.<\/p>\n\n\n\n<p>Popular pages, or those with more links and <a href=\"https:\/\/ahrefs.com\/blog\/google-pagerank\/\">PageRank<\/a>, will generally receive priority over other pages. Remember that Google has to prioritize your pages for crawling in some way, and links are an easy way to determine which pages on your site are more popular. It\u2019s not just your site though, it\u2019s all pages on all sites on the internet that Google has to figure out how to prioritize.<\/p>\n<p>You can use the <strong>Best by links<\/strong>&nbsp;report in <a href=\"https:\/\/ahrefs.com\/site-explorer\">Site Explorer<\/a>&nbsp;as an indication of which pages are likely to be crawled more often. It also shows you when Ahrefs last crawled your&nbsp;pages.<\/p>\n\n\n\n\n\n<figure class=\"wp-block-image is-style-default\"><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/6-best-by-links.jpg\" alt=\"this report shows pages with more links which may get more crawl budget\"><\/figure>\n\n\n\n<p>Google also looks at which pages they are serving in their index the most often when determining crawl demand. They want to crawl pages they have to serve more often to make sure they\u2019re up to&nbsp;date.<\/p>\n<p>According to Google Webmaster Trends Analyst Gary Illyes, ~60% of the internet is duplicate content.<\/p>\n<p><\/p>\n<blockquote class=\"twitter-tweet\">\n<p dir=\"ltr\" lang=\"en\">Google\u2019s crawling process is highly focused on removing duplication because 60% of the internet is duplicate <a href=\"https:\/\/twitter.com\/methode?ref_src=twsrc%5Etfw\">@methode<\/a> <a href=\"https:\/\/twitter.com\/hashtag\/seodaydk?src=hash&amp;ref_src=twsrc%5Etfw\">#seodaydk<\/a> <a href=\"https:\/\/t.co\/OJ9OkP74DU\">pic.twitter.com\/OJ9OkP74DU<\/a><\/p>\n\u2014 Lily Ray (@lilyraynyc) <a href=\"https:\/\/twitter.com\/lilyraynyc\/status\/1509176261884747781?ref_src=twsrc%5Etfw\">March 30,&nbsp;2022<\/a><\/blockquote>\n<p><script src=\"https:\/\/platform.twitter.com\/widgets.js\" async charset=\"utf-8\"><\/script><\/p>\n<p>Most of these duplicates are technical in nature, and they can waste crawl budget. Here are some examples.<\/p>\n<p><!-- \/wp:post-content -->\n\n<!-- wp:list --><\/p>\n<ul>\n<li><strong>HTTP and <\/strong><a href=\"https:\/\/ahrefs.com\/blog\/what-is-https\/\"><strong>HTTPS<\/strong><\/a><strong> variants <span style=\"font-weight: 400;\">\u2013<\/span><\/strong> Examples: http:\/\/www.example.com and https:\/\/www.example.com.<\/li>\n<li><strong>Non-www and www variants <span style=\"font-weight: 400;\">\u2013 Examples: <\/span><\/strong>http:\/\/example.com and http:\/\/www.example.com.<\/li>\n<li><strong>URLs with and without <\/strong><a href=\"https:\/\/ahrefs.com\/blog\/trailing-slash\/\"><strong>trailing slashes<\/strong><\/a> <strong><span style=\"font-weight: 400;\">\u2013 Examples: <\/span><\/strong>https:\/\/example.com\/page\/ and https:\/\/example.com\/page.<\/li>\n<li><strong>URLs with and without capital letters<\/strong> <strong><span style=\"font-weight: 400;\">\u2013 Examples:<\/span><\/strong> https:\/\/example.com\/page\/ and https:\/\/example.com\/Page\/.<\/li>\n<li><strong>Default versions of the page, such as index pages<\/strong> <strong><span style=\"font-weight: 400;\">\u2013 Examples:<\/span><\/strong> https:\/\/www.example.com\/, https:\/\/www.example.com\/index.htm, https:\/\/www.example.com\/index.html, https:\/\/www.example.com\/index.php, https:\/\/www.example.com\/default.htm, etc.<\/li>\n<li><a href=\"https:\/\/ahrefs.com\/blog\/url-parameters\/\"><strong>URL parameters<\/strong><\/a> <strong><span style=\"font-weight: 400;\">\u2013 <\/span><\/strong>Examples: example.com?parameter=whatever. These may exist because of tracking codes, faceted navigation, sorting content, session IDs,&nbsp;etc.&nbsp;<\/li>\n<\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p>There\u2019s also a concept of staleness. If Google sees that a page isn\u2019t changing, they will crawl the page less frequently. For instance, if they crawl a page and see no changes after a day, they may wait three days before crawling again, ten days the next time, 30 days, 100 days, etc. There\u2019s no actual set period they will wait between crawls, but it will become more infrequent over time. However, if Google sees large changes on the site as a whole or a site move, they will typically increase the crawl rate, at least temporarily.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:heading {\"level\":3} -->\n<h3>Crawl rate&nbsp;limit<\/h3>\n<!-- \/wp:heading -->\n\n<!-- wp:paragraph -->\n<p>Crawl rate limit is how much crawling your website can support. Websites have a certain amount of crawling they can take before having issues with the stability of the server like slowdowns or errors. Most crawlers will back off crawling if they start to see these issues so they do not harm the&nbsp;site.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Google will adjust based on the crawl health of the site. If the site is fine with more crawling, then the limit will increase. If the site is having issues, then Google will slow down the rate at which they&nbsp;crawl.<\/p>\n<p>Google will slow down their crawling if they receive too many 5xx (server errors) or 429 (too many requests) <a href=\"https:\/\/ahrefs.com\/blog\/http-status-codes\/\">HTTP status codes<\/a>. They\u2019ll also slow down if the connect times for the server slow&nbsp;down<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"I want Google to crawl faster\" data-section=\"crawl-faster\">\n<!-- \/wp:paragraph -->\n\n<!-- wp:heading {\"level\":2} -->\n<h2>I want Google to crawl faster \/ smarter<\/h2>\n<!-- \/wp:heading -->\n\n<!-- wp:paragraph -->\n<\/div><\/div>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>There are a few things you can do to make sure your site can support additional crawling and increase your site\u2019s crawl demand. Let\u2019s look at some of those options.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:heading {\"level\":3} -->\n<h3>Speed up your server \/ increase resources<\/h3>\n<!-- \/wp:heading -->\n\n<!-- wp:paragraph -->\n<p>The way Google crawls pages is basically to download resources and then process them on their end. Your <a href=\"https:\/\/ahrefs.com\/blog\/core-web-vitals\/\" data-ahr=\"https:\/\/ahrefs.com\/blog\/advanced-pagespeed-guide\/\">page speed<\/a> as a user perceives it isn\u2019t quite the same. What will impact crawl budget is how fast Google can connect and download resources which has more to do with the server and resources. Check your logs for any 5xx or 429 errors.<\/p>\n<h3>Add important pages to sitemaps<\/h3>\n<p>Make sure that any page that you want crawled and indexed is included in your <a href=\"https:\/\/ahrefs.com\/blog\/how-to-create-a-sitemap\/\">sitemap<\/a>.&nbsp;<\/p>\n<h3>Get rid of duplicate content<\/h3>\n<p>Earlier, we talked about how the web is 60% duplicate and that many of those extra pages are technical in nature. If you remove these duplicates, Google can focus on crawling unique URLs. For instance, you may want to block faceted navigation URLs with robots.txt or use a # instead of a ? for the parameter since the # is typically ignored by servers and crawlers.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:heading {\"level\":3} -->\n<h3>Get more links, external &amp; internal<\/h3>\n<!-- \/wp:heading -->\n\n<!-- wp:paragraph -->\n<p>Remember that crawl demand is generally based on popularity or links. You can increase your budget by increasing the amount of <a href=\"https:\/\/ahrefs.com\/seo\/link-building\" data-ahr=\"https:\/\/ahrefs.com\/blog\/link-building\/\">external links<\/a>&nbsp;and\/or <a href=\"https:\/\/ahrefs.com\/blog\/link-opportunities\/\">internal links<\/a>. Internal links are easier since you control the site. You can find suggested internal links in the <strong>Link Opportunities<\/strong>&nbsp;report in <a href=\"https:\/\/ahrefs.com\/site-audit\">Site Audit<\/a>,&nbsp;which also includes a tutorial explaining how it&nbsp;works.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:image -->\n<figure class=\"wp-block-image is-style-default\"><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/7-link-opportunities.jpg\" alt=\"adding more internal links can help with crawl budget\"><\/figure>\n<!-- \/wp:image -->\n\n<!-- wp:heading {\"level\":3} -->\n<h3>Fix redirected links<\/h3>\n<!-- \/wp:heading -->\n\n<!-- wp:paragraph -->\n<p>Keeping links to redirected pages on your site active will have a small impact on crawl budget. Typically, the pages linked here will have a fairly low priority because they probably haven\u2019t changed in a while, but cleaning up any issues is good for website maintenance in general and will help your crawl budget a&nbsp;bit.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>You can find redirected (3xx) links on your site easily in the <strong>Internal pages <\/strong>report in Site&nbsp;Audit.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:image -->\n<figure class=\"wp-block-image is-style-default\"><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/8-site-audit.png\" alt=\"fixing redirected links can help with your crawl budget\"><\/figure>\n<!-- \/wp:image -->\n\n<!-- wp:paragraph -->\n<p>For redirected links in the sitemap, check the <strong>All issues<\/strong> report for the \u201c3XX redirect in sitemap\u201d issue.<\/p>\n<h3>Host resources on a different hostname<\/h3>\n<p>If you use a different subdomain or a CDN to host your resources, this shifts crawl budget concerns to the place that is hosting the resources. It can take the burden off your own server resources and free up crawl budget.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:heading {\"level\":3} -->\n<h3>Use GET instead of POST where you&nbsp;can<\/h3>\n<!-- \/wp:heading -->\n\n<!-- wp:paragraph -->\n<p>This one is a little more technical in that it involves HTTP Request methods. Don\u2019t use POST requests where GET requests work. It\u2019s basically GET (pull) vs POST (push). POST requests aren\u2019t cached so they do impact crawl budget, but GET requests can be cached.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:heading {\"level\":3} -->\n<h3>Use the Indexing API<\/h3>\n<!-- \/wp:heading -->\n\n<!-- wp:paragraph -->\n<p>If you need pages crawled faster, check if you\u2019re eligible for Google\u2019s <a href=\"https:\/\/developers.google.com\/search\/apis\/indexing-api\/v3\/quickstart\">Indexing API<\/a>. Currently this is only available for a few use cases like job postings or live videos.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Bing also has an <a href=\"https:\/\/www.bing.com\/webmasters\/url-submission-api\">Indexing API<\/a>&nbsp;that\u2019s available to everyone.<\/p>\n<h3>304 (Not Modified)<\/h3>\n<p>Googlebot doesn\u2019t always send an If-Modified-Since or If-None-Match HTTP request header. If they do send it and your server responds with a 304 (Not Modified) response with no body content, then Google will reuse the content from the last time they crawled.<br><br>This will use a bit less server resources, meaning Google may be able to crawl your site more. However, it is a technically complex setup and it just doesn\u2019t feel like it\u2019s worth it. You\u2019re likely better off working on your caching setup rather than even attempting this approach.<\/p>\n<h3>What may&nbsp;work<\/h3>\n<p>This is a bit iffy as to whether it will help your crawl budget or&nbsp;not.<\/p>\n<ul>\n<li><strong>Nofollow.<\/strong> In the past <a href=\"https:\/\/ahrefs.com\/blog\/nofollow-links\/\">nofollow links<\/a> wouldn\u2019t have used crawl budget. However, nofollow is now treated as a hint so Google may choose to crawl these links. They also may find the links as followed on other pages on the&nbsp;web.<\/li>\n<\/ul>\n<!-- \/wp:paragraph -->\n\n<!-- wp:heading {\"level\":3} -->\n<h3>What won\u2019t&nbsp;work<\/h3>\n<!-- \/wp:heading -->\n\n<!-- wp:paragraph -->\n<p>There are a few things people sometimes try that won\u2019t actually help with your crawl budget.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul>\n<li><strong>Small changes to the site.<\/strong>&nbsp;Making small changes on pages like updating dates, spaces, or punctuation in hopes of getting pages crawled more often. Google is pretty good at determining whether changes are significant or not, so these small changes aren\u2019t likely to have any impact on crawling.<\/li>\n<li><strong>Noindex<\/strong>. Google still has to crawl pages to see the <a href=\"https:\/\/ahrefs.com\/seo\/glossary\/noindex-tag\">noindex tag<\/a>. Unless you were already hitting the crawl limit for your site, Google won\u2019t reallocate the crawl budget to other&nbsp;pages.<\/li>\n<li><strong>Crawl-delay directive in robots.txt.<\/strong>&nbsp;This directive will slow down many bots. However Googlebot doesn\u2019t use it so it won\u2019t have an impact. We do respect this at Ahrefs, so if you ever need to slow down our crawling you can add a crawl delay in your <a href=\"https:\/\/ahrefs.com\/blog\/robots-txt\/\">robots.txt<\/a>&nbsp;file.<\/li>\n<li><strong>Removing third-party scripts.<\/strong>&nbsp;Third-party party scripts don\u2019t count against your crawl budget, so removing them won\u2019t&nbsp;help.<\/li>\n<\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"I want Google to crawl slower\" data-section=\"crawl-slower\">\n<!-- \/wp:paragraph -->\n\n<!-- wp:heading {\"level\":2} -->\n<h2>I want Google to crawl slower<\/h2>\n<!-- \/wp:heading -->\n\n<!-- wp:paragraph -->\n<\/div><\/div>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>There are just a couple good ways to make Google crawl slower. There are a few other adjustments you could technically make like slowing down your website, but they\u2019re not methods I would recommend.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:heading {\"level\":3} -->\n<h3>Slow adjustment, but guaranteed (deprecated)<\/h3>\n<!-- \/wp:heading -->\n\n<!-- wp:paragraph -->\n<p>The main control Google gave us to crawl slower was a <a href=\"https:\/\/www.google.com\/webmasters\/tools\/settings\">rate limiter<\/a>&nbsp;within Google Search Console, but this is now deprecated. You used to be able to slow down the crawl rate with the tool, but it took up to two days to take effect.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:image -->\n<figure class=\"wp-block-image is-style-default\"><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/9-search-console-limit-crawl.png\" alt=\"the crawl rate limiter in GSC can make Google crawl slower\"><\/figure>\n<!-- \/wp:image -->\n\n<!-- wp:heading {\"level\":3} -->\n<h3>Fast adjustment, but with&nbsp;risks<\/h3>\n<!-- \/wp:heading -->\n\n<!-- wp:paragraph -->\n<p>If you need a more immediate solution, you can take advantage of Google\u2019s crawl rate adjustments related to your site health. If you serve Googlebot a \u2018503 Service Unavailable\u2019 or \u2018429 Too Many Requests\u2019 status codes on pages, they will start to crawl slower or may stop crawling temporarily. You don\u2019t want to do this longer than a few days though or they may start to drop pages from the index.<br><br>Sometimes large changes to a site can also trigger Google to crawl faster. Changes come with risks, so it\u2019s not something I\u2019d typically recommend if your only reason is that you want Google to re-crawl some&nbsp;pages.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:heading {\"level\":2} -->\n<h2>Final thoughts<\/h2>\n<!-- \/wp:heading -->\n\n<!-- wp:paragraph -->\n<p>The rate of crawling isn\u2019t going to impact your rankings. I want to reiterate that crawl budget isn\u2019t something for most people to worry about.&nbsp; If you do have concerns, I hope this guide was useful.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>I typically only look into it when there are issues with pages not getting crawled and indexed, I need to explain why someone shouldn\u2019t be worried about it, or I happen to see something that concerns me in the crawl stats report in Google Search Console.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Have questions? Let me know on <a href=\"https:\/\/twitter.com\/patrickstox\">Twitter<\/a>.<\/p>\n<!-- \/wp:paragraph -->","protected":false},"excerpt":{"rendered":"<p>More crawling doesn\u2019t mean you\u2019ll rank better, but if your pages aren\u2019t crawled and indexed they aren\u2019t going to rank at&nbsp;all.&nbsp; Most sites don\u2019t need to worry about crawl budget, but there are few cases where you may want to<span class=\"ellipsis\">\u2026<\/span><\/p>\n<div class=\"read-more\">Read more \u203a<\/div>\n<p><!-- end of .read-more --><\/p>\n","protected":false},"author":150,"featured_media":133608,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"wp_typography_post_enhancements_disabled":false,"footnotes":""},"categories":[329],"tags":[],"coauthors":[377],"class_list":["post-133600","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technical-seo","odd"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>When Should You Worry About Crawl Budget?<\/title>\n<meta name=\"description\" content=\"Crawl budget is how fast and how many pages a search engine wants to crawl on your site.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ahrefs.com\/blog\/crawl-budget\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"When Should You Worry About Crawl Budget?\" \/>\n<meta property=\"og:description\" content=\"More crawling doesn&#039;t mean you&#039;ll rank better, but not being crawled means you can&#039;t rank.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ahrefs.com\/blog\/crawl-budget\/\" \/>\n<meta property=\"og:site_name\" content=\"SEO Blog by Ahrefs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Ahrefs\/\" \/>\n<meta property=\"article:author\" content=\"patrickstox\" \/>\n<meta property=\"article:published_time\" content=\"2021-05-28T14:15:20+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-03-14T03:37:17+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/fb-crawl-budget.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"990\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Patrick Stox\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:description\" content=\"More crawling doesn&#039;t mean you&#039;ll rank better, but not being crawled means you can&#039;t rank.\" \/>\n<meta name=\"twitter:creator\" content=\"@patrickstox\" \/>\n<meta name=\"twitter:site\" content=\"@ahrefs\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/crawl-budget\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/crawl-budget\\\/\"},\"author\":{\"name\":\"Patrick Stox\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/14bf754248f3c561786477e4e5fd2067\"},\"headline\":\"When Should You Worry About Crawl Budget?\",\"datePublished\":\"2021-05-28T14:15:20+00:00\",\"dateModified\":\"2025-03-14T03:37:17+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/crawl-budget\\\/\"},\"wordCount\":2573,\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/crawl-budget\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/05\\\/fb-crawl-budget.png\",\"articleSection\":[\"Technical SEO\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/crawl-budget\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/crawl-budget\\\/\",\"name\":\"When Should You Worry About Crawl Budget?\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/crawl-budget\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/crawl-budget\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/05\\\/fb-crawl-budget.png\",\"datePublished\":\"2021-05-28T14:15:20+00:00\",\"dateModified\":\"2025-03-14T03:37:17+00:00\",\"description\":\"Crawl budget is how fast and how many pages a search engine wants to crawl on your site.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ahrefs.com\\\/blog\\\/crawl-budget\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/crawl-budget\\\/#primaryimage\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/05\\\/fb-crawl-budget.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/05\\\/fb-crawl-budget.png\",\"width\":1920,\"height\":990,\"caption\":\"crawl budget\"},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"name\":\"SEO Blog by Ahrefs\",\"description\":\"Link Building Strategies &amp; SEO Tips\",\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\",\"name\":\"Ahrefs\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"width\":2048,\"height\":768,\"caption\":\"Ahrefs\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/Ahrefs\\\/\",\"https:\\\/\\\/x.com\\\/ahrefs\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/ahrefs\\\/\",\"https:\\\/\\\/www.youtube.com\\\/c\\\/ahrefscom\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/14bf754248f3c561786477e4e5fd2067\",\"name\":\"Patrick Stox\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/11\\\/Screenshot-2019-11-06-at-00.57.29.pngbade1fd182f70b6825c334271c12533e\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/11\\\/Screenshot-2019-11-06-at-00.57.29.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/11\\\/Screenshot-2019-11-06-at-00.57.29.png\",\"caption\":\"Patrick Stox\"},\"description\":\"Patrick Stox is a Product Advisor, Technical SEO, &amp; Brand Ambassador at Ahrefs. He was the lead author for the SEO chapter of the 2021 Web Almanac and a reviewer for the 2022 SEO chapter. He also co-wrote the SEO Book For Beginners by Ahrefs and was the Technical Review Editor for The Art of SEO 4th Edition. He\u2019s an organizer for the Triangle SEO Meetup, the Tech SEO Connect conference, he runs a Technical SEO Slack group, and is a moderator for \\\/r\\\/TechSEO on Reddit.\",\"sameAs\":[\"https:\\\/\\\/patrickstox.com\\\/\",\"patrickstox\",\"https:\\\/\\\/x.com\\\/patrickstox\"],\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/author\\\/patrick-stox\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"When Should You Worry About Crawl Budget?","description":"Crawl budget is how fast and how many pages a search engine wants to crawl on your site.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ahrefs.com\/blog\/crawl-budget\/","og_locale":"en_US","og_type":"article","og_title":"When Should You Worry About Crawl Budget?","og_description":"More crawling doesn't mean you'll rank better, but not being crawled means you can't rank.","og_url":"https:\/\/ahrefs.com\/blog\/crawl-budget\/","og_site_name":"SEO Blog by Ahrefs","article_publisher":"https:\/\/www.facebook.com\/Ahrefs\/","article_author":"patrickstox","article_published_time":"2021-05-28T14:15:20+00:00","article_modified_time":"2025-03-14T03:37:17+00:00","og_image":[{"width":1920,"height":990,"url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/fb-crawl-budget.png","type":"image\/png"}],"author":"Patrick Stox","twitter_card":"summary_large_image","twitter_description":"More crawling doesn't mean you'll rank better, but not being crawled means you can't rank.","twitter_creator":"@patrickstox","twitter_site":"@ahrefs","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ahrefs.com\/blog\/crawl-budget\/#article","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/crawl-budget\/"},"author":{"name":"Patrick Stox","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/14bf754248f3c561786477e4e5fd2067"},"headline":"When Should You Worry About Crawl Budget?","datePublished":"2021-05-28T14:15:20+00:00","dateModified":"2025-03-14T03:37:17+00:00","mainEntityOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/crawl-budget\/"},"wordCount":2573,"publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/crawl-budget\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/fb-crawl-budget.png","articleSection":["Technical SEO"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/ahrefs.com\/blog\/crawl-budget\/","url":"https:\/\/ahrefs.com\/blog\/crawl-budget\/","name":"When Should You Worry About Crawl Budget?","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/crawl-budget\/#primaryimage"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/crawl-budget\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/fb-crawl-budget.png","datePublished":"2021-05-28T14:15:20+00:00","dateModified":"2025-03-14T03:37:17+00:00","description":"Crawl budget is how fast and how many pages a search engine wants to crawl on your site.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ahrefs.com\/blog\/crawl-budget\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/crawl-budget\/#primaryimage","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/fb-crawl-budget.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/05\/fb-crawl-budget.png","width":1920,"height":990,"caption":"crawl budget"},{"@type":"WebSite","@id":"https:\/\/ahrefs.com\/blog\/#website","url":"https:\/\/ahrefs.com\/blog\/","name":"SEO Blog by Ahrefs","description":"Link Building Strategies &amp; SEO Tips","publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ahrefs.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ahrefs.com\/blog\/#organization","name":"Ahrefs","url":"https:\/\/ahrefs.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","width":2048,"height":768,"caption":"Ahrefs"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Ahrefs\/","https:\/\/x.com\/ahrefs","https:\/\/www.linkedin.com\/company\/ahrefs\/","https:\/\/www.youtube.com\/c\/ahrefscom"]},{"@type":"Person","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/14bf754248f3c561786477e4e5fd2067","name":"Patrick Stox","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.pngbade1fd182f70b6825c334271c12533e","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2019\/11\/Screenshot-2019-11-06-at-00.57.29.png","caption":"Patrick Stox"},"description":"Patrick Stox is a Product Advisor, Technical SEO, &amp; Brand Ambassador at Ahrefs. He was the lead author for the SEO chapter of the 2021 Web Almanac and a reviewer for the 2022 SEO chapter. He also co-wrote the SEO Book For Beginners by Ahrefs and was the Technical Review Editor for The Art of SEO 4th Edition. He\u2019s an organizer for the Triangle SEO Meetup, the Tech SEO Connect conference, he runs a Technical SEO Slack group, and is a moderator for \/r\/TechSEO on Reddit.","sameAs":["https:\/\/patrickstox.com\/","patrickstox","https:\/\/x.com\/patrickstox"],"url":"https:\/\/ahrefs.com\/blog\/author\/patrick-stox\/"}]}},"as_json":null,"json_reviewers":[114],"_links":{"self":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/133600","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/users\/150"}],"replies":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/comments?post=133600"}],"version-history":[{"count":0,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/133600\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media\/133608"}],"wp:attachment":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media?parent=133600"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/categories?post=133600"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/tags?post=133600"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/coauthors?post=133600"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}