{"id":16962,"date":"2017-09-07T01:56:01","date_gmt":"2017-09-07T09:56:01","guid":{"rendered":"https:\/\/ahrefs.com\/blog\/?p=16962"},"modified":"2022-11-17T06:49:48","modified_gmt":"2022-11-17T11:49:48","slug":"crawling-javascript","status":"publish","type":"post","link":"https:\/\/ahrefs.com\/blog\/crawling-javascript\/","title":{"rendered":"Ahrefs crawlers are now rendering web pages and executing JavaScript"},"content":{"rendered":"
Nearly three years ago Google officially announced<\/a> that they were \u201crendering a substantial number of web pages\u201d with JavaScript in order to \u201cinterpret what a typical browser running JavaScript would see.\u201d <\/div>\n

Well, unfortunately, Ahrefs was only crawling backlinks found in HTML up until 2017. <\/p>\n

Which bothered us quite a lot, especially since the SEO community was putting a lot of faith in our backlink data, even calling us \u201cthe gold standard\u201d:<\/p>\n

<\/p>\n

So earlier this year Dmitry<\/a> (our Founder & CEO) decided that Ahrefs has to be the first among its rivals to crawl JavaScript links.<\/p>\n

Which we achieved with ease last month.<\/p>\n

\u201cJS\u201d tag means that the link was found while executing JavaScript on a target page<\/p><\/div>\n

In case you\u2019re new to JavaScript<\/h2>\n

In simple terms, JavaScript is the code that is being executed by your browser when you load a webpage, which will often modify its looks and content (that is, if the page has any JavaScript in its source code in the first place).<\/p>\n

This \u201cbrowser rendering\u201d imposed a certain challenge for Google crawlers back in the days, because they didn\u2019t use browsers and didn\u2019t look at the web pages the way humans do. So if some backlink was hidden in the part of the page, that was generated by JS, they could not see it.<\/p>\n

But that is no longer the case for quite a few years now.<\/p>\n

Does Google \u201ccount\u201d links in JS?<\/h2>\n

According to Mariya Moeva<\/a> (a Search Quality Team member at Google) links found in JS are treated by Google the same way as links in plain HTML.<\/p>\n

And, as we all know, link profile is one of the three major Google ranking factors<\/a>.<\/p>\n

Does Ahrefs render JS the same way as Google?<\/h2>\n

Unfortunately, Google is not giving us a lot of information about the way they handle JavaScript. <\/p>\n

But the guys from Onely have recently performed a great experiment<\/a> to study how Google (and some other search engines<\/a>) handled different JS frameworks.<\/p>\n

They created a simple website - http:\/\/jsseo.expert<\/a> - where each page was built with a different JS framework, and let different search engines crawl it. <\/p>\n

Google and Ask were the only search engines that were able to execute JS from most of these frameworks:<\/p>\n

<\/p>\n

We accepted the challenge and crawled this same website with our JS crawler, which rendered all of the pages that were there (as you can see in the image above).<\/p>\n

So, I guess, it would be safe to say that Ahrefs crawls JS in the same way that Google does.<\/p>\n

But it doesn\u2019t only come down to different JS frameworks, right? There are quite a few more things to consider when rendering web pages. Like the ones, outlined by Will Critchlow in his recent article at Moz<\/a>:<\/p>\n

<\/p>\n

The answers to these questions for Ahrefs JS crawler are pretty much identical.<\/p>\n

Does Ahrefs execute JS on all web pages?<\/h2>\n

As of today, we will only execute JavaScript if a page has more than 15 referring domains pointing at it. And we have ~400 servers dedicated to doing just that.<\/p>\n

Like Dmitry<\/a> said earlier, when interviewed for SEL<\/a>:<\/p>\n

\u201cTo execute JS for every page at our scale [would] require 10,000-15,000 servers, and we believe our customers are not ready to pay for that yet.\u201d <\/div><\/blockquote>\n

And even with that restriction, today we crawl about 6 Billion pages per day<\/strong> and execute JS on about 30 Million of them.<\/p>\n

As a result, we find about 250 Million \u201cJS\u201d links per day (these are the links that were discovered only after executing JS, they weren\u2019t in the raw HTML of a page).<\/p>\n

<\/p>\n

Closing thoughts<\/h2>\n

One of our core values at Ahrefs has always been providing the best and the most accurate data for digital marketing professionals<\/a>. <\/p>\n

We know that our customers heavily rely on Ahrefs data for making all sorts of business and marketing decisions. So we\u2019re going to continue improving the integrity and scale of our data in every way possible.<\/p>\n

And, as always, if you have any feedback about this update, please let us know about it in the comments section.<\/p>\n","protected":false},"excerpt":{"rendered":"

Well, unfortunately, Ahrefs was only crawling backlinks found in HTML up until 2017.  Which bothered us quite a lot, especially since the SEO community was putting a lot of faith in our backlink data, even calling us \u201cthe gold standard\u201d: So\u2026<\/span><\/p>\n

Read more \u203a<\/div>\n

<\/p>\n","protected":false},"author":81,"featured_media":16988,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"wp_typography_post_enhancements_disabled":false,"footnotes":""},"categories":[333],"tags":[],"coauthors":[338],"class_list":["post-16962","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-product-blog","odd"],"yoast_head":"\nAhrefs crawlers are now rendering web pages and executing JavaScript<\/title>\n<meta name=\"description\" content=\"Ahrefs crawler (aka AhrefsBot) is now able to render web pages and execute JavaScript. This means even more links found and reported to you!\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ahrefs.com\/blog\/crawling-javascript\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Ahrefs crawlers are now rendering web pages and executing JavaScript\" \/>\n<meta property=\"og:description\" content=\"Ahrefs crawler (aka AhrefsBot) is now able to render web pages and execute JavaScript. This means even more links found and reported to you!\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ahrefs.com\/blog\/crawling-javascript\/\" \/>\n<meta property=\"og:site_name\" content=\"SEO Blog by Ahrefs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Ahrefs\/\" \/>\n<meta property=\"article:author\" content=\"timsoulo\" \/>\n<meta property=\"article:published_time\" content=\"2017-09-07T09:56:01+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-11-17T11:49:48+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2017\/09\/00-javascript-in-AHrefs.png\" \/>\n\t<meta property=\"og:image:width\" content=\"951\" \/>\n\t<meta property=\"og:image:height\" content=\"495\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Tim Soulo\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@timsoulo\" \/>\n<meta name=\"twitter:site\" content=\"@ahrefs\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/ahrefs.com\/blog\/crawling-javascript\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/crawling-javascript\/\"},\"author\":{\"name\":\"Tim Soulo\",\"@id\":\"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/c0bf22f67b0ff8f8221976119abac64e\"},\"headline\":\"Ahrefs crawlers are now rendering web pages and executing JavaScript\",\"datePublished\":\"2017-09-07T09:56:01+00:00\",\"dateModified\":\"2022-11-17T11:49:48+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/crawling-javascript\/\"},\"wordCount\":715,\"commentCount\":18,\"publisher\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/crawling-javascript\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2017\/09\/00-javascript-in-AHrefs.png\",\"articleSection\":[\"Product Blog\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/ahrefs.com\/blog\/crawling-javascript\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ahrefs.com\/blog\/crawling-javascript\/\",\"url\":\"https:\/\/ahrefs.com\/blog\/crawling-javascript\/\",\"name\":\"Ahrefs crawlers are now rendering web pages and executing JavaScript\",\"isPartOf\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/crawling-javascript\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/crawling-javascript\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2017\/09\/00-javascript-in-AHrefs.png\",\"datePublished\":\"2017-09-07T09:56:01+00:00\",\"dateModified\":\"2022-11-17T11:49:48+00:00\",\"description\":\"Ahrefs crawler (aka AhrefsBot) is now able to render web pages and execute JavaScript. This means even more links found and reported to you!\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ahrefs.com\/blog\/crawling-javascript\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ahrefs.com\/blog\/crawling-javascript\/#primaryimage\",\"url\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2017\/09\/00-javascript-in-AHrefs.png\",\"contentUrl\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2017\/09\/00-javascript-in-AHrefs.png\",\"width\":951,\"height\":495},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ahrefs.com\/blog\/#website\",\"url\":\"https:\/\/ahrefs.com\/blog\/\",\"name\":\"SEO Blog by Ahrefs\",\"description\":\"Link Building Strategies & SEO Tips\",\"publisher\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ahrefs.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/ahrefs.com\/blog\/#organization\",\"name\":\"Ahrefs\",\"url\":\"https:\/\/ahrefs.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png\",\"contentUrl\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png\",\"width\":2048,\"height\":768,\"caption\":\"Ahrefs\"},\"image\":{\"@id\":\"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/Ahrefs\/\",\"https:\/\/x.com\/ahrefs\",\"https:\/\/www.linkedin.com\/company\/ahrefs\/\",\"https:\/\/www.youtube.com\/c\/ahrefscom\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/c0bf22f67b0ff8f8221976119abac64e\",\"name\":\"Tim Soulo\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/05\/Tim_profile_pic.jpg8f04cfcbb173737c084ee81a923c32ba\",\"url\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/05\/Tim_profile_pic.jpg\",\"contentUrl\":\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/05\/Tim_profile_pic.jpg\",\"caption\":\"Tim Soulo\"},\"description\":\"Tim Soulo is CMO at Ahrefs. With over 15 years in the digital marketing industry, Tim is the author of many data-driven studies and actionable marketing frameworks, some of which gained industry-wide recognition. He enjoys challenging conventional marketing knowledge and optimizing for simplicity and efficiency.\",\"sameAs\":[\"https:\/\/www.timsoulo.com\/\",\"timsoulo\",\"https:\/\/x.com\/timsoulo\"],\"url\":\"https:\/\/ahrefs.com\/blog\/author\/tim-soulo\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Ahrefs crawlers are now rendering web pages and executing JavaScript","description":"Ahrefs crawler (aka AhrefsBot) is now able to render web pages and execute JavaScript. This means even more links found and reported to you!","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ahrefs.com\/blog\/crawling-javascript\/","og_locale":"en_US","og_type":"article","og_title":"Ahrefs crawlers are now rendering web pages and executing JavaScript","og_description":"Ahrefs crawler (aka AhrefsBot) is now able to render web pages and execute JavaScript. This means even more links found and reported to you!","og_url":"https:\/\/ahrefs.com\/blog\/crawling-javascript\/","og_site_name":"SEO Blog by Ahrefs","article_publisher":"https:\/\/www.facebook.com\/Ahrefs\/","article_author":"timsoulo","article_published_time":"2017-09-07T09:56:01+00:00","article_modified_time":"2022-11-17T11:49:48+00:00","og_image":[{"width":951,"height":495,"url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2017\/09\/00-javascript-in-AHrefs.png","type":"image\/png"}],"author":"Tim Soulo","twitter_card":"summary_large_image","twitter_creator":"@timsoulo","twitter_site":"@ahrefs","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ahrefs.com\/blog\/crawling-javascript\/#article","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/crawling-javascript\/"},"author":{"name":"Tim Soulo","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/c0bf22f67b0ff8f8221976119abac64e"},"headline":"Ahrefs crawlers are now rendering web pages and executing JavaScript","datePublished":"2017-09-07T09:56:01+00:00","dateModified":"2022-11-17T11:49:48+00:00","mainEntityOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/crawling-javascript\/"},"wordCount":715,"commentCount":18,"publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/crawling-javascript\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2017\/09\/00-javascript-in-AHrefs.png","articleSection":["Product Blog"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/ahrefs.com\/blog\/crawling-javascript\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/ahrefs.com\/blog\/crawling-javascript\/","url":"https:\/\/ahrefs.com\/blog\/crawling-javascript\/","name":"Ahrefs crawlers are now rendering web pages and executing JavaScript","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/crawling-javascript\/#primaryimage"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/crawling-javascript\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2017\/09\/00-javascript-in-AHrefs.png","datePublished":"2017-09-07T09:56:01+00:00","dateModified":"2022-11-17T11:49:48+00:00","description":"Ahrefs crawler (aka AhrefsBot) is now able to render web pages and execute JavaScript. This means even more links found and reported to you!","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ahrefs.com\/blog\/crawling-javascript\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/crawling-javascript\/#primaryimage","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2017\/09\/00-javascript-in-AHrefs.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2017\/09\/00-javascript-in-AHrefs.png","width":951,"height":495},{"@type":"WebSite","@id":"https:\/\/ahrefs.com\/blog\/#website","url":"https:\/\/ahrefs.com\/blog\/","name":"SEO Blog by Ahrefs","description":"Link Building Strategies & SEO Tips","publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ahrefs.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ahrefs.com\/blog\/#organization","name":"Ahrefs","url":"https:\/\/ahrefs.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","width":2048,"height":768,"caption":"Ahrefs"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Ahrefs\/","https:\/\/x.com\/ahrefs","https:\/\/www.linkedin.com\/company\/ahrefs\/","https:\/\/www.youtube.com\/c\/ahrefscom"]},{"@type":"Person","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/c0bf22f67b0ff8f8221976119abac64e","name":"Tim Soulo","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/05\/Tim_profile_pic.jpg8f04cfcbb173737c084ee81a923c32ba","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/05\/Tim_profile_pic.jpg","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/05\/Tim_profile_pic.jpg","caption":"Tim Soulo"},"description":"Tim Soulo is CMO at Ahrefs. With over 15 years in the digital marketing industry, Tim is the author of many data-driven studies and actionable marketing frameworks, some of which gained industry-wide recognition. He enjoys challenging conventional marketing knowledge and optimizing for simplicity and efficiency.","sameAs":["https:\/\/www.timsoulo.com\/","timsoulo","https:\/\/x.com\/timsoulo"],"url":"https:\/\/ahrefs.com\/blog\/author\/tim-soulo\/"}]}},"_links":{"self":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/16962","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/users\/81"}],"replies":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/comments?post=16962"}],"version-history":[{"count":0,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/16962\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media\/16988"}],"wp:attachment":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media?parent=16962"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/categories?post=16962"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/tags?post=16962"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/coauthors?post=16962"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}