Ahrefs crawlers are now rendering web pages and executing JavaScript

Tim Soulo

Tim is the guy responsible for marketing and product strategy at Ahrefs. But most importantly he’s the chief evangelist of the company. Learn more about Tim

Article stats

  • Referring domains 19
Data from Content Explorer tool.
    Nearly three years ago Google officially announced that they were “rendering a substantial number of web pages” with JavaScript in order to “interpret what a typical browser running JavaScript would see.”

    Well, unfortunately, Ahrefs was only crawling backlinks found in HTML up until 2017.

    Which bothered us quite a lot, especially since the SEO community was putting a lot of faith in our backlink data, even calling us “the gold standard”:

    So earlier this year Dmitry (our Founder & CEO) decided that Ahrefs has to be the first among its rivals to crawl JavaScript links.

    Which we achieved with ease last month.

    JS” tag means that the link was found while executing JavaScript on a target page

    In case you’re new to JavaScript

    In simple terms, JavaScript is the code that is being executed by your browser when you load a webpage, which will often modify its looks and content (that is, if the page has any JavaScript in its source code in the first place).

    This “browser rendering” imposed a certain challenge for Google crawlers back in the days, because they didn’t use browsers and didn’t look at the web pages the way humans do. So if some backlink was hidden in the part of the page, that was generated by JS, they could not see it.

    But that is no longer the case for quite a few years now.

    Does Google “count” links in JS?

    According to Mariya Moeva (a Search Quality Team member at Google) links found in JS are treated by Google the same way as links in plain HTML.

    And, as we all know, link profile is one of the three major Google ranking factors.

    Does Ahrefs render JS the same way as Google?

    Unfortunately, Google is not giving us a lot of information about the way they handle JavaScript.

    But guys from Elephate have recently performed a great experiment to study how Google (and some other search engines) handled different JS frameworks.

    They created a simple website — http://jsseo.expert — where each page was built with a different JS framework, and let different search engines crawl it.

    Google and Ask were the only search engines that were able to execute JS from most of these frameworks:

    We accepted the challenge and crawled this same website with our JS crawler, which rendered all of the pages that were there (as you can see in the image above).

    So, I guess, it would be safe to say that Ahrefs crawls JS in the same way that Google does.

    But it doesn’t only come down to different JS frameworks, right? There are quite a few more things to consider when rendering web pages. Like the ones, outlined by Will Critchlow in his recent article at Moz:

    The answers to these questions for Ahrefs JS crawler are pretty much identical.

    Does Ahrefs execute JS on all web pages?

    As of today, we will only execute JavaScript if a page has more than 15 referring domains pointing at it. And we have ~400 servers dedicated to doing just that.

    Like Dmitry said earlier, when interviewed for SEL:

    “To execute JS for every page at our scale [would] require 10,000–15,000 servers, and we believe our customers are not ready to pay for that yet.”

    And even with that restriction, today we crawl about 6 Billion pages per day and execute JS on about 30 Million of them.

    As a result, we find about 250 Million “JS” links per day (these are the links that were discovered only after executing JS, they weren’t in the raw HTML of a page).

    Closing thoughts

    One of our core values at Ahrefs has always been providing the best and the most accurate data for digital marketing professionals.

    We know that our customers heavily rely on Ahrefs data for making all sorts of business and marketing decisions. So we’re going to continue improving the integrity and scale of our data in every way possible.

    And, as always, if you have any feedback about this update, please let us know about it in the comments section.

    Tim Soulo

    Tim is the guy responsible for marketing and product strategy at Ahrefs. But most importantly he’s the chief evangelist of the company. Learn more about Tim

    Article stats

    • Referring domains 19
    Data from Content Explorer tool.

    Shows how many different websites are linking to this piece of content. As a general rule, the more websites link to you, the higher you rank in Google.

    Shows estimated monthly search traffic to this article according to Ahrefs data. The actual search traffic (as reported in Google Analytics) is usually 3-5 times bigger.

    Get notified of new articles

    46,617 marketers are already subscribed to Ahrefs blog. Leave your email to get our weekly newsletter.

    • great article

    • Wawan Kurniawan

      awesome

    • Дмитрий Буланов

      Thank you for your work

    • luisandroid

      thank you for your work, greetings and keep it up 😉

    • Amine Marghich

      great announcement!

    • Wow — you guys are on the bleeding edge! Great work!

    • AHREFS. Most awesome SEO software EVER.

    • Oleg

      Awesome! You are the best guys. Keep up with good work!

    • This is awesome! Thank you for the constant updates!

    • great info Tim!! we have seen some sites using ReactJS getting properly indexed by Google..

    • great

    • Mejsny

      Woow, well done!

    • Hi its very usefoul!

    • nice development, congratulations.

    • That was really Awesome. thanks!

    • But that is no longer the case for quite a few years now.

    • Ben Alfrey

      Interesting. When I render pages in Search Console I can still see that Googlebot struggles to crawl and render JS map functions effectively. Can only assume that this must be an error with the Googlebot rendering in Search Console as according to this study there should be no issues