Ahrefs crawlers are now rendering web pages and executing JavaScript

Well, unfortunately, Ahrefs was only crawling backlinks found in HTML up until 2017.

Which bothered us quite a lot, especially since the SEO community was putting a lot of faith in our backlink data, even calling us “the gold standard”:

So earlier this year Dmitry (our Founder & CEO) decided that Ahrefs has to be the first among its rivals to crawl JavaScript links.

Which we achieved with ease last month.

In simple terms, JavaScript is the code that is being executed by your browser when you load a webpage, which will often modify its looks and content (that is, if the page has any JavaScript in its source code in the first place).

This "browser rendering" imposed a certain challenge for Google crawlers back in the days, because they didn’t use browsers and didn’t look at the web pages the way humans do. So if some backlink was hidden in the part of the page, that was generated by JS, they could not see it.

But that is no longer the case for quite a few years now.

According to Mariya Moeva (a Search Quality Team member at Google) links found in JS are treated by Google the same way as links in plain HTML.

And, as we all know, link profile is one of the three major Google ranking factors.

Unfortunately, Google is not giving us a lot of information about the way they handle JavaScript.

But the guys from Onely have recently performed a great experiment to study how Google (and some other search engines) handled different JS frameworks.

They created a simple website - http://jsseo.expert - where each page was built with a different JS framework, and let different search engines crawl it.

Google and Ask were the only search engines that were able to execute JS from most of these frameworks:

We accepted the challenge and crawled this same website with our JS crawler, which rendered all of the pages that were there (as you can see in the image above).

So, I guess, it would be safe to say that Ahrefs crawls JS in the same way that Google does.

But it doesn’t only come down to different JS frameworks, right? There are quite a few more things to consider when rendering web pages. Like the ones, outlined by Will Critchlow in his recent article at Moz:

The answers to these questions for Ahrefs JS crawler are pretty much identical.

As of today, we will only execute JavaScript if a page has more than 15 referring domains pointing at it. And we have ~400 servers dedicated to doing just that.

Like Dmitry said earlier, when interviewed for SEL:

And even with that restriction, today we crawl about 6 Billion pages per day and execute JS on about 30 Million of them.

As a result, we find about 250 Million "JS" links per day (these are the links that were discovered only after executing JS, they weren't in the raw HTML of a page).

One of our core values at Ahrefs has always been providing the best and the most accurate data for digital marketing professionals.

We know that our customers heavily rely on Ahrefs data for making all sorts of business and marketing decisions. So we’re going to continue improving the integrity and scale of our data in every way possible.

And, as always, if you have any feedback about this update, please let us know about it in the comments section.