New in September: Traffic value, JS crawling, “delayed” links & more

Tim Soulo is CMO at Ahrefs. With over 15 years in the digital marketing industry, Tim is the author of many data-driven studies and actionable marketing frameworks, some of which gained industry-wide recognition. He enjoys challenging conventional marketing knowledge and optimizing for simplicity and efficiency.

I had to skip reports for July and August, because the only two things we were working on were the release of new Rank Tracker and the upcoming release of Site Audit tool.

But we made quite a few cool updates in September:

1. Ahrefs now crawls links in JS

This is a pretty big deal actually, so it was already announced with a dedicated article:

Ahrefs crawlers are now rendering web pages and executing JavaScript

Right now we only execute JS on pages that have over 15 referring domains pointing at them.

But that is 30 Million pages every 24 hours!

That said, we’re the only SEO tool that can crawl JS at scale as of today. Which we’re insanely proud of.

2. Ahrefs’ Big Data in numbers

We often say that Ahrefs tools are powered by Big Data, but we rarely give you updates on what these numbers are.

So we allocated a bit of time last month to create a cool page that would list all the most important numbers behind Ahrefs:

Ahrefs in numbers

For the time being, it is just a snapshot of our data (and infrastructure) as of September.

But because we continuously improve our service and all these numbers grow consistently, we’ll probably develop an API to pull the most recent data every time you load that page.

3. “Traffic value” metric

Every marketer knows that traffic by itself is a vanity metric (in most cases).

So we’ve added a monetary value right next to the amount of organic search traffic that Ahrefs estimates for any website or URL:

“Traffic value” basically shows how much you would have to pay using AdWords to get the same amount of visits from the same search queries that this website ranks for organically (learn more).

Some of the search queries have very high Cost Per Click, because a lot of people want to place their ads in the search results. But other search queries are of no interest to any advertisers, so the CPC is practically 0.

This is why you might see websites with a small amount of traffic that costs a lot, as well as websites with huge traffic that is dirt cheap.

4. “Delayed” new backlinks

In Site Explorer tool, under “Backlinks > New” report, you may have seen a new tag:

Delayed

To explain what it means, I will have to tell you a little bit about how Ahrefs crawler works:

Let’s say we have crawled 1 page and discovered links to 10 other pages on it.

We won’t go and crawl all these 10 pages right away, because each of them will probably have another 10 links, which we’ll have to also immediately crawl. And each of those will have another 10, and so on…

So what we do instead is save all the URLs that we have found on that page to our “smart scheduler.”

This “smart scheduler” then uses a bunch of factors to determine which of these URLs have to be crawled first and which ones can be crawled later.

Sidenote.

One of the main factors that will influence our decision to crawl an unknown URL is (obviously) how often we find links to that page from other pages that we crawl.

But even though we didn’t crawl a certain URL right away, we have recorded the date when we first discovered it.

And this is how this Delayed tag was born.

The whole purpose of “New Backlinks” report in Site Explorer is to show you the most recent backlinks.

But it may happen that you get a link from a very low quality page that our “smart scheduler” will not feed to our crawler for quite a while. So by the time we crawl this page and discover your backlink, it won’t really be “new.”

This is why we decided to add a Delayed tag to such backlinks.

It indicates that we have discovered the linking page more than 90 days ago, but only crawled it just now and discovered that it links to you.

5. More data on “Top Pages”

One of my favourite reports in Site Explorer tool is “Organic search > Top pages.”

It helps me to quickly assess which pages bring the most traffic to the websites of my competitors.

But that report was just a bit incomplete. So we’ve added two more columns to it:

The first column shows how much search traffic each page gets compared to the total search traffic of that website.
And the second shows how many other websites link to each page.

Both numbers are insanely valuable. Now you can see the value of each page relative to other pages of that website and you can also understand how many backlinks you’ll have to get in order to replicate that result on your own website.

Sidenote.

The number of referring domains (RD) that you see in this report is a cached number. Ahrefs stores organic traffic data and backlink data in two separate indexes and there’s a big technical challenge to pull data from both of them simultaneously. So we had to cache RD values for this report.

6. Improvements in Content Gap

Content Gap tool shows you what keywords your competitors are ranking for in search that your own website doesn’t rank for.

But what it didn’t show was the actual URLs of the pages, that ranked for these keywords.

Which we have fixed!

By clicking on website’s ranking position for each search query, you’ll go straight to the ranking page.

But not only that!

You can also click on a small caret to the right from this number and you’ll see a popup that has quick links to other Ahrefs reports for that URL as well as some of its core metrics:

That’s it!

We’re hoping that you will enjoy these new updates to Ahrefs tools that were made in September.

And if you have any questions or feedback, please do not hesitate to let us know about it!