Crawler FAQs

These are some frequently asked questions on our support chat that might help with your final exam.

Question: Why hasn’t Ahrefs found my link? I see it in Google Search Console.

Answer: Our scheduler prioritizes the most authoritative pages that have a higher quality of backlinks pointing to them.

It’s important to also understand the difference between discovery vs. crawling/parsing.

One of the times our crawler discovers new pages is when our parser discovers a link on a page. So the higher the authority of a page, the easier it is for a link on it to get discovered.

In general, Ahrefs will likely crawl your page if it comes from a high-authority domain.

If you want to get it crawled faster, one way to do it is to raise crawl demand by getting high-quality backlinks to the page.

Question: Why has my link been reported as lost?

Answer: Lost links occur on two occasions:

1. When something happens to the referring page (like a 404)

2. When something happens with the link, while the referring page remains live and unchanged (like a swap/removed link)

Question: Why is Ahrefs’ link data different from other tools?

Answer: Our method of counting links is different from other tools.

We trust all redirects and consolidate to the canonical URL.

Links we try not to store:

• Links with URL parameters

• Links from pages in infinite crawl paths

Links we don’t store:

• Links in PDFs or other docs

• Links in iframes

• Links from pages not indexed

• Same links from multiple IPs

• Multiple links to the same page from a single page