Ahrefs Bots

IP addresses

We crawl from publicly published IP ranges. You can pull our IP addresses as IP ranges or Individual IPs. You can find information on how to whitelist our IPs in the help article.

Reverse DNS

The reverse DNS suffix of the IPs hostname is always ahrefs.com or ahrefs.net.

Website status

You can check the status of your website as it's seen by our bots and whether it can be crawled by them:

AhrefsBot indexes fresh, accurate information about websites and their content, and how they link to each other. This data is incredibly useful and can be harnessed in many ways:

• AhrefsBot powers Yep—an independent, privacy-focused search engine. Being included in Yep’s index helps site owners reach a new audience.
• AhrefsBot feeds data into Ahrefs toolset. Website owners can create a free Ahrefs webmaster account and verify domain ownership to unlock site analytics, including access to in-depth backlink data, website performance metrics, and content change monitoring. Ahrefs also offers a suite of free SEO tools that anyone can use without creating an account.
• AhrefsSiteAudit powers our Site Audit tool. Site Audit checks websites for technical and on-page issues such as broken links, slow performance, security misconfigurations, and SEO pitfalls. By crawling and rendering pages, we help identify improvements that can boost visibility, loading speed, and overall user experience. Ahrefs also provides the option to run Site Audit for free on verified websites, helping site owners discover and fix technical issues, all without incurring any charges

Obeying Robots.txt

Both bots strictly respect robots.txt on both disallow and allow rules, as well as crawl-delay directives. Only verified site owners can allow AhrefsSiteAudit crawler to disobey robots.txt on their site so they can check for issues on the site sections normally disallowed for crawling.

Crawl-delay is strictly followed when requesting HTML pages, ensuring that we do not exceed the specified rate limits. However, it cannot be followed when rendering JavaScript. When our bots render a page, they may request multiple assets (e.g., images, scripts, stylesheets) simultaneously, which can result in more frequent requests appearing in server logs than allowed by the crawl-delay setting. This behavior mimics a real user's experience, as modern web pages often require multiple resources to be loaded at once for proper rendering and functionality.

Caching assets

During crawls, we cache frequently requested files (images, CSS, JS) to minimize repeated fetches, which reduces bandwidth consumption and server load.

Load management

If we encounter non-200 status codes, especially 4xx or 5xx errors, we automatically reduce our crawling speed for that site. This ensures minimal stress on sites that may be experiencing outages or high server load.

Transparent practices

We understand that hosting providers, CDNs, and CMS platforms may want to manage how bots interact with their customers’ sites. Our publicly available IP addresses and user-agent strings let you or your service providers quickly verify legitimate Ahrefs traffic. We’re committed to being transparent about our crawling activities to foster trust and collaboration. If you have any concerns, reach out to [email protected] and we'll do our best to help.