Sign in
Sign up

AhrefsSiteAudit Crawler

Learn about the web crawler that powers Ahrefs' Site Audit tool.

AhrefsBot

AhrefsSiteAudit

Version
6.1
Bot type
Good (Identifies itself, has an official moniker)
Category
SEO
Obeys robots.txt
Yes by default (website owners can request to disobey robots.txt on their sites)
Obeys crawl delay
Yes by default (website owners can request to disobey crawl delay on their sites)
Desktop user-agent string
Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)
Mobile user-agent string
Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.5359.128 Mobile Safari/537.36 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)
Reverse DNS suffix
ahrefs.com
IP address range
Current list, API documentation

Website status

Check the status of your website and whether it can be crawled

What is AhrefsSiteAudit crawler?

AhrefsSiteAudit is a web crawler that powers Ahrefs' Site Audit tool. Ahrefs users can use Site Audit to analyze websites and find both technical SEO and on-page SEO issues.

This bot can crawl any website unless disallowed, and prevents excessive load on website servers by limiting crawling to 1 request per 2 seconds by default. If you're a website owner, you can crawl your own sites at higher speeds and also allow AhrefsSiteAudit crawler to ignore robots.txt. To do that, you'll need to verify ownership in Site Audit tool.

Questions? We have answers

  • What is AhrefsSiteAudit bot doing on my website?

    If you see AhrefsSiteAudit in your server log, this means that someone did an analysis of your website using Ahrefs' Site Audit tool. This bot crawls a website and follows its internal links from one URL to another. It also checks website resources such as CSS files, JS files, and images.AhrefsSiteAudit crawler does not add numbers to your Google Analytics traffic.

  • Does it respect robots.txt file?

    Yes. By default.Only verified owners can allow AhrefsSiteAudit crawler to disobey robots.txt on their site.

  • How do I control AhrefsSiteAudit crawler on my website?

    Since AhrefsSiteAudit bot strictly follows the robots.txt file on your website, you're in full control of what it does.If you want to stop AhrefsSiteAudit from visiting your site, add these two lines to the robots.txt file on your server:
    User-agent: AhrefsSiteAuditDisallow: /
    You can also allow or disallow crawling on any part of your website with your robots.txt file. Read more about robots.txt at www.robotstxt.org.AhrefsSiteAudit bot checks your robots.txt file before each new crawl.If your robots.txt contains errors, AhrefsSiteAudit bot won’t be able to recognize your commands and it'll continue crawling your website the way it did before.If you think that AhrefsSiteAudit bot is misbehaving on your website or if you have any questions, please reach out to [email protected] and we'll do our best to help.

Have other questions?

Якщо у вас виникли додаткові запитання,
contact us, щоб ми могли допомогти.