AhrefsSiteAudit Crawler

Learn about the web crawler that powers Ahrefs' Site Audit tool.

Bot type
Good (Identifies itself, has an official moniker)
Robots.txt befolgen.
Yes by default (website owners can request to disobey robots.txt on their sites)
Obeys crawl delay
Yes by default (website owners can request to disobey crawl delay on their sites)
Desktop user-agent string
Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)
Mobile user-agent string
Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.5359.128 Mobile Safari/537.36 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)

Website status

Prüfe den Status deiner Website und ob sie gecrawled werden kann

What is AhrefsSiteAudit crawler?

AhrefsSiteAudit is a web crawler that powers Ahrefs' Site Audit tool. Ahrefs users can use Site Audit to analyze websites and find both technical SEO and on-page SEO issues.

This bot can crawl any website unless disallowed, and prevents excessive load on website servers by limiting crawling to 1 request per 2 seconds by default. If you're a website owner, you can crawl your own sites at higher speeds and also allow AhrefsSiteAudit crawler to ignore robots.txt. To do that, you'll need to verify ownership in Site Audit tool.

Questions? We have answers

  • What is AhrefsSiteAudit bot doing on my website?

    Falls du AhrefsSiteAudit in deinen Server-Logs siehst, bedeutet das, dass jemand deine Website mit dem Ahrefs Site Audit Tool analysiert hat. Dieser Bot crawled eine Website und folgt internen Links von einer URL zur anderen. Er prüft außerdem Ressourcen wie CSS-, JS- und Bilddateien.AhrefsSiteAudit crawler does not add numbers to your Google Analytics traffic.

  • Does it respect robots.txt file?

    Yes. By default.Only verified owners can allow AhrefsSiteAudit crawler to disobey robots.txt on their site.

  • How do I control AhrefsSiteAudit crawler on my website?

    Since AhrefsSiteAudit bot strictly follows the robots.txt file on your website, you're in full control of what it does.If you want to stop AhrefsSiteAudit from visiting your site, add these two lines to the robots.txt file on your server:
    User-agent: AhrefsSiteAuditDisallow: /
    You can also allow or disallow crawling on any part of your website with your robots.txt file. Read more about robots.txt at www.robotstxt.org.AhrefsSiteAudit bot checks your robots.txt file before each new crawl.If your robots.txt contains errors, AhrefsSiteAudit bot won’t be able to recognize your commands and it'll continue crawling your website the way it did before.If you think that AhrefsSiteAudit bot is misbehaving on your website or if you have any questions, please reach out to [email protected] and we'll do our best to help.

Du hast andere Fragen?

Falls du weitere Fragen hast,
kontaktiere uns, damit wir dir helfen können.

© 2024 Ahrefs Pte. Ltd. (201227417H) 16 Raffles Quay, #33-03 Hong Leong Building, Singapore 048581