{"id":179611,"date":"2024-09-04T08:10:30","date_gmt":"2024-09-04T13:10:30","guid":{"rendered":"https:\/\/ahrefs.com\/blog\/?p=179611"},"modified":"2024-09-05T06:10:39","modified_gmt":"2024-09-05T11:10:39","slug":"awt-website-crawler","status":"publish","type":"post","link":"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/","title":{"rendered":"The Game-Changer in Free Website Crawlers You Need to Know&nbsp;About"},"content":{"rendered":"<div class=\"intro-txt\"> If you own or manage a small website and have some knowledge of SEO, you\u2019re likely using website crawler software (aka SEO crawler) to audit and monitor your&nbsp;site.&nbsp;<\/div>\n<p>For smaller websites with up to a few hundred pages, you\u2019re likely using tools like Screaming Frog, Seobility, or WebSite Auditor, with Screaming Frog being arguably the most popular.<\/p>\n<div id=\"attachment_179615\" style=\"width: 790px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-179615\" class=\"size-full wp-image-179615\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/SEO-Crawlers-search-volumes-in-ahrefs-keywords-explorer.png\" alt width=\"780\" height=\"233\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/SEO-Crawlers-search-volumes-in-ahrefs-keywords-explorer.png 780w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/SEO-Crawlers-search-volumes-in-ahrefs-keywords-explorer-680x203.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/SEO-Crawlers-search-volumes-in-ahrefs-keywords-explorer-768x229.png 768w\" sizes=\"auto, (max-width: 780px) 100vw, 780px\"><p id=\"caption-attachment-179615\" class=\"wp-caption-text\">The search volume data from Ahrefs\u2019 <a href=\"https:\/\/app.ahrefs.com\/keywords-explorer\" target=\"_blank\" rel=\"noopener\">Keywords Explorer<\/a><\/p><\/div>\n<p>So why would you even consider an alternative?<\/p>\n<p>Let\u2019s be honest: nearly every free site crawler does a pretty decent job at identifying fundamental SEO issues for small websites, such as broken links between pages, problematic titles and descriptions, or slow&nbsp;pages.<\/p>\n<p>However, if you want to detect more advanced SEO issues or need to crawl a larger website, you\u2019d need a paid SEO audit&nbsp;tool.<\/p>\n<p>\u2026unless you\u2019re using the <a href=\"https:\/\/ahrefs.com\/webmaster-tools\" target=\"_blank\" rel=\"noopener\">Ahrefs Webmaster Tools (aka AWT)<\/a>.<\/p>\n<p>Here\u2019s why the Site Audit in AWT beats any other other free (and often even premium) website crawler:<\/p>\n<ul>\n<li>5,000 crawled pages per project per month (with unlimited projects)<\/li>\n<li>Runs in the cloud and has a web interface<\/li>\n<li>User-friendly interface<\/li>\n<li>Pro-level customizability of issues and crawls<\/li>\n<li>Automated scheduled crawls with email alerts<\/li>\n<li>3 months of crawl history<\/li>\n<li>Crawl comparison<\/li>\n<li>Backlink metrics<\/li>\n<li>Organic traffic metrics<\/li>\n<li>And so much&nbsp;more!<\/li>\n<\/ul>\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div> There\u2019s one important thing to note: the free Ahrefs Webmaster Tools only work for websites <strong>where you can prove ownership<\/strong>. You won\u2019t be able to audit a client\u2019s website with no access to it. You\u2019ll need at least the <a href=\"https:\/\/ahrefs.com\/blog\/starter-plan\/\" target=\"_blank\" rel=\"noopener\">Starter plan<\/a> with Ahrefs to do&nbsp;this.&nbsp;<\/div>\n<p>Now, let me introduce the unique features of Ahrefs Webmaster Tools that you won\u2019t find in any other free SEO checker:<\/p>\n<h2>1. Generous crawl limits<\/h2>\n<p>To my knowledge, no other free SEO crawler offers such generous limits.<\/p>\n\n<table id=\"tablepress-329\" class=\"tablepress tablepress-id-329 tablepress-responsive tablepress-ahrefs-width-full\">\n<thead>\n<tr class=\"row-1 odd\">\n\t<th class=\"column-1\">SEO Crawler<\/th><th class=\"column-2\">Limits<\/th>\n<\/tr>\n<\/thead>\n<tbody class=\"row-hover\">\n<tr class=\"row-2 even\">\n\t<td class=\"column-1\">Screaming Frog<\/td><td class=\"column-2\">500\/crawl<\/td>\n<\/tr>\n<tr class=\"row-3 odd\">\n\t<td class=\"column-1\">Website Auditor: 500 per&nbsp;crawl<\/td><td class=\"column-2\">500\/crawl<\/td>\n<\/tr>\n<tr class=\"row-4 even\">\n\t<td class=\"column-1\">SEOBility<\/td><td class=\"column-2\">1,000\/crawl<\/td>\n<\/tr>\n<tr class=\"row-5 odd\">\n\t<td class=\"column-1\">Ahrefs Webmaster Tools (AWT)<\/td><td class=\"column-2\">5,000\/month<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<!-- #tablepress-329 from cache -->\n<p>You get 5,000 crawl credits per project per&nbsp;month.<\/p>\n<p>These credits are spent on crawling HTML pages only. Redirects, images, and external URLs don\u2019t even&nbsp;count.<\/p>\n<p>This means you can crawl a 300-page website every second day. For example, the English part of the Ahrefs Blog could be fully crawled 4 times per month with 5,000 crawl credits.<br>\n<img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-179622\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Internal-pages-crawled-with-Ahrefs-Webmaster-Tools.png\" alt width=\"1485\" height=\"639\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Internal-pages-crawled-with-Ahrefs-Webmaster-Tools.png 1485w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Internal-pages-crawled-with-Ahrefs-Webmaster-Tools-680x293.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Internal-pages-crawled-with-Ahrefs-Webmaster-Tools-768x330.png 768w\" sizes=\"auto, (max-width: 1485px) 100vw, 1485px\"><\/p>\n<h2>2. User-friendly reports<\/h2>\n<p>Despite its popularity, Screaming Frog has a bit of a learning curve. It can take quite some time to get used to their Excel-2001-styled spreadsheet reports or even to find the \u201cIssues\u201d report in the first&nbsp;place.<\/p>\n<p>The Site Audit in AWT has an informative Overview page. It features a clearly visible health score that indicates the number of important issues on your website and a list of the top issues.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-179623\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Ahrefs-Site-Audit-project-Overview.png\" alt width=\"1294\" height=\"910\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Ahrefs-Site-Audit-project-Overview.png 1294w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Ahrefs-Site-Audit-project-Overview-604x425.png 604w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Ahrefs-Site-Audit-project-Overview-768x540.png 768w\" sizes=\"auto, (max-width: 1294px) 100vw, 1294px\"><\/p>\n<p>Unlike many other SEO crawlers, the Site Audit in AWT provides an explanation of the detected issue and a brief recommendation on how to fix&nbsp;it.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-179624\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Explanation-of-the-issue-and-information-on-how-to-fix-it.png\" alt=\"Explanation of the SEO issue and information on how to fix it\" width=\"1100\" height=\"726\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Explanation-of-the-issue-and-information-on-how-to-fix-it.png 1100w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Explanation-of-the-issue-and-information-on-how-to-fix-it-644x425.png 644w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Explanation-of-the-issue-and-information-on-how-to-fix-it-768x507.png 768w\" sizes=\"auto, (max-width: 1100px) 100vw, 1100px\"><\/p>\n<h2>3. Customizable issues<\/h2>\n<p>The Site Audit in Ahrefs Webmaster Tools checks your website for over 170 potential technical and on page SEO issues that can harm your website\u2019s performance in search results.<\/p>\n<p>These issues are categorized by their importance into three categories:<\/p>\n<ul>\n<li>Errors<\/li>\n<li>Warnings<\/li>\n<li>Notices<\/li>\n<\/ul>\n<p>This is typical for many SEO audit tools, although most of them report much fewer issues.<\/p>\n<div id=\"attachment_179625\" style=\"width: 1275px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-179625\" class=\"size-full wp-image-179625\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Preview-of-the-issues-report-in-the-overview.png\" alt=\"Preview of the issues report in the overview\" width=\"1265\" height=\"435\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Preview-of-the-issues-report-in-the-overview.png 1265w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Preview-of-the-issues-report-in-the-overview-680x234.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Preview-of-the-issues-report-in-the-overview-768x264.png 768w\" sizes=\"auto, (max-width: 1265px) 100vw, 1265px\"><p id=\"caption-attachment-179625\" class=\"wp-caption-text\">The preview of the issues report in the Overview<\/p><\/div>\n<p>One unique feature of Ahrefs Webmaster Tools is the ability to customize the criteria that trigger issues and even create new ones based on your priorities.<\/p>\n<p>For instance, if you disagree with our recommended Title tag character limit, you can easily modify it to suit your preferences.<br>\n<img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-179626\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Using-the-advanced-filters-to-customize-issues.png\" alt=\"Using the advanced filters to customize issues\" width=\"1500\" height=\"357\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Using-the-advanced-filters-to-customize-issues.png 1500w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Using-the-advanced-filters-to-customize-issues-680x162.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Using-the-advanced-filters-to-customize-issues-768x183.png 768w\" sizes=\"auto, (max-width: 1500px) 100vw, 1500px\"><\/p>\n<p>The wide range of data points available for pages and links enables you to create custom issues that extend far beyond the default set.<\/p>\n<p>For example, I\u2019ve set up an alert for my project that notifies me whenever someone edits a page to insert an external link. This allows me to react quickly if one of the writers adds a link to an unintended website or even if the website is hacked for the purpose of link injections.<br>\n<img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-179627\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Creating-cusom-issues-in-Site-Audit.png\" alt=\"Creating custom issues in Site Audit\" width=\"1115\" height=\"311\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Creating-cusom-issues-in-Site-Audit.png 1115w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Creating-cusom-issues-in-Site-Audit-680x190.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Creating-cusom-issues-in-Site-Audit-768x214.png 768w\" sizes=\"auto, (max-width: 1115px) 100vw, 1115px\"><\/p>\n<p>And if you find the 170+ predetermined SEO issues overwhelming, you can simply disable any of them at the project level.<br>\n<img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-179628\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/It-is-possible-to-disable-unwanted-issues-for-a-project.png\" alt=\"It is possible to disable unwanted issues for a project\" width=\"1286\" height=\"427\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/It-is-possible-to-disable-unwanted-issues-for-a-project.png 1286w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/It-is-possible-to-disable-unwanted-issues-for-a-project-680x226.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/It-is-possible-to-disable-unwanted-issues-for-a-project-768x255.png 768w\" sizes=\"auto, (max-width: 1286px) 100vw, 1286px\"><\/p>\n<h2>4. Cloud-based crawling<\/h2>\n<p>Screaming Frog is a classic tool for website audits. Even its free version, limited to 500 pages per crawl, can satisfy the needs of many webmasters and&nbsp;SEOs.<\/p>\n<p>However, it must be installed on your PC or laptop, using your computer\u2019s resources and Internet connection.<\/p>\n<p>Another serious drawback is that the free version doesn\u2019t allow you to save a crawl to revisit it later. The same limitation applies to WebsiteAuditor.<\/p>\n<p>With Ahrefs Webmaster Tools, the crawl runs in the cloud, meaning you can start it and access the results from any location at any time \u2013 even from your <strong>smartphone<\/strong>.<br>\n<img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-179629\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/The-web-interface-of-the-Site-Audit-in-Ahrefs-Webmaster-Tools.png\" alt=\"The web interface of the Site Audit in Ahrefs Webmaster Tools\" width=\"1504\" height=\"1104\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/The-web-interface-of-the-Site-Audit-in-Ahrefs-Webmaster-Tools.png 1504w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/The-web-interface-of-the-Site-Audit-in-Ahrefs-Webmaster-Tools-579x425.png 579w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/The-web-interface-of-the-Site-Audit-in-Ahrefs-Webmaster-Tools-768x564.png 768w\" sizes=\"auto, (max-width: 1504px) 100vw, 1504px\"><\/p>\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div> Seobility is the only free cloud-based SEO crawler with decent limits that I know of. However, they only show a fraction of data in their reports, hiding ~90% behind a paywall.&nbsp;<\/div>\n<h2>5. Three months of history and side-by-side crawl comparison<\/h2>\n<p>In Ahrefs Webmaster Tools, SEO audit crawl data is stored on our servers for three months.<\/p>\n<div id=\"attachment_179630\" style=\"width: 1407px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-179630\" class=\"size-full wp-image-179630\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Crawl-history-log.png\" alt width=\"1397\" height=\"668\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Crawl-history-log.png 1397w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Crawl-history-log-680x325.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Crawl-history-log-768x367.png 768w\" sizes=\"auto, (max-width: 1397px) 100vw, 1397px\"><p id=\"caption-attachment-179630\" class=\"wp-caption-text\">The list of recent crawls for one of my projects.<\/p><\/div>\n<p>This allows you to access crawls anytime, from anywhere, enabling you to compare them and track which aspects of your website\u2019s SEO have improved or declined over time. Check the <strong>Change column<\/strong> for significant deviations.<\/p>\n<div id=\"attachment_179631\" style=\"width: 1286px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-179631\" class=\"size-full wp-image-179631\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Changes-in-the-issues-over-time.png\" alt width=\"1276\" height=\"474\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Changes-in-the-issues-over-time.png 1276w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Changes-in-the-issues-over-time-680x253.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Changes-in-the-issues-over-time-768x285.png 768w\" sizes=\"auto, (max-width: 1276px) 100vw, 1276px\"><p id=\"caption-attachment-179631\" class=\"wp-caption-text\">The changes in the SEO issues between the two crawls<\/p><\/div>\n<p>This is particularly useful when you need to investigate the cause of an issue and determine when it occurred.<\/p>\n<div id=\"attachment_179632\" style=\"width: 1507px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-179632\" class=\"size-full wp-image-179632\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Issue-history-over-time.png\" alt width=\"1497\" height=\"433\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Issue-history-over-time.png 1497w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Issue-history-over-time-680x197.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Issue-history-over-time-768x222.png 768w\" sizes=\"auto, (max-width: 1497px) 100vw, 1497px\"><p id=\"caption-attachment-179632\" class=\"wp-caption-text\">The number of pages affected by an issue over&nbsp;time<\/p><\/div>\n<h2>6. Page history and code\/text comparison<\/h2>\n<p>Our <a href=\"https:\/\/ahrefs.com\/robot\/site-audit\" target=\"_blank\" rel=\"noopener\">SiteAudit bot<\/a> crawls and stores <strong>each page\u2019s<\/strong> HTML code and text, allowing you to see changes in a single page between two crawls.<\/p>\n<p>The <strong>comparison<\/strong> feature highlights what has been added, removed, or edited in terms of raw HTML, rendered HTML, or page&nbsp;text.<\/p>\n<p>For example, we\u2019ve recently updated our \u201c<a href=\"https:\/\/ahrefs.com\/blog\/top-google-searches\/\" target=\"_blank\" rel=\"noopener\">Top Google Searches<\/a>\u201d page on our blog to increase its chances of ranking higher. By using the comparison feature under the <strong>View source<\/strong> tab, I can access the older version if I want to restore it or see exactly what was changed.<\/p>\n<div id=\"attachment_179633\" style=\"width: 1050px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-179633\" class=\"size-full wp-image-179633\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Viewing-the-changes-on-a-page.png\" alt=\"Viewing the changes on a page\" width=\"1040\" height=\"1076\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Viewing-the-changes-on-a-page.png 1040w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Viewing-the-changes-on-a-page-411x425.png 411w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Viewing-the-changes-on-a-page-768x795.png 768w\" sizes=\"auto, (max-width: 1040px) 100vw, 1040px\"><p id=\"caption-attachment-179633\" class=\"wp-caption-text\">Viewing the changes in the page content.<\/p><\/div>\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div> Here, I need to take a moment to remind you that we\u2019re still talking about a 100% free&nbsp;tool!&nbsp;<\/div>\n<h2>7. Integrated SEO metrics<\/h2>\n<p>In AWT, you can view the most important SEO metrics for every URL crawled, such&nbsp;as:<\/p>\n<ul>\n<li>Organic search traffic<\/li>\n<li>The number of raking keywords<\/li>\n<li>Top keyword<\/li>\n<li>The number of backlinks<\/li>\n<li>And so&nbsp;on<\/li>\n<\/ul>\n<p>These metrics, integrated into the report tables, can help you prioritize your efforts in addressing SEO issues. Pages with the highest traffic and those ranking for the most important keywords typically require your immediate attention.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-179638\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Prioritising-SEO-fixes-based-on-page-metrics.png\" alt=\"Prioritizing SEO fixes based on page metrics\" width=\"1502\" height=\"810\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Prioritising-SEO-fixes-based-on-page-metrics.png 1502w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Prioritising-SEO-fixes-based-on-page-metrics-680x367.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Prioritising-SEO-fixes-based-on-page-metrics-768x414.png 768w\" sizes=\"auto, (max-width: 1502px) 100vw, 1502px\"><\/p>\n<p>We\u2019ve also added notifications about drops in traffic and search rankings to the audit results so you can <a href=\"https:\/\/ahrefs.com\/academy\/how-to-use-ahrefs\/site-audit\/page-explorer#:~:text=Monitor%20content%20decay\">monitor content decay<\/a> proactively and reverse-engineer what caused the drop in the first&nbsp;place.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-179639\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Traffic-related-issues.png\" alt=\"Traffic-related issues\" width=\"671\" height=\"215\"><\/p>\n<h2>8. Internal link opportunities<\/h2>\n<p>This is one of the more advanced features and may require some explanation.<\/p>\n<p><a href=\"https:\/\/ahrefs.com\/seo\/glossary\/internal-link\" target=\"_blank\" rel=\"noopener\">Internal links<\/a> can significantly influence search rankings:<\/p>\n<ol>\n<li>They boost the internal \u201cimportance\u201d of a page on your website, which can influence the page rankings and crawl priority.<\/li>\n<li>With relevant anchor texts, they signal the topic of the page to Google.<\/li>\n<\/ol>\n<p>But let\u2019s face it, finding relevant pages for internal linking can be quite time-consuming. Not if you\u2019re using&nbsp;AWT!<\/p>\n<p>Our site audit analyzes the top 10 keywords for which each page on your site ranks and then searches for mentions of these exact terms on all crawled pages.<\/p>\n<p>Based on this data, you\u2019ll receive automatic recommendations for internal links between the pages on your website.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-179640\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Internal-link-opportunities-shown-in-Ahrefs-Site-Audit.jpg\" alt=\"Internal link opportunities shown in Ahrefs Site Audit\" width=\"1608\" height=\"831\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Internal-link-opportunities-shown-in-Ahrefs-Site-Audit.jpg 1608w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Internal-link-opportunities-shown-in-Ahrefs-Site-Audit-680x351.jpg 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Internal-link-opportunities-shown-in-Ahrefs-Site-Audit-768x397.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/Internal-link-opportunities-shown-in-Ahrefs-Site-Audit-1536x794.jpg 1536w\" sizes=\"auto, (max-width: 1608px) 100vw, 1608px\"><\/p>\n<h2>9. A steady flow of new helpful features<\/h2>\n<p>We\u2019re constantly improving all our products, and the free Ahrefs Webmaster Tools are no exception.<\/p>\n<p>The two upcoming features we\u2019re most excited about&nbsp;are:<\/p>\n<ul>\n<li><strong>Always-on audit<\/strong> will allow you to monitor and recrawl selected pages of your site with no need to run a complete website crawl every&nbsp;time.<\/li>\n<li><a href=\"https:\/\/ahrefs.com\/patches\/\" data-ahr=\"https:\/\/ahrefs.com\/blog\/site-audit-patches\/\"><strong>Patches<\/strong><\/a> are intended to give you an opportunity to fix some on-page SEO issues straight from the Site Audit reports - no developers needed.<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-179641\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/The-upcoming-Patches-feature.png\" alt=\"The upcoming Patches feature\" width=\"1500\" height=\"650\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/The-upcoming-Patches-feature.png 1500w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/The-upcoming-Patches-feature-680x295.png 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/The-upcoming-Patches-feature-768x333.png 768w\" sizes=\"auto, (max-width: 1500px) 100vw, 1500px\"><\/p>\n<h2>Wrapping it&nbsp;up<\/h2>\n<p>There\u2019s so much more Ahrefs Webmaster Tools can do for you that I didn\u2019t cover to avoid overwhelming you, such&nbsp;as:<\/p>\n<ul>\n<li>Segmentation<\/li>\n<li>Structured data checks<\/li>\n<li>Localization issues<\/li>\n<li>Content duplication<\/li>\n<li>Backlink analysis<\/li>\n<li>And more<\/li>\n<\/ul>\n<p>With all these incredibly useful features, AWT is 100% free, with no hidden fees, so just <a href=\"https:\/\/ahrefs.com\/webmaster-tools\" target=\"_blank\" rel=\"noopener\">give it a try<\/a>.<\/p>\n<p>Just remember, you can only use it on<strong> websites that you own or manage<\/strong>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>For smaller websites with up to a few hundred pages, you\u2019re likely using tools like Screaming Frog, Seobility, or WebSite Auditor, with Screaming Frog being arguably the most popular. So why would you even consider an alternative? Let\u2019s be honest:<span class=\"ellipsis\">\u2026<\/span><\/p>\n<div class=\"read-more\">Read more \u203a<\/div>\n<p><!-- end of .read-more --><\/p>\n","protected":false},"author":116,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"wp_typography_post_enhancements_disabled":false,"footnotes":""},"categories":[333],"tags":[],"coauthors":[356],"class_list":["post-179611","post","type-post","status-publish","format-standard","hentry","category-product-blog","odd"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>A Free Website Crawler from Ahrefs<\/title>\n<meta name=\"description\" content=\"For smaller websites with up to a few hundred pages, you\u2019re likely using tools like Screaming Frog, Seobility, or WebSite Auditor, with Screaming Frog\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"A Free Website Crawler from Ahrefs\" \/>\n<meta property=\"og:description\" content=\"For smaller websites with up to a few hundred pages, you\u2019re likely using tools like Screaming Frog, Seobility, or WebSite Auditor, with Screaming Frog\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/\" \/>\n<meta property=\"og:site_name\" content=\"SEO Blog by Ahrefs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Ahrefs\/\" \/>\n<meta property=\"article:author\" content=\"nick.churick\" \/>\n<meta property=\"article:published_time\" content=\"2024-09-04T13:10:30+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-09-05T11:10:39+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/SEO-Crawlers-search-volumes-in-ahrefs-keywords-explorer.png\" \/>\n<meta name=\"author\" content=\"Nick Churick\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@nickchurick\" \/>\n<meta name=\"twitter:site\" content=\"@ahrefs\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/awt-website-crawler\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/awt-website-crawler\\\/\"},\"author\":{\"name\":\"Nick Churick\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/f1e225898abcef1c1f7525ae6462663f\"},\"headline\":\"The Game-Changer in Free Website Crawlers You Need to Know&nbsp;About\",\"datePublished\":\"2024-09-04T13:10:30+00:00\",\"dateModified\":\"2024-09-05T11:10:39+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/awt-website-crawler\\\/\"},\"wordCount\":1479,\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/awt-website-crawler\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2024\\\/09\\\/the-game-changer-in-free-website-crawlers-by-nick-churick-product-blog.jpg\",\"articleSection\":[\"Product Blog\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/awt-website-crawler\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/awt-website-crawler\\\/\",\"name\":\"A Free Website Crawler from Ahrefs\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/awt-website-crawler\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/awt-website-crawler\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2024\\\/09\\\/SEO-Crawlers-search-volumes-in-ahrefs-keywords-explorer.png\",\"datePublished\":\"2024-09-04T13:10:30+00:00\",\"dateModified\":\"2024-09-05T11:10:39+00:00\",\"description\":\"For smaller websites with up to a few hundred pages, you\u2019re likely using tools like Screaming Frog, Seobility, or WebSite Auditor, with Screaming Frog\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ahrefs.com\\\/blog\\\/awt-website-crawler\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/awt-website-crawler\\\/#primaryimage\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2024\\\/09\\\/SEO-Crawlers-search-volumes-in-ahrefs-keywords-explorer.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2024\\\/09\\\/SEO-Crawlers-search-volumes-in-ahrefs-keywords-explorer.png\",\"width\":780,\"height\":233,\"caption\":\"The search volume data from Ahrefs' Keywords Explorer\"},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"name\":\"SEO Blog by Ahrefs\",\"description\":\"Link Building Strategies &amp; SEO Tips\",\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\",\"name\":\"Ahrefs\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"width\":2048,\"height\":768,\"caption\":\"Ahrefs\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/Ahrefs\\\/\",\"https:\\\/\\\/x.com\\\/ahrefs\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/ahrefs\\\/\",\"https:\\\/\\\/www.youtube.com\\\/c\\\/ahrefscom\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/f1e225898abcef1c1f7525ae6462663f\",\"name\":\"Nick Churick\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2017\\\/03\\\/Avatar.jpg4a190d34fef4979fae4357edcb2a7466\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2017\\\/03\\\/Avatar.jpg\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2017\\\/03\\\/Avatar.jpg\",\"caption\":\"Nick Churick\"},\"description\":\"Nick is one of our Product Marketers and coincidentally he's also a pretty skilled writer. So there you have it - he's now a regular contributor to our blog.\",\"sameAs\":[\"nick.churick\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/nick-churick-570a4195\\\/\",\"https:\\\/\\\/x.com\\\/nickchurick\"],\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/author\\\/nick-churick\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"A Free Website Crawler from Ahrefs","description":"For smaller websites with up to a few hundred pages, you\u2019re likely using tools like Screaming Frog, Seobility, or WebSite Auditor, with Screaming Frog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/","og_locale":"en_US","og_type":"article","og_title":"A Free Website Crawler from Ahrefs","og_description":"For smaller websites with up to a few hundred pages, you\u2019re likely using tools like Screaming Frog, Seobility, or WebSite Auditor, with Screaming Frog","og_url":"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/","og_site_name":"SEO Blog by Ahrefs","article_publisher":"https:\/\/www.facebook.com\/Ahrefs\/","article_author":"nick.churick","article_published_time":"2024-09-04T13:10:30+00:00","article_modified_time":"2024-09-05T11:10:39+00:00","og_image":[{"url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/SEO-Crawlers-search-volumes-in-ahrefs-keywords-explorer.png","type":"","width":"","height":""}],"author":"Nick Churick","twitter_card":"summary_large_image","twitter_creator":"@nickchurick","twitter_site":"@ahrefs","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/#article","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/"},"author":{"name":"Nick Churick","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/f1e225898abcef1c1f7525ae6462663f"},"headline":"The Game-Changer in Free Website Crawlers You Need to Know&nbsp;About","datePublished":"2024-09-04T13:10:30+00:00","dateModified":"2024-09-05T11:10:39+00:00","mainEntityOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/"},"wordCount":1479,"publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/the-game-changer-in-free-website-crawlers-by-nick-churick-product-blog.jpg","articleSection":["Product Blog"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/","url":"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/","name":"A Free Website Crawler from Ahrefs","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/#primaryimage"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/SEO-Crawlers-search-volumes-in-ahrefs-keywords-explorer.png","datePublished":"2024-09-04T13:10:30+00:00","dateModified":"2024-09-05T11:10:39+00:00","description":"For smaller websites with up to a few hundred pages, you\u2019re likely using tools like Screaming Frog, Seobility, or WebSite Auditor, with Screaming Frog","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ahrefs.com\/blog\/awt-website-crawler\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/awt-website-crawler\/#primaryimage","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/SEO-Crawlers-search-volumes-in-ahrefs-keywords-explorer.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/09\/SEO-Crawlers-search-volumes-in-ahrefs-keywords-explorer.png","width":780,"height":233,"caption":"The search volume data from Ahrefs' Keywords Explorer"},{"@type":"WebSite","@id":"https:\/\/ahrefs.com\/blog\/#website","url":"https:\/\/ahrefs.com\/blog\/","name":"SEO Blog by Ahrefs","description":"Link Building Strategies &amp; SEO Tips","publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ahrefs.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ahrefs.com\/blog\/#organization","name":"Ahrefs","url":"https:\/\/ahrefs.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","width":2048,"height":768,"caption":"Ahrefs"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Ahrefs\/","https:\/\/x.com\/ahrefs","https:\/\/www.linkedin.com\/company\/ahrefs\/","https:\/\/www.youtube.com\/c\/ahrefscom"]},{"@type":"Person","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/f1e225898abcef1c1f7525ae6462663f","name":"Nick Churick","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2017\/03\/Avatar.jpg4a190d34fef4979fae4357edcb2a7466","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2017\/03\/Avatar.jpg","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2017\/03\/Avatar.jpg","caption":"Nick Churick"},"description":"Nick is one of our Product Marketers and coincidentally he's also a pretty skilled writer. So there you have it - he's now a regular contributor to our blog.","sameAs":["nick.churick","https:\/\/www.linkedin.com\/in\/nick-churick-570a4195\/","https:\/\/x.com\/nickchurick"],"url":"https:\/\/ahrefs.com\/blog\/author\/nick-churick\/"}]}},"as_json":null,"json_reviewers":[],"_links":{"self":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/179611","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/users\/116"}],"replies":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/comments?post=179611"}],"version-history":[{"count":0,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/179611\/revisions"}],"wp:attachment":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media?parent=179611"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/categories?post=179611"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/tags?post=179611"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/coauthors?post=179611"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}