{"id":140730,"date":"2022-04-11T04:16:34","date_gmt":"2022-04-11T09:16:34","guid":{"rendered":"https:\/\/ahrefs.com\/blog\/?p=140730"},"modified":"2024-06-16T17:17:29","modified_gmt":"2024-06-16T22:17:29","slug":"log-file-analysis","status":"publish","type":"post","link":"https:\/\/ahrefs.com\/blog\/log-file-analysis\/","title":{"rendered":"How to Do an SEO Log File Analysis [Template Included]"},"content":{"rendered":"\n<div class=\"intro-txt\">Log files&nbsp;have been receiving increasing recognition from technical SEOs over the past five years, and for a good reason.<\/div>\n\n\n\n<p>They\u2019re the most&nbsp;trustworthy&nbsp;source of information to understand the URLs that search engines have crawled,&nbsp;which can be critical information to help diagnose problems with technical SEO.<\/p>\n\n\n\n<p>Google itself&nbsp;recognizes&nbsp;their&nbsp;importance, releasing new features in Google Search Console&nbsp;and&nbsp;making it easy to see samples of data that would previously only be available by analyzing&nbsp;logs.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/1-crawl-stats.png\" alt=\"Crawl stats report; key data above and line graph showing trend of crawl requests below \"><\/figure>\n<\/div>\n\n\n<p>In addition, Google Search Advocate John Mueller has publicly stated how much good information log files&nbsp;hold.<\/p>\n\n\n\n<p>https:\/\/twitter.com\/JohnMu\/status\/717455167344521217<\/p>\n\n\n\n<p>With all this hype around the data in log files, you may want to understand logs better, how to analyze&nbsp;them, and whether the sites&nbsp;you\u2019re working on will&nbsp;benefit from&nbsp;them.<\/p>\n\n\n\n<p>This article will answer&nbsp;all of that and more. Here\u2019s&nbsp;what we\u2019ll be discussing:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"#1-server-log-file\">What is a server log&nbsp;file?<\/a><\/li>\n<li><a href=\"#2-log-files-benefit-seo\">How log files benefit SEO<\/a><\/li>\n<li><a href=\"#3-access-log-files\">How to access your log&nbsp;files<\/a><\/li>\n<li><a href=\"#4-analyze-log-files\">How to analyze your log&nbsp;files<\/a><\/li>\n<\/ul>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"First, what is a server log file?\" data-section=\"1-server-log-file\">\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h.bvk2852m6of7\">First, what is a server log&nbsp;file?<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>A server log file is a file created and updated by a server that records the activities it has performed. A popular server log file is an <strong>access log file<\/strong>, which holds a history of HTTP requests to the server (by both users and&nbsp;bots).<\/p>\n\n\n\n<p>When a non-developer mentions a log file, access logs are the ones they\u2019ll&nbsp;usually be referring to.<\/p>\n\n\n\n<p>Developers, however, find themselves spending more time looking at error logs, which report issues encountered by the server.<\/p>\n\n\n\n<p>The above is important: If you request logs from a developer, the first thing they\u2019ll ask is, \u201cWhich ones?\u201d<\/p>\n\n\n\n<p>Therefore, <strong>always be specific with log file requests<\/strong>. If&nbsp;you want logs to analyze&nbsp;crawling, <strong>ask for access logs<\/strong>.<\/p>\n\n\n\n<p>Access log files contain lots of information about each request made to the server, such as the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>IP addresses<\/li>\n<li>User agents<\/li>\n<li>URL path<\/li>\n<li>Timestamps (when the bot\/browser made the request)<\/li>\n<li>Request type (GET or&nbsp;POST)<\/li>\n<li><a href=\"https:\/\/ahrefs.com\/blog\/http-status-codes\/\">HTTP status codes<\/a><\/li>\n<\/ul>\n\n\n\n<p>What&nbsp;servers include in access logs varies by the server type&nbsp;and sometimes what developers have&nbsp;configured the server to store in log files. Common formats for log files include the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Apache format <\/strong>\u2013&nbsp;This is used by Nginx and Apache servers.<\/li>\n<li><strong>W3C format <\/strong>\u2013&nbsp;This is used by Microsoft IIS servers.<\/li>\n<li><strong>ELB format <\/strong>\u2013&nbsp;This is used by Amazon Elastic Load Balancing.<\/li>\n<li><strong>Custom formats <\/strong>\u2013&nbsp;Many servers&nbsp;support outputting a custom log format.<\/li>\n<\/ul>\n\n\n\n<p>Other forms&nbsp;exist, but these are the main ones you\u2019ll encounter.<\/p>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"How log files benefit SEO\" data-section=\"2-log-files-benefit-seo\">\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h.uv0ti7nltevk\">How log files benefit SEO<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>Now that we\u2019ve got a basic&nbsp;understanding of log files, let\u2019s see&nbsp;how&nbsp;they benefit SEO.<\/p>\n\n\n\n<p>Here are some key&nbsp;ways:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Crawl monitoring <\/strong>\u2013&nbsp;You can see the URLs search engines crawl and use this to spot crawler traps, look out for crawl budget wastage, or better understand how quickly content changes are picked up.<\/li>\n<li><strong>Status code reporting <\/strong>\u2013&nbsp;This is particularly useful for prioritizing&nbsp;fixing errors. Rather&nbsp;than knowing you\u2019ve got a 404, you can see precisely&nbsp;how many times a user\/search engine is visiting the 404&nbsp;URL.<\/li>\n<li><strong>Trends analysis <\/strong>\u2013&nbsp;By monitoring&nbsp;crawling over time to a URL, page type\/site section, or your entire site, you can spot changes and investigate potential causes.<\/li>\n<li><strong>Orphan page discovery <\/strong>\u2013&nbsp;You can cross-analyze&nbsp;data from log files and a site crawl you run yourself to discover orphan pages.<\/li>\n<\/ul>\n\n\n\n<p>All sites&nbsp;will benefit from log file analysis to some degree, but&nbsp;the amount of benefit <strong>varies massively<\/strong>&nbsp;depending on site&nbsp;size.<\/p>\n\n\n\n<p>This is as&nbsp;log files&nbsp;primarily benefit sites by helping you<strong>&nbsp;better <\/strong><strong>manage<\/strong><strong>&nbsp;crawling<\/strong>. <a>Google <\/a><a>itself<\/a> <a>states<\/a>&nbsp;managing the crawl budget is&nbsp;something larger-scale&nbsp;or frequently changing sites&nbsp;will benefit from.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/2-google-recommendations.jpg\" alt=\"Excerpt of Google article \"><\/figure>\n<\/div>\n\n\n<p>The same is true for log file&nbsp;analysis.<\/p>\n\n\n\n<p>For example, smaller sites can likely use the \u201cCrawl stats\u201d&nbsp;data provided in Google Search Console and receive all of the benefits mentioned above\u2014without ever needing to touch a log&nbsp;file.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/3-crawl-stats.gif\" alt=\"Gif of Crawl stats report being scrolled down gradually \"><\/figure>\n<\/div>\n\n\n<p>Yes, Google won\u2019t provide you&nbsp;with <strong>all URLs<\/strong>&nbsp;crawled (like with log files), and the trends analysis is limited to three&nbsp;months of&nbsp;data.<\/p>\n\n\n\n<p>However, smaller sites that change infrequently also need less ongoing technical SEO. It\u2019ll&nbsp;likely suffice to have a site auditor discover and diagnose issues.<\/p>\n\n\n\n<p>For example, a cross-analysis from a site crawler, XML sitemaps, Google Analytics, and Google Search Console will likely&nbsp;discover all orphan pages.<\/p>\n\n\n\n<p>You can&nbsp;also use a site auditor to discover error status codes from internal links.<\/p>\n\n\n\n<p>There are a few key reasons I\u2019m pointing this&nbsp;out:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Access log files <strong>aren\u2019t easy to get a hold of<\/strong>&nbsp;(more on this&nbsp;next).<\/li>\n<li>For small sites that change infrequently, the benefit of log files isn\u2019t&nbsp;as much, meaning SEO focuses will&nbsp;likely&nbsp;go&nbsp;elsewhere.<\/li>\n<\/ul>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"How to access your log files\" data-section=\"3-access-log-files\">\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h.e4tsk6i4z6k7\">How to access your log&nbsp;files<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>In most cases, to analyze&nbsp;log files, you\u2019ll first have to request <strong>access <\/strong>to&nbsp;log&nbsp;files from a developer.<\/p>\n\n\n\n<p>The developer is then likely going to have a few issues, which they\u2019ll bring to your attention.&nbsp;These include:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Partial data <\/strong>\u2013&nbsp;Log files can include partial data scattered across multiple servers. This usually happens when developers use various servers, such as an origin server, load balancers, and a CDN. Getting an accurate picture of&nbsp;all logs will likely&nbsp;mean compiling the access logs from all servers.<\/li>\n<li><strong>File size <\/strong>\u2013&nbsp;Access log files for high-traffic sites can end up in terabytes, if not petabytes, making them hard to transfer.<\/li>\n<li><strong>Privacy\/compliance <\/strong>\u2013&nbsp;Log files include user IP addresses that&nbsp;are&nbsp;personally identifiable information (PII). User information may need removing before it&nbsp;can be shared&nbsp;with you.<\/li>\n<li><strong>Storage history <\/strong>\u2013&nbsp;Due to file size, developers may have configured access logs to be&nbsp;stored for a few days only, making them not useful for spotting trends and issues.<\/li>\n<\/ul>\n\n\n\n<p>These&nbsp;issues will bring to question whether storing, merging, filtering, and transferring log files are&nbsp;worth the dev effort, especially if developers&nbsp;already have a long list of priorities (which is often the&nbsp;case).<\/p>\n\n\n\n<p>Developers will likely put the onus&nbsp;on the SEO to explain\/build a case for&nbsp;why developers should invest time in this, which you will need to prioritize&nbsp;among other SEO focuses.<\/p>\n\n\n\n<p><strong>These issues are<\/strong><strong>&nbsp;precisely why log file analysis doesn\u2019t happen frequently.<\/strong><\/p>\n\n\n\n<p>Log files you receive from developers are also&nbsp;often formatted in unsupported&nbsp;ways by popular log file analysis tools, making analysis more difficult.<\/p>\n\n\n\n<p>Thankfully, there are software solutions that simplify this process. My favorite&nbsp;is <a href=\"https:\/\/logflare.app\">Logflare<\/a>, a <a href=\"https:\/\/www.cloudflare.com\/apps\/\">Cloudflare app<\/a>&nbsp;that can store log files in a <a>BigQuery database<\/a>&nbsp;that you&nbsp;own.<\/p>\n\n\n\n<div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"How to analyze your log files\" data-section=\"4-analyze-log-files\">\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h.er2rl5bbqktl\">How to analyze your log&nbsp;files<\/h2>\n\n\n\n<\/div><\/div>\n\n\n\n<p>Now it\u2019s time to start analyzing&nbsp;your logs.<\/p>\n\n\n\n<p>I\u2019m going to show you how to do this in the context of Logflare specifically; however, the tips on how to use log data will work with any&nbsp;logs.<\/p>\n\n\n\n<p>The template I\u2019ll share shortly also works with any logs. You\u2019ll&nbsp;just need to make sure the columns in the data sheets match&nbsp;up.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h.ytqc8k41yvpl\">1. Start by setting up Logflare (optional)<\/h3>\n\n\n\n<p>Logflare is simple to set up.&nbsp;And with the BigQuery integration, it stores data long&nbsp;term. You\u2019ll&nbsp;own the data, making it easily accessible for everyone.<\/p>\n\n\n\n<p>There\u2019s one&nbsp;difficulty.&nbsp;You need to swap out your <a href=\"https:\/\/developers.cloudflare.com\/dns\/zone-setups\/full-setup\/setup\/\">domain name servers to use Cloudflare ones<\/a>&nbsp;and manage your DNS&nbsp;there.<\/p>\n\n\n\n<p>For most, this is fine. However, if you\u2019re working with a more enterprise-level site, it\u2019s unlikely you can convince&nbsp;the server infrastructure team&nbsp;to change&nbsp;the name servers to simplify log analysis.<\/p>\n\n\n\n<p>I won\u2019t go&nbsp;through every step on how to get Logflare working.&nbsp;But to get started, all you need to do is head to the Cloudflare Apps part of your dashboard.<\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-140713\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/4-cloudflare-app-463x425.png\" alt=\"&quot;Apps&quot; in a sidebar \" width=\"370.4\" height=\"340\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/4-cloudflare-app-463x425.png 463w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/4-cloudflare-app.png 510w\" sizes=\"(max-width: 463px) 100vw, 463px\"><\/p>\n\n\n\n\n\n\n\n<p>And then search for Logflare.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/5-logflare.png\" alt=\"&quot;Logflare&quot; appearing in search field on top-right corner, and the app appearing below in the results \"><\/figure>\n<\/div>\n\n\n<p>The setup past this point is self-explanatory&nbsp;(create an account, give your project a name, choose the data to send, etc.). The only extra part I recommend following is <a href=\"https:\/\/logflare.app\/guides\/bigquery-setup\">Logflare\u2019s guide to&nbsp;setting up BigQuery<\/a>.<\/p>\n\n\n\n<p>Bear in mind, however, that <a>BigQuery does have a cost<\/a>&nbsp;that\u2019s based on the queries you do and&nbsp;the amount of data you&nbsp;store.<\/p>\n\n\n\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div>&nbsp;It\u2019s worth noting that one significant&nbsp;advantage of the BigQuery backend is that you <strong>own the data<\/strong>. That means you can circumvent PII issues by configuring Logflare not to send PII like IP addresses and&nbsp;delete PII from BigQuery using an SQL&nbsp;query.<\/div>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h.hwqk3brpqqe4\">2. Verify Googlebot<\/h3>\n\n\n\n<p>We\u2019ve&nbsp;now stored log files&nbsp;(via Logflare or an alternative method). Next, we need to extract logs precisely&nbsp;from the user agents we want to analyze. For most, this will be <a href=\"https:\/\/ahrefs.com\/blog\/googlebot\/\">Googlebot<\/a>.<\/p>\n\n\n\n<p>Before we do that, we have another hurdle to jump across.<\/p>\n\n\n\n<p>Many bots pretend to&nbsp;be Googlebot to get past firewalls (if you have one). In addition, some auditing tools do the same to&nbsp;get an accurate reflection of the content your site returns for the user agent, which is essential if your&nbsp;server returns different HTML for Googlebot, e.g., if you\u2019ve set up <a>dynamic rendering<\/a>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h.3s0cdnwcmju3\">I\u2019m not using Logflare<\/h4>\n\n\n\n<p>If you aren\u2019t using Logflare, identifying Googlebot will require a reverse DNS lookup to verify the request did come from Google.<\/p>\n\n\n\n<p>Google has a handy guide <a>on validating Googlebot manually here<\/a>.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/6-validating-googlebot-manually.png\" alt=\"Excerpt of Google article\"><\/figure>\n<\/div>\n\n\n<p>You can do this on a one-off basis, using a <a href=\"https:\/\/mxtoolbox.com\/ReverseLookup.aspx\">reverse IP lookup tool<\/a>&nbsp;and checking the domain name returned.<\/p>\n\n\n\n<p>However, we need to do this in bulk for all rows in our log files. This also requires <a>you to match IP addresses from a list<\/a>&nbsp;provided&nbsp;by Google.<\/p>\n\n\n\n<p>The easiest way to do this is by using server firewall rule sets maintained by third parties&nbsp;that block fake bots (resulting in fewer\/no&nbsp;fake Googlebots&nbsp;in your log files). A&nbsp;popular one for Nginx is \u201c<a href=\"https:\/\/github.com\/mitchellkrogza\/nginx-ultimate-bad-bot-blocker\">Nginx Ultimate Bad Bot Blocker<\/a>.\u201d<\/p>\n\n\n\n<p>Alternatively, something you\u2019ll note <a>on the list of Googlebot IPs<\/a>&nbsp;is the IPV4 addresses all begin with \u201c66.\u201d<\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-140702\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/7-googlebot-ips-550x425.png\" alt=\"List of IPV4 addresses\" width=\"412.5\" height=\"319\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/7-googlebot-ips-550x425.png 550w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/7-googlebot-ips.png 556w\" sizes=\"(max-width: 550px) 100vw, 550px\"><\/p>\n\n\n\n\n\n\n\n<p>While it&nbsp;won\u2019t be 100% accurate, you can&nbsp;also check for Googlebot by filtering for IP addresses starting&nbsp;with \u201c6\u201d&nbsp;when analyzing&nbsp;the data within your&nbsp;logs.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h.1gk1vbnco57c\">I\u2019m using Cloudflare\/Logflare<\/h4>\n\n\n\n<p>Cloudflare\u2019s pro plan (currently $20\/month) has built-in firewall&nbsp;features that can block fake Googlebot requests from accessing your&nbsp;site.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/8-cloudflare-pricings.png\" alt=\"Cloudflare pricing \"><\/figure>\n<\/div>\n\n\n<p>Cloudflare disables these features by default, but you can find them&nbsp;by heading to <em>Firewall &gt; Managed Rules<\/em><em>&nbsp;&gt; <\/em><em>enabling <\/em><em>\u201c<\/em><em>Cloudflare Specials<\/em><em>\u201d <\/em><em>&gt; <\/em><em>select <\/em><em>\u201c<\/em><em>Advanced<\/em>\u201d:<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/9-cloudflare-specials.jpg\" alt=\"Webpage showing &quot;Managed Rules&quot;\"><\/figure>\n<\/div>\n\n\n<p>Next, change the search type from \u201cDescription\u201d&nbsp;to \u201cID\u201d&nbsp;and search for \u201c100035.\u201d<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/10-id.png\" alt=\"List of description IDs\"><\/figure>\n<\/div>\n\n\n<p>Cloudflare will now present you&nbsp;with a list of options to block fake search bots. Set the relevant ones to \u201cBlock,\u201d&nbsp;and Cloudflare will check all requests from search bot user agents are legitimate, keeping your log files&nbsp;clean.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h.3zpvhg93221h\">3. Extract data from log&nbsp;files<\/h3>\n\n\n\n<p>Finally, we now have access to log files, and we know the log files accurately reflect genuine&nbsp;Googlebot requests.<\/p>\n\n\n\n<p>I recommend analyzing&nbsp;your log files within Google Sheets\/Excel to start with because&nbsp;you\u2019ll likely be used to spreadsheets, and it\u2019s simple&nbsp;to cross-analyze&nbsp;log files with other sources like a site&nbsp;crawl.<\/p>\n\n\n\n<p>There is no one right way to do this. You can use the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/www.gnu.org\/software\/grep\/manual\/grep.html\">grep<\/a><\/li>\n<li><a href=\"https:\/\/www.splunk.com\">Splunk<\/a><\/li>\n<li><a href=\"https:\/\/logz.io\">logz.io<\/a><\/li>\n<li><a href=\"https:\/\/www.elastic.co\/elastic-stack\/\">ELK stack<\/a><\/li>\n<\/ul>\n\n\n\n<p>You can also do this within a Data Studio report. I find Data Studio helpful for monitoring data over time, and Google Sheets\/Excel is better for a one-off analysis when technical auditing.<\/p>\n\n\n\n<p>Open BigQuery and head to your project\/dataset.<\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-140706\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/11-project-dataset.png\" alt=\"Sidebar showing project dataset \" width=\"459.2\" height=\"235\"><\/p>\n\n\n\n\n\n\n\n<p>Select the \u201cQuery\u201d&nbsp;dropdown and open it in a new&nbsp;tab.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/12-query-new-tab.png\" alt=\"&quot;Query&quot; dropdown showing 2 options: new tab or split tab\"><\/figure>\n<\/div>\n\n\n<p>Next, you\u2019ll need to write some SQL to extract the data you\u2019ll be analyzing. To&nbsp;make this easier, first copy the contents of the FROM part of the&nbsp;query.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/13-from.png\" alt=\"FROM part of the query\"><\/figure>\n<\/div>\n\n\n<p>And then you can add that within the query I\u2019ve written for you&nbsp;below:<\/p>\n\n\n\n<p><code>SELECT DATE(timestamp) AS Date, req.url AS URL, req_headers.cf_connecting_ip AS IP, req_headers.user_agent AS User_Agent, resp.status_code AS Status_Code, resp.origin_time AS Origin_Time, resp_headers.cf_cache_status AS Cache_Status, resp_headers.content_type AS Content_Type<\/code><\/p>\n\n\n\n<p><code>FROM `[Add Your from address here]`,<\/code><\/p>\n\n\n\n<p><code>UNNEST(metadata) m,<\/code><\/p>\n\n\n\n<p><code>UNNEST(m.request) req,<\/code><\/p>\n\n\n\n<p><code>UNNEST(req.headers) req_headers,<\/code><\/p>\n\n\n\n<p><code>UNNEST(m.response) resp,<\/code><\/p>\n\n\n\n<p><code>UNNEST(resp.headers) resp_headers<\/code><\/p>\n\n\n\n<p><code>WHERE DATE(timestamp) &gt;= \"2022-01-03\" AND (req_headers.user_agent LIKE '%Googlebot%' OR req_headers.user_agent LIKE '%bingbot%')<\/code><\/p>\n\n\n\n<p><code>ORDER BY timestamp DESC<\/code><\/p>\n\n\n\n<p>This query selects all the columns of data that are&nbsp;useful for log file analysis for SEO purposes. It also only pulls data for Googlebot and Bingbot.<\/p>\n\n\n\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div> If there are other bots you want to analyze, just add another OR req_headers.user_agent LIKE \u2018%bot_name%\u2019 within the WHERE statement. You can also easily change the start date&nbsp;by updating the WHERE DATE(timestamp) &gt;= \u201c2022-03-03\u201d line.<\/div>\n\n\n\n\n\n<p>Select \u201cRun\u201d&nbsp;at the top.&nbsp;Then choose to save the results.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/14-query-results.png\" alt=\"Button to &quot;save results&quot; \"><\/figure>\n<\/div>\n\n\n<p>Next, save the data to a CSV in Google Drive (this is the best option due to the larger file&nbsp;size).<\/p>\n\n\n\n<p>And then, once BigQuery has run the job and saved the file, open the file with Google Sheets.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h.8w47pxehvxjg\">4. Add to Google Sheets<\/h3>\n\n\n\n<p>We\u2019re now going to start with some analysis. I recommend using my Google Sheets template.&nbsp;But I\u2019ll explain what I\u2019m doing, and you can build the report yourself if you&nbsp;want.<\/p>\n\n\n\n<p><a href=\"https:\/\/docs.google.com\/spreadsheets\/d\/1_cEZkmUFpnnZSowd9Wo-6Ux5gf7zotUCyUhhDH__1vE\/copy?usp=sharing\">Here is my template.<\/a><\/p>\n\n\n\n<p>The template consists of two data tabs to copy and paste your data into, which I then&nbsp;use&nbsp;for all other tabs using the Google Sheets <a>QUERY function<\/a>.<\/p>\n\n\n\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div> If you want to see how I\u2019ve&nbsp;completed the reports that we\u2019ll run through after setting up, select the first cell in each&nbsp;table.<\/div>\n\n\n\n<p>To start with, copy and paste the output of your export from BigQuery into the \u201cData - Log files\u201d&nbsp;tab.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/15-data-log-files.jpg\" alt=\"Output from BigQuery \"><\/figure>\n<\/div>\n\n\n<p>Note that there are multiple columns added to the end of the sheet (in darker grey) to make analysis a little easier (like the bot name&nbsp;and&nbsp;first URL directory).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h.kn1x417v31m2\">5. Add Ahrefs data<\/h3>\n\n\n\n<p>If you have a site auditor, I recommend adding more&nbsp;data to the Google Sheet. Mainly, you should add&nbsp;these:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Organic traffic<\/li>\n<li>Status codes<\/li>\n<li>Crawl depth<\/li>\n<li>Indexability<\/li>\n<li>Number of internal links<\/li>\n<\/ul>\n\n\n\n<p>To get this data out of Ahrefs\u2019&nbsp;<a href=\"https:\/\/ahrefs.com\/site-audit\">Site Audit<\/a>, head to <strong>Page Explorer<\/strong>&nbsp;and select \u201cManage Columns.\u201d<\/p>\n\n\n\n<p>I then recommend adding the columns shown&nbsp;below:<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/16-columns.png\" alt=\"Columns to add\"><\/figure>\n<\/div>\n\n\n<p>Then export all of that&nbsp;data.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/17-export-data.png\" alt=\"Options to export to CSV \"><\/figure>\n<\/div>\n\n\n<p>And copy and paste into the \u201cData - Ahrefs\u201d&nbsp;sheet.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h.aq6vxnivbd4g\">6. Check for status codes<\/h3>\n\n\n\n<p>The first thing we\u2019ll analyze is <a href=\"https:\/\/ahrefs.com\/blog\/http-status-codes\/\">status codes<\/a>. This data will answer whether search bots are wasting crawl budget on non-200 URLs.<\/p>\n\n\n\n<p><strong>Note that t<\/strong><strong>his doesn\u2019t always point toward an&nbsp;issue.<\/strong><\/p>\n\n\n\n<p>Sometimes,&nbsp;Google can crawl old 301s for many years. However, it can highlight an issue if you\u2019re internally linking to many non-200 status codes.<\/p>\n\n\n\n<p>The \u201cStatus Codes - Overview\u201d&nbsp;tab has a <a>QUERY function<\/a>&nbsp;that&nbsp;summarizes the log file data and displays the results in a&nbsp;chart.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/18-log-file-data-chart.png\" alt=\"Pie chart showing summary of log file data for status codes \"><\/figure>\n<\/div>\n\n\n<p>There is also a dropdown to&nbsp;filter by bot type and&nbsp;see which ones are hitting non-200 status codes the&nbsp;most.<\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-140718\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/19-status-codes-485x425.png\" alt=\"Table showing status codes and corresponding hits; above, dropdown to filter results by bot type\" width=\"339.5\" height=\"297\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/19-status-codes-485x425.png 485w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/19-status-codes.png 610w\" sizes=\"(max-width: 485px) 100vw, 485px\"><\/p>\n\n\n\n\n\n\n\n<p>Of course, this report alone&nbsp;doesn\u2019t help us solve the issue, so I\u2019ve added another tab, \u201cURLs - Overview.\u201d<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/20-status-codes-by-url.jpg\" alt=\"List of URLs with corresponding data like status codes, organic traffic, etc\"><\/figure>\n<\/div>\n\n\n<p>You can use this to filter for URLs that return non-200 status codes. As I\u2019ve also included data from Ahrefs\u2019&nbsp;<a href=\"https:\/\/ahrefs.com\/site-audit\">Site Audit<\/a>, you can see whether you\u2019re internally linking to any of those non-200 URLs in the \u201cInlinks\u201d&nbsp;column.<\/p>\n\n\n\n<p>If you see a lot of internal links to the URL, you can then use the <strong>Internal link opportunities <\/strong>report&nbsp;to spot these incorrect internal links by simply copying and pasting the URL in the search bar with \u201cTarget page\u201d&nbsp;selected.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/21-internal-link-opportunities.jpg\" alt=\"Excerpt of Internal link opportunities report results \"><\/figure>\n<\/div>\n\n\n<h3 class=\"wp-block-heading\" id=\"h.xvxfihbks7b0\">7. Detect crawl budget wastage<\/h3>\n\n\n\n<p>The best way to highlight <a href=\"https:\/\/ahrefs.com\/blog\/crawl-budget\/\">crawl budget<\/a>&nbsp;wastage from log files that isn\u2019t&nbsp;due to crawling non-200 status codes is to find frequently crawled non-indexable URLs (e.g., they\u2019re canonicalized&nbsp;or noindexed).<\/p>\n\n\n\n<p>Since we\u2019ve added data from our log files and Ahrefs\u2019&nbsp;<a href=\"https:\/\/ahrefs.com\/site-audit\">Site Audit<\/a>, spotting these URLs is straightforward.<\/p>\n\n\n\n<p>Head to the \u201cCrawl budget wastage\u201d&nbsp;tab, and you\u2019ll find highly crawled HTML files that return a 200 but are non-indexable.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/22-crawl-budget-waste.jpg\" alt=\"List of URLs with corresponding data like hits, etc\"><\/figure>\n<\/div>\n\n\n<p>Now that you have this data,&nbsp;you\u2019ll want to investigate why the bot is crawling the&nbsp;URL. Here are some common&nbsp;reasons:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>It\u2019s&nbsp;internally linked to.<\/li>\n<li>It\u2019s&nbsp;incorrectly included&nbsp;in XML sitemaps.<\/li>\n<li>It&nbsp;has&nbsp;links from external sites.<\/li>\n<\/ul>\n\n\n\n<p>It\u2019s common for larger sites, especially those with <a href=\"https:\/\/ahrefs.com\/blog\/faceted-navigation\/\">faceted navigation<\/a>, to link to many non-indexable URLs internally.<\/p>\n\n\n\n<p>If the hit numbers in this report are very high and you believe you\u2019re wasting your crawl budget, you\u2019ll likely need to remove internal links to the URLs or <a href=\"https:\/\/ahrefs.com\/blog\/robots-txt\/\">block crawling with the robots.txt<\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h.uh1nml7tbwfa\">8. Monitor important URLs<\/h3>\n\n\n\n<p>If you have specific URLs on your site that are incredibly important to you, you may want to watch&nbsp;how often search engines crawl&nbsp;them.<\/p>\n\n\n\n<p>The \u201cURL monitor\u201d&nbsp;tab does just that, plotting the daily trend of hits for&nbsp;up to five URLs that you can&nbsp;add.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/23-daily-trend-of-hits.jpg\" alt=\"Line graph showing daily trend of hits for 4 URLs\"><\/figure>\n<\/div>\n\n\n<p>You can also filter by bot type, making it easy to monitor how often Bing or Google crawls&nbsp;a URL.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/24-url-monitoring.png\" alt=\"URL monitoring with dropdown option to filter by bot type \"><\/figure>\n<\/div>\n\n\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div> You can&nbsp;also use this report to check URLs you\u2019ve recently redirected.&nbsp;Simply add the old URL and new URL in the dropdown and see how quickly Googlebot notices the change.<\/div>\n\n\n\n<p>Often, the&nbsp;advice here is that it\u2019s a bad thing if Google doesn\u2019t crawl a URL frequently. That simply isn\u2019t the&nbsp;case.<\/p>\n\n\n\n<p>While Google tends to crawl popular URLs more frequently, it will likely <a>crawl a URL less if it doesn\u2019t change often<\/a>.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/25-staleness.jpg\" alt=\"Excerpt of Google article \"><\/figure>\n<\/div>\n\n\n<p>Still, it\u2019s helpful to&nbsp;monitor URLs like this if you need content changes picked up quickly, such as&nbsp;on a news site\u2019s homepage.<\/p>\n\n\n\n<p>In fact, if you notice Google is recrawling a URL too frequently, I\u2019ll&nbsp;advocate for trying to help it&nbsp;better manage crawl rate by doing things like adding &lt;lastmod&gt; to XML sitemaps. Here\u2019s what it&nbsp;looks like:<\/p>\n\n\n\n<p><code>&lt;?xml version=\"1.0\" encoding=\"UTF-8\"?&gt;<\/code><\/p>\n\n\n\n<p><code>&lt;urlset xmlns=\"http:\/\/www.sitemaps.org\/schemas\/sitemap\/0.9\"&gt;<\/code><\/p>\n\n\n\n<p>&nbsp; <code>&lt;url&gt;<\/code><\/p>\n\n\n\n<p>&nbsp; &nbsp; <code>&lt;loc&gt;https:\/\/www.logexample.com\/example&lt;\/loc&gt;<\/code><\/p>\n\n\n\n<p>&nbsp; &nbsp; <code>&lt;lastmod&gt;2022-10-04&lt;\/lastmod&gt;<\/code><\/p>\n\n\n\n<p>&nbsp; <code>&lt;\/url&gt;<\/code><\/p>\n\n\n\n<p><code>&lt;\/urlset&gt;<\/code><\/p>\n\n\n\n<p>You can then update the &lt;lastmod&gt; attribute whenever the content of the page changes, signaling&nbsp;Google to recrawl.<\/p>\n\n\n\n<div class=\"recommendation\"><div class=\"recommendation-title\">TIP<\/div><div class=\"recommendation-content\">\n\n\n\n<p>Be aware Google has provided mixed feedback on this attribute. In 2015, Gary Ilysses <a href=\"https:\/\/www.seroundtable.com\/google-lastmod-xml-sitemap-20579.html\">stated it\u2019s mostly ignored<\/a>.&nbsp;In 2017, <a href=\"https:\/\/www.seroundtable.com\/google-url-last-modification-date-24319.html\">John indicated it is used<\/a>.&nbsp;Recently, in 2022, Gary stated, \u201cWe are just not going to use it.\u201d <a>Google\u2019s XML sitemap documentation<\/a>&nbsp;suggests the attribute is used. But if it isn\u2019t accurate, it will be ignored.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/26-google-documentation-on-attribute.jpg\" alt=\"Excerpt of Google article \"><\/figure>\n<\/div>\n\n\n<\/div><\/div>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h.evsdk1t1vjf4\">9. Find orphan URLs<\/h3>\n\n\n\n<p>Another way to use log files is to discover orphan URLs, i.e., URLs&nbsp;that you want search engines to crawl and index but haven\u2019t internally linked to.<\/p>\n\n\n\n<p>We can do this by checking for 200 status code HTML URLs with no internal links found by Ahrefs\u2019 <a href=\"https:\/\/ahrefs.com\/site-audit\">Site Audit<\/a>.<\/p>\n\n\n\n<p>You can see the report I\u2019ve created for this named \u201cOrphan URLs.\u201d<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/27-orphan-urls.jpg\" alt=\"List of URLs with corresponding data like hits, etc\"><\/figure>\n<\/div>\n\n\n<p>There is one caveat here. As Ahrefs hasn\u2019t discovered these URLs but Googlebot has, these URLs may not be URLs we want to link to because they\u2019re non-indexable.<\/p>\n\n\n\n<p>I recommend copying and pasting&nbsp;these URLs using the \u201cCustom URL list\u201d&nbsp;functionality when setting up crawl sources for your Ahrefs project.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/28-custom-url.png\" alt=\"Page to set up crawl sources; text field to enter custom URLs \"><\/figure>\n<\/div>\n\n\n<p>This way, Ahrefs will now consider these orphan URLs found in your log files and report any issues to you in your next&nbsp;crawl:<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/29-issues.png\" alt=\"List of issues \"><\/figure>\n<\/div>\n\n\n<h3 class=\"wp-block-heading\" id=\"h.fjk3etk16wgp\">10. Monitor crawling by directory<\/h3>\n\n\n\n<p>Suppose&nbsp;you\u2019ve implemented structured URLs that indicate how you\u2019ve organized&nbsp;your site (e.g., \/features\/feature-page\/).<\/p>\n\n\n\n<p>In that case,&nbsp;you can also analyze&nbsp;log files based on the directory to see if Googlebot is crawling certain sections of the site more than others.<\/p>\n\n\n\n<p>I\u2019ve implemented this kind of analysis in the \u201cDirectories - Overview\u201d&nbsp;tab of the Google Sheet.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/30-directories.png\" alt=\"Table showing list of directories with corresponding data like organic traffic, inlinks, etc\"><\/figure>\n<\/div>\n\n\n<p>You can see I\u2019ve also included data on the number of internal links to the directories, as well as total organic traffic.<\/p>\n\n\n\n<p>You can use this to see whether Googlebot is spending more time crawling low-traffic directories than high-value&nbsp;ones.<\/p>\n\n\n\n<p>But again, bear in mind this may occur, as some URLs within specific&nbsp;directories change more often than others. Still, it\u2019s worth further investigating if you spot an odd&nbsp;trend.<\/p>\n\n\n\n<p>In addition to this report, there is also a \u201cDirectories - Crawl trend\u201d&nbsp;report if you want to see the crawl trend per directory for your&nbsp;site.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/31-crawl-trend.png\" alt=\"Line graph showing crawl trend per directory\"><\/figure>\n<\/div>\n\n\n<h3 class=\"wp-block-heading\" id=\"h.hewu3o8buy0a\">11. View Cloudflare cache ratios<\/h3>\n\n\n\n<p>Head to the \u201cCF cache status\u201d&nbsp;tab, and you\u2019ll see a summary of how often Cloudflare is caching your files on the&nbsp;edge servers.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/32-cache-status-summary.png\" alt=\"Bar chart showing how often Cloudflare is caching files on the edge servers\"><\/figure>\n<\/div>\n\n\n<p>When Cloudflare caches content (HIT in the above chart), the request no longer goes to your origin server and is served directly from its global CDN. This results in better <a href=\"https:\/\/ahrefs.com\/blog\/core-web-vitals\/\">Core Web Vital<\/a><a href=\"https:\/\/ahrefs.com\/blog\/core-web-vitals\/\">s<\/a>, especially for global sites.<\/p>\n\n\n\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div>&nbsp;It\u2019s also worth having a caching setup on your origin server (such as Varnish, Nginx FastCGI, or Redis full-page cache). This is so that even when Cloudflare hasn\u2019t cached a URL, you\u2019ll still benefit from some caching.<\/div>\n\n\n\n<p>If you see&nbsp;a large amount of \u201cMiss\u201d&nbsp;or \u201cDynamic\u201d&nbsp;responses, I recommend investigating further to understand why Cloudflare isn\u2019t caching content. Common causes can&nbsp;be:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>You\u2019re linking to URLs with parameters in them<\/strong> \u2013 Cloudflare, by default, passes requests with <a href=\"https:\/\/ahrefs.com\/blog\/url-parameters\/\">parameters<\/a> to your origin server, as they\u2019re likely dynamic.<\/li>\n<li><strong>Your cache expiry times are too low<\/strong>&nbsp;\u2013&nbsp;If you set short cache lifespans, it\u2019s likely more users will receive uncached content.<\/li>\n<li><strong>You aren\u2019t preloading your cache<\/strong>&nbsp;\u2013&nbsp;If you need your&nbsp;cache to expire often (as content changes frequently), rather than letting users hit uncached&nbsp;URLs, use a preloader bot that will prime the cache, <a href=\"https:\/\/patrickmn.com\/projects\/ocp\/\">such as Optimus Cache Preloader<\/a>.<\/li>\n<\/ul>\n\n\n\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div>&nbsp;I thoroughly&nbsp;recommend setting up HTML edge-caching via Cloudflare, which significantly reduces TTFB. You can do this easily with WordPress and&nbsp;Cloudflare\u2019s <a href=\"https:\/\/blog.cloudflare.com\/automatic-platform-optimizations-starting-with-wordpress\/\">Automatic Platform Optimization<\/a>.<\/div>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h.y2ccufmmlkja\">12. Check which bots crawl your site the&nbsp;most<\/h3>\n\n\n\n<p>The final report (found in the \u201cBots - Overview\u201d&nbsp;tab) shows you which bots crawl your site the&nbsp;most:<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/33-bots-overview.png\" alt=\"Pie chart showing Googlebot crawls site the most, as compared to Bingbot \"><\/figure>\n<\/div>\n\n\n<p>In the \u201cBots - Crawl trend\u201d&nbsp;report, you can see how that trend has changed over&nbsp;time.<\/p>\n\n\n\n\n<div class=\"wp-block-image is-style-default aligncenter\">\n<figure><img decoding=\"async\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/34-bots-crawl-trend.png\" alt=\"Stacked bar chart showing how crawl trend changes over time\"><\/figure>\n<\/div>\n\n\n<p>This report can help check if&nbsp;there\u2019s&nbsp;an increase in bot activity on your site. It\u2019s also helpful when&nbsp;you\u2019ve recently made a significant change, such as&nbsp;a <a href=\"https:\/\/ahrefs.com\/blog\/website-migration\/\">URL migration<\/a>, and want to see if bots have increased their crawling to collect new&nbsp;data.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h.45wi30lwdimf\">Final thoughts<\/h2>\n\n\n\n<p>You should now have a good idea of the analysis you can do with your log files when auditing a site. Hopefully, you\u2019ll find it easy to use my template and do this analysis yourself.<\/p>\n\n\n\n<p>Anything unique you\u2019re doing with your log files that I haven\u2019t mentioned? <a href=\"https:\/\/twitter.com\/SamUnderwoodUK\">Tweet me.<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>They\u2019re the most&nbsp;trustworthy&nbsp;source of information to understand the URLs that search engines have crawled,&nbsp;which can be critical information to help diagnose problems with technical SEO. Google itself&nbsp;recognizes&nbsp;their&nbsp;importance, releasing new features in Google Search Console&nbsp;and&nbsp;making it easy to see samples of<span class=\"ellipsis\">\u2026<\/span><\/p>\n<div class=\"read-more\">Read more \u203a<\/div>\n<p><!-- end of .read-more --><\/p>\n","protected":false},"author":162,"featured_media":140735,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"wp_typography_post_enhancements_disabled":false,"footnotes":""},"categories":[329],"tags":[],"coauthors":[423],"class_list":["post-140730","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technical-seo","odd"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>How to Do an SEO Log File Analysis [Template Included]<\/title>\n<meta name=\"description\" content=\"Log file analysis sheds light on how search engines crawl URLs on a site and can be critical in helping diagnose technical SEO issues.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ahrefs.com\/blog\/log-file-analysis\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Do an SEO Log File Analysis [Template Included]\" \/>\n<meta property=\"og:description\" content=\"Boost your technical SEO skills.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ahrefs.com\/blog\/log-file-analysis\/\" \/>\n<meta property=\"og:site_name\" content=\"SEO Blog by Ahrefs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Ahrefs\/\" \/>\n<meta property=\"article:published_time\" content=\"2022-04-11T09:16:34+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-06-16T22:17:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/fb-log_file_analysis.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Sam Underwood\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:description\" content=\"Boost your technical SEO skills.\" \/>\n<meta name=\"twitter:creator\" content=\"@SamUnderwoodUK\" \/>\n<meta name=\"twitter:site\" content=\"@ahrefs\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/log-file-analysis\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/log-file-analysis\\\/\"},\"author\":{\"name\":\"Sam Underwood\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/7a4e7e6b862119be5f43a0c5783888d5\"},\"headline\":\"How to Do an SEO Log File Analysis [Template Included]\",\"datePublished\":\"2022-04-11T09:16:34+00:00\",\"dateModified\":\"2024-06-16T22:17:29+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/log-file-analysis\\\/\"},\"wordCount\":3754,\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/log-file-analysis\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/04\\\/fb-log_file_analysis.png\",\"articleSection\":[\"Technical SEO\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/log-file-analysis\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/log-file-analysis\\\/\",\"name\":\"How to Do an SEO Log File Analysis [Template Included]\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/log-file-analysis\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/log-file-analysis\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/04\\\/fb-log_file_analysis.png\",\"datePublished\":\"2022-04-11T09:16:34+00:00\",\"dateModified\":\"2024-06-16T22:17:29+00:00\",\"description\":\"Log file analysis sheds light on how search engines crawl URLs on a site and can be critical in helping diagnose technical SEO issues.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ahrefs.com\\\/blog\\\/log-file-analysis\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/log-file-analysis\\\/#primaryimage\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/04\\\/fb-log_file_analysis.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/04\\\/fb-log_file_analysis.png\",\"width\":1200,\"height\":630},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"name\":\"SEO Blog by Ahrefs\",\"description\":\"Link Building Strategies &amp; SEO Tips\",\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\",\"name\":\"Ahrefs\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"width\":2048,\"height\":768,\"caption\":\"Ahrefs\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/Ahrefs\\\/\",\"https:\\\/\\\/x.com\\\/ahrefs\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/ahrefs\\\/\",\"https:\\\/\\\/www.youtube.com\\\/c\\\/ahrefscom\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/7a4e7e6b862119be5f43a0c5783888d5\",\"name\":\"Sam Underwood\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/09\\\/sam-underwood.jpg0aa3181310d847d8be2845189fc2b794\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/09\\\/sam-underwood.jpg\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/09\\\/sam-underwood.jpg\",\"caption\":\"Sam Underwood\"},\"description\":\"Sam Underwood is a solo consultant helping ecommerce businesses grow their organic revenue via on-page SEO, technical SEO, and content strategies.\",\"sameAs\":[\"https:\\\/\\\/www.samunderwood.co.uk\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/samunderwooduk\\\/\",\"https:\\\/\\\/x.com\\\/SamUnderwoodUK\"],\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/author\\\/sam-underwood\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to Do an SEO Log File Analysis [Template Included]","description":"Log file analysis sheds light on how search engines crawl URLs on a site and can be critical in helping diagnose technical SEO issues.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ahrefs.com\/blog\/log-file-analysis\/","og_locale":"en_US","og_type":"article","og_title":"How to Do an SEO Log File Analysis [Template Included]","og_description":"Boost your technical SEO skills.","og_url":"https:\/\/ahrefs.com\/blog\/log-file-analysis\/","og_site_name":"SEO Blog by Ahrefs","article_publisher":"https:\/\/www.facebook.com\/Ahrefs\/","article_published_time":"2022-04-11T09:16:34+00:00","article_modified_time":"2024-06-16T22:17:29+00:00","og_image":[{"width":1200,"height":630,"url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/fb-log_file_analysis.png","type":"image\/png"}],"author":"Sam Underwood","twitter_card":"summary_large_image","twitter_description":"Boost your technical SEO skills.","twitter_creator":"@SamUnderwoodUK","twitter_site":"@ahrefs","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ahrefs.com\/blog\/log-file-analysis\/#article","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/log-file-analysis\/"},"author":{"name":"Sam Underwood","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/7a4e7e6b862119be5f43a0c5783888d5"},"headline":"How to Do an SEO Log File Analysis [Template Included]","datePublished":"2022-04-11T09:16:34+00:00","dateModified":"2024-06-16T22:17:29+00:00","mainEntityOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/log-file-analysis\/"},"wordCount":3754,"publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/log-file-analysis\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/fb-log_file_analysis.png","articleSection":["Technical SEO"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/ahrefs.com\/blog\/log-file-analysis\/","url":"https:\/\/ahrefs.com\/blog\/log-file-analysis\/","name":"How to Do an SEO Log File Analysis [Template Included]","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/log-file-analysis\/#primaryimage"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/log-file-analysis\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/fb-log_file_analysis.png","datePublished":"2022-04-11T09:16:34+00:00","dateModified":"2024-06-16T22:17:29+00:00","description":"Log file analysis sheds light on how search engines crawl URLs on a site and can be critical in helping diagnose technical SEO issues.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ahrefs.com\/blog\/log-file-analysis\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/log-file-analysis\/#primaryimage","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/fb-log_file_analysis.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2022\/04\/fb-log_file_analysis.png","width":1200,"height":630},{"@type":"WebSite","@id":"https:\/\/ahrefs.com\/blog\/#website","url":"https:\/\/ahrefs.com\/blog\/","name":"SEO Blog by Ahrefs","description":"Link Building Strategies &amp; SEO Tips","publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ahrefs.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ahrefs.com\/blog\/#organization","name":"Ahrefs","url":"https:\/\/ahrefs.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","width":2048,"height":768,"caption":"Ahrefs"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Ahrefs\/","https:\/\/x.com\/ahrefs","https:\/\/www.linkedin.com\/company\/ahrefs\/","https:\/\/www.youtube.com\/c\/ahrefscom"]},{"@type":"Person","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/7a4e7e6b862119be5f43a0c5783888d5","name":"Sam Underwood","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/09\/sam-underwood.jpg0aa3181310d847d8be2845189fc2b794","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/09\/sam-underwood.jpg","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2021\/09\/sam-underwood.jpg","caption":"Sam Underwood"},"description":"Sam Underwood is a solo consultant helping ecommerce businesses grow their organic revenue via on-page SEO, technical SEO, and content strategies.","sameAs":["https:\/\/www.samunderwood.co.uk\/","https:\/\/www.linkedin.com\/in\/samunderwooduk\/","https:\/\/x.com\/SamUnderwoodUK"],"url":"https:\/\/ahrefs.com\/blog\/author\/sam-underwood\/"}]}},"as_json":null,"json_reviewers":[150,149],"_links":{"self":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/140730","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/users\/162"}],"replies":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/comments?post=140730"}],"version-history":[{"count":0,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/140730\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media\/140735"}],"wp:attachment":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media?parent=140730"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/categories?post=140730"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/tags?post=140730"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/coauthors?post=140730"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}