{"id":198700,"date":"2026-06-15T05:27:40","date_gmt":"2026-06-15T10:27:40","guid":{"rendered":"https:\/\/ahrefs.com\/blog\/?p=198700"},"modified":"2026-06-15T05:45:27","modified_gmt":"2026-06-15T10:45:27","slug":"llmstxt-study","status":"publish","type":"post","link":"https:\/\/ahrefs.com\/blog\/llmstxt-study\/","title":{"rendered":"97% of llms.txt Files Never Get Read (137,000 Sites Analyzed)"},"content":{"rendered":"<div class=\"intro-txt\">Everyone has an opinion on llms.txt, but when it comes to actual evidence we have only single-site logs or the odd small-scale experiment.<\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Using <a href=\"https:\/\/ahrefs.com\/web-analytics\">Ahrefs Web Analytics<\/a> and <a href=\"https:\/\/ahrefs.com\/bot-analytics\">Bot Analytics<\/a>, we analyzed the server logs and live traffic of 137K domains, plus the user agents hitting all of&nbsp;them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s what we&nbsp;found.<\/p>\n\n\n<div class=\"recommendation\"><div class=\"recommendation-title\">Top findings<\/div><div class=\"recommendation-content\"><a id=\"post-198700-_ul8fvq5268dg\"><\/a>\n<ul>\n<li><strong>28%<\/strong> of 137K domains using Ahrefs Web Analytics publish an llms.txt file.<\/li>\n<li><strong>97%<\/strong> of those files received zero traffic in May 2026. Nothing fetched them at&nbsp;all.<\/li>\n<li><strong>96%<\/strong> of the requests that did reach llms.txt files came from&nbsp;bots.<\/li>\n<li><strong>19.5%<\/strong> of fetches came from named AI tools (of the 3% of files that weren\u2019t ignored). GPTBot is top and Claude-Code is second, ahead of every AI search and assistant bot.<\/li>\n<li><strong>12%<\/strong> of fetches come from the industry studying itself: GEO\/AEO tools, llms.txt checker tools, and researchers.<\/li>\n<li><strong>Zero<\/strong> requests came from AI bots for llms.txt files that don\u2019t exist. They never go looking.<\/li>\n<li>The<strong> Chrome Lighthouse llms.txt audit<\/strong> produced roughly 1 in 1,000 fetches.<\/li>\n<\/ul>\n<\/div><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">In late May 2026, Google took both sides of the llms.txt argument in under a&nbsp;week.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Its new <a href=\"https:\/\/developers.google.com\/search\/docs\/fundamentals\/ai-optimization-guide\">guide on optimizing for generative AI features<\/a> told site owners, in a section literally titled \u201cmythbusting\u201d, that machine-readable files like llms.txt aren\u2019t needed to appear in generative AI search.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2048\" height=\"865\" class=\"wp-image-198701\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-text-excerpt-from-mythbusting-generative-ai-sea.jpg\" alt=\"A text excerpt from &quot;Mythbusting generative AI search: what you don't need to do.&quot; Highlighted text states you don't need special files or markup for generative AI search.\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-text-excerpt-from-mythbusting-generative-ai-sea.jpg 2048w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-text-excerpt-from-mythbusting-generative-ai-sea-680x287.jpg 680w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-text-excerpt-from-mythbusting-generative-ai-sea-768x324.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-text-excerpt-from-mythbusting-generative-ai-sea-1536x649.jpg 1536w\" sizes=\"auto, (max-width: 2048px) 100vw, 2048px\"><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Days later, the Chrome team shipped an llms.txt check inside <a href=\"https:\/\/developer.chrome.com\/docs\/lighthouse\/agentic-browsing\/llms-txt\">Lighthouse\u2019s experimental Agentic Browsing audits<\/a>, with documentation explaining that without the file, agents may spend more time crawling a site to understand its structure<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2048\" height=\"1434\" class=\"wp-image-198702\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-webpage-titled-llms-txt-on-chrome-for-develope.png\" alt=\"A webpage titled &quot;llms.txt&quot; on Chrome for developers, under a &quot;Lighthouse > Agentic browsing audits&quot; section.\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-webpage-titled-llms-txt-on-chrome-for-develope.png 2048w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-webpage-titled-llms-txt-on-chrome-for-develope-607x425.png 607w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-webpage-titled-llms-txt-on-chrome-for-develope-768x538.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-webpage-titled-llms-txt-on-chrome-for-develope-1536x1076.png 1536w\" sizes=\"auto, (max-width: 2048px) 100vw, 2048px\"><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When <a href=\"https:\/\/www.linkedin.com\/posts\/lily-ray-44755615_got-some-interesting-answers-from-john-mueller-ugcPost-7462840528913494019-V_Iw\/\">Lily Ray pressed Google\u2019s John Mueller on the contradiction<\/a>, he explained that llms.txt is \u201cnot done for search.\u201d It\u2019s a \u201ctemporary crutch, perhaps to save some tokens\u201d for AI coding tools parsing developer documentation\u2014not something non-developer sites need to worry&nbsp;about.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">He also stated that site owners who check their logs will find very little AI agent traffic.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1179\" height=\"1991\" class=\"wp-image-198703\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-screenshot-of-a-twitter-thread-from-john-mueller.jpg\" alt=\"A screenshot of a Twitter thread from John Mueller. The highlighted text says, &quot;even with more agentic traffic in the future (and if you check your logs, you\u2019re not getting a lot of that at the moment).&quot;\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-screenshot-of-a-twitter-thread-from-john-mueller.jpg 1179w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-screenshot-of-a-twitter-thread-from-john-mueller-252x425.jpg 252w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-screenshot-of-a-twitter-thread-from-john-mueller-768x1297.jpg 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-screenshot-of-a-twitter-thread-from-john-mueller-910x1536.jpg 910w\" sizes=\"auto, (max-width: 1179px) 100vw, 1179px\"><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This is something we decided to&nbsp;test.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;<\/p>\n\n\n<div class=\"recommendation\"><div class=\"recommendation-title\">What llms.txt is (and what it&nbsp;isn\u2019t)<\/div><div class=\"recommendation-content\"> Before we go any further, let\u2019s clear up what llms.txt actually <em>is.<\/em> Llms.txt is a single index file, written in markdown, placed at a site\u2019s root. Proposed by <a href=\"https:\/\/llmstxt.org\/\">Jeremy Howard<\/a>, co-founder of Answer.AI and fast.ai, in 2024, it summarizes what a site is and links its most important content. The idea being that LLMs and agents can use this information to orient themselves without crawling everything. The \u201cAI visibility\u201d framing around llms.txt came later on, attached by the SEO industry as adoption spread on the speculation that AI platforms would reward the file. Two things it is often confused with, and&nbsp;isn\u2019t.\n<ul>\n<li>It is not the practice of publishing markdown copies of your web pages, <a href=\"https:\/\/www.jonoalderson.com\/conjecture\/more-than-words\/\">a separate tactic with its own problems<\/a>.<\/li>\n<li>And despite the filename, it is not a robots.txt-style directive: it controls nothing and blocks nothing.<\/li>\n<\/ul>\n<p>This study measures the index file, and only the index&nbsp;file.&nbsp;<\/p><\/div><\/div>\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-198700-_hymdtq22qz2f\"><\/a><div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"Methodology\" data-section=\"methodology\"> Methodology&nbsp;<\/div><\/div><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Our study focuses on 137,210 domains in <a href=\"https:\/\/ahrefs.com\/web-analytics\">Ahrefs Web Analytics<\/a> that received traffic in May&nbsp;2026.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We checked each domain root for an llms.txt returning HTTP 200, then used <a href=\"https:\/\/ahrefs.com\/bot-analytics\">Ahrefs Bot Analytics<\/a> to examine every request to \/llms.txt paths across the population, split by HTTP response (200 vs 404) and classified by channel and individual user&nbsp;agent.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To rule out soft 404s and phantom files, we also confirmed each file was actual Markdown rather than HTML, and screened titles and content for error signals like \u201c404\u201d or \u201cPage not&nbsp;found\u201d<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It\u2019s important to&nbsp;note:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ahrefs Web Analytics customers skew more technical and SEO-aware than the web at large, so treat the 28% adoption figure as an upper&nbsp;bound.<\/li>\n\n\n\n<li>We did not explicitly study whether a file was well-formed against the llms.txt specification.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-198700-_h9vbc9rn8pwf\"><\/a><div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"28% of domains publish llms.txt\" data-section=\"domains-publish\"> 28% of domains publish llms.txt <\/div><\/div><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Google Search\u2019s guidance says you can skip it, the Chrome team audits for it, and Mueller calls it a stopgap for coding tools.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So amid all the mixed messages, how widespread is llms.txt actually? Among the 137K domains in our study, 28% publish these&nbsp;files.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">More than one in four domains (38,000) in our population have adopted llms.txt, despite the fact that no major AI platform has ever committed to reading it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Adoption has been driven by speculation that AI platforms may start consuming the file, rather than by any confirmation that they <em>do<\/em>.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1900\" height=\"1700\" class=\"wp-image-198704\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/pie-chart-28-of-sites-publish-valid-llms-txt-38.png\" alt=\"Pie chart: 28% of sites publish valid llms.txt (38,360 domains), 72% do not (98,640 domains).\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/pie-chart-28-of-sites-publish-valid-llms-txt-38.png 1900w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/pie-chart-28-of-sites-publish-valid-llms-txt-38-475x425.png 475w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/pie-chart-28-of-sites-publish-valid-llms-txt-38-768x687.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/pie-chart-28-of-sites-publish-valid-llms-txt-38-1536x1374.png 1536w\" sizes=\"auto, (max-width: 1900px) 100vw, 1900px\"><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-198700-_ab0yi3iavo7i\"><\/a><div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"97% of llms.txt files receive zero requests\" data-section=\"llmstxt-files\"> 97% of llms.txt files receive zero requests&nbsp;<\/div><\/div><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Almost every llms.txt file in our study is unread.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Of the ~38,000 domains with a valid file, 97% saw no requests for it whatsoever in&nbsp;May.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">No bots. No humans. Nothing.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1900\" height=\"1550\" class=\"wp-image-198705\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bar-chart-shows-ahrefs-study-of-137k-domains-97.png\" alt=\"Bar chart shows ahrefs\u2019 study of 137K domains. 97% of llm.txt files are never requested.\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bar-chart-shows-ahrefs-study-of-137k-domains-97.png 1900w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bar-chart-shows-ahrefs-study-of-137k-domains-97-521x425.png 521w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bar-chart-shows-ahrefs-study-of-137k-domains-97-768x627.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bar-chart-shows-ahrefs-study-of-137k-domains-97-1536x1253.png 1536w\" sizes=\"auto, (max-width: 1900px) 100vw, 1900px\"><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The remaining 3% (1.1K domains) received all of the llms.txt traffic we measured.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Our data suggests John Mueller is right. Not only will you find very little<em> AI traffic<\/em> as a result of this file\u2014you will find very little traffic, period.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you publish an llms.txt file today, the most likely outcome by far is that nothing ever fetches it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The 3% of files that do get read, though, get read by interesting visitors.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We\u2019ll focus on them for the rest of the&nbsp;study.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-198700-_4motgyjbtt89\"><\/a><div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"96% of requests to llms.txt files come from bots\" data-section=\"requests-llmstxt\"> 96% of requests to llms.txt files come from&nbsp;bots&nbsp;<\/div><\/div><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Llms.txt files are written for machines, and machines are nearly the only things reading them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Across the files that received traffic, 96% of requests came from&nbsp;bots.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Humans accounted for 4%, and a chunk of those appear to be SEOs sharing llms.txt links in chat apps, where unfurl bots dutifully fetch&nbsp;them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Slackbot alone fetched llms.txt files more often than PerplexityBot did.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Perplexity is one of the AI search engines llms.txt was seemingly designed to help, so finding that a chat app\u2019s link-preview bot outfetched it speaks volumes about how much real AI search interest these files are actually generating.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-198700-_s8tuwyriw9rh\"><\/a><div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"77% of the bots reading llms.txt aren't from AI tools\" data-section=\"bots-reading\"> 77% of the bots reading llms.txt aren\u2019t from AI&nbsp;tools&nbsp;<\/div><\/div><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Many sites publish llms.txt precisely because they think it will improve their chances of appearing in ChatGPT answers, or landing Perplexity citations, or winning an AI Overview.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But our data tells a different story: 77% of the bots fetching llms.txt aren\u2019t AI tools at&nbsp;all.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To understand which bots <em>were<\/em> requesting llms.txt, we classified every user agent into twelve categories.<\/p>\n\n<p><style>\n.cat-table { width:100%; border-collapse:collapse; font-family: -apple-system, BlinkMacSystemFont, \"Segoe UI\", Roboto, Arial, sans-serif; font-size:15px; line-height:1.6; }\n.cat-table thead th { font-size:11px; font-weight:500; color:#6b6b6b; letter-spacing:0.06em; text-align:left; padding:14px 16px; border-bottom:1px solid #d4d4d4; }\n.cat-table thead th.num { text-align:right; }\n.cat-table tbody tr { border-bottom:0.5px solid #e8e8e8; }\n.cat-table tbody tr:last-child { border-bottom:none; }\n.cat-table td { padding:18px 16px; vertical-align:top; }\n.cat-name { display:block; font-size:16px; font-weight:500; color:#1a1a1a; margin-bottom:4px; }\n.cat-desc { display:block; font-size:14px; color:#555; line-height:1.5; }\n.cat-examples { display:block; font-size:13px; color:#888; margin-top:6px; font-style:italic; }\n.cat-num { text-align:right; font-size:15px; color:#1a1a1a; white-space:nowrap; }\n.badge { display:inline-block; padding:3px 10px; border-radius:999px; font-size:12px; font-weight:500; white-space:nowrap; }\n.b-seo { background:#E6F1FB; color:#0C447C; }\n.b-other { background:#F1EFE8; color:#444441; }\n.b-gen { background:#FAEEDA; color:#854F0B; }\n.b-tech { background:#FAECE7; color:#993C1D; }\n.b-ai { background:#EEEDFE; color:#3C3489; }\n.b-meta { background:#FBEAF0; color:#72243E; }\n.b-soc { background:#E1F5EE; color:#085041; }\n.cat-footnote { font-size:13px; color:#6b6b6b; margin-top:14px; line-height:1.55; font-family: -apple-system, BlinkMacSystemFont, \"Segoe UI\", Roboto, Arial, sans-serif; }\n<\/style><\/p>\n<table class=\"cat-table\">\n<thead>\n<tr>\n<th>CATEGORY<\/th>\n<th>TYPE<\/th>\n<th class=\"num\">REQUESTS<\/th>\n<th class=\"num\">% OF&nbsp;TOTAL<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><span class=\"cat-name\">SEO audit tools<\/span><span class=\"cat-desc\">Crawl sites for traditional SEO health checks, with no specific interest in llms.txt<\/span><span class=\"cat-examples\">e.g. SiteAuditBot, WebPageTest<\/span><\/td>\n<td><span class=\"badge b-seo\">Auditing<\/span><\/td>\n<td class=\"cat-num\">4,776<\/td>\n<td class=\"cat-num\">21.7%<\/td>\n<\/tr>\n<tr>\n<td><span class=\"cat-name\">Other and unidentified<\/span><span class=\"cat-desc\">Anonymous SDK defaults and bots whose purpose or operator we could not determine<\/span><span class=\"cat-examples\">e.g. node, satoric-indexer<\/span><\/td>\n<td><span class=\"badge b-other\">Unknown<\/span><\/td>\n<td class=\"cat-num\">3,278<\/td>\n<td class=\"cat-num\">14.9%<\/td>\n<\/tr>\n<tr>\n<td><span class=\"cat-name\">General web crawlers<\/span><span class=\"cat-desc\">Index the web for search and product discovery, with no stated AI-agent use case<\/span><span class=\"cat-examples\">e.g. Googlebot, Amazonbot<\/span><\/td>\n<td><span class=\"badge b-gen\">Crawling<\/span><\/td>\n<td class=\"cat-num\">2,871<\/td>\n<td class=\"cat-num\">13.1%<\/td>\n<\/tr>\n<tr>\n<td><span class=\"cat-name\">Tech profiling tools<\/span><span class=\"cat-desc\">Crawl sites to identify technology stacks and business intelligence data<\/span><span class=\"cat-examples\">e.g. BuiltWith, Dataprovider<\/span><\/td>\n<td><span class=\"badge b-tech\">Profiling<\/span><\/td>\n<td class=\"cat-num\">2,546<\/td>\n<td class=\"cat-num\">11.6%<\/td>\n<\/tr>\n<tr>\n<td><span class=\"cat-name\">AI agents &amp; agentic infrastructure<\/span><span class=\"cat-desc\">AI agents acting on a user\u2019s behalf, plus the crawlers and tooling built to serve them<\/span><span class=\"cat-examples\">e.g. Claude-Code, IbouBot<\/span><\/td>\n<td><span class=\"badge b-ai\">AI<\/span><\/td>\n<td class=\"cat-num\">2,302<\/td>\n<td class=\"cat-num\">10.5%<\/td>\n<\/tr>\n<tr>\n<td><span class=\"cat-name\">GEO\/AEO tools<\/span><span class=\"cat-desc\">Scan websites and score their readiness for AI search and agent discovery<\/span><span class=\"cat-examples\">e.g. CairrotReadinessBot, AuditMetricBot<\/span><\/td>\n<td><span class=\"badge b-meta\">Studying llms.txt<\/span><\/td>\n<td class=\"cat-num\">1,278<\/td>\n<td class=\"cat-num\">5.8%<\/td>\n<\/tr>\n<tr>\n<td><span class=\"cat-name\">AI training crawlers<\/span><span class=\"cat-desc\">Collect data for model building<\/span><span class=\"cat-examples\">e.g. GPTBot, ClaudeBot<\/span><\/td>\n<td><span class=\"badge b-ai\">AI<\/span><\/td>\n<td class=\"cat-num\">1,179<\/td>\n<td class=\"cat-num\">5.3%<\/td>\n<\/tr>\n<tr>\n<td><span class=\"cat-name\">llms.txt discoverability bots<\/span><span class=\"cat-desc\">Specifically scan, validate, or catalogue llms.txt files<\/span><span class=\"cat-examples\">e.g. LLMS-Txt-Scanner, txtfeed-bot<\/span><\/td>\n<td><span class=\"badge b-meta\">Studying llms.txt<\/span><\/td>\n<td class=\"cat-num\">793<\/td>\n<td class=\"cat-num\">3.6%<\/td>\n<\/tr>\n<tr>\n<td><span class=\"cat-name\">Service and social bots<\/span><span class=\"cat-desc\">Fetch URLs to generate link previews in messaging apps and social platforms<\/span><span class=\"cat-examples\">e.g. Slackbot, Skype URI Preview<\/span><\/td>\n<td><span class=\"badge b-soc\">Social<\/span><\/td>\n<td class=\"cat-num\">645<\/td>\n<td class=\"cat-num\">2.9%<\/td>\n<\/tr>\n<tr>\n<td><span class=\"cat-name\">Research bots<\/span><span class=\"cat-desc\">Crawl for academic or investigative purposes, including security research<\/span><span class=\"cat-examples\">e.g. prompt-injection-survey, ResearchProject<\/span><\/td>\n<td><span class=\"badge b-meta\">Studying llms.txt<\/span><\/td>\n<td class=\"cat-num\">585<\/td>\n<td class=\"cat-num\">2.7%<\/td>\n<\/tr>\n<tr>\n<td><span class=\"cat-name\">AI assistants<\/span><span class=\"cat-desc\">Browse the web on behalf of a user in response to a single query<\/span><span class=\"cat-examples\">e.g. ChatGPT-User, Claude-User<\/span><\/td>\n<td><span class=\"badge b-ai\">AI<\/span><\/td>\n<td class=\"cat-num\">559<\/td>\n<td class=\"cat-num\">2.5%<\/td>\n<\/tr>\n<tr>\n<td><span class=\"cat-name\">AI retrieval bots<\/span><span class=\"cat-desc\">Fetch pages to answer live user queries in AI search products<\/span><span class=\"cat-examples\">e.g. OAI-SearchBot, PerplexityBot<\/span><\/td>\n<td><span class=\"badge b-ai\">AI<\/span><\/td>\n<td class=\"cat-num\">233<\/td>\n<td class=\"cat-num\">1.1%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p class=\"cat-footnote\">* SEO audit tools includes Ahrefs\u2019 own crawlers (SiteAuditBot, Ahrefs Bot, and Ahrefs Site Audit), which together account for 2,334 requests (10.6% of total). Excluding them, third-party SEO audit tools account for 2,442 requests (11.1%). Bot categories sum to 96% of total requests; the remaining 4% (930 requests) came from humans.<\/p>\n\n<p>Individually, no AI bot category makes the top&nbsp;four.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">SEO audit tools (21.7%), Other and unidentified (14.9%), General web crawlers (13.1%), and Tech profiling tools (11.6%) all send more requests than any one AI&nbsp;bot.<\/p>\n\n\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div> That top category also contains Chrome\u2019s Lighthouse audit, the check that reignited the llms.txt debate. It made just 22 requests\u2014roughly 1 in 1,000.<\/div>\n\n\n\n<p class=\"wp-block-paragraph\">The biggest standalone AI category, AI agents, sits in fifth place at 10.5%.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But when you combine the four AI categories (training crawlers, retrieval bots, assistants, and agents), AI bots become the largest single bucket at 19.5%.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The bot traffic splits into three stories:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>AI bots consuming the file (19.5%)<\/li>\n\n\n\n<li>A long tail of anonymous scrapers (14.9%)<\/li>\n\n\n\n<li>An industry auditing it (12.1%)<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">We\u2019ll dig into a couple of those&nbsp;below.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-198700-_582zaol1vxmz\"><\/a><div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"19.5% of requests come from AI bots \" data-section=\"requests-come\"> 19.5% of requests come from AI&nbsp;bots&nbsp;<\/div><\/div><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Of the requests that <em>do<\/em> reach llms.txt files, named AI bots account for 19.5%.<\/p>\n\n\n\n\n\n<p class=\"wp-block-paragraph\">While AI bots are the largest identifiable readership of llms.txt, the breakdown by AI <em>bot type<\/em> shows the file isn\u2019t serving the AI tools most people have in&nbsp;mind.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We group them four&nbsp;ways:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"font-claude-response-body whitespace-normal break-words pl-2\"><strong>AI agents &amp; agentic infrastructure<\/strong> that act on a user\u2019s behalf, or crawl to serve the agents that&nbsp;do.<\/li>\n\n\n\n<li><strong>AI training crawlers<\/strong> that collect data for model building<\/li>\n\n\n\n<li><strong>AI assistants<\/strong> that browse the web on behalf of a user in real&nbsp;time<\/li>\n\n\n\n<li><strong>AI retrieval bots<\/strong> that fetch pages to answer live user queries in AI platforms<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s how they size&nbsp;up\u2026<\/p>\n<div id=\"attachment_198743\" style=\"width: 3385px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-198743\" class=\"wp-image-198743 size-full\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_by_type-10.png\" alt=\"Bar chart showing AI bot requests. Agents (blue) 10.5%, Training crawlers (orange) 5.3%, Assistants (red) 2.5%, Retrieval bots (green) 1.1%. GPTBot is highest at 4.51%.\" width=\"3375\" height=\"7762\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_by_type-10.png 3375w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_by_type-10-185x425.png 185w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_by_type-10-768x1766.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_by_type-10-668x1536.png 668w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_by_type-10-890x2048.png 890w\" sizes=\"auto, (max-width: 3375px) 100vw, 3375px\"><p id=\"caption-attachment-198743\" class=\"wp-caption-text\">*statespace-indexer: operator identified as Statespace (agentic infrastructure), IP ranges unconfirmed.<\/p><\/div>\n\n\n\n\n\n<div class=\"sidenote\"><div class=\"sidenote-title\">Sidenote.<\/div> Quick reminder: This analysis covers the 3% of files that received any requests at all, not to the total 137K domains. That equates to roughly 1.1K domains and 22K requests in total\u2014so we\u2019re still only studying a tiny pool. Also, \u201cfetched\u201d doesn\u2019t mean \u201cread\u201d. Many bots may have fetched the llms.txt file without ever acting on what\u2019s inside. Every figure in this study is therefore a ceiling on actual llms.txt consumption. For instance, 19.5% of requests from AI is the most generous possible reading. Actual AI consumption is somewhere at or below&nbsp;this.<\/div>\n<h3 class=\"wp-block-heading\"><a id=\"post-198700-_ygikr07qtzr8\"><\/a>The agentic web is the real consumer, sending 10.5% of requests<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">AI agents, and the infrastructure built to serve them, drive 10.5% of llms.txt requests\u2014more than any other type of AI&nbsp;bot.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This finding lines up with a hunch that many in the industry already had.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We heard earlier from John Mueller that llms.txt works best as reference material for AI coding agents.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/www.linkedin.com\/feed\/update\/urn:li:activity:7463206622803202050\/\">Chris Long<\/a>, Founder of <a href=\"https:\/\/nectivdigital.com\/\">Nectiv<\/a>, has also stated that, even if llms.txt doesn\u2019t help you in Google search, the file has utility if your customers \u201care using Claude Code to source recommendations\u201d<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-198761 size-full\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/Chris-Long.png\" alt=\"LinkedIn post by Chris Long about LLMs.txt and its relevance to SEO beyond Google Search, with highlighted text.\" width=\"1634\" height=\"2728\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/Chris-Long.png 1634w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/Chris-Long-255x425.png 255w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/Chris-Long-768x1282.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/Chris-Long-920x1536.png 920w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/Chris-Long-1227x2048.png 1227w\" sizes=\"auto, (max-width: 1634px) 100vw, 1634px\"><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Our <a href=\"https:\/\/ahrefs.com\/bot-analytics\">Bot Analytics<\/a> data supports both&nbsp;ideas.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We see llms.txt files being fetched far less by the search and AI bots that are seemingly responsible for visibility, and far more by the agentic tools that seek out structured information and\/or act on a user\u2019s behalf.<\/p>\n<div id=\"attachment_198736\" style=\"width: 3385px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-198736\" class=\"wp-image-198736 size-full\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_agents-2.png\" alt=\"Bar chart showing the share of verified AI bot requests from various agents, totaling 10.5%. &quot;statespace-indexer&quot; leads with 3.52%.\" width=\"3375\" height=\"4652\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_agents-2.png 3375w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_agents-2-308x425.png 308w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_agents-2-768x1059.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_agents-2-1114x1536.png 1114w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_agents-2-1486x2048.png 1486w\" sizes=\"auto, (max-width: 3375px) 100vw, 3375px\"><p id=\"caption-attachment-198736\" class=\"wp-caption-text\">*statespace-indexer: operator identified as Statespace (agentic infrastructure), IP ranges unconfirmed.<\/p><\/div>\n\n\n\n\n\n<p class=\"wp-block-paragraph\">Aside from statespace-indexer and GPTBot, Claude-Code (Anthropic\u2019s coding agent), out-fetched every AI retrieval bot, every AI assistant, and every AI training crawler.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><a id=\"post-198700-_uhlv7dffb3j1\"><\/a>Training crawlers are the second-largest AI category at 5.3%<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Llms.txt files feed training corpora more than they feed AI search retrieval.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In fact, AI training crawlers fetch llms.txt nearly 5X more than AI retrieval bots.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-198742\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_training_crawlers-4.png\" alt=\"Bar chart showing 5.3% of AI bot requests come from AI training crawlers. GPTBot is 4.51%, ClaudeBot 0.8%.\" width=\"3375\" height=\"3302\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_training_crawlers-4.png 3375w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_training_crawlers-4-434x425.png 434w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_training_crawlers-4-768x751.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_training_crawlers-4-1536x1503.png 1536w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_training_crawlers-4-2048x2004.png 2048w\" sizes=\"auto, (max-width: 3375px) 100vw, 3375px\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So if llms.txt were to in any way impact your brand\u2019s AI visibility, it would likely be upstream\u2014not at the point of retrieval.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Of all training crawlers, GPTBot is far and away the biggest fetcher of llms.txt.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You won\u2019t find a Gemini crawler in this list, because it doesn\u2019t exist.<\/p>\n<p>Google trains and grounds Gemini on content fetched by regular Googlebot, and <a href=\"https:\/\/developers.google.com\/crawling\/docs\/crawlers-fetchers\/google-common-crawlers?sjid=18113437194454589947-EU#google-extended\">Google-Extended, the opt-out publishers use<\/a>, is a robots.txt token rather than a crawler with its own user&nbsp;agent.<\/p>\n<p>Googlebot did fetch llms.txt files ~900 times in May, but Googlebot routinely fetches any URL it discovers on a site as part of normal search indexing, so those fetches don\u2019t indicate special interest in llms.txt\u2014it\u2019s crawling the file the same way it crawls a sitemap or any other&nbsp;page.<\/p>\n<p>Whether any of that content then feeds Gemini is invisible to&nbsp;us.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><a id=\"post-198700-_w63t4g9h1jsr\"><\/a>AI retrieval bots barely register, with 1.1% of total requests<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">According to our data, AI retrieval bots account for just 1.1% of AI bot requests.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Even when taken together with AI assistants and AI training crawlers, these bots still count for only 8.9% of requests (1.6% less than AI agents).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">OAI-SearchBot, PerplexityBot, and Claude\u2019s search crawler combined made only a couple of hundred fetches across thousands of&nbsp;sites.<\/p>\n\n\n\n\n\n<p class=\"wp-block-paragraph\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-198738\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_retrieval_bots-2.png\" alt=\"Bar chart showing that 1.1% of AI bot requests come from AI retrieval bots. OAI-SearchBot leads with 0.74%.\" width=\"3375\" height=\"2857\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_retrieval_bots-2.png 3375w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_retrieval_bots-2-502x425.png 502w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_retrieval_bots-2-768x650.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_retrieval_bots-2-1536x1300.png 1536w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/bot_requests_retrieval_bots-2-2048x1734.png 2048w\" sizes=\"auto, (max-width: 3375px) 100vw, 3375px\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you are planning on generating an llms.txt in hopes of boosting your AI citations, you may want to think&nbsp;again.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-198700-_rpatpc841p0a\"><\/a><div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"12% of requests come from tools studying llms.txt, not consuming it\" data-section=\"requests-come\"> 12% of requests come from tools studying llms.txt, not consuming it&nbsp;<\/div><\/div><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A whole ecosystem has formed around auditing, scoring, validating, and studying the llms.txt standard, before we\u2019ve even established whether any major AI platform actually reads&nbsp;it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Three categories account for 12% of all requests combined.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1900\" height=\"1700\" class=\"wp-image-198712\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/pie-chart-showing-12-of-requests-study-the-llms-t.png\" alt=\"Pie chart showing 12% of requests study the llms.txt standard. Research bots: 2.7%, llms.txt discoverability: 3.6%, GEO\/AEO tools: 5.8%.\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/pie-chart-showing-12-of-requests-study-the-llms-t.png 1900w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/pie-chart-showing-12-of-requests-study-the-llms-t-475x425.png 475w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/pie-chart-showing-12-of-requests-study-the-llms-t-768x687.png 768w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/pie-chart-showing-12-of-requests-study-the-llms-t-1536x1374.png 1536w\" sizes=\"auto, (max-width: 1900px) 100vw, 1900px\"><\/figure>\n\n\n\n\n\n<h3 class=\"wp-block-heading\"><a id=\"post-198700-_5zzz8bp1o6up\"><\/a>GEO\/AEO tools send 5.8% of requests<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Commercial tools scan websites and score their readiness for AI search and agent discovery, with llms.txt presence as one of many signals.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The most active, CairrotReadinessBot, belongs to Cairrot, a WordPress-focused AEO platform launched in late&nbsp;2025.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Then you have the mainstream website builders like Framer, Lovable, and Wix all baking AI-readiness checks into their products.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Lms.txt adoption has become a platform default before it\u2019s even become a webmaster decision.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><a id=\"post-198700-_r05pje2mw2qb\"><\/a>llms.txt discoverability bots cover 3.6% of requests<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">There\u2019s an ecosystem of tools that catalog the llms.txt files that almost nobody else&nbsp;reads.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Dedicated scanners, validators, and directories built solely for llms.txt files send more requests than AI retrieval bots and AI assistants.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><a id=\"post-198700-_aziy1dlp871g\"><\/a>Research bots send 2.7% of requests<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The largest single research crawler in the dataset identifies itself as prompt-injection-survey\/1.0.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Someone is systematically studying llms.txt as a prompt injection opportunity that AI agents are designed to ingest and&nbsp;trust.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The security implications of agents trusting llms.txt files at scale have barely been discussed, and yet potential bad actors are already on the&nbsp;case.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-198700-_kolmxinefjkw\"><\/a><div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"Zero AI bots \u201cgo looking\u201d for llms.txt files that don't exist\" data-section=\"zero-bots\"> Zero AI bots \u201cgo looking\u201d for llms.txt files that don\u2019t&nbsp;exist&nbsp;<\/div><\/div><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">AI tools never go looking for llms.txt files that aren\u2019t there, so publishing one does not put you on any AI&nbsp;radar.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We analyzed every request to \/llms.txt paths that returned a 404 and found the cleanest split we\u2019ve seen in bot data: where on the one hand valid files drew 96% bot traffic, missing files drew 98% human traffic, and the <strong>AI bot<\/strong> share of those 404s was&nbsp;zero.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The people probing for absent llms.txt files are humans typing the URL into a browser, presumably SEOs checking on competitors.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This kills the assumption that AI systems actively hunt for llms.txt files, and that a site without one is missing a knock at the&nbsp;door.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">AI tools fetch llms.txt when a link, an index, or a user instruction tells them it exists.<\/p>\n<div class=\"recommendation\"><div class=\"recommendation-title\">How to check your own llms.txt bot traffic<\/div><div class=\"recommendation-content\">\n<p class=\"font-claude-response-body break-words whitespace-normal\">If you want to see which bots are actually hitting your <code class=\"bg-text-200\/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]\">llms.txt<\/code> file, head to <a href=\"https:\/\/ahrefs.com\/bot-analytics\">Ahrefs Bot Analytics<\/a>&nbsp;and add a filter for <strong>Page URL \u2192 Contains \u2192 llms.txt<\/strong>, then hit <strong>Apply<\/strong>.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-198781\" src=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/studying-llmstxt-fetches-in-bot-analytics.png\" alt=\"studying llmstxt fetches in Ahrefs bot analytics\" width=\"1097\" height=\"1054\" srcset=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/studying-llmstxt-fetches-in-bot-analytics.png 1097w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/studying-llmstxt-fetches-in-bot-analytics-442x425.png 442w, https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/studying-llmstxt-fetches-in-bot-analytics-768x738.png 768w\" sizes=\"auto, (max-width: 1097px) 100vw, 1097px\"><\/p>\n<p class=\"font-claude-response-body break-words whitespace-normal\">This narrows everything down to requests hitting your <code class=\"bg-text-200\/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]\">llms.txt<\/code> file (or any pages with \u201cllms.txt\u201d in the URL, like blog posts about&nbsp;it).<\/p>\n<p>We don\u2019t have an llms.txt file on the Ahrefs site but we are getting some bots hitting that page, as indicated by the 404 status.<\/p>\n<p class=\"font-claude-response-body break-words whitespace-normal\">From there, you can&nbsp;check:<\/p>\n<ul class=\"[li_&amp;]:mb-0 [li_&amp;]:mt-1 [li_&amp;]:gap-1 [&amp;:not(:last-child)_ul]:pb-1 [&amp;:not(:last-child)_ol]:pb-1 list-disc flex flex-col gap-1 pl-8 mb-3\">\n<li class=\"font-claude-response-body whitespace-normal break-words pl-2\"><strong>Visits over time.<\/strong> Toggle between <em>By bot<\/em> and <em>By category<\/em> to see whether traffic is climbing, flat, or spiking.&nbsp;<\/li>\n<li class=\"font-claude-response-body whitespace-normal break-words pl-2\"><strong>The Bots table.<\/strong> See <em>which<\/em> exact bots are fetching the&nbsp;file.<\/li>\n<li class=\"font-claude-response-body whitespace-normal break-words pl-2\"><strong>Last status in Crawled pages.<\/strong> Check the status code. A <code class=\"bg-text-200\/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]\">404<\/code> on <code class=\"bg-text-200\/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]\">\/llms.txt<\/code> means bots are <em>asking<\/em> for a file that isn\u2019t&nbsp;there.<\/li>\n<\/ul>\n<p class=\"font-claude-response-body break-words whitespace-normal\">You can also use the <strong>AI bots<\/strong> filter at top of the page to strip out other crawlers and see only the LLM-related ones.<\/p>\n<p class=\"font-claude-response-body break-words whitespace-normal\">And, remember, a bot requesting your <code class=\"bg-text-200\/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]\">llms.txt<\/code> isn\u2019t proof anything <em>read<\/em> or <em>acted on<\/em> it. It only tells you the file was fetched.<\/p>\n<\/div><\/div>\n\n\n\n<h2 class=\"wp-block-heading\"><a id=\"post-198700-_g50q97ut55a3\"><\/a><div class=\"post-nav-link clearfix\" id=\"section1\"><a class=\"subhead-anchor\" data-tip=\"tooltip__copielink\" rel=\"#section1\"><svg width=\"19\" height=\"19\" viewBox=\"0 0 14 14\" style><g fill=\"none\" fill-rule=\"evenodd\"><path d=\"M0 0h14v14H0z\" \/><path d=\"M7.45 9.887l-1.62 1.621c-.92.92-2.418.92-3.338 0a2.364 2.364 0 0 1 0-3.339l1.62-1.62-1.273-1.272-1.62 1.62a4.161 4.161 0 1 0 5.885 5.884l1.62-1.62L7.45 9.886zM5.527 5.135L7.17 3.492c.92-.92 2.418-.92 3.339 0 .92.92.92 2.418 0 3.339L8.866 8.473l1.272 1.273 1.644-1.643A4.161 4.161 0 1 0 5.897 2.22L4.254 3.863l1.272 1.272zm-.66 3.998a.749.749 0 0 1 0-1.06l2.208-2.206a.749.749 0 1 1 1.06 1.06L5.928 9.133a.75.75 0 0 1-1.061 0z\" style \/><\/g><\/svg><\/a><div class=\"link-text\" data-anchor=\"So, should you create an llms.txt file?\" data-section=\"should-create\"> So, should you create an llms.txt file?&nbsp;<\/div><\/div><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">If your goal is showing up in ChatGPT, Perplexity, or AI Overviews, an llms.txt file is largely decoration.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">AI search bots barely fetch them, no AI system goes looking for them, and 97% of existing files attract no readers of any&nbsp;kind.<\/p>\n<p>And remember that <strong>requests<\/strong> are the generous measure. Whether bots act on what they fetch is another question<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here are the pros and cons, side-by-side.<\/p>\n\n\n\n<p><style>\n.pc-table { width:100%; border-collapse:collapse; font-family: -apple-system, BlinkMacSystemFont, \"Segoe UI\", Roboto, Arial, sans-serif; font-size:15px; line-height:1.6; table-layout:fixed; }\n.pc-table thead th { text-align:left; padding:14px 16px; border-bottom:1px solid #d4d4d4; }\n.pc-table tbody tr { border-bottom:0.5px solid #e8e8e8; }\n.pc-table tbody tr:last-child { border-bottom:none; }\n.pc-table td { padding:18px 16px; vertical-align:top; width:50%; font-size:14px; color:#1a1a1a; line-height:1.55; }\n.pc-badge { display:inline-block; padding:3px 12px; border-radius:999px; font-size:12px; font-weight:500; letter-spacing:0.06em; white-space:nowrap; }\n.pc-pros { background:#E1F5EE; color:#085041; }\n.pc-cons { background:#FAECE7; color:#993C1D; }\n.pc-lead { font-weight:500; }\n<\/style><\/p>\n<table class=\"pc-table\">\n<thead>\n<tr>\n<th><span class=\"pc-badge pc-pros\">PROS<\/span><\/th>\n<th><span class=\"pc-badge pc-cons\">CONS<\/span><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><span class=\"pc-lead\">Publishing llms.txt is cheap,<\/span> and platforms like Wix will increasingly do it for&nbsp;you.<\/td>\n<td><span class=\"pc-lead\">The base rate is brutal:<\/span> 97% of existing llms.txt files attract no readers of any&nbsp;kind.<\/td>\n<\/tr>\n<tr>\n<td><span class=\"pc-lead\">The closest thing to an intended audience in our data is coding agents.<\/span> If your customers use coding agents, or if agents act on your site, the file stands a real chance of being&nbsp;read.<\/td>\n<td><span class=\"pc-lead\">It won\u2019t help your AI search visibility today.<\/span> AI retrieval bots barely fetch these files, and no AI system goes looking for one you haven\u2019t published.<\/td>\n<\/tr>\n<tr>\n<td><span class=\"pc-lead\">It may futureproof your strategy.<\/span> Google has made it clear that the future of search is agentic. If agents end up mediating AI search, rather than retrieval bots fetching pages directly, llms.txt could start influencing AI visibility through the agent&nbsp;layer.<\/td>\n<td><span class=\"pc-lead\">Publishing is only half the job.<\/span> Agents fetch llms.txt when directed, not speculatively, so an unlinked file is unlikely to get picked up.<\/td>\n<\/tr>\n<tr>\n<td>&nbsp;<\/td>\n<td><span class=\"pc-lead\">It\u2019s a security risk.<\/span> Agents are built to trust this file, and potential bad actors are already probing llms.txt for prompt injection. A stale or compromised file misleads every agent that reads&nbsp;it.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n<p class=\"wp-block-paragraph\">My verdict: the cons outweigh the pros right now. If you want to show up in <a href=\"https:\/\/ahrefs.com\/blog\/category\/ai-search\/\">AI search<\/a>, there are more reliable ways to improve your visibility than this&nbsp;file.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But if you\u2019re still toying with the idea of generating llms.txt, here are the steps you should take:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Check your own logs before investing further.<\/strong> A 97% chance of zero readership is the base&nbsp;rate.<\/li>\n\n\n\n\n\n<li><strong>Get a website-building platform to do it for you.<\/strong> Wix already generates these files, and Framer and Lovable are scanning for them. Within a year, having an llms.txt may be as much a CMS default as having a sitemap. If the payoff is uncertain, it makes sense to keep the effort minimal.<\/li>\n\n\n\n<li><strong>Route agents to it.<\/strong> Link the file from your HTML, reference it in your docs, or mention it anywhere agents receive instructions about your site. Agents fetch llms.txt when directed, not speculatively.<\/li>\n\n\n\n<li><strong>Offset the prompt injection risk by treating llms.txt like code.<\/strong> Version-control it, restrict who can edit it, set an alert for unauthorized changes, keep the content to plain links and descriptions (nothing instruction-shaped), only link to resources you control, and review anything a platform auto-generates on your behalf.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">This study answers how many sites publish llms.txt, and who reads&nbsp;it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But there are a couple of other questions worthy of further research that were beyond the scope of this&nbsp;study:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Do agents fetch developer-docs more often?<\/strong> Is Claude-Code\u2019s llms.txt interest concentrated on documentation paths like \/docs\/ and \/api\/, as Mueller\u2019s framing predicts?<\/li>\n\n\n\n<li><strong>Do bots actually <em>act<\/em> on what they read?<\/strong> When an AI agent fetches llms.txt, does it then fetch the resources the file links to? SEO consultant <a href=\"https:\/\/uk.linkedin.com\/in\/david-mcsweeney-79840154\">David McSweeney<\/a>, Founder of <a href=\"https:\/\/queryburst.com\/\">Queryburst<\/a>, is already running an experiment along these lines: he\u2019s serving AI user agents a compressed, agent-friendly summary of his test sites, complete with instructions for requesting deeper content, and tracking whether any agent actually follows through. His results are <a href=\"https:\/\/www.linkedin.com\/feed\/update\/urn:li:activity:7470130850219663361\/\">worth following<\/a>.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Mueller called llms.txt a temporary crutch.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But that crutch seems to already have its own supply chain: platforms generating llms.txt files, an industry auditing them, and security researchers studying them, all before the \u201creaders\u201d actually showed up.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Either we\u2019re watching the early scaffolding of a real standard, or we\u2019re watching the SEO industry prove it can productize anything. Our money is on a bit of&nbsp;both.<\/p>\n\n\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Using Ahrefs Web Analytics and Bot Analytics, we analyzed the server logs and live traffic of 137K domains, plus the user agents hitting all of&nbsp;them. Here\u2019s what we&nbsp;found. In late May 2026, Google took both sides of the llms.txt argument<span class=\"ellipsis\">\u2026<\/span><\/p>\n<div class=\"read-more\">Read more \u203a<\/div>\n<p><!-- end of .read-more --><\/p>\n","protected":false},"author":197,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"wp_typography_post_enhancements_disabled":false,"footnotes":""},"categories":[469,414],"tags":[],"coauthors":[464,467],"class_list":["post-198700","post","type-post","status-publish","format-standard","hentry","category-ai-search","category-data-studies","odd"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.7 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>97% of llms.txt Files Never Get Read (137,000 Sites Analyzed)<\/title>\n<meta name=\"description\" content=\"We analyzed 137K sites &amp; found 97% of llms.txt files are never read. See why bots ignore these files &amp; what this means for your AI visibility\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ahrefs.com\/blog\/llmstxt-study\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"97% of llms.txt Files Never Get Read (137,000 Sites Analyzed)\" \/>\n<meta property=\"og:description\" content=\"We analyzed 137K sites &amp; found 97% of llms.txt files are never read. See why bots ignore these files &amp; what this means for your AI visibility\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ahrefs.com\/blog\/llmstxt-study\/\" \/>\n<meta property=\"og:site_name\" content=\"SEO Blog by Ahrefs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Ahrefs\/\" \/>\n<meta property=\"article:published_time\" content=\"2026-06-15T10:27:40+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-06-15T10:45:27+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-text-excerpt-from-mythbusting-generative-ai-sea.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2048\" \/>\n\t<meta property=\"og:image:height\" content=\"865\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Louise Linehan, Xibeijia Guan\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@ahrefs\" \/>\n<meta name=\"twitter:site\" content=\"@ahrefs\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/llmstxt-study\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/llmstxt-study\\\/\"},\"author\":{\"name\":\"Louise Linehan\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/444b3643c35b16b94b763446c5562388\"},\"headline\":\"97% of llms.txt Files Never Get Read (137,000 Sites Analyzed)\",\"datePublished\":\"2026-06-15T10:27:40+00:00\",\"dateModified\":\"2026-06-15T10:45:27+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/llmstxt-study\\\/\"},\"wordCount\":3578,\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/llmstxt-study\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/06\\\/97-of-llms.txt-files-never-get-by-louise-linehan-data-studies.jpg\",\"articleSection\":[\"AI Search\",\"Data &amp; Studies\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/llmstxt-study\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/llmstxt-study\\\/\",\"name\":\"97% of llms.txt Files Never Get Read (137,000 Sites Analyzed)\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/llmstxt-study\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/llmstxt-study\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/06\\\/a-text-excerpt-from-mythbusting-generative-ai-sea.jpg\",\"datePublished\":\"2026-06-15T10:27:40+00:00\",\"dateModified\":\"2026-06-15T10:45:27+00:00\",\"description\":\"We analyzed 137K sites & found 97% of llms.txt files are never read. See why bots ignore these files & what this means for your AI visibility\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ahrefs.com\\\/blog\\\/llmstxt-study\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/llmstxt-study\\\/#primaryimage\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/06\\\/a-text-excerpt-from-mythbusting-generative-ai-sea.jpg\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/06\\\/a-text-excerpt-from-mythbusting-generative-ai-sea.jpg\",\"width\":2048,\"height\":865,\"caption\":\"A text excerpt from \\\"Mythbusting generative AI search: what you don't need to do.\\\" Highlighted text states you don't need special files or markup for generative AI search.\"},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"name\":\"SEO Blog by Ahrefs\",\"description\":\"Link Building Strategies &amp; SEO Tips\",\"publisher\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#organization\",\"name\":\"Ahrefs\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/06\\\/ahrefs-logo.png\",\"width\":2048,\"height\":768,\"caption\":\"Ahrefs\"},\"image\":{\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/Ahrefs\\\/\",\"https:\\\/\\\/x.com\\\/ahrefs\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/ahrefs\\\/\",\"https:\\\/\\\/www.youtube.com\\\/c\\\/ahrefscom\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/#\\\/schema\\\/person\\\/444b3643c35b16b94b763446c5562388\",\"name\":\"Louise Linehan\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2024\\\/08\\\/Louise-Linehan.jpg02b05bbed9b25ec9b04e39f0d88f15b0\",\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2024\\\/08\\\/Louise-Linehan.jpg\",\"contentUrl\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/wp-content\\\/uploads\\\/2024\\\/08\\\/Louise-Linehan.jpg\",\"caption\":\"Louise Linehan\"},\"description\":\"Louise is a Content Marketer at Ahrefs. Over the past ten years, she has held senior content positions at SaaS brands: Pi Datametrics, BuzzSumo, and Cision. By day, she writes about content and SEO; by night, you'll find her playing football or screaming down the mic at karaoke.\",\"sameAs\":[\"https:\\\/\\\/www.linkedin.com\\\/in\\\/louise-linehan\\\/\"],\"url\":\"https:\\\/\\\/ahrefs.com\\\/blog\\\/author\\\/louise-linehan\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"97% of llms.txt Files Never Get Read (137,000 Sites Analyzed)","description":"We analyzed 137K sites & found 97% of llms.txt files are never read. See why bots ignore these files & what this means for your AI visibility","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ahrefs.com\/blog\/llmstxt-study\/","og_locale":"en_US","og_type":"article","og_title":"97% of llms.txt Files Never Get Read (137,000 Sites Analyzed)","og_description":"We analyzed 137K sites & found 97% of llms.txt files are never read. See why bots ignore these files & what this means for your AI visibility","og_url":"https:\/\/ahrefs.com\/blog\/llmstxt-study\/","og_site_name":"SEO Blog by Ahrefs","article_publisher":"https:\/\/www.facebook.com\/Ahrefs\/","article_published_time":"2026-06-15T10:27:40+00:00","article_modified_time":"2026-06-15T10:45:27+00:00","og_image":[{"width":2048,"height":865,"url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-text-excerpt-from-mythbusting-generative-ai-sea.jpg","type":"image\/jpeg"}],"author":"Louise Linehan, Xibeijia Guan","twitter_card":"summary_large_image","twitter_creator":"@ahrefs","twitter_site":"@ahrefs","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ahrefs.com\/blog\/llmstxt-study\/#article","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/llmstxt-study\/"},"author":{"name":"Louise Linehan","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/444b3643c35b16b94b763446c5562388"},"headline":"97% of llms.txt Files Never Get Read (137,000 Sites Analyzed)","datePublished":"2026-06-15T10:27:40+00:00","dateModified":"2026-06-15T10:45:27+00:00","mainEntityOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/llmstxt-study\/"},"wordCount":3578,"publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/llmstxt-study\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/97-of-llms.txt-files-never-get-by-louise-linehan-data-studies.jpg","articleSection":["AI Search","Data &amp; Studies"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/ahrefs.com\/blog\/llmstxt-study\/","url":"https:\/\/ahrefs.com\/blog\/llmstxt-study\/","name":"97% of llms.txt Files Never Get Read (137,000 Sites Analyzed)","isPartOf":{"@id":"https:\/\/ahrefs.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ahrefs.com\/blog\/llmstxt-study\/#primaryimage"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/llmstxt-study\/#primaryimage"},"thumbnailUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-text-excerpt-from-mythbusting-generative-ai-sea.jpg","datePublished":"2026-06-15T10:27:40+00:00","dateModified":"2026-06-15T10:45:27+00:00","description":"We analyzed 137K sites & found 97% of llms.txt files are never read. See why bots ignore these files & what this means for your AI visibility","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ahrefs.com\/blog\/llmstxt-study\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/llmstxt-study\/#primaryimage","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-text-excerpt-from-mythbusting-generative-ai-sea.jpg","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2026\/06\/a-text-excerpt-from-mythbusting-generative-ai-sea.jpg","width":2048,"height":865,"caption":"A text excerpt from \"Mythbusting generative AI search: what you don't need to do.\" Highlighted text states you don't need special files or markup for generative AI search."},{"@type":"WebSite","@id":"https:\/\/ahrefs.com\/blog\/#website","url":"https:\/\/ahrefs.com\/blog\/","name":"SEO Blog by Ahrefs","description":"Link Building Strategies &amp; SEO Tips","publisher":{"@id":"https:\/\/ahrefs.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ahrefs.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ahrefs.com\/blog\/#organization","name":"Ahrefs","url":"https:\/\/ahrefs.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2023\/06\/ahrefs-logo.png","width":2048,"height":768,"caption":"Ahrefs"},"image":{"@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Ahrefs\/","https:\/\/x.com\/ahrefs","https:\/\/www.linkedin.com\/company\/ahrefs\/","https:\/\/www.youtube.com\/c\/ahrefscom"]},{"@type":"Person","@id":"https:\/\/ahrefs.com\/blog\/#\/schema\/person\/444b3643c35b16b94b763446c5562388","name":"Louise Linehan","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/Louise-Linehan.jpg02b05bbed9b25ec9b04e39f0d88f15b0","url":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/Louise-Linehan.jpg","contentUrl":"https:\/\/ahrefs.com\/blog\/wp-content\/uploads\/2024\/08\/Louise-Linehan.jpg","caption":"Louise Linehan"},"description":"Louise is a Content Marketer at Ahrefs. Over the past ten years, she has held senior content positions at SaaS brands: Pi Datametrics, BuzzSumo, and Cision. By day, she writes about content and SEO; by night, you'll find her playing football or screaming down the mic at karaoke.","sameAs":["https:\/\/www.linkedin.com\/in\/louise-linehan\/"],"url":"https:\/\/ahrefs.com\/blog\/author\/louise-linehan\/"}]}},"as_json":null,"as_tables":null,"as_images":null,"json_reviewers":[194],"as_post_info":null,"as_sticky":null,"_links":{"self":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/198700","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/users\/197"}],"replies":[{"embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/comments?post=198700"}],"version-history":[{"count":3,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/198700\/revisions"}],"predecessor-version":[{"id":198814,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/posts\/198700\/revisions\/198814"}],"wp:attachment":[{"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/media?parent=198700"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/categories?post=198700"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/tags?post=198700"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/ahrefs.com\/blog\/wp-json\/wp\/v2\/coauthors?post=198700"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}