We Spent 5 Minutes Adding llms.txt to Our Site. Here Is What We Know So Far.

Everyone in the content and SEO world is suddenly talking about llms.txt. Some say it is the robots.txt of the AI era. Others say it is completely pointless. At Scribble, we did not want to take either side without trying it first. So we set it up ourselves. It took 5 minutes. Here is an honest look at what llms.txt is, what the current evidence shows, and why we still think you should add it, even though we cannot promise it will do anything yet.

What is llms.txt?

llms.txt is a plain text file, written in Markdown, that sits at the root of your website (for example, yoursite.com/llms.txt). It summarises what your site is about and links to your most important pages, all in a format that is easy for AI tools to read without crawling your entire site.

Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed the format in 2024. His original goal was practical: instead of making an AI agent wade through your whole website to understand what you do, you hand it a clean, structured summary upfront. Almost like a welcome note you leave at the front door.

A few things llms.txt is not:

The "AI visibility" angle arrived later, pushed by the SEO and marketing industry as the format spread. No major AI platform has officially confirmed they factor it into answers. But that is not quite the same as saying it is worthless.

What does the data say?

In June 2026, Ahrefs analysed server logs across 137,000 domains and measured every request hitting llms.txt files. The headline number got a lot of attention: 97% of llms.txt files received zero traffic in May 2026. Of the 38,000 domains that published a valid file, only around 1,100 saw any requests at all.

There is another finding worth sitting with. When Ahrefs checked requests to llms.txt paths that returned a 404 error (meaning no file existed there), the AI bot share was zero. AI tools do not proactively hunt for the file. They fetch it when something tells them it exists.

Google's guidance adds more context. In late May 2026, Google published an AI optimisation guide that stated machine-readable files like llms.txt are not required to appear in generative AI search results. When SEO professional Lily Ray raised this with Google's John Mueller, he clarified that llms.txt is "not done for search" and described it as a shortcut for AI coding tools parsing developer documentation.

So yes, the data is sobering.

However, the fact that 97% of files get no traffic does not mean the file is useless. It could mean the ecosystem has not caught up yet. The format is two years old. The tooling is still early. AI search itself is still early. Judging llms.txt today is a bit like judging XML sitemaps in 2001 before Google formally supported them.

Who is actually reading it?

Of the small fraction of files that do get any traffic, the breakdown by bot type is genuinely interesting.

Source: Ahrefs analysis of 137K domains, June 2026

AI retrieval bots like PerplexityBot and OAI-SearchBot make up just 1.1% of requests. That is the disappointing part for brands hoping for immediate citation benefits. But look at the AI agents and agentic tools category at 10.5%. That segment includes tools like Anthropic's Claude Code, which outfetched every AI retrieval bot and every AI assistant bot combined. As agentic AI tools become more common, files like llms.txt are exactly what they will look for first.

This is the future Scribble is betting on, not the present.

One genuine risk worth flagging: research bots probing llms.txt files for prompt injection vulnerabilities already exist in the wild. Because AI agents are designed to trust this file, a poorly maintained or outdated llms.txt can mislead every agent that reads it. Keep the file simple and under version control.

So should you add it?

Scribble's answer is yes, with the right expectations.

We added ours in 5 minutes. If your CMS or site platform generates it automatically (Wix and Framer already do), it takes even less. The upside is uncertain but the downside is almost nothing.

Here is how the tradeoffs actually look:

The key shift in mindset is this: do not add llms.txt because you expect it to drive AI citations today. Add it because you want to be ready if and when it does. If AI platforms start using llms.txt to log crawl frequency and track what actions agents take after reading your content, the brands who already have clean, maintained files will have a head start.

That future is not guaranteed. But it costs almost nothing to prepare for it.

If you do publish one, a few quick rules: link to it from your site so discovery bots can find it, keep descriptions factual and plain rather than instruction-shaped, and treat it like code by versioning it and restricting who can edit it.

Here is Scribble’s llms.txt file for reference: https://scribble.network/llms.txt

What else builds AI visibility right now

While llms.txt catches up to its promise, here is what Scribble knows actually moves the needle for AI visibility today.

• Earn mentions on the sources AI tools pull from. AI retrieval bots fetch live web pages, not index files. Getting your brand referenced on high-authority, well-linked pages is what drives citations in ChatGPT and Perplexity.

• Write content that directly answers specific questions. AI tools retrieve pages that match what a user is asking. A well-structured FAQ or a clear product comparison will earn more AI citations than any technical file you publish at your root.

• Keep your site crawlable. Googlebot, GPTBot, ClaudeBot, and PerplexityBot all need access to your pages. Blocking them in robots.txt removes you from both training data and live retrieval indexes.

• Get named in trusted third-party editorial sources. Your brand name appearing in review sites, industry publications, and editorial content carries more weight with AI tools than any on-site optimisation move you make.

If you are not sure how visible your brand is in AI search today, start by Checking your GEO Score to see how you're performing across leading AI platforms, identify the queries and competitors that matter most, and get actionable recommendations to improve your AI visibility.