llms.txt Explained: What It Is, and Whether You Actually Need One

Every few months, a new file promises to get your site into ChatGPT's answers. Right now that file is llms.txt. The pitch is simple: drop a markdown file at the root of your site and the AI engines will understand you better and cite you more. It sounds plausible, adoption is climbing, and plenty of plugins and agencies now sell it as an AI-SEO must-have.

So does it actually work? The honest answer, backed by the current evidence: llms.txt is real, cheap to add, and genuinely useful for one specific audience, but it does close to nothing for the thing most people add it for, which is getting cited by ChatGPT or Google's AI Overviews. This guide explains what it is, how to make one, and when it is actually worth your time.

What llms.txt actually is

llms.txt is a proposed convention: a single markdown file at the root of your site (/llms.txt) that hands large language models a clean, curated version of your most important content. It was proposed by Jeremy Howard (of Answer.AI and fast.ai) in September 2024.

The problem it tries to solve is real. An LLM has a limited context window and cannot read your entire site. Your HTML is also full of navigation, scripts, and markup that waste tokens and bury the actual content. llms.txt is meant to give the model a concise markdown map of what matters, so it can use your content at inference time without crawling and parsing everything.

One thing it is not: robots.txt. Those two files are opposites. robots.txt controls whether bots may access your pages. llms.txt does the reverse, inviting models in and handing them a curated summary. One is a gate, the other is a guidebook.

llms.txt vs llms-full.txt vs robots.txt

Three files get confused constantly. Here is the difference:

Think of it this way: llms.txt is the table of contents, llms-full.txt is the whole book in one clean file. Note that only llms.txt is defined by the core spec; llms-full.txt is a companion convention that grew around it.

What an llms.txt file looks like

The format is deliberately simple markdown, in this order: an H1 with your name (the only required part), a blockquote summary, optional context, then H2 sections listing links as [Title](URL): description. A short example:

# Sublim Analytics
> Privacy-first, cookieless web analytics for teams that want complete data without a consent banner.

Sublim measures 100% of your traffic, includes behavioral analytics, and is hosted in the EU.

## Docs
- [Getting started](https://sublimanalytics.com/docs/start): install the script in one line
- [Tracking events](https://sublimanalytics.com/docs/events): measure goals and conversions

## Guides
- [Web analytics for marketers](https://sublimanalytics.com/blog/guides/web-analytics-for-marketers): what to measure and why

You can write it by hand, generate it from your CMS (SEO plugins like Yoast can output one), or run your site through a generator or crawler tool. It is a static file, so there is no ongoing maintenance beyond keeping the links current.

Does it actually work?

Here is the part most guides skip, because it undercuts the pitch. If your goal is to get cited more often by ChatGPT, Perplexity, or Google's AI Overviews, the current evidence says llms.txt does not move the needle.

• Adoption is thin. A study by SE Ranking across 300,000 domains found only about 10% had an llms.txt file, more than a year after the idea launched. The same analysis found no correlation between having the file and being cited by AI. Removing it from their model actually improved predictions, meaning the file added noise, not signal.
• The major AI engines do not use it. Google's Gary Illyes confirmed that Google Search does not support llms.txt and is not pursuing it. John Mueller compared it to the old keywords meta tag: no AI service has said it uses llms.txt, and server logs show the bots do not even request the file.
• The reason is structural. llms.txt is a signal you write about yourself and fully control. Search engines learned long ago that self-declared signals are trivial to game and therefore worthless for ranking. That is exactly why the keywords meta tag died. As Mueller put it, why would a model trust what a site claims about itself when it can just read the site directly?

So when a plugin or agency tells you adding llms.txt will get you into AI answers, treat that as a claim, not a fact.

Where llms.txt genuinely helps: the agentic web

There is a real use case, just not the hyped one. The tools that actually read /llms.txt and /llms-full.txt today are AI coding agents and developer tools: Cursor, Claude Code, GitHub Copilot, Windsurf, Cline, and MCP-based integrations. Point one of these at a documentation site and it will look for llms.txt to load clean context fast, instead of scraping rendered HTML.

That makes llms.txt genuinely worth it if you publish:

• Developer documentation or an API reference.
• A technical product that people use from inside coding assistants.
• Anything meant to be consumed by AI agents, not just human readers.

For a docs site, llms.txt is a small, high-leverage addition. For a marketing blog hoping to rank in ChatGPT, it is close to a no-op. (Worth noting: Chrome's Lighthouse recently added an experimental "agentic" audit that checks for llms.txt, so the convention is not dead, it is just aimed at agents rather than search.)

So should your site have one?

A simple way to decide:

• You have developer docs, an API, or an agent-facing product: yes. It is cheap, and the agent ecosystem actually uses it.
• You run a marketing or content site and want AI citations: it will not hurt (it is a static file), but do not expect traffic from it, and do not prioritize it over real content work.
• You are short on time: skip it and spend the hour on something that moves the needle.

The honest framing: llms.txt is low cost, low risk, occasionally useful, and heavily oversold. Add it if it fits your case, but set your expectations from the evidence, not the hype.

What actually gets you cited by AI (do this instead)

If the real goal is showing up in AI answers, the levers are the same ones that have always earned citations: genuinely useful content, a clear structure the model can parse, and being referenced by other credible sources. AI engines read your actual pages, so the work lives on the pages, not in a manifest you control.

And there is a step almost everyone skips: measure whether any of it works. AI referral traffic is small but converts well, and most of it hides in "Direct" because AI tools strip the referrer. If you cannot see your AI traffic, you cannot tell whether your GEO efforts (llms.txt included) do anything at all. Our guide on how to track AI traffic from ChatGPT, Perplexity, and Gemini covers exactly how to surface it.

How Sublim helps

Sublim classifies visits from AI assistants as their own acquisition source instead of dumping them into Direct, and it captures the full referrer wherever it exists. Because it is cookieless, it also counts the visitors a consent-gated tool loses, and it never samples away a channel as small as AI referrals. So when you experiment with GEO tactics, you can actually measure whether they bring visits and whether those visitors convert. That turns "should I add llms.txt?" from a guess into a number you can watch.

The bottom line

llms.txt is a clean idea with a narrow reality. It is a curated markdown file that helps AI agents and documentation tools load your content, and for docs and developer products that is a real, if modest, benefit. What it is not is a shortcut to getting cited by ChatGPT or Google's AI Overviews: the major engines do not use it, adoption is thin, and self-declared signals do not earn trust.

Add it if you have docs or an agent audience. Otherwise, put the effort into content worth citing, and measure your AI traffic so you actually know what is working.