llms.txt is a proposed web standard: a plain Markdown file served at your domain root (yourdomain.com/llms.txt) that gives large language models a curated, token-efficient index of your most important pages. Where robots.txt sets permissions and sitemap.xml lists inventory, llms.txt supplies context β a one-line description of what your site is, followed by an annotated reading list an AI agent can parse in milliseconds. Proposed by Jeremy Howard of Answer.AI in September 2024, it remains a community convention in 2026: widely published, but not yet officially consumed by any major AI provider.
Key takeaways
llms.txtis a curated Markdown index at your site root. The spec at llmstxt.org requires only an H1 with your site name; the summary blockquote and annotated link sections are recommended, not mandatory.- It answers a different question than
robots.txt(may you crawl this?) orsitemap.xml(what exists here?): it tells a model what to read first, and why. - The honest 2026 status: published files grew 8.8x in twelve months, yet 97% of them receive zero requests from AI bots, and Google says it does not use the file at all.
- The realistic upside is agentic retrieval. Assistants and shopping agents that fetch your site live during a conversation benefit from a clean Markdown map instead of heavy HTML.
- It costs about an hour to implement. Treat it as one low-cost line in a broader GEO checklist, not a visibility strategy on its own.
How llms.txt works
The format, defined at llmstxt.org, is deliberately minimal. A valid file contains, in order:
- An H1 with the site or project name β the only required element, e.g.
# GEOly AI. - A blockquote summary directly below it:
> GEO data platform for DTC brands, covering seven AI engines. - Optional paragraphs of free-form context the model should know before reading further.
- H2 sections grouping annotated links, one per line:
- [Quick start](/docs/quick-start): Install and run your first audit. - An optional section literally titled
Optional, for links an agent may skip when its context budget runs short.
Two companion conventions travel with it. llms-full.txt inlines the complete text of your key pages into a single file, so an agent gets everything in one fetch. The spec also suggests serving Markdown twins of important pages by appending .md to their URLs.
The case for Markdown is arithmetic. A typical e-commerce page delivers hundreds of kilobytes of navigation, scripts, and cookie banners wrapped around a few paragraphs of substance. Language models pay for every token they ingest, so a curated plain-text index is the difference between an agent grounding itself in your actual positioning and drowning in your header menu.
llms.txt vs robots.txt vs sitemap.xml
The three root files serve different audiences, and they get confused constantly.
robots.txtspeaks to crawlers about permission: which paths GPTBot, ClaudeBot, or Googlebot may fetch. It is the enforcement layer, and unlikellms.txt, major AI crawlers demonstrably respect it.sitemap.xmlspeaks to indexers about inventory: every URL you want discovered, with no sense of priority or narrative.llms.txtspeaks to language models about meaning: a curated shortlist, usually ten to fifty links, explaining what you sell, what you stand for, and where the authoritative answers live.
None replaces the others β and they need to agree. A store that blocks GPTBot in robots.txt while publishing an llms.txt is sending contradictory signals to the same company.
Who actually reads llms.txt in 2026?
This is where most guides oversell, so here are the observable facts. Google has said plainly that it does not use llms.txt for Search or its AI features; John Mueller compared it to the keywords meta tag β self-declared, easy to game, therefore ignored. Neither OpenAI nor Anthropic has committed to fetching it for training or indexing. And a ppc.land analysis of over three million monitored sites found that while published files grew from about 4,000 to 36,000 in a year, 97% of them received no AI-bot requests at all.
So why bother? Because crawling and retrieval are different behaviors. Training crawlers ignore the file today. But when an agent visits your site live β an answer engine grounding a response mid-conversation, or a shopping agent comparing products in an agentic commerce flow β it reads whatever helps it finish the task cheaply, and a curated Markdown index is exactly that. The file is a hedge on agent traffic, not a lever on today's rankings. At an hour of work, the hedge is cheap.

How to implement llms.txt properly
- Draft the file in plain Markdown: H1 name, blockquote summary, then two to four H2 sections β documentation, products or collections, policies (shipping, returns, warranty), company.
- Curate ruthlessly. Ten to fifty links with one-line descriptions. If you are pasting your sitemap, you have missed the point.
- Serve it at
/llms.txtwith a plain-text content type, and confirm yourrobots.txtand CDN rules allow AI user agents to fetch it. - Keep it current. Regenerate when products launch or pricing changes. Shopify merchants can automate this: GEOly's LLMs.txt Master app scans the store for high-performing product and policy pages and regenerates the file as the catalog changes.
- Verify consumption directly: filter server logs for
/llms.txtrequests from GPTBot, ClaudeBot, and PerplexityBot. - Measure outcomes, not the artifact. Machine readability β including
llms.txtpresence, structured data, and crawler access β is one section of GEOly's 29-point GEO audit. The numbers that tell you whether any of it worked are citation rate, mention rate, and your AIGVR score trending across engines; the full metrics stack is here.

Common mistakes
- Dumping every URL into the file. That is a second sitemap; the entire value is editorial judgment about what matters.
- Letting it rot. An
llms.txtpointing at discontinued products or old pricing is worse than none β you are feeding agents stale facts with high confidence. - Expecting a ranking lift. There is no evidence the file moves Google results or AI Overviews, and anyone claiming otherwise is selling something.
- Contradicting your own signals by publishing the file while blocking the same companies' crawlers.
- Treating it as the whole job. Structured data and content worth citing carry far more observable weight in what AI engines actually say about you.
FAQ
Does llms.txt improve my rankings in Google or ChatGPT?
There is no evidence it does. Google explicitly does not use it, and no LLM provider has confirmed reading it for indexing. Its plausible value is making live agent visits cheaper and more accurate, which can indirectly support mentions and citations β so measure those, not the file.
What is the difference between llms.txt and llms-full.txt?
llms.txt is the index: links plus one-line descriptions. llms-full.txt inlines the full content of those pages into one large file, trading a single convenient fetch against a size that can blow an agent's context budget. For most DTC sites, the index alone is the right starting point.
Do I still need robots.txt and sitemap.xml if I publish llms.txt?
Yes. They do three different jobs β permissions, inventory, and context β and robots.txt is the only one of the three that AI companies demonstrably honor today. Keep all three consistent with each other.
How do I know if AI bots are actually reading my llms.txt?
Filter your server or CDN logs for requests to /llms.txt by AI user agents. Most sites will find few or none, which matches the industry-wide data. The more meaningful signal is downstream: whether engines mention and cite your domain more over time, which is what per-engine monitoring exists to track.

