# What is llms.txt? An honest guide to the AI-readable web standard (2026)

- Canonical: https://www.geoly.ai/blog/llms-txt
- Published: 2026-07-05
- Updated: 2026-07-05
- Author: [GEOly AI](https://www.geoly.ai/blog/author/geoly-ai)
- Tags: [Glossary](https://www.geoly.ai/blog/tag/glossary), [GEO](https://www.geoly.ai/blog/tag/geo), [AEO](https://www.geoly.ai/blog/tag/aeo)

> **Summary:** llms.txt is a proposed web standard — a curated Markdown index at your domain root that tells AI models which pages matter most; it takes an hour to ship, has no proven ranking weight in 2026, but hedges your brand for the era of live-browsing AI agents.

`llms.txt` is a proposed web standard: a plain Markdown file served at your domain root (`yourdomain.com/llms.txt`) that gives large language models a curated, token-efficient index of your most important pages. Where `robots.txt` sets permissions and `sitemap.xml` lists inventory, `llms.txt` supplies context — a one-line description of what your site is, followed by an annotated reading list an AI agent can parse in milliseconds. Proposed by Jeremy Howard of Answer.AI in September 2024, it remains a community convention in 2026: widely published, but not yet officially consumed by any major AI provider.

## Key takeaways

- `llms.txt` is a curated Markdown index at your site root. The spec at llmstxt.org requires only an H1 with your site name; the summary blockquote and annotated link sections are recommended, not mandatory.
- It answers a different question than `robots.txt` (may you crawl this?) or `sitemap.xml` (what exists here?): it tells a model what to read first, and why.
- The honest 2026 status: published files grew 8.8x in twelve months, yet 97% of them receive zero requests from AI bots, and Google says it does not use the file at all.
- The realistic upside is agentic retrieval. Assistants and shopping agents that fetch your site live during a conversation benefit from a clean Markdown map instead of heavy HTML.
- It costs about an hour to implement. Treat it as one low-cost line in a broader [GEO](https://www.geoly.ai/blog/what-is-geo) checklist, not a visibility strategy on its own.

## How llms.txt works

The format, defined at [llmstxt.org](https://llmstxt.org), is deliberately minimal. A valid file contains, in order:

1. An H1 with the site or project name — the only required element, e.g. `# GEOly AI`.
2. A blockquote summary directly below it: `> GEO data platform for DTC brands, covering seven AI engines.`
3. Optional paragraphs of free-form context the model should know before reading further.
4. H2 sections grouping annotated links, one per line: `- [Quick start](/docs/quick-start): Install and run your first audit.`
5. An optional section literally titled `Optional`, for links an agent may skip when its context budget runs short.

Two companion conventions travel with it. `llms-full.txt` inlines the complete text of your key pages into a single file, so an agent gets everything in one fetch. The spec also suggests serving Markdown twins of important pages by appending `.md` to their URLs.

The case for Markdown is arithmetic. A typical e-commerce page delivers hundreds of kilobytes of navigation, scripts, and cookie banners wrapped around a few paragraphs of substance. Language models pay for every token they ingest, so a curated plain-text index is the difference between an agent grounding itself in your actual positioning and drowning in your header menu.

## llms.txt vs robots.txt vs sitemap.xml

The three root files serve different audiences, and they get confused constantly.

- `robots.txt` speaks to crawlers about permission: which paths GPTBot, ClaudeBot, or Googlebot may fetch. It is the enforcement layer, and unlike `llms.txt`, major [AI crawlers](https://www.geoly.ai/blog/ai-crawlers) demonstrably respect it.
- `sitemap.xml` speaks to indexers about inventory: every URL you want discovered, with no sense of priority or narrative.
- `llms.txt` speaks to language models about meaning: a curated shortlist, usually ten to fifty links, explaining what you sell, what you stand for, and where the authoritative answers live.

None replaces the others — and they need to agree. A store that blocks GPTBot in `robots.txt` while publishing an `llms.txt` is sending contradictory signals to the same company.

## Who actually reads llms.txt in 2026?

This is where most guides oversell, so here are the observable facts. Google has said plainly that it does not use `llms.txt` for Search or its AI features; John Mueller [compared it to the keywords meta tag](https://www.searchenginejournal.com/googles-mueller-says-llms-txt-cant-help-llms-differentiate-sites/579304/) — self-declared, easy to game, therefore ignored. Neither OpenAI nor Anthropic has committed to fetching it for training or indexing. And a [ppc.land analysis](https://ppc.land/llms-txt-adoption-rises-8-8x-but-97-of-files-get-zero-ai-requests/) of over three million monitored sites found that while published files grew from about 4,000 to 36,000 in a year, 97% of them received no AI-bot requests at all.

So why bother? Because crawling and retrieval are different behaviors. Training crawlers ignore the file today. But when an agent visits your site live — an answer engine [grounding a response](https://www.geoly.ai/blog/grounding-queries) mid-conversation, or a shopping agent comparing products in an [agentic commerce](https://www.geoly.ai/blog/agentic-commerce) flow — it reads whatever helps it finish the task cheaply, and a curated Markdown index is exactly that. The file is a hedge on agent traffic, not a lever on today's rankings. At an hour of work, the hedge is cheap.

![Cross-platform visibility matrix comparing brand mentions across ChatGPT, Gemini, Google AI Overview, AI Mode and Perplexity — Source: GEOly AI (app.geoly.ai)](https://gslkfhamqhkktynnjqng.supabase.co/storage/v1/object/public/cms-media/geoly-cross-platform-ai-engine-matrix.png)

## How to implement llms.txt properly

1. Draft the file in plain Markdown: H1 name, blockquote summary, then two to four H2 sections — documentation, products or collections, policies (shipping, returns, warranty), company.
2. Curate ruthlessly. Ten to fifty links with one-line descriptions. If you are pasting your sitemap, you have missed the point.
3. Serve it at `/llms.txt` with a plain-text content type, and confirm your `robots.txt` and CDN rules allow AI user agents to fetch it.
4. Keep it current. Regenerate when products launch or pricing changes. Shopify merchants can automate this: GEOly's LLMs.txt Master app scans the store for high-performing product and policy pages and regenerates the file as the catalog changes.
5. Verify consumption directly: filter server logs for `/llms.txt` requests from GPTBot, ClaudeBot, and PerplexityBot.
6. Measure outcomes, not the artifact. Machine readability — including `llms.txt` presence, structured data, and crawler access — is one section of [GEOly's](https://www.geoly.ai/blog/what-is-geoly-ai) 29-point [GEO audit](https://www.geoly.ai/blog/geo-audit). The numbers that tell you whether any of it worked are citation rate, mention rate, and your [AIGVR score](https://www.geoly.ai/blog/aigvr-score) trending across engines; the full [metrics stack is here](https://www.geoly.ai/blog/ai-search-visibility-metrics-kpis).

![Citation source analysis: source type distribution and the domains AI engines cite most — Source: GEOly AI (app.geoly.ai)](https://gslkfhamqhkktynnjqng.supabase.co/storage/v1/object/public/cms-media/geoly-ai-citation-source-analysis.png)

## Common mistakes

- Dumping every URL into the file. That is a second sitemap; the entire value is editorial judgment about what matters.
- Letting it rot. An `llms.txt` pointing at discontinued products or old pricing is worse than none — you are feeding agents stale facts with high confidence.
- Expecting a ranking lift. There is no evidence the file moves Google results or [AI Overviews](https://www.geoly.ai/blog/ai-overviews), and anyone claiming otherwise is selling something.
- Contradicting your own signals by publishing the file while blocking the same companies' crawlers.
- Treating it as the whole job. [Structured data](https://www.geoly.ai/blog/structured-data-ai-search) and content worth [citing](https://www.geoly.ai/blog/ai-citations) carry far more observable weight in what AI engines actually say about you.

## FAQ

### Does llms.txt improve my rankings in Google or ChatGPT?

There is no evidence it does. Google explicitly does not use it, and no LLM provider has confirmed reading it for indexing. Its plausible value is making live agent visits cheaper and more accurate, which can indirectly support mentions and citations — so measure those, not the file.

### What is the difference between llms.txt and llms-full.txt?

`llms.txt` is the index: links plus one-line descriptions. `llms-full.txt` inlines the full content of those pages into one large file, trading a single convenient fetch against a size that can blow an agent's context budget. For most DTC sites, the index alone is the right starting point.

### Do I still need robots.txt and sitemap.xml if I publish llms.txt?

Yes. They do three different jobs — permissions, inventory, and context — and `robots.txt` is the only one of the three that AI companies demonstrably honor today. Keep all three consistent with each other.

### How do I know if AI bots are actually reading my llms.txt?

Filter your server or CDN logs for requests to `/llms.txt` by AI user agents. Most sites will find few or none, which matches the industry-wide data. The more meaningful signal is downstream: whether engines mention and cite your domain more over time, which is what [per-engine monitoring](https://www.geoly.ai/blog/track-brand-mentions-in-ai-search) exists to track.
