The llms.txt Guide for 2026: What It Is, What It Isn't, and When to Bother

WHY THIS GUIDE EXISTS

Search "llms.txt" in May 2026 and you get two opposing camps. One camp insists every site needs llms.txt immediately or risk vanishing from AI search. The other camp publishes server-log studies showing zero major AI crawler actually fetches the file, and concludes the standard is dead on arrival.

Both camps are partly right and mostly wrong. llms.txt is real, low-cost, and worth publishing for a specific class of sites — but the magical-thinking version ("publish this and rank in ChatGPT") is not how it works. This guide separates the protocol from the promotional noise.

If you're new to optimizing for AI agents, the Agent-First SEO Playbook covers the broader landscape and the Agent-Ready Scorecard gives you a maturity model. This piece zooms in on llms.txt specifically: what it is, where it fits, and when it actually moves a needle.

WHAT LLMS.TXT ACTUALLY IS

llms.txt is a single Markdown file you place at the root of your domain — https://yoursite.com/llms.txt. It contains a curated, hierarchical summary of the resources on your site that you want AI systems to be able to find and cite. Unlike a sitemap (which is a flat XML list of every URL) or robots.txt (which is a directive file for crawlers), llms.txt is human and machine readable Markdown that an LLM can parse directly into its context window.

The spec defines a strict structure. The file opens with a single H1 that names the site or project. Below that comes a blockquote with a one-paragraph summary. After the summary, you organize content under H2 sections — typically things like "Documentation", "Tutorials", "API Reference", "Blog" — each containing a bullet list of links with short descriptions.

There are two variants. llms.txt is compact and link-only, designed to fit comfortably in a context window. llms-full.txt embeds the actual content of every linked page directly in the file, so an agent can pull the entire site corpus in a single request. Documentation platforms like Mintlify and GitBook auto-generate both versions; for a hand-coded site, llms.txt alone is usually plenty.

THE 2026 ADOPTION REALITY

The honest picture in May 2026 is mixed. Independent crawl studies place llms.txt adoption at roughly 10% of scanned domains — a meaningful minority but nowhere near universal. Documentation-heavy sites adopt at higher rates because the format genuinely fits how their content is consumed by IDE agents and coding assistants. Marketing sites and small-business pages adopt at far lower rates because the format adds work without obvious return.

On the consumer side, the major AI vendors have been quiet about whether they actively read llms.txt at inference time. Anthropic listed llms.txt and llms-full.txt in its own developer documentation, which is the closest thing to an endorsement from a frontier lab. OpenAI, Google, and Perplexity have not made comparable public statements as of this writing. Server-log analyses from Search Engine Land and others have repeatedly shown that the named AI crawlers (GPTBot, ClaudeBot, PerplexityBot) rarely if ever request the file by name.

That sounds damning until you remember how AI assistants actually work. When you ask ChatGPT "what does loaded.ch do?", the system does not necessarily fetch loaded.ch/llms.txt in real time. It pulls from an indexed corpus assembled by training crawlers, plus retrieval over a search index. llms.txt is most likely to influence what shows up in those indexes, not to be hit on the synchronous request path. That is genuinely hard to measure from the publisher side, which is why the studies look bleak even when the file might be doing useful work.

WHEN LLMS.TXT IS WORTH PUBLISHING

There is a clear pattern in who benefits from llms.txt in 2026 and who does not. The benefit scales with how often AI assistants are asked technical questions about your site, and how badly raw HTML scraping fails on your content.

Developer tool companies are the highest-leverage adopters. If your product has documentation, an SDK, or an API reference, AI coding assistants are already trying to ingest it — often badly. A clean llms.txt plus llms-full.txt can dramatically improve how Cursor, Claude Code, and similar tools answer questions about your library. Stripe, Anthropic, Mintlify-hosted docs, and Mastercard's developer platform have all published examples worth studying.

E-commerce and content publishers see less direct benefit because the volume and update cadence make llms-full.txt impractical to maintain, and AI assistants are usually not parsing product catalogs through this channel. For these sites, structured data (Schema.org, Product, Offer, Article) and an agent-callable interface deliver more measurable lift than llms.txt will.

Small businesses, local services, and marketing sites occupy a middle ground. Publishing a 30-line llms.txt is cheap, and it can help a future AI assistant answer questions like "what does this company do" with your phrasing rather than scraped fragments. That is worth the half-hour even if the direct measurement is fuzzy.

A 30-MINUTE IMPLEMENTATION

For most sites, publishing a useful llms.txt is a four-step exercise. The version below assumes a small-to-medium site with a handful of priority pages. Larger documentation sites should adopt a generator (Mintlify, GitBook, or a custom build script) instead of hand-editing.

Step one: list the pages an AI assistant should know about. Be ruthless. The goal is not coverage; it is a curated short list. Five to twenty links is typical. If you list a hundred, you have a sitemap, not an llms.txt.

Step two: write a one-paragraph summary of the site that an LLM can quote verbatim. Three to five sentences. Lead with what the site is and who it is for, then what makes it distinctive. This summary is the single highest-leverage piece of the file because it is the most likely fragment to end up in an AI answer.

Step three: group the links under H2 headings. Common buckets are "Core documentation", "Tutorials", "Reference", "Blog", "About". Each bullet should be [link text](url): short description. Keep descriptions to one line.

Step four: serve the file at /llms.txt with content type text/markdown or text/plain. Do not gate it behind authentication, do not redirect it through a CDN that rewrites paths, and link to it from your homepage footer if you want crawlers to discover it through normal hyperlinks rather than the well-known path.

# OpenHermit
 
> OpenHermit is an open-source JavaScript library that auto-injects
> W3C WebMCP attributes into existing websites so AI agents can
> discover and operate page actions without DOM scraping. Built by
> loaded.ch, a Swiss digital product studio.
 
## Core documentation
- [Getting started](https://openhermit.com/docs/quickstart): install
  the client script and verify agent discovery in 5 minutes.
- [How it works](https://openhermit.com/docs/architecture): the
  attribute injection pipeline and the WebMCP bridge.
- [Configuration](https://openhermit.com/docs/configuration): per-site
  rules for which elements to expose.
 
## Tutorials
- [WebMCP tutorial](https://openhermit.com/blog/webmcp-tutorial):
  the raw navigator.modelContext API, no library required.
- [Three paths to agent-ready](https://openhermit.com/blog/three-paths):
  pick OpenAPI, WebMCP, or platform detection.
 
## Background
- [How agents interact with sites](https://openhermit.com/blog/how-agents-interact)
- [Agent-ready scorecard](https://openhermit.com/blog/agent-ready-scorecard)
 
## About
- [loaded.ch](https://loaded.ch): the studio behind OpenHermit.
- [Contact](https://openhermit.com/contact): bw@expat-savvy.ch

That file is roughly 30 lines, takes a careful editor about half an hour to write, and is far more useful to a downstream AI assistant than a 5,000-URL sitemap.

WHERE LLMS.TXT FITS IN THE STACK

The mistake the magical-thinking camp makes is treating llms.txt as the entire AI strategy. It is not. llms.txt answers the question "what does this site contain". A complete agent-readiness stack also answers "is this site allowed to be crawled" (robots.txt + AI Preferences), "what does this individual page mean" (Schema.org structured data), and "what can an agent do on this page right now" (WebMCP, OpenAPI, or platform detection). Each layer is a different question, and each layer fails differently when missing.

Treat llms.txt as the discovery and orientation layer. It is cheap to publish, modestly useful for assistants that do parse it, and probably contributes to higher-quality citations even when the file is not fetched on the live request path. But if your only investment in agent-readiness is llms.txt, you have given assistants a map to a city they cannot enter. The interaction layer is where the meaningful traffic and revenue actually move, and that is where structured data and agent-callable tools earn their keep. The Three Paths post breaks down those interaction options.

For OpenHermit users specifically, llms.txt is complementary, not redundant. OpenHermit handles the action layer — turning your existing HTML buttons, forms, and product cards into discoverable WebMCP tools — while llms.txt describes what your site is at a higher level. A site running both gives an agent a clean orientation file plus a working set of callable actions, which is the configuration most likely to convert agent traffic. The Agent-Driven ROI post covers how to actually measure that conversion.

FAQ

THE BOTTOM LINE

llms.txt is a small, low-cost, modestly useful addition to a site's agent-readiness stack. It is not a silver bullet, not a permission system, and not a replacement for structured data or an action interface. The right way to think about it in May 2026 is as a 30-minute investment that improves how AI assistants describe your site without unlocking any new revenue path on its own.

Publish a clean version. Keep the summary paragraph tight. Keep the link list short and curated. Then move on to the layers that actually move agent revenue: structured data, an action layer, and analytics that distinguish agent traffic from human traffic. Those are the levers that compound.

# Make your website agent-ready
> npm install @openhermit/client-script
> Ready to be discovered by AI agents.