RebilderLearn

llms.txt: what it is, how to write one, and whether it works

Updated

llms.txt is a plain markdown file at the root of your site that gives AI systems a compact map of what you have and where it lives. It takes about ten minutes to write and costs nothing to serve.

It is also the most over-sold artifact in this field, so this guide leads with the evidence rather than the enthusiasm: what the file is, exactly what shape it takes, where it goes, what the measurements say it does, and what to do instead when it turns out not to be the mechanism.

What is llms.txt?

llms.txt is a markdown file served at /llms.txt that describes your site to AI systems in a form they can read cheaply: who you are, one line about what you do, and curated lists of links to the pages that matter. Think of it as a table of contents written for a reader with a token budget, rather than a sitemap written for a crawler with none.

It is a community convention, not a standard from a standards body, and no vendor is contractually obliged to read it. That is worth knowing before you plan around it. What it costs you is one static file; what it buys you is that a system which does look will find a clean index instead of guessing your structure from your navigation.

FileAudienceWhat it says
robots.txtCrawlersWhat you may and may not fetch
sitemap.xmlSearch crawlersEvery URL, with change metadata
llms.txtAI systemsThe curated subset that matters, in prose

The distinction from a sitemap is curation. A sitemap lists everything; an llms.txt lists what you would point a new colleague at. Publishing three hundred URLs in it defeats the purpose, because the file is only useful while it is cheaper to read than the site.

The llms.txt format, exactly

The shape is small enough to state completely: an H1 with the site name, a blockquote with a one-line description, then H2 sections containing markdown link lists. A link may carry a note after a colon.

llms.txt example
# Basecamp Supply Co

> Independent outdoor gear shop. Packs, tents and trail equipment, shipped from Colorado.

## Products

- [Alpine Trail Pack 28L](https://basecamp-supply.example/products/alpine-trail-pack-28l): $148.00, in stock
- [Ridge Tent 2P](https://basecamp-supply.example/products/ridge-tent-2p): $329.00, in stock

## Policies

- [Shipping](https://basecamp-supply.example/policies/shipping): Free US shipping over $75
- [Returns](https://basecamp-supply.example/policies/returns): 60-day returns, unused with tags
  • One H1, at the top. Your site or organisation name, nothing else.
  • One blockquote, immediately after. A single sentence a stranger could repeat back accurately.
  • H2 sections with link lists. Group by what a reader would be looking for: products, docs, policies, guides.
  • Absolute https URLs. A relative link in a file being read out of context is a dead link.
  • Notes carry facts, not adjectives. "$148.00, in stock" earns its bytes; "our most popular pack" does not.

You can write one by hand, build one in the browser with the free generator, or have the gateway emit it from the same source of truth your pages render from. The last option is the one that does not drift: a hand-maintained index disagrees with the site within a quarter, and a disagreeing index is worse than none.

Does llms.txt actually work?

On its own, the evidence says no, and this is the part most guides skip. A 300k-domain study in mid-2026 found llms.txt alone produced no measurable citation lift. The same study measured content negotiation, answering the Accept header with clean text, at roughly 4.2x more effective for accurate retrieval.

Content negotiation vs llms.txt, accurate retrieval
~4.2x
300k-domain study, mid-2026
Context reduction, gateway markdown vs product-page HTML
98.6%
Rebilder reference capture

The reason is structural rather than a matter of adoption. llms.txt tells a system where your pages are. It does nothing about what those pages return when the system goes and fetches one. If a product page answers with ninety kilobytes of theme HTML, a perfect index has done nothing except deliver the assistant more efficiently to the same wall.

What moves the number instead

If llms.txt is the index, content negotiation is the mechanism. An agent sends Accept: text/markdown; a server that understands the header answers with clean structured text rather than a rendered page. Same substance, different format, and the difference is measurable:

MeasureBefore: full product-page HTMLAfter: gateway markdown
Bytes91,2261,315 (−98.6%)
Approx tokens~22,789~328
First price locationCharacter offset 23,185, inside a JSON blobLine 4

Roughly 69× less context for the same facts, and the facts arrive front-loaded rather than recovered by markup archaeology. The full guide to this work covers the other levers, but the ordering is stable: unblock the fetch, answer the header, front-load the facts, then publish the index.

Four ways an llms.txt goes wrong

  • It drifts. Written by hand in January, wrong by April. Generate it from the same data your pages render from, or accept that it will be a list of stale links.
  • It lists everything. A file with hundreds of URLs stops being a curated map and becomes a sitemap in a worse format. Curate to what you would show a new colleague.
  • It is unreachable. A CDN rule, an auth redirect, or a framework route that only matches after JavaScript will all make it a 404 to the systems it is for. Fetch it from a terminal to be sure.
  • It is treated as the strategy. The measurable failure mode: the file gets published, the pages stay unreadable, and six months later the conclusion drawn is that AI search optimization does not work.

Frequently asked questions

What is llms.txt?

A plain markdown file served at /llms.txt that gives AI systems a compact, curated map of your site: an H1 with your name, a one-line description in a blockquote, and H2 sections of markdown links to the pages that matter. It is a community convention rather than a formal standard.

Where do I put llms.txt?

At the root of your domain, reachable at https://example.com/llms.txt, served as text/plain or text/markdown and returning a 2xx to an unauthenticated request. Test it with a plain fetch rather than assuming, since CDN rules and auth redirects are the usual reasons it is silently unreachable.

Does llms.txt improve AI search rankings or citations?

The evidence says not on its own. A 300k-domain study in mid-2026 found no measurable citation lift from llms.txt alone, while content negotiation measured roughly 4.2x more effective for accurate retrieval. Publish llms.txt because it is a cheap, useful index; treat answering Accept: text/markdown as the mechanism.

What is the difference between llms.txt and robots.txt?

robots.txt is a permission file: it tells crawlers what they may fetch. llms.txt is a content file: it tells AI systems what is worth fetching and where it is. They answer different questions and you want both.

Is llms.txt the same as a sitemap?

No, and the difference is curation. A sitemap enumerates every URL for a crawler with no cost pressure. An llms.txt lists the subset a reader with a token budget should start from, in prose, with notes. A three-hundred-link llms.txt has become a sitemap and lost the point.

How do I generate an llms.txt file?

Write it by hand from the shape above, build it in the browser with the free generator, or have your gateway emit it from the same source of truth your pages render from. The generated route is the one that does not drift, which matters more than the initial effort saved.