BlogAI search

llms.txt: what it is, and whether it matters for AI visibility

Benjamin Libor1 September 202610 min read

`llms.txt` is a proposed text file at the root of a website that lists its most important pages, in Markdown, so that AI assistants can find the right content without parsing the whole site. It is cheap to add and does no harm, but at the time of writing none of the major assistants has documented that it reads the file when answering questions, so it is not a lever for AI visibility. Add it if you have documentation or a developer audience and an hour to spare; put the real effort into crawlable pages, clear answers and presence on the sites assistants already cite.

In short

llms.txt is a Markdown file at /llms.txt with a short description of the site and a curated list of links, proposed in 2024 as a guide for AI systems.
No major assistant (ChatGPT, Perplexity, Gemini, Claude, Google's AI Overviews) has documented using it to choose sources for answers. Treat claims that it boosts visibility with suspicion.
What moves AI visibility is the same as before: pages crawlers can read, answers stated plainly near the top, and mentions on third-party sites.
It is worth an hour for documentation sites and developer tools, where the audience and their tools may use it directly.
Keep it accurate if you add it. A stale llms.txt that lists dead pages is worse than none.

What llms.txt is

The llms.txt proposal describes a plain Markdown file served at the root of a domain, for example https://www.fernwick.example/llms.txt. The idea is that an AI system with a limited amount of reading time can fetch one small file and learn what the site is, which pages matter, and where to find clean versions of them, instead of crawling hundreds of pages of navigation and marketing.

The proposed structure is simple:

A top-level heading with the site or project name.
A blockquote with a one-paragraph summary of what it is.
Optional paragraphs with anything an AI should know (who it's for, what it is not).
One or more sections, each a heading followed by a list of links, each link with a short description of what the page contains.
An optional section named "Optional" for pages that can be skipped when reading time is short.

The proposal also suggests two companions. Markdown versions of pages: serve a clean .md version of each important page at the same address with .md appended, so a system that wants the content gets it without the page chrome. And llms-full.txt: one large file with the full text of every important page concatenated, for systems that would rather read everything at once.

llms.txt is a curated table of contents for machines, written by the site owner. robots.txt says where crawlers may go; sitemap.xml lists every page; llms.txt says which pages are worth reading and why.

Who proposed it, and why

The proposal came from Jeremy Howard, co-founder of Answer.AI and fast.ai, in September 2024, and is published at llmstxt.org. The reasoning: language models have a limited context window and cannot read a whole site at once; HTML pages are full of navigation, scripts and boilerplate that waste that window; and a short, human-curated file would let a model get the essentials of a site quickly, in the same way a well-written README helps a developer.

The use case the proposal has in mind is mostly developer tooling. A coding assistant that needs to know how a library works benefits from a file that says "here is the quick start, here is the API reference, here are the examples". Documentation platforms adopted it first, and a number of them now generate llms.txt and .md page versions automatically. Several AI companies publish an llms.txt for their own developer documentation.

What is known about adoption by the AI companies

This is the part that matters for a marketer, so it deserves plain language.

At the time of writing, OpenAI, Google, Anthropic and Perplexity document their crawlers and, in some cases, how their search features choose pages. None of that documentation says the company fetches /llms.txt when answering a user's question or uses it to rank or select sources. A Google Search representative has said publicly that, as far as Google is aware, no AI system was using the file. Crawl logs from sites that have adopted it, as reported by their owners, mostly show the file fetched rarely, and there is no documented link between having the file and being cited.

Three things are true at once:

The file is read by some tools. Coding assistants and developer agents, and people who paste the link into a chat, do use it. That is a real, if narrow, audience.
The major assistants have not said they use it for answers. Their search features work from a web index built by crawling ordinary pages, and they cite ordinary pages.
The situation can change. A company could announce support at any time, and the cost of being ready is low.

So the honest position is: llms.txt is a harmless, cheap signal with no documented effect on whether ChatGPT, Perplexity, Gemini or AI Overviews cite you. Anyone selling it as a visibility lever is ahead of the evidence.

What actually moves AI visibility

If you have one hour for AI visibility this week, the file is not where it goes. The work that measurably changes whether assistants find and cite your pages is unglamorous and well understood.

Pages the crawlers can read

The assistants' search features rely on their own crawlers: OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity, Googlebot for AI Overviews. If a crawler is blocked, challenged by bot protection, or served an empty page that fills in with JavaScript, nothing else matters. AI crawlers explained covers what to allow and how to check which bots actually visit.

Clear answers near the top

An assistant quotes the page that states the answer plainly. A page that answers one question in its first paragraph, with a date and an author, under a title that matches how people ask the question, gets quoted. A page of slogans does not. Writing content that AI answers quote sets out how to write that way.

Presence on third-party pages

For "best tools for" and "X vs Y" questions, assistants cite review sites, comparison articles and forum threads far more than brand sites. An accurate listing on the review sites that matter in your category, honest comparison pages, and real answers in real threads move those questions. Why AI answers lean on Reddit, reviews and forums explains the mechanism.

Ranking in ordinary search

Every assistant that searches the web reads from search results. Pages that rank for a question are the pages that get read. Your SEO work is still the foundation.

A worked example makes the priority clear. Allsite, a made-up website builder, has two hours this month. Option A: write a careful llms.txt. Option B: find that OAI-SearchBot is being served a bot challenge by the firewall, allowlist it, and add a "Last updated" date and a two-sentence answer to the top of its five most-searched help pages. Option B changes what ChatGPT can read and quote. Option A changes nothing measurable today.

When it is worth doing anyway

There are good reasons to add the file, as long as nobody expects a visibility jump from it.

You have documentation. Developers and their coding assistants do use llms.txt and .md pages. If your product has an API, a CLI or an integration guide, the file serves a real audience, and your documentation platform may generate it for free.
Your audience is technical. Developers notice the file, and its absence, and some judge a tool by it.
It takes an hour. Writing a curated list of your twenty most important pages with one-line descriptions is quick, and the exercise itself is useful: it forces you to decide which pages matter.
It is ready if support arrives. If an assistant starts reading the file, you are already there.
It is harmless. The file has no effect on search rankings, crawling or anything else. The only risk is letting it go stale.

Skip it, for now, if you have no documentation, a small marketing site and a long list of unfixed basics. Come back to it when the basics are done.

A short example

Here is what a minimal llms.txt for Fernwick, a made-up scheduling tool, would contain. Each line below is one line of the file.

# Fernwick
> Fernwick is a scheduling tool for client-facing teams. Customers book meetings from a link that shows only the host's free times; the meeting lands in both calendars.
Fernwick has a free plan for one calendar and paid plans for teams. The docs below cover setup, the API and integrations.
## Docs
- [Getting started](https://www.fernwick.example/docs/start.md): create a booking link and connect a calendar in ten minutes
- [Booking rules](https://www.fernwick.example/docs/rules.md): buffers, limits per day, notice periods and round-robin
- [API reference](https://www.fernwick.example/docs/api.md): endpoints, authentication and webhooks
## Product
- [Pricing](https://www.fernwick.example/pricing.md): plans, limits and what is free
- [Fernwick vs Allsite scheduling](https://www.fernwick.example/compare/allsite.md): an honest comparison
## Optional
- [Changelog](https://www.fernwick.example/changelog.md): releases by month

A few rules for writing it well:

Keep it under a few hundred lines. The point is curation; a dump of the sitemap defeats it.
Write the descriptions as facts, not slogans: what the page contains, who it is for.
Link to .md versions if you can serve them; otherwise link to the normal pages.
Put the pages you most want quoted first within each section.
Review it whenever pricing, plans or the product's shape change. Add a line to your release checklist.

If you also add llms-full.txt, generate it from the same list rather than maintaining two files by hand.

The recommendation

Add llms.txt if you have documentation or a technical audience, and spend about an hour on it. Have your documentation platform generate it where possible. Do not expect it to change your AI visibility, and do not let it displace the work that does: crawlers allowed and verified, answers stated plainly at the top of the pages people search for, accurate listings and honest comparisons on the sites assistants cite, and ordinary rankings.

Then measure. Ask your customers' questions in each assistant monthly, note whether you are named and which pages are cited, and watch the trend. If an assistant announces support for llms.txt, you will see it in the citations before you read it in a press release. What is AI visibility? explains how to set up that measurement.

Questions people ask

Does ChatGPT read llms.txt?

OpenAI's documentation describes its crawlers and ChatGPT search, and does not say that ChatGPT fetches or uses llms.txt when answering. At the time of writing there is no documented effect of the file on whether ChatGPT cites a page. A user can paste the file's address into ChatGPT and it will read it, as it would any page.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers which parts of a site they may or may not fetch, and the major crawlers honour it. llms.txt is a reading guide for AI systems, listing pages worth reading with descriptions; it grants and forbids nothing. Blocking a crawler in robots.txt still blocks it, whatever llms.txt says.

Does llms.txt help SEO?

There is no indication that it affects search rankings. Google has not announced any use of the file, and it is not part of any documented ranking signal. It will not hurt, either.

Should I add llms.txt to my marketing site?

If you have documentation or a developer audience, yes: it takes an hour and serves the coding assistants and developers who do read it. If you have a small marketing site and unfixed basics (blocked crawlers, pages without clear answers, missing review-site listings), do those first.

Where does llms.txt go on the website?

At the root of the domain, served as plain text: https://www.yourdomain.com/llms.txt. If you serve Markdown versions of pages, they go at the page's own address with .md appended, and llms-full.txt goes next to llms.txt.

Written by Benjamin Libor, founder, echo.

Something wrong or out of date? Write to hello@echo-aeo.com and we'll fix it.