Skip to content
EchoVerseTen focused tools

Standards

llms.txt vs robots.txt: which do you need?

Short answer

You need robots.txt. It is a long-established standard that crawlers actually obey, and it controls whether AI systems can reach your pages at all. llms.txt is a newer proposed convention for summarising a site to language models; it is cheap to add, but no major AI provider has publicly confirmed using it for retrieval, so treat it as optional.

By The EchoVerse teamUpdated 4 min read

The honest comparison

robots.txt llms.txt
Status Long-established, widely obeyed Proposed convention
Location /robots.txt /llms.txt
Purpose Controls crawler access Summarises the site for language models
Confirmed use by major AI providers Yes Not publicly confirmed
Consequence of omitting it Loss of control over crawling Almost certainly none

This is the part most write-ups soften. robots.txt determines whether an AI system can fetch your pages. llms.txt is a proposal — a markdown file listing your key pages with short descriptions — and while it costs almost nothing to publish, there is no public confirmation from the major providers that it influences retrieval or citation.

Anyone presenting llms.txt as a requirement, or claiming measurable ranking benefit from it, is ahead of the evidence.

Do robots.txt properly first

This is the file with real consequences. Confirm that the AI crawlers you want to be visible to are not blocked, that your sitemap is declared, and that no platform layer is injecting rules you did not write:


User-agent: *
Allow: /
Disallow: /admin
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Then add llms.txt if you like

It is a few minutes of work and the downside is zero. Keep it short, link only pages that genuinely matter, and describe each one plainly:


# Example Co

> One sentence on what this site is and who it serves.

## Pages

- [Pricing](/pricing): Plans, limits, and what each tier includes.
- [Docs](/docs): API reference and integration guides.

Treat it as a low-cost bet, not a ranking factor. The work that reliably affects whether you get cited is upstream of both files: server-rendered content, answers placed near the top of the page, passages that survive being read out of context, and claims a system can attribute to you.

The check worth doing today

Fetch both files as they are actually served, rather than trusting the copy in your repo:


curl -s https://example.com/robots.txt
curl -s https://example.com/llms.txt

A surprising number of sites discover their robots.txt is not what they committed, because a CDN, a hosting platform, or a security product is generating or rewriting it.

Common questions

Is llms.txt an official standard?
No. It is a proposed convention. It has some adoption among developer-facing sites, but no major AI provider has publicly confirmed using it for retrieval or ranking. Add it if you want; do not prioritise it over robots.txt or over page structure.
Will llms.txt get me cited by ChatGPT or Perplexity?
There is no public evidence that it will. Citations follow from being crawlable and from having clean, self-contained, answer-first passages. llms.txt is cheap to add but should not displace that work.
What happens if I have neither file?
With no robots.txt, crawlers assume they may crawl everything, which is often fine but means you have no control. With no llms.txt, in all likelihood nothing happens at all.
All articles