Glossary
What is llms.txt?
llms.txt is a proposed web standard — published by Jeremy Howard in September 2024 — for a markdown file at a site's root that gives language models a concise, curated map of the site: what it is, and which pages matter, with links. It addresses LLMs' limited context windows the way robots.txt addresses crawler permissions.
Last updated:
The format is deliberately simple: an H1 with the site or project name (the only required element), an optional blockquote summary, and H2-sectioned lists of markdown links to the pages worth reading. A companion convention serves clean markdown versions of pages at the same URL with .md appended, and many sites also publish an llms-full.txt containing full page text in one file.
Adoption is voluntary on both sides — a site can publish the file, and an AI system can choose to read it — so its value today is strongest as low-cost GEO groundwork: it makes a site legible to answer engines and agents in one request, with the site's own framing of what matters.
For vendors preparing for agent-led evaluation, llms.txt is the passive end of a spectrum that ends at an agent door: llms.txt tells a visiting agent what to read, while an interactive endpoint can answer the agent's actual questions. AgentDoor publishes both for its own site (llms.txt and llms-full.txt) and treats them as table stakes for any vendor that expects buyers' agents to visit.
Sources: [1]
Frequently asked questions
- Is llms.txt the same as robots.txt?
- No. robots.txt tells crawlers what they may access; llms.txt tells language models what is worth reading and how the site is organised. One is permissions, the other is curation — sites typically publish both.
Skip the discovery calls.
Describe what you need and let AgentDoor's agents interview the vendors for you — you get a decision-ready shortlist, not six meetings.
We'll reach out at launch. No spam, ever.