What is llms.txt? Does your site really need it?

llms.txt is a proposed file, placed at the root of a website (/llms.txt), that gives large language models a Markdown summary of what the site is and where its most important pages live. As of today, no major AI provider has officially said it uses the file when generating answers, and Google states plainly that its search products ignore it. For most sites, then, llms.txt is not a must-have but a low-cost, low-return experiment; for products with technical documentation it can deliver real value.

Key takeaways

llms.txt is a community proposal published by Jeremy Howard on 3 September 2024; no standards body has ratified it.
The file is plain Markdown: the only required element is an H1 title, followed by a summary, optional notes and lists of links.
Google's official guide to generative AI features says llms.txt will neither help nor harm visibility in Google Search, because Search ignores it.
The clearest use case is API and product documentation that developers feed to coding assistants.
What drives AI visibility is still content quality, crawlability and how other sources talk about you, and you cannot know whether the file helps unless you measure.

What exactly is llms.txt?

The proposal is a short specification published at llmstxt.org. The idea: HTML pages are cluttered with navigation, cookie banners and scripts that waste a model's context window, so llms.txt offers a human-readable table of contents that says "start here". For a one-paragraph definition, see our llms.txt glossary entry.

The file structure

According to the specification, sections appear in this order:

An H1 heading with the name of the project or site (the only required section).
A blockquote summarising what the site does.
Optional detail paragraphs without headings.
H2 sections, each holding a list of links in the form [name](url): short note.
By convention, an H2 section called "Optional" for secondary links a model can skip when context is tight.

An example for a hypothetical e-commerce software vendor:

# Example Shop Software
> Cloud-based e-commerce platform for small businesses.

## Docs
- [Installation](https://example.com/docs/install.md): opening your first store
- [API reference](https://example.com/docs/api.md): REST endpoints

## Optional
- [Changelog](https://example.com/changelog.md)

The proposal also recommends offering clean Markdown versions of useful pages at the same URL with .md appended (for example page.html.md). Many documentation platforms additionally generate an llms-full.txt that concatenates all content into one file; this is a widespread community habit, not part of the core specification.

How is it different from robots.txt and sitemap.xml?

The three files are often confused, but they do different jobs:

File	Purpose	Audience	Status
robots.txt	Says which bot may crawl what	Crawlers (Googlebot, GPTBot, etc.)	Widely adopted protocol
sitemap.xml	Lists all indexable URLs	Search engines	Widely adopted protocol
llms.txt	Offers a curated, summarised content map	LLMs and agents looking things up at inference time	Proposal; support not guaranteed

The crucial point: llms.txt is not access control. If you want to block or allow an AI crawler, you do that in robots.txt; OpenAI, for instance, documents that crawlers such as GPTBot and OAI-SearchBot respect robots.txt rules. llms.txt only says "if you are here, prioritise these".

Do AI systems actually read llms.txt?

The honest answer: the evidence is thin. In April 2025, Google's John Mueller noted that, as far as he knew, none of the AI services had said they use llms.txt, that server logs showed they did not even request it, and he compared it to the old keywords meta tag. Google Search Central's official guide to generative AI features is even clearer: you do not need new machine-readable files, AI text files or Markdown to appear in Google Search; Search ignores them, and adding one will neither help nor harm.

The picture is not entirely one-sided. Lighthouse, the Chrome team's auditing tool, added an "Agentic Browsing" category in 2026 that includes llms.txt as an experimental check. But that audit only looks at whether a server error occurs when the file is requested; if the file does not exist (404), the result is "not applicable". Lighthouse does not penalise a missing file, and the category makes no claims about rankings or citations.

Meanwhile, many software companies publish llms.txt for their own documentation. That usage is mostly user-initiated: a developer pastes the file's URL into a coding assistant, or an IDE plugin pulls it in as context. We explain how AI systems actually choose sources in which sources does AI cite.

Does your site need llms.txt?

The decision depends on your type of site and your resources.

When it makes sense

Developer-facing products: if you have an API, SDK or complex software docs, your users already work with coding assistants. A clean llms.txt plus Markdown versions can help those assistants answer with accurate, current information.
Teams preparing for AI agents: if you expect agents to complete tasks on your site, offering a structured entry point is a reasonable experiment.

When it is not a priority

Local businesses, corporate brochure sites, small online shops: here visibility depends on content quality, reviews, directories and mentions on third-party sources.
Sites with unresolved technical SEO problems: if pages cannot be crawled, sit behind JavaScript or load slowly, fix those first.

A hypothetical example: a dental clinic in Manchester that spends a day on llms.txt will likely gain far less than one that publishes a page clearly answering questions like "how much do dental implants cost" and gets listed in reputable health directories. For the broader strategy, see our guide what is GEO.

How to create llms.txt, step by step

Keep the process small and measurable:

Define the goal: which questions should a model be able to answer correctly? Product definition, pricing logic, documentation?
Pick 10–30 critical pages at most: list the pages that carry the most value and will stay current, not the whole site.
Write the summary: in the blockquote, describe your brand in one factual sentence without hype.
Offer Markdown versions where possible: generate .md versions of key pages and point the links to them.
Publish at the root: the file should load at https://yourdomain.com/llms.txt as text/plain or text/markdown, without server errors.
Automate maintenance: a hand-kept llms.txt full of stale links can be worse than none.
Measure: watch server logs for who requests the file, and compare your visibility in AI answers before and after.

How do you measure the effect?

The most common mistake is to add llms.txt and assume you are now visible in AI. Measuring the effect needs two layers of data. First, server logs: they show which user agents request the file and how often. No requests means no effect. Second, answer-level measurement: regularly asking AI assistants real user questions in your target market and recording how many answers mention your brand and which sources they show.

Methodology matters here. AI answers vary even for the same prompt; a single screenshot is not evidence. You need an approach that excludes failed answers from the denominator and compares competitors on the same sample. We set out those principles in how to measure AI visibility and on GeoDog's methodology page. If you want a baseline for your own brand, you can run a free scan; re-running the same question set a few weeks after publishing llms.txt is the most robust way to see its real impact.

The academic picture is similar. Aggarwal et al.'s "GEO: Generative Engine Optimization" (KDD 2024) showed that content-level interventions, such as adding citations and statistics, can affect visibility in generative engine answers; there is no comparable evidence yet for file-level signals like llms.txt. See also GEO vs SEO.

FAQ

Can adding llms.txt hurt my SEO?

According to Google's official guidance, no: Google Search ignores the file, so it neither raises nor lowers your rankings. The only risk is an outdated file misleading tools that do read it.

Can I block AI bots with llms.txt?

No. Access control happens in robots.txt (and at server level if needed). llms.txt only suggests content; it blocks nothing.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a short map made of links. llms-full.txt usually merges the full documentation text into a single file; it is not in the core specification but has become common, especially on documentation platforms.

What is llms.txt and does your site really need it?