What is llms.txt? It is a plain Markdown file placed at the root of a website (yoursite.com/llms.txt) that gives large language models a short, curated map of your most important content. Instead of an AI tool crawling dozens of cluttered HTML pages, it can read one clean file that says what your site is about and where the key pages live.
The idea was proposed in September 2024 by Jeremy Howard of Answer.AI. It has spread quickly, especially on developer documentation sites, and several WordPress SEO plugins can now generate the file for you.
The honest picture is mixed, though. llms.txt is a proposal, not an official web standard, and major search engines have not said they rely on it. This guide explains what the file is, what it can and cannot do, and whether it is worth adding to your site.
In short: llms.txt is an optional Markdown file at your site root that summarizes your site and links to its most useful pages for AI tools. It does not control crawling like robots.txt and is not a ranking factor in Google. It is cheap to add, can help AI tools that read it, and is easy to generate on WordPress.
What is llms.txt, exactly?
The proposal describes a simple, human-readable format. A typical llms.txt file contains:
- An H1 with the name of the site or project (the only required part).
- A short summary in a blockquote describing what the site offers.
- Optional paragraphs with extra context, such as who the content is for.
- H2 sections, each with a list of links to key pages and a one-line note on what each page covers.
- An optional section (often titled “Optional”) for secondary links that can be skipped when context space is limited.
Because it is Markdown, the file is readable by people as well as machines. It is a curated table of contents, not a full copy of your site.
Here is a minimal example for a small business site:
# Acme Plumbing
> Licensed residential plumbing in Austin, Texas. Emergency repairs, water heaters and repiping.
## Services
- [Emergency plumbing](https://example.com/emergency/): 24/7 call-outs, typical response times and pricing approach
- [Water heater installation](https://example.com/water-heaters/): tank and tankless options, brands we install
## Company
- [About us](https://example.com/about/): licensing, service area and team
- [Contact](https://example.com/contact/): phone, booking form and hours
## Optional
- [Blog](https://example.com/blog/): maintenance tips and guidesSome sites also publish an llms-full.txt file, which contains the full text of key pages in Markdown in a single document. This is a community convention rather than part of the original proposal.
Why llms.txt was proposed
Language models have limited context windows, and web pages are noisy. Navigation, cookie banners, scripts, ads and footers all surround the content that actually matters. When an AI assistant fetches a page to answer a question, much of what it reads is clutter.
llms.txt tries to solve that by giving AI tools a clean entry point: here is what this site is, here are the pages that matter, and here is what each one covers. It is particularly useful for documentation, where developers increasingly ask AI coding assistants about libraries and APIs.
llms.txt vs robots.txt vs sitemap.xml
People often confuse these three files. They do different jobs.
| File | Purpose | Format | Who reads it |
|---|---|---|---|
| robots.txt | Tells crawlers which paths they may or may not crawl | Plain text directives | Search engine and AI crawlers that respect it |
| sitemap.xml | Lists all indexable URLs to help discovery | XML | Search engines |
| llms.txt | Curated summary and key links for AI tools | Markdown | AI tools and agents that choose to read it |
The important point: llms.txt does not grant or block access. If you want to stop AI crawlers from using your content, that is a robots.txt question (and depends on each crawler respecting it). llms.txt is a suggestion about what to read, not a permission system. Google’s documentation on robots.txt covers how crawl control actually works.
Does llms.txt help SEO or AI search visibility?
Knowing the answer to “what is llms.txt” only gets you halfway. The next question is whether it makes a difference, and this is where expectations need to be realistic.
What we know
- llms.txt is not an official standard from any standards body.
- Google representatives have said publicly that Google Search does not use llms.txt, and it is not a ranking factor.
- Adoption is strongest on developer documentation sites, where AI coding tools and agents are known to fetch it when pointed at a site.
- AI assistants that browse the web on demand can read the file if they request it, like any other public URL.
What is uncertain
No major AI search provider has published clear documentation saying it uses llms.txt to choose which sites to cite. Anyone claiming the file will directly increase your AI citations is guessing.
So why add one?
Because the cost is close to zero and the possible benefits are real, if modest:
- AI agents and assistants that do read it get a clean summary of your business and the right pages to cite.
- It forces you to decide which pages actually represent your site, which is a useful content exercise on its own.
- If adoption grows, you are already set up.
What earns visibility in AI answers is still the same work that earns visibility in search: clear, well-structured, accurate content that answers real questions, on a fast and crawlable site. llms.txt is a small supporting step, not a strategy.
How to add llms.txt to your website
On WordPress with an SEO plugin
Several WordPress SEO plugins now generate llms.txt automatically. All in One SEO (AIOSEO), for example, includes an LLMs.txt setting under its Sitemaps menu. According to AIOSEO’s documentation, the feature is enabled by default, and you can choose which post types and taxonomies to include, exclude specific pages, and optionally generate an llms-full.txt version. Check that the output reflects your most important pages rather than every post on the site.
If you use AIOSEO, open the setting, review the generated file at yoursite.com/llms.txt, and trim it so it highlights your core service, product and about pages.
Manually on any site
You can also create the file by hand:
- Write the file in Markdown using the structure shown above.
- Name it llms.txt and upload it to your site’s root directory (the same place robots.txt lives), or add it through your platform’s static file feature.
- Visit yoursite.com/llms.txt in a browser to confirm it loads as plain text.
- Update it whenever you add or remove key pages.
On hosted platforms like Webflow, Framer or Shopify, options vary. Some support uploading root files or have apps for it. Others require a workaround such as a redirect to a hosted file.
What to include (and what to leave out)
A good llms.txt file is short and selective. Include:
- A one-paragraph summary of who you are, what you offer and who you serve.
- Your main service or product pages.
- Pricing, if public.
- About and contact pages.
- Your best guides or documentation, with a note on what each answers.
Leave out:
- Every blog post. Pick the ones that represent your expertise.
- Thin pages such as tag archives, author archives and paginated listings.
- Private, staging or login-only URLs.
- Marketing fluff. Write the summary like a reference entry, not an ad.
A practical recommendation
If your site runs on WordPress with AIOSEO or a similar plugin, turn the feature on, review the output and clean it up. It takes a few minutes. If you are on another platform, a hand-written file is a reasonable 30-minute task.
Then spend the real effort where it counts: content that clearly answers the questions your customers ask, structured headings, FAQ sections, accurate business details and fast pages. If you want help with that broader work, the TNCFlow growth marketing team works on SEO and AI search visibility for business sites, from technical setup to content strategy.
Frequently asked questions
Is llms.txt required?
No. It is an optional proposal, not a web standard. Your site will be crawled and indexed normally without it.
Does Google use llms.txt?
Google has said Google Search does not use llms.txt, and it is not a ranking factor. Its value today is mainly for AI tools and agents that choose to read it.
Where do I put the llms.txt file?
At the root of your domain, so it is available at yoursite.com/llms.txt, the same location as robots.txt. Many WordPress SEO plugins serve it at that address automatically.
Can llms.txt block AI from using my content?
No. llms.txt only suggests what to read. To restrict AI crawlers, use robots.txt rules for the specific user agents, keeping in mind that compliance depends on each crawler.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a short index with links and descriptions. llms-full.txt is a community convention that puts the full text of key pages into one Markdown file, so an AI tool can read everything without following links.
