What is llms.txt — the new standard that helps AI engines understand your site

A new file for a new era

You almost certainly know about robots.txt — the file that tells search engine crawlers which pages they can and cannot access. And you probably know about sitemap.xml — the file that lists all your pages so crawlers can find them efficiently.

llms.txt is the newest addition to this family. Proposed in late 2024 and adopted by a growing number of websites aiming for visibility in AI-powered search, it is a plain text file that describes your site's content in a format meant for Large Language Models (LLMs).

It is a convention, not a standard. Nobody is obliged to read it, and — as the section below sets out — one major engine has said plainly that it does not.

What robots.txt and sitemaps cannot do

robots.txt and sitemap.xml were designed for traditional web crawlers. They answer two questions: can you crawl this page? and does this page exist? They say nothing about what the content means, how it is structured, or which parts are most important.

AI assistants face a harder problem than traditional crawlers. They do not just index pages — they understand them. When a user asks "what's the best way to configure Schema.org for a hotel website?", the AI needs to find not just a page that contains those words, but a page that actually addresses that question with accurate, trustworthy content.

llms.txt is an attempt to give AI systems that orientation before they dive into your individual pages. It answers: what is this site about, who runs it, and where is the most important content? — for any system that chooses to look.

The format

llms.txt follows a simple Markdown-like structure:

# Acme Hotel Belgrade

> A 4-star hotel in Belgrade, Serbia, offering luxury rooms and conference facilities.

## Key pages

- [Home](https://acme-hotel.com/): Hotel overview and booking
- [Rooms](https://acme-hotel.com/rooms): Room types and rates
- [Restaurant](https://acme-hotel.com/restaurant): Dining options
- [Contact](https://acme-hotel.com/contact): Address and directions

## About

Acme Hotel has been operating since 1999. We accommodate both leisure and business travellers.

The file starts with a level-1 heading (the site name), a blockquote description, and then structured sections linking to the most important pages. The intent is that an AI reads it as a map — a high-level understanding of your site before it decides which pages to crawl.

Who actually reads llms.txt?

This is where most articles on the subject overstate the case, so here is what can and cannot be said.

Google does not use llms.txt. Google has stated this publicly and directly. It plays no part in Google Search or in AI Overviews, and any guide telling you that Googlebot starts by reading your llms.txt is wrong.

For every other provider, we have no confirmation either way. OpenAI, Anthropic, Perplexity and the rest have not published a commitment to read llms.txt, and we are not going to state another company's crawler behaviour as fact on their behalf. If you find a claim that a named assistant reads it, check whether it points at a source from that company.

What is true is that the file costs nothing to publish, sits at a predictable address, and is trivially readable by any agent that decides to fetch it — including the AI tools people use to ask questions about a specific website. Treat it as an emerging convention you make available, not as a channel with a guaranteed audience.

Why AI crawlers benefit from it

Without llms.txt, an agent arrives at your site cold. It crawls your homepage, follows some links, reads some pages, and builds its own picture of what you offer. It may miss your best content if it is buried in page 3 of a category archive. It may misidentify what your site is primarily about.

With llms.txt, a curated introduction is at least available: your most authoritative pages, and a precise description of who you are and what you do. Where an agent does read it, that is one less thing left to inference.

How AI Boost for Joomla handles llms.txt

Writing and maintaining llms.txt by hand requires you to keep it in sync with your site's actual content — updating it whenever you add major new pages or change your site's focus.

AI Boost for Joomla generates llms.txt dynamically and serves it at yoursite.com/llms.txt. It automatically includes your organisation name, description, and key pages from your Joomla menu and sitemap. When you add new content or update your organisation details, llms.txt updates with it.

The full-site index /llms-full.txt and Markdown versions of your pages — for agents that prefer plain text to HTML — come with it.