AI BoostAI Boost
Get AI Boost
robots.txtAI Crawlers

What robots.txt is, and how to handle AI crawlers in it

robots.txt tells crawlers where they may go. AI companies now run several crawlers each, for different jobs, and blocking the wrong one costs you.

9 May 2026Updated 22 October 20264 min read

A file from 1994

robots.txt dates from 1994, when Martijn Koster proposed the Robots Exclusion Protocol. The IETF published it as RFC 9309 in 2022. A crawler that follows it fetches /robots.txt before anything else and keeps to the rules it finds there.

The syntax is small:

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

User-agent: * addresses every crawler. Disallow: /admin/ asks them to stay out of everything under /admin/. Allow: / says the rest is fine.

People get two things wrong about it. It is a request, not a lock, and a crawler that ignores it can still fetch your pages. It also controls crawling, not indexing. A blocked page can still appear in Google if other sites link to it, and a noindex tag is the tool for keeping it out.

One company, several crawlers

AI companies do not run one crawler each. Several run separate user-agents for separate jobs, and the job decides what a block costs you.

OpenAI is the clearest case:

  • GPTBot collects pages that may be used to train OpenAI's models.
  • OAI-SearchBot crawls for ChatGPT search, the answers that come with linked sources.
  • ChatGPT-User fetches a page when a person in ChatGPT asks about it. OpenAI says robots.txt rules may not apply to it, because a person started the fetch.
  • OAI-AdsBot checks pages submitted as ChatGPT ads against OpenAI's ad policies. OpenAI says what it collects is not used to train its models.

Anthropic has the same split, with ClaudeBot for training, Claude-SearchBot for search and Claude-User for fetches a person asked for. Perplexity runs PerplexityBot for its search index and Perplexity-User for pages a person asks about, and says Perplexity-User generally ignores robots.txt rules.

Meta splits its crawlers too. meta-externalagent collects pages to train its AI models, meta-webindexer indexes pages for Meta AI search, and meta-externalfetcher fetches a link a person asked for and may bypass robots.txt. FacebookBot is no longer on Meta's list of crawlers.

Some names are not crawlers at all. Google-Extended is a token you address in robots.txt to tell Google whether pages Googlebot already crawls may be used to train Gemini models and to ground Gemini's answers. Blocking it does not remove you from Google Search or from AI Overviews. Applebot-Extended does the same job for Apple's models.

Then there are the dataset and training crawlers of companies without a search product you would care about, such as CCBot from Common Crawl and Bytespider from ByteDance.

The mistake that costs visibility

Many site owners added Disallow: / for GPTBot after OpenAI announced it in 2023, to keep their text out of training. That does what they meant. It does not remove the site from ChatGPT search, which uses OAI-SearchBot.

The reverse mistake is the expensive one. Block OAI-SearchBot, or paste a list that blocks every OpenAI name, and OpenAI says your pages will not appear as sources in ChatGPT search answers.

So decide per job.

  • Crawlers that build the index AI search answers from, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot and DuckAssistBot, are the ones that can send a reader to your page. Most sites want them in.
  • Training crawlers such as GPTBot, ClaudeBot, CCBot and Bytespider, plus the Google-Extended token, decide whether your text feeds the next model. That is a business decision, and OpenAI, Anthropic and Google each keep it separate from their search crawler.
  • Fetchers that act for one person, such as ChatGPT-User, Perplexity-User and meta-externalfetcher, may not obey a block at all. OpenAI, Perplexity and Meta each say so on their own crawler pages.
  • SEO tools such as AhrefsBot and SemrushBot crawl to build backlink and keyword databases. Blocking them changes nothing in Google and hides your site from competitors' research, and from your own audits if you use those tools.

Point crawlers at your sitemap

robots.txt can also say where your sitemap lives:

Sitemap: https://yoursite.com/sitemap.xml

Any crawler that reads the file then knows where your full list of pages is.

Joomla's robots.txt

The robots.txt that comes with Joomla keeps crawlers out of system folders such as /administrator/ and /cache/. It says nothing about AI crawlers, and it cannot know about user-agents that appear after you install it. Keeping it current by hand means editing a file on the server every time a new one turns up.

robots.txt in AI Boost for Joomla

AI Boost for Joomla switches robots.txt on by itself the first time it runs, so AI search and answer crawlers can read your site. It keeps one marked block at the end of the file, creating the file if there is none. Your own lines stay as you wrote them, and switching the feature off removes the block again.

You answer 25 AI crawlers one by one, allow or block, and 12 crawlers from SEO tools the same way. Every one starts at Allow. The AI crawlers sit in four groups by job, training, AI search, live browsing and ad checks, and the SEO tools' crawlers in a fifth. Each group has Allow all and Block all. The AI list labels each crawler's job, so GPTBot shows as training, OAI-SearchBot as search and ChatGPT-User as a page fetched because a person asked. It covers:

  • GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot from OpenAI
  • ClaudeBot, Claude-SearchBot and Claude-User from Anthropic
  • PerplexityBot and Perplexity-User from Perplexity
  • Amzn-SearchBot, Amzn-User and Amazonbot from Amazon
  • MistralAI-Index, MistralAI-User and MistralAI-Training from Mistral
  • meta-externalagent, meta-webindexer and meta-externalfetcher from Meta
  • Google-Extended, Applebot-Extended, bingbot, DuckAssistBot, YouBot, CCBot and Bytespider

A blocked crawler gets a User-agent line and Disallow: /. An allowed crawler gets no line at all, so the block never loosens a rule you wrote yourself. While the whole file, your own lines included, lets every search and browsing crawler in, one comment in the block says AI Boost lets AI search and answer crawlers read this site. The block also carries a Content-Signal line, which AI Boost starts at search=yes, ai-input=yes and leaves training for you to answer. When the sitemap is on, the block also carries a Sitemap: line. AI Boost rewrites the block when a page on your site is served and the result would differ from the file, so you never edit it by hand.

Crawlers that ignore robots.txt are outside its reach. Stopping those takes a firewall or your host.