All docs

Robots.txt

Before a crawler reads your site, it reads /robots.txt, a plain file in your site's root that says who may come in. Search engines, AI companies and SEO tools all send their own crawlers, and each one looks for its own name in that file.

The crawlers do different jobs, and that is the whole reason to answer them one by one. One collects your pages to train next year's AI model. Another fetches one page this second because a person asked an assistant about you. A third builds a competitor's backlink report. You may well want the second and not the first.

Robots.txt is the last screen under SEO in the sidebar. It has three numbered sections.

Let AI Boost write into the file

Section 01, robots.txt Management. The Preview robots.txt button sits above the Let AI Boost write into robots.txt switch, which is on. Below it the Usage Signals panel, with Write usage signals into robots.txt on, Search indexing and Real-time AI answers at Yes, allowed, AI model training at No, not allowed, and the robots.txt preview box showing the Content-Signal line those answers write.

Let AI Boost write into robots.txt is already on. AI Boost switches it on once, the first time it runs, on a new site and on one that updates, because AI search and answer crawlers need to read your site before they can cite it. The section then says so in one line, "AI Boost switched on robots.txt so AI search and answer crawlers can read your site. Training crawlers are your choice." The line goes once you save a change on this screen.

AI Boost switches it on only that once. If you switch it off, it stays off. Remove all settings on Backup & Migration switches it on again, as on a new install. A crawler you had already blocked stays blocked. Nothing else on this screen reaches a crawler while the switch is off, and Usage Signals and AI Crawler Rules each say so.

AI Boost then keeps one marked section at the end of your robots.txt, as soon as there is something to write in it. It never touches a line outside that section, so your own rules, and the rules Joomla ships with, stay exactly as they are. Switch it off and the section goes, usage signals included, leaving the rest of the file as it was.

AI Boost writes the file while it serves a visitor, not while you work in the administrator. After a change, AI Boost rewrites the section on the next visit to your site.

Press Preview robots.txt to read the file your answers produce before any crawler does. The window gives the file's address with an Open button, counts the lines and bytes in AI Boost's section, the User-agent groups and the Sitemap lines, and shows the file itself. Under it, it flags four problems when it finds them:

  • There is no robots.txt on the site at all.
  • A rule AI Boost did not write tells every crawler to stay out of the whole site.
  • The XML Sitemap is on, but the file does not name it.
  • The file treats some crawlers differently from your answers here, because another extension or a rule written by hand decides for them.

Usage signals

The Usage Signals panel inside the first section answers a different question, what a crawler may do with what it reads. Switch on Write usage signals into robots.txt and three answers appear.

  • Search indexing, for building a search index and showing links and short excerpts in search results. An answer an AI writes counts under Real-time AI answers.
  • Real-time AI answers, for feeding a page into an assistant's answer to a question asked right now.
  • AI model training, for using your content to train or fine-tune a model.

Each takes Yes, allowed, No, not allowed or No preference, which writes nothing. AI Boost adds a short group for every crawler to its section, ending in one line that carries each answer set to Yes or No. The robots.txt preview box under the answers shows that line as you change them.

When AI Boost switches robots.txt on, it switches this panel on too. It sets Search indexing and Real-time AI answers to Yes, allowed, unless you had already answered them, and leaves AI model training at No preference. Until you change them, the line reads Content-Signal: search=yes, ai-input=yes. Training is yours to answer.

The hotel allows the first two and refuses training, and the line reads:

Content-Signal: search=yes, ai-input=yes, ai-train=no

The format is Content Signals, published by Cloudflare. The IETF's AIPREF working group is drafting a different line, Content-Usage, which AI Boost does not write. Like everything in robots.txt the line is a request. A crawler that honours it follows it, and one that does not is not stopped by it. To ask one crawler to stay out altogether, use the per-crawler answers below.

AI Crawler Rules

Section 02, AI Crawler Rules. The Training AI models and building datasets group, with Allow all and Block all beside its heading and nine crawlers, GPTBot to Bytespider with MistralAI-Training among them, each set to Allow. Google-Extended carries the note that it decides whether Gemini trains on your pages and uses them in Gemini app answers. Below them the heading of the AI search and answers group.

This section names the 25 AI crawlers AI Boost can write a rule for, in four groups by the job they do. Each group says under its heading what its crawlers do. Every crawler starts at Allow.

GroupCrawlersWhat a block does
Training AI models and building datasetsGPTBot, ClaudeBot, MistralAI-Training, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, BytespiderAsks them not to use your pages for training models or building datasets. Google-Extended decides whether Gemini trains on your pages and uses them in Gemini app answers. Google-Extended and Applebot-Extended fetch nothing themselves. They tell Google and Apple how to use what their search crawlers already read, so your place in Google Search, AI Overviews and Apple's search does not change.
AI search and answersOAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot, MistralAI-Index, meta-webindexer, bingbot, DuckAssistBot, YouBotYou drop out of the answers those assistants give.
Live browsing at a reader's requestChatGPT-User, Claude-User, Perplexity-User, Amzn-User, MistralAI-User, meta-externalfetcherAsks them not to open your page when a reader asks an assistant to. Claude-User follows the block. OpenAI, Perplexity and Meta say theirs may not, because a person asked for the page.
Ad checksOAI-AdsBotIt visits only pages submitted as ads in ChatGPT, to check they are safe. If you do not advertise in ChatGPT, it does not visit.
The nine crawlers of the AI search and answers group, OAI-SearchBot to YouBot with Amzn-SearchBot and MistralAI-Index among them and a note under bingbot. Then the Live browsing at a reader's request group with six crawlers, ChatGPT-User to meta-externalfetcher with Amzn-User and MistralAI-User among them, and the Ad checks group with OAI-AdsBot. Every one is set to Allow.

Each crawler takes Allow or Block, and each group has Allow all and Block all. Answer a group at a time. A blanket answer treats the assistant fetching your page for a reader this second the same as the archive collecting it for next year.

Block is a request in robots.txt. Honest crawlers follow it; a firewall stops the rest. And a firewall can stop a crawler you allow here. If your site answers through Cloudflare, its bot settings decide as well, and Sites behind Cloudflare says which ones keep AI search.

One row carries a note worth reading. bingbot is Microsoft's AI crawler and also the crawler behind ordinary Bing search, so blocking it takes you out of Bing's results as well.

Googlebot and Applebot are not on the list, and AI Boost never writes a rule for them. Blocking Googlebot takes your site out of Google Search and AI Overviews, and blocking Applebot takes it out of Siri and Spotlight answers. If a line written elsewhere in your robots.txt blocks either, AI Access reports it as a Problem.

SEO tools

Section 03, SEO tools. The SEO research tools group, with Allow all and Block all beside its heading and every crawler set to Allow. Screaming Frog SEO Spider, Sitebulb and SiteAuditBot carry a note that they are auditing tools you run against your own site, and PetalBot a note that it also indexes pages for a web search engine.

The last section holds one group, SEO research tools. It takes the same answers for 12 crawlers that build backlink and ranking databases for SEO tools, such as AhrefsBot and SemrushBot. They are not search engines, so blocking them costs you no ranking and no visitors, and on a small server it takes their traffic off it.

Read the notes before Block all. Screaming Frog SEO Spider, Sitebulb and SiteAuditBot may be tools you run against your own site, and blocking them blocks your own reports. PetalBot also indexes pages for Huawei's search engine, so blocking it takes you out of that engine's results.

What AI Boost writes

The hotel's file keeps Joomla's own lines and ends with this section:

# BEGIN AI Boost for Joomla - managed block (aiboostnow.com)
# Written by AI Boost from your settings. Change your settings, not this block. Anything you write outside it is left alone.

# AI crawlers
# AI Boost lets AI search and answer crawlers read this site.

# SEO tools and audit crawlers
# None of these is being asked to stay out.

# How this site's owner asks automated systems to use its content.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no

# Sitemap
Sitemap: https://example.com/sitemap.xml
# END AI Boost for Joomla - managed block

The comment under the AI crawlers heading is written only while the whole file lets every AI search and answer crawler in, your own lines included. Block GPTBot and a group appears under it:

# GPTBot (OpenAI) - collects pages to train a model on
User-agent: GPTBot
Disallow: /

Block a search or browsing crawler, such as PerplexityBot, here or in a line of your own, and the comment goes. So does a line of yours such as User-agent: * with Disallow: /. Among the crawler rules, AI Boost writes a group only for a crawler you block. It never writes an Allow line, so its section cannot loosen a rule you wrote yourself. The Sitemap line appears while Enable XML Sitemap is on, on the XML Sitemap screen.

Check it on your site

Open /robots.txt on your domain. AI Boost writes its section when the next page of your site is served, so if it is missing, open any page once and look again. If it is still missing, your host may not let AI Boost write the file, and the error log on Debug says so. A CDN in front of the site can also serve an old copy for a while.

To see what the file does to each AI crawler, open AI Access and press Run AI Access Check. It reads the robots.txt your site serves and gives every crawler one of three verdicts, Allowed, Blocked or Blocked by you. Blocked by you means the rule is one AI Boost wrote on your answer. Show the rule opens the line that decided it, including a line you wrote yourself outside AI Boost's section.