robots.txt & AI Crawlers

Overview

robots.txt is the access control file for web crawlers. Every well-behaved crawler — from Googlebot to GPTBot — checks this file before indexing your site. With AI crawlers now a distinct group from search crawlers, getting your robots.txt right matters for both traditional SEO and AI search visibility.

Enabling Dynamic robots.txt

Go to Crawler Rules → Robots & AI Crawlers and set Enable Dynamic robots.txt to Yes.

AI Boost serves a dynamically generated robots.txt at yoursite.com/robots.txt. This replaces any static robots.txt file you may have.

Note: If you have a physical robots.txt file in your Joomla site root, remove it. The dynamic version from AI Boost takes precedence, but removing the static file avoids confusion.

What the Generated robots.txt Includes

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /tmp/
Disallow: /installation/
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

AI Boost writes an explicit rule for each of the 17 AI crawlers it knows about, so none of them is accidentally caught by a broad Disallow rule aimed at Joomla's system paths.

The AI Crawler Rules card

Crawler Rules → Robots & AI Crawlers lists the 17 AI crawlers AI Boost can write a rule for. Each one is a single allow/block switch — no file editing.

Training bots collect content to train a model; answering bots fetch a page to answer a question a person has just asked. Blocking a training bot while allowing an answering bot is a common choice — it keeps you visible in AI answers while opting out of model training.

These are AI crawlers, not ordinary search crawlers. Googlebot and the classic Bing, Yandex or Baidu search crawlers are governed by the general rules at the top of the file, not by this card.

The Scraper Rules card

Crawler Rules → Scraper Rules blocks 11 SEO and audit crawlers in one click — Ahrefs, Semrush, Dotbot, Majestic, BLEXBot, Moz, Screaming Frog, Sitebulb, SE Ranking, Serpstat and PetalBot. Blocking them does not affect how your site appears in search or in AI answers; it only stops third-party SEO tools from profiling your site.

Staging and development copies

When you are working on a staging or development copy of your site, you do not want it competing with the live site.

Go to Output Conflicts and switch on Disable all AI Boost output. From that moment the copy publishes nothing from AI Boost: no structured data, no OpenGraph tags, no canonical links, no sitemap and no llms.txt — /sitemap.xml and /llms.txt return 404. Switch it back off when the site goes live.

This switch turns off AI Boost's own output. It is not a robots.txt block, so it does not by itself stop a search engine crawling the staging site — for that use your host's or Joomla's access controls (an HTTP password on the staging domain is the reliable option), or add a Disallow: / rule of your own in the custom-rules box below.

There is no separate "Staging Mode" setting, and AI Boost does not try to guess whether a domain is production or not.

Custom robots.txt Rules

Crawler Rules → Robots & AI Crawlers has a custom-rules box at the bottom. Anything you put there is appended to the generated file, so you can add your own User-agent / Disallow / Allow lines for specific crawlers or paths.

Troubleshooting

robots.txt showing old content: Clear Joomla's cache (System → Clear Cache) and reload yoursite.com/robots.txt.

AI crawlers still blocked: If you previously had a static robots.txt with Disallow: /, check that the static file has been removed from the server root. Some caching plugins or CDNs may also cache the old robots.txt — flush your CDN cache.

Google Search Console robots.txt fetcher error: The GSC robots.txt tester fetches the file with a specific user agent. Ensure your hosting does not block the GSC user agent. AI Boost's dynamic robots.txt serves correctly to all user agents.