AI BoostAI Boost
Get AI Boost
FeaturesPricingDocsToolsBlogFAQGet AI Boost
robots.txtAI Crawlers

What is robots.txt — and why AI crawlers need special rules in 2026

robots.txt is the access control file for web crawlers. With AI crawlers now a separate group from search crawlers, getting your robots.txt right matters more than it used to.

9 May 20266 min read

The original gatekeeping file

Every website that has ever had a robots.txt file at its root has been using a standard that dates back to 1994. The Robots Exclusion Protocol (REP) was one of the first informal agreements of the web: crawlers should check /robots.txt before indexing a site and respect the rules they find there.

The file syntax is simple:

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

User-agent: * means "all crawlers". Disallow: /admin/ means "do not crawl anything under /admin/". Allow: / means "everything else is fine".

Simple in principle. More consequential in 2026.

The AI crawler explosion

In 2024 and 2025, the number of distinct web crawlers grew dramatically. Every major AI company deployed its own crawler:

  • GPTBot — OpenAI (ChatGPT, GPT-4)
  • ClaudeBot — Anthropic (Claude)
  • PerplexityBot — Perplexity
  • Google-Extended — Google's opt-out control for Gemini and Vertex AI training
  • Applebot-Extended — Apple AI features
  • Bytespider — ByteDance (TikTok AI)
  • CCBot — Common Crawl (training data)
  • DataForSeoBot, SemrushBot, AhrefsBot — SEO tools

...and dozens more. Some are search crawlers. Some are training data collectors. Some are fact-checking systems.

For many sites, the question is no longer just "which pages can Google access?" but "which of these crawlers do I want to allow, and under what conditions?" AI Boost ships a per-bot rule for 17 of them — the ones with a documented user-agent and a real reason to treat them separately.

The two failure modes

Failure mode 1: Blocking AI crawlers that bring traffic. Some site owners, concerned about AI systems using their content, added Disallow: / for GPTBot and others. The problem: GPTBot is not just used for training data — it is also used when ChatGPT Browse answers user questions by reading live web pages. Blocking GPTBot means your site cannot be cited in ChatGPT answers.

Failure mode 2: Allowing all crawlers unconditionally. An unrestricted robots.txt allows every crawler, including aggressive training data harvesters that may stress your server, bypass your terms of service, and use your content in ways you have not authorised.

A more useful approach is to separate the two jobs: allow the answering crawlers — the ones that fetch a page to answer a question someone has just asked, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot and DuckAssistBot — while deciding separately about the training crawlers, such as GPTBot, ClaudeBot, CCBot and Google-Extended. Note that GPTBot and ChatGPT-User are different user-agents doing different jobs, so blocking one does not block the other.

The Sitemap declaration

robots.txt also serves a secondary purpose: it should declare where your sitemap lives:

Sitemap: https://yoursite.com/sitemap.xml

That one line points every crawler that reads robots.txt at your complete list of pages.

Joomla's default robots.txt

Joomla installs a static robots.txt by default. It is outdated — written for an era when Googlebot and Bingbot were the only crawlers worth considering. It does not mention AI crawlers at all.

Updating it manually means editing a file on your server and maintaining it yourself every time a new crawler appears.

How AI Boost for Joomla handles robots.txt

AI Boost for Joomla replaces Joomla's static robots.txt with a dynamically generated one. When a crawler requests yoursite.com/robots.txt, the plugin serves a file that includes:

  • Standard Joomla path restrictions (/administrator/, /cache/, etc.)
  • An explicit rule for each of the 17 AI crawlers it knows about, allow or block, as you set them
  • One-click blocking for 11 SEO and audit crawlers
  • A Sitemap: declaration pointing to your dynamic sitemap

The file is regenerated automatically when you save plugin settings, and cached for performance. You never need to touch the server file again.

For site owners who want full control, the plugin provides a custom robots.txt field where you can add your own rules that are appended to the generated output.