All docs
Sites behind Cloudflare
Checked against Cloudflare's own pages on 22 October 2026. Cloudflare changed these settings twice in 2026, so if your dashboard shows a choice this page does not name, trust the dashboard.
Two things decide which crawlers reach a site that answers through Cloudflare. Your robots.txt asks each crawler to come in or stay out. Cloudflare's bot settings turn a request away before it reaches your server. So a crawler your robots.txt allows can still be stopped, and AI Boost cannot see that from inside your site. When your site answers through Cloudflare, AI Access adds a line to its first row, "This site answers through Cloudflare. Its bot settings can stop a crawler that robots.txt allows."
Search, Agent and Training
Cloudflare sorts AI crawlers into three groups, in your domain's Security settings. Search crawlers index your content to answer questions later. Agents act for a person right now, such as an assistant opening your page for a reader. Training crawlers take your content to train or fine-tune a model.
| Setting | Choose | What another choice costs |
|---|---|---|
| Search | Allow | Block takes your pages out of AI search answers. |
| Agent | Allow | Block means an assistant asked to open your page gets nothing. |
| Training | Allow, or Disallow AI Training | Block also stops Googlebot, Bingbot and Applebot. |
Never set Training to Block
Since 15 September 2026, Training's Block also stops Googlebot, Bingbot and Applebot, because each of them crawls for search and for AI at once. Block on pages with ads does the same on those pages. Your site then drops out of Google Search, Bing and Apple's search, not only out of AI training.
To refuse training, choose Disallow AI Training. Cloudflare then adds a no-training line to the robots.txt it serves, keeps Googlebot, Bingbot and Applebot in for search, and blocks the crawlers that only collect training data. Cloudflare moved a Training choice of Block made before that date to Disallow AI Training, so read what your domain shows now.
Cloudflare puts its lines above yours in the robots.txt it serves. AI Boost writes no Allow line, so nothing in its section cancels them. AI Access reads the file on your server, so it does not see the lines Cloudflare adds.
Bot Fight Mode challenges PerplexityBot
Bot Fight Mode, Cloudflare's bot protection on Free plans, answers traffic that looks like a known bot with a challenge. Cloudflare removed Perplexity from its verified bots in August 2025, and with Bot Fight Mode on, PerplexityBot gets challenged. Your pages can then drop out of Perplexity's answers while your robots.txt allows it.
Cloudflare says a WAF custom rule cannot skip Bot Fight Mode. If Perplexity's answers matter to you, switch Bot Fight Mode off. Super Bot Fight Mode, Cloudflare says, supports Skip rules.
Check it on your site
Open /robots.txt on your domain in a browser. That is the file crawlers get, with Cloudflare's lines in it. Then run AI Access in AI Boost. It reads the file on your server, so a block that shows in the browser and not on AI Access came from Cloudflare, or from a cache in front of your site.
Cloudflare's own pages
- Bot Fight Mode, last updated 3 August 2026.
- The Search, Agent and Training settings, 1 July 2026.
- Training's Block and Disallow AI Training, 15 September 2026.
- Bot Preference Sync, the lines Cloudflare adds to robots.txt, 21 August 2026.
- Perplexity removed from the verified bots, 4 August 2025.
New to AI Boost? Start with the Getting started guide, or read how AI Boost writes your crawler rules in Robots.txt.