All docs

AI Access

An AI assistant can only answer from your site if its crawler can fetch your pages. Several things can stop it, and most of them are invisible from the front of the site. A line in robots.txt that another extension or a past developer wrote. An instruction your server attaches to every reply. A noindex left on the home page after a redesign. AI Access looks for each of these and reports what it found. It changes no setting and gives no score. Its request for your home page opens the page as a visit does, so AI Boost may bring its own section of robots.txt up to date at that moment.

It is the second screen under AI Visibility in the sidebar, after AI Discovery. You decide which crawlers may come in on Robots.txt. This screen reads the file back and tells you what it says, including the lines AI Boost did not write.

Run the check

AI Access before the first check, with the Run AI Access Check button at the top right and section 01, AI Access Check, saying no check has run yet

Until you press Run AI Access Check at the top right, section 01, AI Access Check, holds a one-line description of the check and a box saying no check has run yet. Press it and your server requests /robots.txt from your own site, the file a crawler gets, then its own home page. The first request carries a one-off parameter to get past a cache. If it fails, or a web page answers at that address, AI Boost reads the robots.txt file in your site's folder instead. It takes a few seconds. Two more sections, 02 and 03, appear under AI Access Check once it has run.

Your server asks for the home page the way a browser does, not in a crawler's name. A firewall or host that turns GPTBot away by its name while it lets browsers in does not show up here, because the check never looks like GPTBot. This screen reports what your robots.txt and your home page say to every crawler.

The result stays on the screen until you press the button again. AI Boost keeps no copy of it, so after you reload the administrator the screen starts empty. Run it again after you change anything on Robots.txt or Page SEO, and after any change on your server.

AI Boost rewrites its section of robots.txt when someone next opens a page of your site, and the check reads the file before it requests the home page. So after a change on Robots.txt, open any page of your site first, or run the check twice.

The first line of the result names the address that was fetched and the time. Look at it before you read on. A check run on a test copy of the site says nothing about the live one.

Under that line, three counts, Problem, Fine and For information, add up the rows of the next section by the word each row ends in. A count of nought carries no colour. The crawler table at the bottom is not part of them.

Access signals on your home page

This section has seven rows, and each ends in Fine, Problem or For information. More rows join them when the check finds one of the blocks described under the table.

RowWhat it looks atWhen it reads Problem
Your home page can be fetchedWhether your server could load its own home pageThe request failed, or your server answered with an error such as 403 or 500
You serve a robots.txtWhether the file existsNever. With no file every crawler may come in, and the row reads For information
AI search crawlersThe crawlers behind AI search and answers, such as OAI-SearchBot and PerplexityBotAny of them is blocked
Crawlers that fetch a page for a personThe crawlers that open a page because a reader asked, such as ChatGPT-User and Claude-UserAny of them is blocked
Crawlers that collect training dataThe crawlers that collect pages to train models, such as GPTBot and ClaudeBotNever. A block here is your choice to make, and the row reads For information
Indexing directives in the response headersAn X-Robots-Tag header on the reply your server sends with the home pageIt asks for noindex, none, nosnippet or noarchive
Indexing directives on the page itselfThe robots meta tag in the home page's headIt asks for noindex, none, nosnippet or noarchive

A crawler row that finds a block names the blocked crawlers, such as 1 of 9 blocked followed by PerplexityBot. A blocked Google-Extended stays in the training row, which then adds "Gemini will not train on your pages or use them in Gemini app answers. Google Search and AI Overviews are not affected."

The header row is the one to read closely. A header never appears in the page source, so a noindex your server sends this way is hard to spot by viewing the page.

If the first row fails, the two directive rows cannot look at the page. They say so and read For information. A failure there often means a firewall stops your server from requesting its own pages. That says little about what a crawler from outside sees, so check the site from another network before you change anything.

Googlebot and Applebot

Each gets its own row only when your robots.txt blocks it, and that row is always a Problem. A Googlebot block reads "robots.txt blocks Googlebot, so your pages drop out of Google Search and AI Overviews." An Applebot block reads "robots.txt blocks Applebot, so your pages drop out of Siri and Spotlight answers." The line that blocks it is shown under the row, such as User-agent: * → Disallow: /. Neither is in the crawler table and neither has a switch on Robots.txt, because AI Boost never writes a rule for them. The block comes from a line written outside AI Boost's section, by hand or by another extension.

Cloudflare

When the reply to the home-page request comes through Cloudflare, the first row adds "This site answers through Cloudflare. Its bot settings can stop a crawler that robots.txt allows." This screen cannot see those settings, so check them in Cloudflare itself, as Sites behind Cloudflare explains. The note changes no count.

nosnippet and noarchive

Either one, in the response headers or on the page, is a Problem, even though the page stays in search results. nosnippet keeps the page out of Google's and Apple's AI answers, and the row reads "This page asks for no snippet, so Google and Apple leave it out of their AI answers." noarchive keeps it out of Copilot answers in Bing, and the row reads "This page asks not to be archived, so Bing leaves it out of Copilot answers." Any other value, such as max-snippet, reads For information and shows the value it found.

A tag for one crawler

A page can address one crawler by name, as in <meta name="googlebot" content="noindex"> or the header X-Robots-Tag: googlebot: noindex. When such a tag asks for noindex, none, nosnippet or noarchive, it is a Problem, and the row names the crawler and what the tag asks for, such as "A meta tag for googlebot on this page asks for nosnippet." A header reads "An X-Robots-Tag header for googlebot on this page asks for noindex." The screen reads tags for Googlebot, bingbot, Applebot, and for the AI search crawlers and the crawlers that fetch a page for a person. A tag for any other name is left unread.

The AI search row does not ask who wrote a block. If you blocked PerplexityBot on Robots.txt yourself, the row still reads Problem, because that crawler cannot read your home page. The crawler table below marks the same crawler as your own decision.

AI crawler access

This section lists every crawler AI Boost knows, the 25 AI crawlers and the 12 SEO tools from Robots.txt. A line above the table says how many are blocked and names them. CCBot, OAI-AdsBot and the SEO tools appear here only, since none of the three crawler rows above counts them.

Each row has a Verdict:

  • Allowed. Nothing in the file keeps this crawler off your home page.
  • Blocked by you. The rule sits in the section AI Boost writes, so the row reports your own answer on Robots.txt back to you.
  • Blocked. A line AI Boost did not write keeps it out.

The verdict is for your home page. A crawler kept out of one folder alone still reads Allowed here.

The Crawler column gives the name a crawler sends when it asks for a page, and the company behind it. Show the rule, in the Detail column, opens a row and Hide the rule closes it. The open row names the group in your robots.txt that governs this crawler and says who decided. That is the rule AI Boost writes for you, a rule AI Boost did not write that blocks the crawler, or a rule AI Boost did not write that allows it. Then it shows the line that decided, when there is one. A crawler with no group of its own follows the * group, and the row says so.

When Cloudflare's part of your robots.txt blocks a crawler, the open row adds "Cloudflare's part of robots.txt blocks this crawler too." A crawler only Cloudflare blocks reads Blocked. One you also blocked on Robots.txt reads Blocked by you. Sites behind Cloudflare explains where those lines come from.

When a group holds both an Allow and a Disallow that reach the home page, the rule with the longer path wins, and Allow wins a tie. The open row then shows a second line, the rule that lost, so you can see why a line you expected to count did not.

A worked example

The Aldwick Lane Hotel blocks GPTBot on Robots.txt and allows everything else. On Crawlers that collect training data its check names GPTBot and reads For information, so that row lands in the For information count. In the table, GPTBot reads Blocked by you, and Show the rule opens this:

User-agent: GPTBot → Disallow: /

Say another extension later adds these lines to the top of the hotel's file:

User-agent: PerplexityBot
Disallow: /

The next check reads Problem on AI search crawlers, with PerplexityBot named, and the Problem count goes up by one. The table shows PerplexityBot as Blocked. Show the rule says AI Boost did not write that rule. AI Boost never edits lines outside its own section, so you remove those two lines yourself, or switch the setting off in the extension that wrote them.

Check it on your site

Open /robots.txt on your domain in a browser. AI Access requests the same address and tries to get past a cache, so if the browser shows something different, a cache may be serving an older copy. Clear the cache and run the check again. If its request failed, AI Access read the file in your site's folder, and that file does not hold what a CDN adds.