AI crawler policy is deliberate

What this means

Your robots.txt names at least one AI crawler, such as GPTBot, ClaudeBot, PerplexityBot or Google-Extended, with an explicit Allow or Disallow under its own User-agent: line. Passing means you decided whether answer engines may read your site. Not passing means the decision was made by whatever template you started from, which is an accident rather than a policy.

How we test it

We fetch your robots.txt, following at most 3 redirects, and look for User-agent: blocks that name an AI crawler (GPTBot, ClaudeBot, Google-Extended, PerplexityBot and similar) followed by an explicit Allow: or Disallow: line. At least one such block passes, whichever way it decides. None found is reported as info, not a failure, because leaving it unstated is a choice we flag rather than grade.

Why it matters

Being cited by answer engines starts with letting them in on purpose. The Fix Kit notes that most old robots.txt templates block the AI crawlers by accident, and that a CDN can block them before robots.txt is ever read, so the file is only half of the decision. If you want to be cited, say so in the file and check the CDN setting; if you want to opt out of training crawlers, say that explicitly too, and keep the search crawlers allowed either way.

The fix

Add an explicit User-agent: block with Allow: / (or Disallow: / if you are opting out) for each of GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot and Applebot-Extended to your robots.txt, as laid out in templates/robots.txt, keeping your existing Disallow: lines for private paths. Then check by hand in Cloudflare under Security, Bots whether the AI crawler block is on, because that setting overrides robots.txt and your code cannot see it. The full runbook is in the Fix Kit, Module 02.

Prompt for your AI

Paste this into the coding agent that built the app. It reproduces the finding and stops before changing anything; the fix steps are in the kit.

Do not change anything yet. This is a verification pass only.
Run: curl -s https://YOUR_DOMAIN/robots.txt and print the whole file.
Run: curl -s https://YOUR_DOMAIN/robots.txt | grep -A1 -iE "GPTBot|ClaudeBot|PerplexityBot|Google-Extended"
For each AI crawler named, tell me whether the line under it is Allow or Disallow.
Tell me whether any Disallow line under User-agent: * would block those crawlers even without a named block.
Find the file in the repo that produces robots.txt and show me its path.
List what it would take to state an explicit policy for each AI crawler, without editing anything.
Verify: the grep prints at least one AI crawler user agent followed by an Allow or Disallow line; if it prints nothing, report that no policy is stated and stop.