AI/LLM scraping restrictionsWhat it checks and how to fix it

What it checks

Reads robots.txt, the robots meta tags and the X-Robots-Tag header for rules aimed at AI crawlers: citation bots such as GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot and Google-Extended, training bots such as CCBot, Bytespider and Applebot-Extended, Cloudflare's Content-Signal, and noai / noimageai directives. The verdict follows the project's AI policy: when it allows AI crawlers, the check passes with no blocking signals and warns when any are present; when it blocks them, it passes with a restriction in place and warns without one.

Why it matters

AI crawlers come in two kinds. Citation bots fetch pages to answer questions in ChatGPT, Perplexity, Claude or Google AI Overviews and link to their sources, so blocking them removes your site from those answers. Training-only bots collect data for model training, and blocking them has no effect on AI search visibility. A rule copied from a template, or a default switched on at your CDN, can block the first kind when you only meant to stop the second.

How to fix it

Decide what you want, then make robots.txt, your meta tags and your CDN settings all say the same thing.

To be cited in AI answers, allow the citation bots explicitly and remove any noai meta tag or X-Robots-Tag: noai header:

# robots.txt
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /

If you use Cloudflare, check Security → Bots → Block AI bots and the managed robots.txt setting: either one can block these crawlers without you editing the file.

To keep content out of model training but stay in AI search, block only the training crawlers:

# robots.txt
User-agent: CCBot
User-agent: Bytespider
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /

To opt out of AI use altogether, disallow the citation bots as well and add the meta tag to every page. Expect your site to drop out of AI-generated answers:

<meta name="robots" content="noai, noimageai">