Zarlana Check my site

AI bot checker

Does your robots.txt let ChatGPT, Claude, Perplexity and Google's AI read your website? Enter your address to see the rule for 18 AI bots, and what each one does.

Three kinds of AI bot

Blocking "AI bots" in one go usually blocks the wrong thing. They do three different jobs:

  • Answer bots (ChatGPT-User, Claude-User, Perplexity-User) open a page because someone asked the assistant a question just now. Block them and the assistant can't read you while it answers.
  • Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) build the index the assistants search. Block them and you drop out of the sources they choose from.
  • Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot and others) collect text to train future models. Blocking them doesn't remove you from today's answers.

A common setup for a business that wants to be recommended: allow the answer and search bots, and decide on the training bots separately.

How to allow or block one bot

Add a group for it to the robots.txt at the root of your site. This blocks OpenAI's training bot and leaves ChatGPT's answer and search bots alone:

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
User-agent: OAI-SearchBot
Allow: /

A bot follows the group that names it, and falls back to User-agent: * only when no group names it. If your site is on Cloudflare, also check Security → Bots: its "Block AI bots" switch can turn some of these bots away before robots.txt is read.

Allowed isn't the same as read

robots.txt says who may come in, not who does. To see which pages ChatGPT and Claude actually open, read your server log with the AI log checker. To see whether they then name you, run a free AI visibility check.