Which crawlers matter for AI visibility?
The main ones are GPTBot and OAI-SearchBot for ChatGPT, ClaudeBot for Claude, PerplexityBot for Perplexity, and Google-Extended for Google's AI features.
Each is identified by its user-agent in your robots.txt, so you can allow or block them individually.
How do I check and allow them?
Open your site's robots.txt (at yourdomain.com/robots.txt) and check none of these user-agents are disallowed. If they are, remove those rules to let them crawl.
This is a site-level setting - it's the one part of AI visibility your content can't fix on its own.
Should you allow AI crawlers at all?
It's a genuine trade-off. Blocking them keeps your content out of AI training and answers, but it also removes any chance of being cited as the source, which is where the traffic and authority now flow.
For most businesses that want to be found in AI search, allowing them is the right call.
Frequently asked questions
Should I allow AI crawlers?
If you want to be cited in AI search, yes. Blocking them protects content from training but also removes you from the answers - and the citations - entirely.
How do I check my robots.txt?
Visit yourdomain.com/robots.txt and look for any Disallow rules under GPTBot, ClaudeBot, PerplexityBot or Google-Extended. An AI-visibility check flags this automatically.
Does blocking AI bots protect my content?
It reduces AI access to it, but it can't fully prevent reuse, and it costs you all AI-search visibility. Weigh that trade-off deliberately.
See how your pages score with AI search
Paste a URL into the free AI Visibility Check - the same check RankMoss runs on every post it writes.
Run the free check