AI Bot Access Checker
See at a glance which AI crawlers and shopping agents your robots.txt allows or blocks — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more. Accidentally blocking them makes you invisible to AI.
How the AI Bot Access Checker works
Why bot access matters
AI shopping starts with discovery, and discovery starts with your robots.txt. If GPTBot, Google-Extended or ClaudeBot are disallowed, you quietly disappear from AI answers and agentic shelves — even a perfect UCP manifest won't help if agents can't crawl you.
This tool fetches your robots.txt and evaluates whether each major AI user-agent can reach your site root, grouped by what they power: answer/search engines, agentic shopping agents, and training crawlers. Blocking training crawlers can be intentional; blocking answer and shopping agents usually isn't.
Tips
- ✓Don't block answer & shopping agents: OAI-SearchBot, PerplexityBot, Google-Extended, ClaudeBot and Mariner drive AI-referred traffic and orders. Allow them.
- ✓Blocking training bots is a valid choice: If you only want to keep content out of training sets, block GPTBot/CCBot/Bytespider while still allowing the search agents.
- ✓Check after every CDN change: Cloudflare, Vercel and WAF rules can block AI bots even when robots.txt allows them. Re-check after infra changes.
AI Bot Access Checker — FAQ
Why would GPTBot being blocked matter?+
If GPTBot and other AI crawlers can't fetch your pages, your store won't be represented in AI answers and agentic shopping surfaces — you become invisible exactly where shoppers are moving.
What's the difference between the bot categories?+
Answer/search bots power AI search and citations; shopping agents act on commerce surfaces; training crawlers collect data for model training. Blocking training bots protects content without hurting AI-search visibility.
Does this check CDN or firewall blocks?+
It evaluates robots.txt rules. CDN/WAF blocks (Cloudflare, Vercel) can also stop bots even when robots.txt allows them — check those separately after infra changes.
Is it free?+
Yes, free and no signup required.