Your `robots.txt` file now decides whether ChatGPT, Claude, Perplexity, and Google's AI answers can see your brand. By the end of this guide you will know exactly which AI user-agents to allow or block, have copy-paste `robots.txt` blocks for each one, and know how to verify the rules are actually working in your server logs.
The hard part is that "block all AI bots" is usually the wrong call. Blocking the crawlers that power AI answers can quietly erase you from the results your customers now read instead of Google. So the goal is not a wall — it is a deliberate policy: welcome the bots that cite you, and decide case by case about the ones that only train on you.
Key takeaways
- `robots.txt` is the primary control for AI crawler access, but it is voluntary — well-behaved bots obey it, and it does not stop scrapers that ignore the standard. - AI bots split into two jobs: training bots that feed a model's static knowledge, and search/RAG bots that fetch live pages to build an answer. Treat them differently. - Blocking search/RAG bots like `OAI-SearchBot` and `PerplexityBot` removes you from the AI answers where they cite sources — this directly costs you visibility. - `Google-Extended` only controls Gemini/Vertex training; it does not remove you from Google AI Overviews, which are served through the normal `Googlebot`. - A directive is not proof. Confirm real bot behavior in server logs before you trust any rule.
Step 1: Know the two kinds of AI crawler
Before editing anything, sort the bots by what they do with your pages.
Model-training bots scrape content to train large language models. The data becomes part of the model's static knowledge. Being included helps with general "what is brand X" style answers, but it does not guarantee a citation. Examples:
- `GPTBot` — OpenAI model training - `ClaudeBot` and `anthropic-ai` — Anthropic model training - `Google-Extended` — Gemini/Vertex AI training - `Applebot-Extended` — Apple Intelligence training - `CCBot` — Common Crawl, a dataset many models train on - `Bytespider` — ByteDance crawler
AI search / RAG bots fetch live pages at answer time to build a grounded, cited response. Blocking these is what actually removes you from AI answers:
- `OAI-SearchBot` — OpenAI's ChatGPT search/browsing fetcher - `PerplexityBot` — Perplexity's indexing/answer crawler - `Googlebot` — also renders AI Overviews and AI Mode
The practical rule: be generous with search/RAG bots, be deliberate with training bots.
Step 2: Locate or create your robots.txt
The file lives at the root of your domain: `https://yourdomain.com/robots.txt`. It must be plain text, served with a `200` status.
- On a custom site, add or edit `robots.txt` in your web root. - On Shopify, the file is generated but editable through `robots.txt.liquid` in your theme. - On WordPress, an SEO plugin (Yoast, Rank Math) usually manages a virtual `robots.txt`.
