configure robots.txt for AI crawlers (GPTBot, ClaudeBot) β€” a practical GEO how-to guide from GEOly AI (geoly.ai)

How to Configure robots.txt for AI Crawlers (GPTBot, ClaudeBot)

Summary

Copy-paste robots.txt blocks for GPTBot, ClaudeBot, OAI-SearchBot and more β€” plus which AI bots to allow so you stay citable, and how to verify it in your logs.

Your `robots.txt` file now decides whether ChatGPT, Claude, Perplexity, and Google's AI answers can see your brand. By the end of this guide you will know exactly which AI user-agents to allow or block, have copy-paste `robots.txt` blocks for each one, and know how to verify the rules are actually working in your server logs.

The hard part is that "block all AI bots" is usually the wrong call. Blocking the crawlers that power AI answers can quietly erase you from the results your customers now read instead of Google. So the goal is not a wall β€” it is a deliberate policy: welcome the bots that cite you, and decide case by case about the ones that only train on you.

Key takeaways

- `robots.txt` is the primary control for AI crawler access, but it is voluntary β€” well-behaved bots obey it, and it does not stop scrapers that ignore the standard. - AI bots split into two jobs: training bots that feed a model's static knowledge, and search/RAG bots that fetch live pages to build an answer. Treat them differently. - Blocking search/RAG bots like `OAI-SearchBot` and `PerplexityBot` removes you from the AI answers where they cite sources β€” this directly costs you visibility. - `Google-Extended` only controls Gemini/Vertex training; it does not remove you from Google AI Overviews, which are served through the normal `Googlebot`. - A directive is not proof. Confirm real bot behavior in server logs before you trust any rule.

Step 1: Know the two kinds of AI crawler

Before editing anything, sort the bots by what they do with your pages.

Model-training bots scrape content to train large language models. The data becomes part of the model's static knowledge. Being included helps with general "what is brand X" style answers, but it does not guarantee a citation. Examples:

- `GPTBot` β€” OpenAI model training - `ClaudeBot` and `anthropic-ai` β€” Anthropic model training - `Google-Extended` β€” Gemini/Vertex AI training - `Applebot-Extended` β€” Apple Intelligence training - `CCBot` β€” Common Crawl, a dataset many models train on - `Bytespider` β€” ByteDance crawler

AI search / RAG bots fetch live pages at answer time to build a grounded, cited response. Blocking these is what actually removes you from AI answers:

- `OAI-SearchBot` β€” OpenAI's ChatGPT search/browsing fetcher - `PerplexityBot` β€” Perplexity's indexing/answer crawler - `Googlebot` β€” also renders AI Overviews and AI Mode

The practical rule: be generous with search/RAG bots, be deliberate with training bots.

Step 2: Locate or create your robots.txt

The file lives at the root of your domain: `https://yourdomain.com/robots.txt`. It must be plain text, served with a `200` status.

- On a custom site, add or edit `robots.txt` in your web root. - On Shopify, the file is generated but editable through `robots.txt.liquid` in your theme. - On WordPress, an SEO plugin (Yoast, Rank Math) usually manages a virtual `robots.txt`.

Each rule set starts with a `User-agent` line, followed by `Allow` and `Disallow` paths. `Disallow: /` blocks everything; `Allow: /` (or no `Disallow`) permits everything.

Step 3: Allow the AI search bots that cite you

If your goal is GEO visibility, explicitly welcome the fetchers that build cited answers. This block permits them across the whole site:

``` User-agent: OAI-SearchBot Allow: /

User-agent: PerplexityBot Allow: /

User-agent: Googlebot Allow: / ```

Leaving these unblocked is the single most important GEO decision in the file. A site that blocks `OAI-SearchBot` will not be pulled into ChatGPT's cited web results, no matter how strong its content is.

Step 4: Decide your policy on training bots

This is a judgment call, not a default. Allowing training bots means your brand facts, positioning, and product names get baked into the model's world knowledge, which helps unprompted brand recall. Blocking them protects proprietary or high-value content from being absorbed without attribution.

To allow training bots everywhere, do nothing β€” no directive means allowed. To block them, add explicit blocks:

``` User-agent: GPTBot Disallow: /

User-agent: ClaudeBot Disallow: /

User-agent: anthropic-ai Disallow: /

User-agent: Google-Extended Disallow: /

User-agent: Applebot-Extended Disallow: /

User-agent: CCBot Disallow: /

User-agent: Bytespider Disallow: / ```

Note on `Google-Extended`: it controls Gemini and Vertex AI training only. Blocking it does not remove you from Google AI Overviews or AI Mode β€” those are served through the standard `Googlebot`. If you disallow `Googlebot`, you lose normal Google search too, so do not use it as an AI opt-out.

Step 5: Mix allow and block for partial access

Most brands want a middle path β€” let bots read the marketing site, keep them out of checkout, accounts, and internal search. Combine path rules per agent:

``` User-agent: GPTBot Allow: / Disallow: /cart Disallow: /account Disallow: /search ```

You can also allow training on your blog but not your docs, or vice versa, by listing the specific paths. Order matters less than specificity: most modern crawlers honor the longest matching rule.

Common mistakes

- Blocking `OAI-SearchBot` or `PerplexityBot` while trying to "stop AI from stealing content." You are only stopping the bots that would have cited and linked you. - Disallowing `Googlebot` to block AI Overviews. That deletes your normal Google rankings; Overviews still run through `Googlebot`. - Assuming `robots.txt` is enforcement. It is a request. Aggressive scrapers ignore it β€” use firewall or WAF rules for real blocking. - Typos in the user-agent token. `GPT-Bot` or `Claudebot` will not match; the strings are case-insensitive but must otherwise be exact. - Putting `robots.txt` in a subfolder. It only works at the domain root.

How to verify

A directive means nothing until you confirm bots are behaving as intended.

- Grep your server access logs for the user-agent strings, for example `grep -i "GPTBot\|OAI-SearchBot\|PerplexityBot\|ClaudeBot" access.log`. You should see the allowed bots hitting pages and the blocked ones absent. - If you run behind Cloudflare, filter analytics by user-agent to watch AI crawler traffic over time. - Run a GEO audit to catch policy mistakes you would miss by eye. GEOly's GEO audit includes 29 AI-readiness checks and flags crawler-access problems β€” for instance an accidental block on a search bot or a `robots.txt` that returns the wrong status. Its Cloudflare Analytics integration and citation-source analysis then show which AI engines are actually fetching and citing you, so you can tell whether an "allow" rule is paying off. You can run this on a free 3-day trial at `app.geoly.ai`. For the bigger picture on why this audit matters, see [why brands should audit AI crawler access now](/blog/why-brands-audit-ai-crawler-access-now).

FAQ

Should I block GPTBot to protect my content? Only if the content is genuinely proprietary and you accept losing presence in the model's knowledge. For most marketing sites, allowing `GPTBot` helps brand recall and costs little.

Will blocking Google-Extended remove me from Google's AI Overviews? No. `Google-Extended` governs Gemini/Vertex training. AI Overviews are served via `Googlebot`, so blocking Google-Extended leaves Overviews untouched.

Does robots.txt actually stop AI scrapers? Compliant crawlers obey it. It is not a security boundary β€” bots that ignore the standard need blocking at the network or WAF layer.

What is the difference between GPTBot and OAI-SearchBot? `GPTBot` gathers data to train OpenAI models. `OAI-SearchBot` fetches live pages so ChatGPT can cite them in a browsing answer. Allow `OAI-SearchBot` if you want to appear in ChatGPT's cited results.

How do I know which bots matter for my brand? Look at where your buyers ask questions, then confirm with data. To turn access into citations after you open the door, follow [best practices to increase AI crawling and citations](/blog/best-practices-increase-ai-crawling-citations), and pair it with an [IndexNow setup](/blog/implement-indexnow-geo) so fresh pages get fetched fast.

GEOly is a GEO data platform built to make this measurable β€” you can read [what GEOly AI is](/blog/what-is-geoly-ai) or browse more guides from [GEOly AI](/blog/author/geoly-ai).

See your brand in AI search

GEOly tracks how ChatGPT, Gemini and Google AI mention, cite and recommend your brand β€” and helps you win the AI shelf.

Start Free Trial

3-day free trial Β· No card required

Add GEOly as a preferred source on Google

Trusted by leading consumer brands

Anker SOLIX
eufy
soundcore
PLAUD
xTool
Ulike
Jackery
Roborock
Dreame
EcoFlow
Insta360
TCL
Beatbot
CASETiFY
Creality
Shokz
Segway
realme

12,000+ brands track & win AI search with GEOly

From Anker SOLIX to xTool β€” the brands above already see how ChatGPT, Gemini and Google AI mention, cite and recommend them. Your brand is being talked about in AI right now. See it.

See Your Brand in AI β€” Free

3-day free trial Β· No card required