# What Are AI Crawlers? GPTBot, ClaudeBot, and the Bots Behind AI Search (2026)

- Canonical: https://www.geoly.ai/blog/ai-crawlers
- Published: 2026-07-05
- Updated: 2026-07-05
- Author: [GEOly AI](https://www.geoly.ai/blog/author/geoly-ai)
- Tags: [Glossary](https://www.geoly.ai/blog/tag/glossary), [robots.txt](https://www.geoly.ai/blog/tag/robots-txt), [GEO](https://www.geoly.ai/blog/tag/geo)

> **Summary:** AI crawlers are bots from OpenAI, Anthropic, Google, and Perplexity that fetch web content to train models or retrieve live answers — and your robots.txt policy toward them decides whether your brand exists inside AI search.

AI crawlers are automated bots run by AI companies — `GPTBot` from OpenAI, `ClaudeBot` from Anthropic, `PerplexityBot`, and a fast-growing list of others — that fetch web content either to train large language models or to retrieve pages in real time when an AI engine answers a question. Where Googlebot indexes URLs so pages can rank as links, AI crawlers determine whether your brand exists inside ChatGPT, Gemini, Perplexity, and Google AI Overviews at all. How you handle them in robots.txt has become one of the most consequential technical decisions in [Generative Engine Optimization (GEO)](https://www.geoly.ai/blog/what-is-geo).

## Key takeaways

- AI crawlers come in three types: training bots that feed future models (`GPTBot`, `ClaudeBot`), search-index bots that power AI search (`OAI-SearchBot`, `PerplexityBot`), and user-triggered fetchers that grab pages live (`ChatGPT-User`, `Perplexity-User`).
- Blocking a training bot keeps your content out of future model versions. Blocking a search bot removes you from AI answers today. Brands competing for [AI visibility](https://www.geoly.ai/blog/ai-visibility) should almost always allow the search and retrieval bots.
- `Google-Extended` controls Gemini training and grounding, but it does not remove you from [AI Overviews](https://www.geoly.ai/blog/ai-overviews) — those run on ordinary Googlebot crawling.
- Robots.txt is a policy signal, not a wall: user-triggered fetchers often ignore it, and some crawlers have a poor compliance record.
- Audit before you assume: Cloudflare has blocked known AI crawlers by default for new domains since mid-2025, so many brands are invisible to AI engines without ever choosing to be.

## Three jobs, three kinds of crawler

Googlebot has one job: map the web so pages can rank. AI crawlers do three different jobs, and each changes what you get back.

1. Training bots collect large corpora for future model versions. `GPTBot`, `ClaudeBot`, and `Meta-ExternalAgent` sit here, along with `CCBot` from the nonprofit Common Crawl, whose dataset many AI labs reuse. Content crawled today may not influence answers for months, until the next model ships.
2. Search-index bots build the retrieval indexes behind AI search. `OAI-SearchBot` powers ChatGPT search, `Claude-SearchBot` feeds Claude's web search, and `PerplexityBot` builds Perplexity's index. These bots decide whether you can be cited in an answer this week.
3. User-triggered fetchers retrieve one specific page because a person asked. `ChatGPT-User`, `Claude-User`, and `Perplexity-User` fire when a question requires a live look at your site — the retrieval step behind [grounding queries](https://www.geoly.ai/blog/grounding-queries).

The output differs too. A traditional crawl earns a ranked link and maybe a click. An AI crawl earns [citations](https://www.geoly.ai/blog/ai-citations) and [brand mentions](https://www.geoly.ai/blog/ai-brand-mentions) inside a synthesized answer — the currency of [zero-click search](https://www.geoly.ai/blog/zero-click-search).

## The AI crawlers to know in 2026

### OpenAI

`GPTBot` collects training data for future GPT models. `OAI-SearchBot` crawls for ChatGPT search and, per [OpenAI's bot documentation](https://platform.openai.com/docs/bots), is not used for training. `ChatGPT-User` fetches pages live during a conversation. The practical split: blocking `GPTBot` affects future models; blocking `OAI-SearchBot` removes you from ChatGPT search results now.

### Anthropic

`ClaudeBot` is the training crawler, `Claude-SearchBot` indexes content for Claude's web search, and `Claude-User` handles user-initiated fetches. The `Claude-Web` agent you may still see in older guides has been retired.

### Google

`Googlebot` remains the workhorse, and it also feeds AI Overviews and AI Mode. `Google-Extended` is a control token, not a separate crawler: disallowing it stops your content from training Gemini and being used for grounding, but per [Google's crawler documentation](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) it affects neither Search rankings nor AI Overviews. Opting out of AI Overviews requires snippet controls or leaving Google's index entirely.

### Perplexity

`PerplexityBot` builds the index behind Perplexity's answer engine. `Perplexity-User` fetches pages when a question needs live web access — and because a human initiated the request, Perplexity states it generally does not honor robots.txt.

### Everyone else worth logging

- `Applebot` powers Siri and Spotlight; `Applebot-Extended` is Apple's opt-out token for AI training only.
- `Amazonbot` feeds Alexa and Amazon's shopping AI, including Rufus.
- `Bytespider`, ByteDance's crawler, has a long record of aggressive crawling and weak robots.txt compliance.
- `CCBot` crawls for Common Crawl; blocking it removes you from a public dataset many labs train on at once.

![Cross-platform visibility matrix comparing brand mentions across ChatGPT, Gemini, Google AI Overview, AI Mode and Perplexity — Source: GEOly AI (app.geoly.ai)](https://gslkfhamqhkktynnjqng.supabase.co/storage/v1/object/public/cms-media/geoly-cross-platform-ai-engine-matrix.png)

## How to set your robots.txt policy

The mechanics are simple — one user-agent group per bot, such as `User-agent: GPTBot` followed by `Disallow: /` to block it, or no disallow rule to permit it. The strategy is where brands go wrong. Three coherent postures exist.

1. Open: allow everything. The default GEO posture for most DTC brands — you want models to learn your products and engines to cite your pages.
2. Selective: allow the search and user bots (`OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`) while disallowing the training bots (`GPTBot`, `ClaudeBot`, `CCBot`, `Meta-ExternalAgent`). Publishers negotiating content licensing use this to stay citable without donating training data.
3. Closed: disallow all AI bots. Your content stays out of the machine, but AI assistants still talk about your category — describing your brand from stale training data and third-party sources you don't control. For a commercial brand, that trade is rarely worth it.

Whichever posture you pick, pair robots.txt with an [llms.txt](https://www.geoly.ai/blog/llms-txt) file that hands AI crawlers a clean map of your key pages, and make sure your [structured data](https://www.geoly.ai/blog/structured-data-ai-search) is server-rendered — most AI crawlers do not execute JavaScript.

## Monitoring crawler access — and what it actually earns you

Start at the server: filter logs or CDN analytics for the user-agent strings above, and check response codes, not just visits — a crawler receiving 403s reads nothing. Spoofing is common, so verify suspicious traffic against the official IP ranges the major labs publish. And since Cloudflare began blocking known AI crawlers by default for new domains, "we never blocked anyone" no longer proves you're crawlable.

Crawl access is only the input; the output you care about is whether engines mention and cite you. That's the sequence GEOly AI is built around: a 29-point [GEO audit](https://www.geoly.ai/blog/geo-audit) covers the access mechanics (per-bot robots.txt rules, llms.txt presence, renderability), then own-brand monitoring tracks the downstream result — an [AIGVR visibility score](https://www.geoly.ai/blog/aigvr-score), mention and citation rates across seven engines, and which domains those engines actually cite in your category. A pattern we see often in audits: `GPTBot` carefully allowed in robots.txt while a WAF rule silently serves `OAI-SearchBot` a 403 — perfect training-bot hygiene, zero presence in ChatGPT search. The [AI search metrics guide](https://www.geoly.ai/blog/ai-search-visibility-metrics-kpis) shows how these KPIs connect; the [GEOly AI overview](https://www.geoly.ai/blog/what-is-geoly-ai) covers the full platform (free 3-day trial at app.geoly.ai).

![Citation source analysis: source type distribution and the domains AI engines cite most — Source: GEOly AI (app.geoly.ai)](https://gslkfhamqhkktynnjqng.supabase.co/storage/v1/object/public/cms-media/geoly-ai-citation-source-analysis.png)

## Common mistakes

- Assuming `Google-Extended` opts you out of AI Overviews. It doesn't — AI Overviews ride on Googlebot.
- Blanket anti-bot rules (WAF policies, rate limits, default CDN settings) that 403 AI search bots without anyone noticing.
- Treating robots.txt as enforcement. It's a request; user-triggered fetchers and badly behaved crawlers fetch anyway.
- Ignoring `CCBot`. Common Crawl feeds many training pipelines simultaneously, so that one decision carries unusual weight.
- Blocking every AI crawler, then wondering why ChatGPT describes your products wrong. Models fall back on stale, third-party data when they can't read you.

## FAQ

### Should I block GPTBot on my site?

For most commercial brands, no. Blocking `GPTBot` keeps your content out of future OpenAI models, which means the next ChatGPT learns about your category from competitors and third parties instead of from you. Blocking mainly makes sense for publishers whose content is the product and who are pursuing licensing deals.

### Do AI crawlers respect robots.txt?

The major training and search bots — `GPTBot`, `ClaudeBot`, `OAI-SearchBot`, `PerplexityBot` — publicly commit to honoring it and generally do. User-triggered fetchers like `Perplexity-User` often don't, by design, since a human requested the page, and `Bytespider` has a notoriously weak record. Verify in your logs rather than trusting the policy.

### What's the difference between GPTBot and OAI-SearchBot?

`GPTBot` gathers training data for future models; its effect surfaces months later, if at all. `OAI-SearchBot` builds the index behind ChatGPT search, so blocking it removes your pages from citable results almost immediately. They're controlled independently in robots.txt — you can allow one and block the other.

### How do I know if AI crawlers are visiting my site?

Filter server logs or CDN analytics for the user-agent strings (`GPTBot`, `ClaudeBot`, `PerplexityBot`, `OAI-SearchBot`) and confirm they get 200 responses. Then measure the downstream effect: [tracking your brand mentions in AI search](https://www.geoly.ai/blog/track-brand-mentions-in-ai-search) tells you whether the crawling converts into answers and citations.
