AI crawlers are automated bots run by AI companies — GPTBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, and a fast-growing list of others — that fetch web content either to train large language models or to retrieve pages in real time when an AI engine answers a question. Where Googlebot indexes URLs so pages can rank as links, AI crawlers determine whether your brand exists inside ChatGPT, Gemini, Perplexity, and Google AI Overviews at all. How you handle them in robots.txt has become one of the most consequential technical decisions in Generative Engine Optimization (GEO).
Key takeaways
AI crawlers come in three types: training bots that feed future models (GPTBot, ClaudeBot), search-index bots that power AI search (OAI-SearchBot, PerplexityBot), and user-triggered fetchers that grab pages live (ChatGPT-User, Perplexity-User).
Blocking a training bot keeps your content out of future model versions. Blocking a search bot removes you from AI answers today. Brands competing for AI visibility should almost always allow the search and retrieval bots.
Google-Extended controls Gemini training and grounding, but it does not remove you from AI Overviews — those run on ordinary Googlebot crawling.
Robots.txt is a policy signal, not a wall: user-triggered fetchers often ignore it, and some crawlers have a poor compliance record.
Audit before you assume: Cloudflare has blocked known AI crawlers by default for new domains since mid-2025, so many brands are invisible to AI engines without ever choosing to be.
Three jobs, three kinds of crawler
Googlebot has one job: map the web so pages can rank. AI crawlers do three different jobs, and each changes what you get back.
Training bots collect large corpora for future model versions. GPTBot, ClaudeBot, and Meta-ExternalAgent sit here, along with CCBot from the nonprofit Common Crawl, whose dataset many AI labs reuse. Content crawled today may not influence answers for months, until the next model ships.
Search-index bots build the retrieval indexes behind AI search. OAI-SearchBot powers ChatGPT search, Claude-SearchBot feeds Claude's web search, and PerplexityBot builds Perplexity's index. These bots decide whether you can be cited in an answer this week.
User-triggered fetchers retrieve one specific page because a person asked. ChatGPT-User, Claude-User, and Perplexity-User fire when a question requires a live look at your site — the retrieval step behind grounding queries.
The output differs too. A traditional crawl earns a ranked link and maybe a click. An AI crawl earns citations and brand mentions inside a synthesized answer — the currency of zero-click search.
The AI crawlers to know in 2026
OpenAI
GPTBot collects training data for future GPT models. OAI-SearchBot crawls for ChatGPT search and, per OpenAI's bot documentation, is not used for training. ChatGPT-User fetches pages live during a conversation. The practical split: blocking GPTBot affects future models; blocking OAI-SearchBot removes you from ChatGPT search results now.
Anthropic
ClaudeBot is the training crawler, Claude-SearchBot indexes content for Claude's web search, and Claude-User handles user-initiated fetches. The Claude-Web agent you may still see in older guides has been retired.
Google
Googlebot remains the workhorse, and it also feeds AI Overviews and AI Mode. Google-Extended is a control token, not a separate crawler: disallowing it stops your content from training Gemini and being used for grounding, but per Google's crawler documentation it affects neither Search rankings nor AI Overviews. Opting out of AI Overviews requires snippet controls or leaving Google's index entirely.
Perplexity
PerplexityBot builds the index behind Perplexity's answer engine. Perplexity-User fetches pages when a question needs live web access — and because a human initiated the request, Perplexity states it generally does not honor robots.txt.
Everyone else worth logging
Applebot powers Siri and Spotlight; Applebot-Extended is Apple's opt-out token for AI training only.
Amazonbot feeds Alexa and Amazon's shopping AI, including Rufus.
Bytespider, ByteDance's crawler, has a long record of aggressive crawling and weak robots.txt compliance.
CCBot crawls for Common Crawl; blocking it removes you from a public dataset many labs train on at once.
Cross-platform visibility matrix comparing brand mentions across ChatGPT, Gemini, Google AI Overview, AI Mode and Perplexity — Source: GEOly AI (app.geoly.ai)
How to set your robots.txt policy
The mechanics are simple — one user-agent group per bot, such as User-agent: GPTBot followed by Disallow: / to block it, or no disallow rule to permit it. The strategy is where brands go wrong. Three coherent postures exist.
Open: allow everything. The default GEO posture for most DTC brands — you want models to learn your products and engines to cite your pages.
Selective: allow the search and user bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) while disallowing the training bots (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent). Publishers negotiating content licensing use this to stay citable without donating training data.
Closed: disallow all AI bots. Your content stays out of the machine, but AI assistants still talk about your category — describing your brand from stale training data and third-party sources you don't control. For a commercial brand, that trade is rarely worth it.
Whichever posture you pick, pair robots.txt with an llms.txt file that hands AI crawlers a clean map of your key pages, and make sure your structured data is server-rendered — most AI crawlers do not execute JavaScript.
Monitoring crawler access — and what it actually earns you
Start at the server: filter logs or CDN analytics for the user-agent strings above, and check response codes, not just visits — a crawler receiving 403s reads nothing. Spoofing is common, so verify suspicious traffic against the official IP ranges the major labs publish. And since Cloudflare began blocking known AI crawlers by default for new domains, "we never blocked anyone" no longer proves you're crawlable.
Crawl access is only the input; the output you care about is whether engines mention and cite you. That's the sequence GEOly AI is built around: a 29-point GEO audit covers the access mechanics (per-bot robots.txt rules, llms.txt presence, renderability), then own-brand monitoring tracks the downstream result — an AIGVR visibility score, mention and citation rates across seven engines, and which domains those engines actually cite in your category. A pattern we see often in audits: GPTBot carefully allowed in robots.txt while a WAF rule silently serves OAI-SearchBot a 403 — perfect training-bot hygiene, zero presence in ChatGPT search. The AI search metrics guide shows how these KPIs connect; the GEOly AI overview covers the full platform (free 3-day trial at app.geoly.ai).
Citation source analysis: source type distribution and the domains AI engines cite most — Source: GEOly AI (app.geoly.ai)
Common mistakes
Assuming Google-Extended opts you out of AI Overviews. It doesn't — AI Overviews ride on Googlebot.
Blanket anti-bot rules (WAF policies, rate limits, default CDN settings) that 403 AI search bots without anyone noticing.
Treating robots.txt as enforcement. It's a request; user-triggered fetchers and badly behaved crawlers fetch anyway.
Ignoring CCBot. Common Crawl feeds many training pipelines simultaneously, so that one decision carries unusual weight.
Blocking every AI crawler, then wondering why ChatGPT describes your products wrong. Models fall back on stale, third-party data when they can't read you.
FAQ
Should I block GPTBot on my site?
For most commercial brands, no. Blocking GPTBot keeps your content out of future OpenAI models, which means the next ChatGPT learns about your category from competitors and third parties instead of from you. Blocking mainly makes sense for publishers whose content is the product and who are pursuing licensing deals.
Do AI crawlers respect robots.txt?
The major training and search bots — GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot — publicly commit to honoring it and generally do. User-triggered fetchers like Perplexity-User often don't, by design, since a human requested the page, and Bytespider has a notoriously weak record. Verify in your logs rather than trusting the policy.
What's the difference between GPTBot and OAI-SearchBot?
GPTBot gathers training data for future models; its effect surfaces months later, if at all. OAI-SearchBot builds the index behind ChatGPT search, so blocking it removes your pages from citable results almost immediately. They're controlled independently in robots.txt — you can allow one and block the other.
How do I know if AI crawlers are visiting my site?
Filter server logs or CDN analytics for the user-agent strings (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot) and confirm they get 200 responses. Then measure the downstream effect: tracking your brand mentions in AI search tells you whether the crawling converts into answers and citations.
