What Are AI Crawlers?
AI crawlers are automated bots operated by AI companies. They visit your website and fetch content. There are two fundamentally different types:
Training Crawlers
Used to train AI models
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- CCBot (Common Crawl)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- Bytespider
Most sites block these to prevent content from being used in model training without compensation.
AI Search Crawlers
Used to answer user queries
- OAI-SearchBot (OpenAI)
- PerplexityBot
- ChatGPT-User
- Meta-ExternalAgent
Many sites allow these because they can drive referral traffic through AI-powered search citations.
Known AI Crawler Agents
Filter by type to see which crawlers you might want to allow or block:
| User-Agent | Company | Type | Suggested |
|---|---|---|---|
| GPTBot | OpenAI | Training | Block |
| OAI-SearchBot | OpenAI | AI Search | Allow |
| ChatGPT-User | OpenAI | Both | Review |
| ClaudeBot | Anthropic | Training | Block |
| anthropic-ai | Anthropic | Training | Block |
| PerplexityBot | Perplexity | AI Search | Allow |
| Perplexity-User | Perplexity | Both | Review |
| CCBot | Common Crawl | Training | Block |
| Bytespider | Bytespider | Training | Block |
| Google-Extended | Training | Block | |
| Applebot-Extended | Apple | Training | Block |
| Meta-ExternalAgent | Meta | Both | Review |
How to Block AI Crawlers in Robots.txt
Add rules like these to your robots.txt file to control AI crawlers:
# Block GPTBot (OpenAI training)
User-agent: GPTBot
Disallow: /
# Block ClaudeBot (Anthropic training)
User-agent: ClaudeBot
Disallow: /
# Block all training crawlers at once
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /Frequently Asked Questions
What is an AI crawler?
An AI crawler is an automated bot used by AI companies to fetch web content. Training crawlers use content to improve AI models. AI search crawlers fetch pages to answer user queries in AI-powered search products.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot is OpenAI's training crawler — it scrapes pages to train ChatGPT models. OAI-SearchBot is OpenAI's search crawler — it fetches pages for ChatGPT's live search and citation features. Most sites allow OAI-SearchBot and block GPTBot.
Does blocking AI crawlers hurt SEO?
No. Blocking AI training crawlers (GPTBot, ClaudeBot, etc.) has no effect on Googlebot or Bingbot. These are completely separate crawlers. Your Google search ranking is unaffected.