Configure Your robots.txt
Allow AI search, block training
AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) can crawl. Training crawlers (GPTBot, ClaudeBot) are blocked.
Block all AI crawlers
Blocks every known AI crawler. Standard search bots (Googlebot, Bingbot) still allowed.
Custom policy
Manually select which crawlers to allow or block below.
Select which crawlers your policy applies to. Or choose a preset above.
Your robots.txt file
About AI Crawlers and robots.txt
AI companies use crawlers to either train models or to answer user queries in AI-powered search. robots.txt allows you to control both types. Understanding the difference is key:
- Training crawlers — Use your content to improve AI models. Blocking them does not affect your search rankings.
- AI search crawlers — Fetch your pages to answer user queries in AI tools. Blocking them prevents your site from appearing in AI search results.
Frequently Asked Questions
Does blocking AI crawlers affect my Google search ranking?
No. Blocking AI training crawlers like GPTBot or ClaudeBot does not affect your Googlebot crawl budget or search rankings. These are separate systems. Googlebot indexes for Google Search; AI training crawlers are used for model training.
Can I allow AI search crawlers but block training crawlers?
Yes. AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are typically allowed because they can drive referral traffic. Training crawlers (GPTBot, ClaudeBot) are blocked to prevent content from being used to train models without compensation.
Is blocking AI crawlers guaranteed to prevent all AI use of my content?
No. robots.txt is a voluntary protocol. Blocking crawlers prevents well-behaved bots but does not stop unauthorized scraping. Determined scrapers can ignore robots.txt. Do not rely on robots.txt as a security measure.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot is OpenAI's training crawler — it scrapes pages to improve ChatGPT models. OAI-SearchBot is OpenAI's search crawler — it fetches pages for ChatGPT's live search citations. Most site owners allow OAI-SearchBot while blocking GPTBot.
What are Claude-User and Claude-SearchBot?
Claude-User is triggered when a user asks Claude to browse a URL — it fetches on behalf of the user, not automatically. Claude-SearchBot is Anthropic's dedicated search crawler for answering user queries in AI search products. Training crawlers (ClaudeBot) are separate and used for model training.