AI Robots.txt Generator

Configure your robots.txt to earn high-value referral traffic from AI search engines while protecting your proprietary content from AI training scrapers.

1. Configure Crawler Rules

Your Domain:

🔍 AI Search Engines (Send Traffic & Citations)

OAI-SearchBot ChatGPT Search engine

PerplexityBot Perplexity AI answer engine

ClaudeBot Anthropic Claude search & web retrieval

🚫 AI Model Training Scrapers (No Direct Citations)

GPTBot OpenAI foundational model training

Google-Extended Gemini model training (not standard Google Search)

Bytespider ByteDance / TikTok LLM training

CCBot Common Crawl raw web scraper

2. Live robots.txt Output

The Critical Difference: Search Crawlers vs. Training Scrapers

Historically, webmasters faced an all-or-nothing choice when blocking bots. Today, leading AI labs operate distinct crawlers. OAI-SearchBot powers search results in ChatGPT, providing direct referral links to your website. Conversely, GPTBot ingests your content to train future models without necessarily attributing your brand. This generator allows you to selectively welcome search traffic while preventing unauthorized model pre-training.

Verify Your Live Robots.txt Deployment with GEOScope

Once deployed, test your live visibility using ScopeKit's GEOScope API on RapidAPI to confirm AI crawler response codes in real time.