SOP 008: AI Bot Access Setup

Fresh

Document Control
SOP ID	SOP-008
Version	1.0
Category	Technical Foundation
Complexity	Low
Source	Chunked, Retrieved, Synthesized

Purpose

AI crawlers need explicit access to index your content. This procedure configures robots.txt and server settings to welcome AI systems while maintaining appropriate access controls.

Key AI Crawlers

┌─────────────────────────────────────────────────────────────────────┐
│                    AI CRAWLER IDENTIFICATION                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│   User-Agent         Company        Purpose                         │
│   ──────────────────────────────────────────────────────            │
│   GPTBot             OpenAI         ChatGPT training/browsing       │
│   ChatGPT-User       OpenAI         ChatGPT web browsing            │
│   Google-Extended    Google         AI training (Gemini, etc.)      │
│   CCBot              Common Crawl   Open dataset (used by many AI)  │
│   anthropic-ai       Anthropic      Claude training                 │
│   PerplexityBot      Perplexity     Perplexity AI search            │
│   Bytespider         ByteDance      TikTok/AI training              │
│   Amazonbot          Amazon         Alexa/AI services               │
│   FacebookBot        Meta           Meta AI training                │
│   Applebot-Extended  Apple          Apple Intelligence              │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Procedure

Step 1: Audit Current robots.txt

Check your current configuration:

curl https://yoursite.com/robots.txt

Look for any of these blocking patterns:

User-agent: GPTBot
Disallow: /          # ⚠️ This blocks OpenAI

User-agent: *
Disallow: /          # ⚠️ This blocks everything

Step 2: Create AI-Friendly robots.txt

Recommended Configuration:

# Default: Allow all crawlers
User-agent: *
Allow: /
Crawl-delay: 1

# ============================================
# AI CRAWLER PERMISSIONS
# ============================================

# OpenAI - GPTBot (ChatGPT training)
User-agent: GPTBot
Allow: /

# OpenAI - ChatGPT Browsing
User-agent: ChatGPT-User
Allow: /

# Google - AI Training (Gemini, Bard, etc.)
User-agent: Google-Extended
Allow: /

# Common Crawl - Open Dataset
User-agent: CCBot
Allow: /

# Anthropic - Claude
User-agent: anthropic-ai
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

# Apple Intelligence
User-agent: Applebot-Extended
Allow: /

# ============================================
# RESTRICTED AREAS
# ============================================

# Block admin and private areas for all
User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/internal/
Disallow: /staging/

# ============================================
# SITEMAP
# ============================================

Sitemap: https://yoursite.com/sitemap.xml

Step 3: Update Server Configuration

Ensure your server doesn't block AI bots by IP or rate limit them excessively.

Nginx Example:

# Allow AI bots reasonable access
# Don't apply aggressive rate limiting to known AI user-agents

location / {
    # Your normal configuration
}

Step 4: Verify Access

Test that AI bots can access your content:

1. Use Google Search Console - Check for crawl errors
2. Review server logs - Look for AI bot requests and 200 responses
3. Test with curl using AI user-agent:

curl -A "GPTBot" https://yoursite.com/important-page
curl -A "anthropic-ai" https://yoursite.com/important-page

Step 5: Monitor AI Crawler Activity

Set up monitoring to track AI crawler visits:

# Example grep for AI bots in access logs
grep -E "(GPTBot|Google-Extended|CCBot|anthropic-ai|PerplexityBot)" access.log

Step 6: Consider Selective Access (Optional)

If you want to allow AI indexing but control training usage:

# Allow browsing/search but not training (where supported)
User-agent: GPTBot
Allow: /
# Note: Some bots support additional directives

# Google allows separate control
User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /  # Blocks AI training specifically

Training vs. Search

Some organizations distinguish between:

• Search/browsing: AI using content to answer queries
• Training: AI using content to improve models

Not all bots support this distinction. GPTBot, for example, is used for both.

Verification Checklist

• [ ] robots.txt updated and deployed
• [ ] All major AI bots explicitly allowed
• [ ] Sitemap URL included
• [ ] Sensitive areas still blocked
• [ ] Server not blocking by IP
• [ ] Rate limiting not too aggressive
• [ ] Access verified via curl/logs

AI Bot Documentation Links

Bot	Documentation
GPTBot	platform.openai.com/docs/gptbot
Google-Extended	developers.google.com/search/docs/crawling-indexing
CCBot	commoncrawl.org/ccbot
Anthropic	Check official Anthropic documentation
PerplexityBot	Check Perplexity documentation

Troubleshooting

Issue	Diagnosis	Solution
AI not indexing content	Check robots.txt	Ensure bot is allowed
403 errors in logs	Server blocking	Check firewall/rate limits
Partial crawling	Crawl budget issues	Improve site structure
AI showing old content	Crawl frequency	Ensure regular access

See Also

• SOP 004 - AI-Crawlable Sites
• Quick Reference - Tool Stack