SOP 004: Making Sites AI-Crawlable
Fresh| Document Control | |
|---|---|
| SOP ID | SOP-004 |
| Version | 1.0 |
| Category | Technical Foundation |
| Complexity | Low |
| Source | The New Structure of AI Era SEO |
Purpose
If AI crawlers cannot fetch your content cleanly, LLMs cannot rely on it. This procedure ensures your site is accessible to AI systems.
The Foundation Layer
┌─────────────────────────────────────────────────────────────────────┐
│ CRAWLABILITY STACK │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Without AI Crawlability With AI Crawlability │
│ ─────────────────────── ──────────────────── │
│ Content hidden in JS ──► Static HTML available │
│ AI bots blocked ──► AI bots welcomed │
│ No structured data ──► Rich schema markup │
│ Slow page load ──► Fast, clean responses │
│ │
│ Result: Invisible Result: Indexed & retrievable │
│ │
└─────────────────────────────────────────────────────────────────────┘Procedure
Step 1: Audit Current Crawl Access
Check which AI bots can currently access your site:
bash
# Check your robots.txt
curl https://yoursite.com/robots.txt
# Look for blocks on these user-agents:
# - GPTBot
# - Google-Extended
# - CCBot
# - anthropic-ai
# - PerplexityBotStep 2: Review robots.txt Configuration
Problematic Configuration:
txt
User-agent: *
Disallow: /
# This blocks everything including AI crawlersAI-Friendly Configuration:
txt
# Allow all crawlers by default
User-agent: *
Allow: /
# Specifically welcome AI crawlers
User-agent: GPTBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: CCBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: PerplexityBot
Allow: /
# Block only what's necessary
Disallow: /admin/
Disallow: /private/Step 3: Ensure Content is Not JavaScript-Dependent
AI crawlers have limited JavaScript execution. Test your pages:
Disable JavaScript in browser
- Chrome: DevTools → Settings → Debugger → Disable JavaScript
- View your pages—can you see the content?
View page source
- Right-click → View Page Source
- Is your main content visible in the HTML?
Use server-side rendering (SSR) if content is hidden
Step 4: Avoid Content in Problematic Formats
| Format | Issue | Solution |
|---|---|---|
| Limited extraction | Provide HTML version | |
| Images of text | Cannot be read | Use actual text |
| Iframes | Often ignored | Embed content directly |
| Heavy JS frameworks | May not render | Use SSR or static generation |
Step 5: Optimize Page Structure
Ensure clean, semantic HTML:
html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="description" content="Clear description">
<title>Clear, descriptive title</title>
</head>
<body>
<header>
<nav><!-- Navigation --></nav>
</header>
<main>
<article>
<h1>Main Heading</h1>
<p>Content in semantic tags...</p>
</article>
</main>
<footer><!-- Footer --></footer>
</body>
</html>Step 6: Verify Crawl Success
Use these tools to verify:
| Tool | Purpose | URL |
|---|---|---|
| Google Search Console | Check indexing status | search.google.com/search-console |
| Screaming Frog | Crawl simulation | screamingfrog.co.uk |
| Fetch as Google | See what Google sees | In Search Console |
Step 7: Monitor AI Bot Traffic
Check server logs for AI crawler activity:
# Common AI bot user-agents to look for:
GPTBot
ChatGPT-User
Google-Extended
CCBot/2.0
anthropic-ai
PerplexityBotVerification Checklist
- [ ] robots.txt allows AI crawlers
- [ ] Content is visible without JavaScript
- [ ] No critical content in PDFs only
- [ ] No text in images
- [ ] Semantic HTML structure in place
- [ ] Page loads in under 3 seconds
- [ ] Server returns 200 status for main pages
- [ ] No redirect loops
Troubleshooting
| Issue | Diagnosis | Solution |
|---|---|---|
| Content not indexed | Check robots.txt | Remove blocking rules |
| Pages render blank | JavaScript dependency | Implement SSR |
| Slow crawling | Poor performance | Optimize page speed |
| Partial content | Lazy loading | Pre-render critical content |