SOP 006: Creating Embedding-Friendly Content
Fresh| Document Control | |
|---|---|
| SOP ID | SOP-006 |
| Version | 1.0 |
| Category | Content Optimization |
| Complexity | Medium |
| Source | Chunked, Retrieved, Synthesized |
Purpose
LLMs embed and score chunks based on coherence and clarity. Clean paragraphs create clean embeddings; mixed concepts create noisy embeddings. This procedure optimizes content for vector representation.
Understanding Embeddings
┌─────────────────────────────────────────────────────────────────────┐
│ EMBEDDING QUALITY │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Noisy Embedding (Mixed Topics) │
│ ────────────────────────────── │
│ "Vector databases are important for GenAI retrieval. They work │
│ alongside BM25, which ranks documents by keyword frequency. │
│ These models also use embeddings to match meaning." │
│ │
│ → Vector scattered across multiple concept spaces │
│ → Poor retrieval matching │
│ │
│ Clean Embedding (Single Topic) │
│ ────────────────────────────── │
│ "Embeddings allow GenAI systems to match queries with │
│ meaningfully similar content — even when the exact words │
│ don't match." │
│ │
│ → Vector focused in single concept space │
│ → Strong retrieval matching │
│ │
└─────────────────────────────────────────────────────────────────────┘Procedure
Step 1: One Idea Per Paragraph
Each paragraph should express exactly one concept:
Before (Mixed Topics):
Vector databases are important for GenAI retrieval. They work
alongside BM25, which ranks documents by keyword frequency.
These models also use embeddings to match meaning.After (Single Topics):
## What Are Embeddings?
Embeddings allow GenAI systems to match queries with meaningfully
similar content — even when the exact words don't match.
## How Embeddings Work with BM25
Hybrid retrieval combines embedding similarity with BM25 keyword
matching. This ensemble approach improves retrieval accuracy.Step 2: Use Declarative Statements
Strong, confident statements create cleaner embeddings:
| Weak (Noisy) | Strong (Clean) |
|---|---|
| "Some experts believe..." | "Research shows..." |
| "It could be argued that..." | "The evidence indicates..." |
| "GPT-4 might handle larger input sizes..." | "GPT-4 supports 128k context windows." |
Step 3: Add Semantic Redundancy
Rephrase key ideas 2-3 times using synonyms to widen your "retrievability footprint":
markdown
## Optimize for Retrieval
To improve your content's visibility in AI systems, focus on
retrieval optimization. Making your content easier for AI systems
to find requires understanding how LLMs select information.
Improving inclusion in LLM responses starts with proper chunking.All three phrases mean the same thing but cover different query patterns.
Step 4: Clarify Ambiguous Entities
Add context to names that could be confused:
| Ambiguous | Clarified |
|---|---|
| "Apple" | "Apple Inc., the technology company" |
| "Claude" | "Claude, Anthropic's AI assistant" |
| "Jordan" | "Michael Jordan, the basketball player" |
| "Transformer" | "Transformer architecture in machine learning" |
Step 5: Use Contextual Anchors
Keep claims and their context together in the same chunk:
Bad (Spread Apart):
"Polarized lenses reduce glare."
[...several paragraphs later...]
"This helps fishermen and drivers."Good (Together):
"Polarized lenses reduce glare from water and glass —
a key benefit for drivers and anglers who rely on
visibility in high-glare environments."Step 6: Structure for Scanning and Retrieval
Use formatting that aids both humans and machines:
markdown
## Key Takeaways
- **Chunk Retrieval**: LLMs select blocks, not pages
- **Embedding Quality**: Clean structure = clean vectors
- **Declarative Language**: Confident statements retrieve better
- **Semantic Redundancy**: Rephrase ideas for broader matchingStep 7: Include Latent Entity Context
Add specifics that help ground your content:
markdown
## Implementation Example
We tested this approach with GPT-4 Turbo (128k context,
released November 2024) using LangChain's retrieval pipeline
(version 0.1.0) in a production environment processing
approximately 10,000 queries daily.Embedding-Friendly Patterns
FAQs and Q&A
markdown
## What is content chunking?
Content chunking is the process of breaking content into
logical blocks of 100-300 tokens. Each chunk should contain
one complete idea with a clear heading that echoes natural
queries.Definitions
markdown
## Glossary: Embedding
An **embedding** is a vector (array of numbers) that represents
the semantic meaning of text. Embeddings allow AI systems to
compare content by meaning rather than exact word matching.Comparisons
markdown
## BM25 vs Semantic Search
| Feature | BM25 | Semantic Search |
|---------|------|-----------------|
| Matching | Keyword frequency | Meaning similarity |
| Synonyms | No support | Full support |
| Context | Limited | Rich understanding |Verification Checklist
- [ ] Each paragraph contains one idea only
- [ ] Declarative, confident language used
- [ ] Key concepts rephrased 2-3 times
- [ ] Ambiguous entities clarified
- [ ] Claims and context kept together
- [ ] Formatting aids scanning
- [ ] Specific dates, versions, numbers included
Testing Embeddings
To test your content's embedding quality:
- Use OpenAI's tokenizer to check token counts
- Test retrieval with tools like LangChain or LlamaIndex
- Query your content in Perplexity to see if it's cited