Agent Docs

Write documentation that AI agents can efficiently consume. Based on Vercel benchmarks and industry standards (AGENTS.md, llms.txt, CLAUDE.md).

The Hybrid Context Hierarchy

Three-layer architecture for optimal agent performance:

Layer 1: Constitution (Inline)

Always in context. 2,000–4,000 tokens max.

CODEBLOCK0

Include:

• - Security rules, architecture constraints
• Build/test/lint commands (top for primacy bias)
• Documentation map (where to find more)

Layer 2: Reference Library (Local Retrieval)

Fetched on demand. 1K–5K token chunks.

• - Framework-specific guides
• Detailed style guides
• API schemas

Layer 3: Research Assistant (External)

Gated by allow-lists. Edge cases only.

• - Latest library updates
• Stack Overflow for obscure errors
• Third-party llms.txt

Why This Works

Vercel Benchmark (2026):

Approach	Pass Rate
Tool-based retrieval	53%
Retrieval + prompting

79% |
| Inline AGENTS.md | 100% |

Root cause: Meta-cognitive failure. Agents don't know what they don't know—they assume training data is sufficient. Inline docs bypass this entirely.

Core Principles

1. Compressed Index > Full Docs

An 8KB compressed index outperforms a 40KB full dump.

Compress to:

• - File paths (where code lives)
• Function signatures (names + types only)
• Negative constraints ("Do NOT use X")

2. Structure for Chunking

RAG systems split at headers. Each section must be self-contained:

CODEBLOCK1

Rules:

• - Front-load key info (chunkers truncate)
• Descriptive headers (agents search by header text)

3. Inline Over Links

Agents can't autonomously browse. Each link = tool call + latency + potential failure.

Approach	Token Load	Agent Success
Full inline	~12K	✅ High
Links only

~2K | ❌ Requires fetching |
| Hybrid | ~4K base | ✅ Best of both |

4. The "Lost in the Middle" Problem

LLMs have U-shaped attention:

• - Strong: Start of context (primacy)
• Strong: End of context (recency)
• Weak: Middle of context

Solution: Put critical rules at TOP of AGENTS.md. Governance first, details later.

5. Signal-to-Noise Ratio

Strip everything that isn't essential:

• - No "Welcome to..." preambles
• No marketing text
• No changelogs in core docs

Formats like llms.txt and AGENTS.md mechanically increase SNR.

llms.txt Standard

Machine-readable doc index for agents:

CODEBLOCK2

Location: /llms.txt at domain root
Companion: /llms-full.txt — full concatenated docs, HTML stripped

Security Considerations

Inline = Trusted

AGENTS.md is part of your codebase. Controlled, version-pinned.

External = Attack Surface

• - Indirect prompt injection via hidden text
• SSRF risks if agents can browse freely
• Dependency on external uptime

Mitigation: Domain allow-lists, human-in-the-loop for external retrieval.

Anti-Patterns

1. 1. Pasting 50 pages — triggers "Lost in the Middle"
2. "See external docs" — agents can't browse autonomously
3. Generic advice — "Write clean code" (use specific constraints)
4. TOC-only docs — indexes without content
5. Trusting retrieval alone — 53% vs 100% pass rate

Advanced Patterns

For detailed guidance on RAG optimization, multi-framework docs, and API templates, see references/advanced-patterns.md.

Validation Checklist

• - [ ] Critical governance at TOP of doc
• [ ] Total inline context under 4K tokens
• [ ] Each H2 section self-contained
• [ ] No external links without inline summary
• [ ] Negative constraints explicit ("Do NOT...")
• [ ] File paths and signatures, not full code