Agent Survey Corpus (arXiv PDFs → text extracts)

Goal: create a small, local reference library so you can learn from real agent surveys when refining:

• - C2 outline structure (paper-like sectioning)
• C4 tables/claims organization
• C5 writing style and density

This is intentionally not part of the pipeline; it is an optional, repo-level toolkit.

Inputs

• - INLINECODE0

Outputs

• - INLINECODE1
• INLINECODE2
• INLINECODE3 (tracked; auto-generated summary)

Workflow

1) Edit ref/agent-surveys/arxiv_ids.txt (one arXiv id per line).
2) Run the downloader to fetch PDFs and extract the first N pages to text.
3) Skim the extracted text under ref/agent-surveys/text/:
- look at section counts (H2), subsection granularity (H3), and how they transition between chapters.
- identify repeated rhetorical patterns you want the pipeline writer to imitate.

Script

Quick Start

• - INLINECODE6
• INLINECODE7

All Options

• - --workspace <dir> (use . to write into repo root)
• INLINECODE10 (default: ref/agent-surveys/arxiv_ids.txt)
• INLINECODE12 (default: 20)
• INLINECODE13 (default: 1.0)
• INLINECODE14 (re-download + re-extract)

Examples

• - Download/extract into repo root ref/:

• - Download/extract into a specific folder (treated as workspace root):

Troubleshooting

• - Download fails / timeout: rerun with a larger --sleep, or try fewer ids.
• Text extract is empty: the PDF may be scanned; try another survey or increase --max-pages.
• Files showing up in git status: PDFs/text are ignored via .gitignore (ref/**/pdfs/, ref/**/text/).