arXiv Search (metadata-first)

Collect an initial paper set with enough metadata to support downstream ranking, taxonomy building, and citation generation.

When online, prefer rich arXiv metadata (categories, arxivid, pdfurl, published/updated, etc.). When offline, accept an export and convert it cleanly.

Load Order

Always read:

• - references/domain_pack_overview.md — how domain packs drive topic-specific behavior

Domain packs (loaded by topic match):

• - assets/domain_packs/llm_agents.json — pinned IDs, query rewrite rules for LLM agent topics

Script Boundary

Use scripts/run.py only for:

• - arXiv API retrieval and XML parsing
• offline export conversion (CSV/JSON/JSONL normalization)
• metadata enrichment via id_list backfill

Do not treat run.py as the place for:

• - hardcoded topic detection or query rewriting (use domain packs)
• domain-specific pinned paper lists (externalize to assets/domain_packs/)

Input

• - queries.md (keywords, excludes, time window)

Outputs

• - papers/papers_raw.jsonl (JSONL; 1 paper per line)

• - Convenience index (optional but generated by the script):

Decision: online vs offline

• - If you have network access: run arXiv API retrieval.
• If not: import an export the user provides (CSV/JSON/JSONL) and normalize fields.
• Hybrid: if you import offline but still have network later, you can enrich missing fields (abstract/authors/categories) via arXiv id_list using --enrich-metadata or queries.md enrich_metadata: true.

Workflow (heuristic)

1. 1. Read queries.md and expand into concrete query strings.
2. Retrieve results (online) or import an export (offline).
3. Normalize every record to include at least:

1. 4. Keep the set broad at this stage; dedupe/ranking comes next.
2. Apply time window and max_results if specified.

Quality checklist

• - [ ] papers/papers_raw.jsonl exists.
• [ ] Each line is valid JSON and contains title, authors, year, url.

Side effects

• - Allowed: create/overwrite papers/papers_raw.jsonl; append notes to STATUS.md.
• Not allowed: write prose sections in output/ before writing is approved.

Script

Quick Start

• - INLINECODE42
• Online: INLINECODE43
• Offline import: INLINECODE44

All Options

• - --query <q>: repeatable; multiple queries are unioned
• INLINECODE46: repeatable; excludes applied after retrieval
• INLINECODE47: cap total retrieved
• INLINECODE48: offline mode (CSV/JSON/JSONL)
• INLINECODE49: best-effort enrich via arXiv id_list (needs network)
• INLINECODE51 also supports: keywords, exclude, time window, max_results, INLINECODE56

Examples

• - Online (multi-query + excludes):

• - Fetch a single paper by arXiv ID (direct id_list fetch):

• - Offline auto-detect (no flags):

• - Offline import + time window (via queries.md):

Troubleshooting

Common Issues

Issue: papers/papers_raw.jsonl is empty

Symptom:

• - Script exits with “No results returned …” or output file is empty.

Causes:

• - Network is blocked (online mode).
• Queries are too narrow or queries.md is empty.

Solutions:

• - Use offline import: place papers/import.csv|json|jsonl in the workspace or pass --input.
• Broaden keywords and reduce excludes in queries.md.
• Run with explicit --query to sanity-check the parser.

Issue: Offline import records miss fields

Symptom:

• - Downstream steps fail because records miss authors/year/abstract/url.

Causes:

• - Export columns don’t match expected fields; upstream export is incomplete.

Solutions:

• - Ensure the export contains at least title, authors, year, url, abstract.
• If you later have network, use --enrich-metadata to backfill missing fields (best effort).

Recovery Checklist

• - [ ] Confirm queries.md has non-empty keywords (or pass --query).
• [ ] If offline: confirm workspace has papers/import.* and rerun.
• [ ] Spot-check 3–5 JSONL lines: valid JSON + required fields.