ArXiv Search Collector

Use this skill when you want model-led query planning and model-led relevance filtering.

Core Principle

Scripts are tools. The model performs the reasoning and decisions:

1. 1. Expand the original topic into multiple focused queries.
2. Run one fetch command per query.
3. Read each query result list and decide keep indexes.
4. Merge kept items and dedupe with one script.

Step 1: Initialize Run

CODEBLOCK0

This creates a run directory with task_meta.json, task_meta.md, query_results/, and query_selection/.

Language Parameter

• - --language must be set manually for each collection run.
• Use the same language value across all collector scripts for consistency.
• If --language is non-English (for example Chinese), generated markdown files are written in that language:

Query Writing Requirements

Follow these rules before running per-query fetch:

1. 1. Determine query count from final target range.

• - Prefer 3 queries for small/medium targets (2-5, 5-10).
• Prefer 4 queries for larger targets (10-50 or above).
• Avoid writing too many low-quality queries.

1. 2. Allocate target budget to each query, then oversample.

• - Let target_max be the upper bound in target range.
• Compute target_per_query = ceil(target_max / query_count).
• Fetch each query with max_results = target_per_query * 2 (or * 3 when recall is more important).
• Example: target 5-10, query count 3 -> target_per_query=4 -> each query fetches 8-12.

1. 3. Keep one original-theme query, then add normalized/synonym expansions.

• - Query 1 keeps original topic wording.
• Remaining queries use normalized terms and close synonyms.
• Prefer concise noun phrases that match arXiv indexing behavior.

1. 4. Use OR inside the same semantic group (synonyms), and AND across groups.

• - Same-group synonyms should be connected with OR to increase recall.

• - Different semantic groups should be connected with AND to keep relevance.

• - Recommended pattern:

Query Examples (arXiv API-ready)

Theme A: LLM applications in Lean 4 formalization

• - INLINECODE33
• INLINECODE34
• INLINECODE35
• INLINECODE36

Theme B: agentic tool use for code generation

• - INLINECODE38
• INLINECODE39
• INLINECODE40

Theme C: multimodal reasoning with retrieval

• - INLINECODE42
• INLINECODE43
• INLINECODE44

Step 2: Fetch One Query at a Time

Model defines queries manually, for example:

• - INLINECODE45
• INLINECODE46
• INLINECODE47

Recommended batch mode (safe defaults, serial execution):

CODEBLOCK1

In batch mode, the script auto-applies:

• - serial API calls
• INLINECODE48
• INLINECODE49
• INLINECODE50
• INLINECODE51
• INLINECODE52
• per-run rate-state file (<run_dir>/.runtime/arxiv_api_state.json) for throttling
• auto max_results from target_range and query count (default oversample x2, cap 60)
• default language/categories from INLINECODE58

Minimal query_plan.json only needs label and query.
See references/query-plan-format.md.
You normally do not need to set fetch-control args manually.

If you need one-by-one manual fetch, run each query:

CODEBLOCK2

Output files:

• - query_results/<label>.json (indexed full metadata list)
• INLINECODE64 (human-readable preview)

Date range is applied directly in arXiv API search_query via submittedDate:[... TO ...].
No second local date-filter pass is performed.

Rate-limit controls in fetch_query_metadata.py:

• - --min-interval-sec (default 5.0)
• INLINECODE70 (default 4)
• INLINECODE72 (default 5.0)
• INLINECODE74 (default 120.0)
• INLINECODE76 (default 1.0)
• INLINECODE78 (optional override; default is <run_dir>/.runtime/arxiv_api_state.json)
• INLINECODE80 to bypass cache and re-fetch

Step 3: Model Filters Relevance

For each query list, the model reads indexed results and decides what to keep.

Use keep specs by index and/or arXiv ID when merging.
To explicitly drop one weak query in later iterations, set that label to an empty keep list in selection-json.

Step 4: Merge and Dedupe

CODEBLOCK3

or with selection-json:

CODEBLOCK4

An empty list means this query label is intentionally dropped (keep 0).

This writes final outputs:

• - INLINECODE84
• INLINECODE85
• INLINECODE86
• INLINECODE87

Step 5: Iterative Retry Loop (Incremental)

If relevance is weak or final count is insufficient after Step 4, iterate:

1. 1. Review papers_index.md and per-paper metadata quality.
2. Adjust query plan (usually broaden with additional synonym OR terms, keep cross-group AND constraints).
3. Fetch additional query results with new labels.
4. Re-run merge in incremental mode:

CODEBLOCK5

Incremental behavior:

• - Previous label selections are loaded from query_selection/selected_by_query.json.
• Labels provided in the new selection-json override previous selections for those labels.
• New labels can be added.
• Old labels can be dropped by setting [].

Stop retrying when:

• - relevance is acceptable, or
• additional broadened queries mainly add low-relevance papers.

If relevant papers are genuinely scarce, it is valid to finish below the original minimum target range.

Notes

• - Keep API concurrency conservative by controlling query count and --max-results.
• Keep per-query fetch serial (no parallel API calls in Stage A).
• Reuse cache by default for identical query/date/request settings; only use --force when necessary.
• Prefer default run-local rate-state so all steps in the same run share one cooldown/throttling state.
• If arXiv API returns 429 Too Many Requests, retry later and/or increase --min-interval-sec.
• Prefer explicit, narrow queries and let the model filter aggressively.
• Use references/io-contract.md for exact files and schema.

Related Skills

This skill is a sub-skill of arxiv-summarizer-orchestrator.

Pipeline position:

1. 1. Step 1 (collection): arxiv-search-collector (this skill)
2. Step 2 (per-paper processing): INLINECODE101
3. Step 3 (batch reporting): INLINECODE102

This skill produces the initial paper-set structure and metadata that Stage B and Stage C depend on.