docs-scraper

CLI tool that scrapes documents from various sources into local PDF files using browser automation.

Installation

CODEBLOCK0

Quick start

Scrape any document URL to PDF:

CODEBLOCK1

Returns local path: INLINECODE0

Basic scraping

Scrape with daemon (recommended, keeps browser warm):
CODEBLOCK2

Scrape with named profile (for authenticated sites):
CODEBLOCK3

Scrape with pre-filled data (e.g., email for DocSend):
CODEBLOCK4

Direct mode (single-shot, no daemon):
CODEBLOCK5

Authentication workflow

When a document requires authentication (login, email verification, passcode):

1. 1. Initial scrape returns a job ID:

1. 2. Retry with data:

Profile management

Profiles store session cookies for authenticated sites.

CODEBLOCK8

Daemon management

The daemon keeps browser instances warm for faster scraping.

CODEBLOCK9

Note: Daemon auto-starts when running scrape commands.

Cleanup

PDFs are stored in ~/.docs-scraper/output/. The daemon automatically cleans up files older than 1 hour.

Manual cleanup:
CODEBLOCK10

Job management

CODEBLOCK11

Supported sources

• - Direct PDF links - Downloads PDF directly
• Notion pages - Exports Notion page to PDF
• DocSend documents - Handles DocSend viewer
• LLM fallback - Uses Claude API for any other webpage

Scraper Reference

Each scraper accepts specific -D data fields. Use the appropriate fields based on the URL type.

DirectPdfScraper

Handles: URLs ending in INLINECODE3

Data fields: None (downloads directly)

Example:

docs-scraper scrape https://example.com/document.pdf

DocsendScraper

Handles: docsend.com/view/*, docsend.com/v/*, and subdomains (e.g., org-a.docsend.com)

URL patterns:

• - Documents: https://docsend.com/view/{id} or INLINECODE8
• Folders: INLINECODE9
• Subdomains: INLINECODE10

Data fields:

Examples:
CODEBLOCK13

Notes:

• - DocSend may require any combination of email, password, and name
• Folders are scraped as a table of contents PDF with document links
• The scraper auto-checks NDA checkboxes when name is provided

NotionScraper

Handles: notion.so/*, INLINECODE15

Data fields:

Examples:
CODEBLOCK14

Notes:

• - Public Notion pages don't require authentication
• Toggle blocks are automatically expanded before PDF generation
• Uses session profiles to persist login across scrapes

LlmFallbackScraper

Handles: Any URL not matched by other scrapers (automatic fallback)

Data fields: Dynamic - determined by Claude analyzing the page

The LLM scraper uses Claude to analyze the page HTML and detect:

• - Login forms (extracts field names dynamically)
• Cookie banners (auto-dismisses)
• Expandable content (auto-expands)
• CAPTCHAs (reports as blocked)
• Paywalls (reports as blocked)

Common dynamic fields:

Examples:
CODEBLOCK15

Notes:

• - Requires ANTHROPIC_API_KEY environment variable
• Field names are extracted from the page's actual form fields
• Limited to 2 login attempts before failing
• CAPTCHAs require manual intervention

Data field summary

*Fields detected dynamically from page analysis

Environment setup (optional)

Only needed for LLM fallback scraper:

CODEBLOCK16

Optional browser settings:
CODEBLOCK17

Common patterns

Archive a Notion page:
CODEBLOCK18

Download protected DocSend:
CODEBLOCK19

Batch scraping with profiles:
CODEBLOCK20

Output

Success: Local file path (e.g., ~/.docs-scraper/output/1706123456-abc123.pdf)
Blocked: Job ID + required credential types

Troubleshooting

• - Timeout: INLINECODE23
• Auth fails: docs-scraper jobs list to check pending jobs
• Disk full: docs-scraper cleanup to remove old PDFs