Anakin - Web Data Extraction

Convert websites into clean data at scale using the anakin-cli. Supports single URL scraping, batch scraping, AI-powered search, and autonomous deep research.

Installation & Authentication

Check status and authentication:

CODEBLOCK0

Output when ready:
CODEBLOCK1

If not installed: INLINECODE0

Always refer to the installation rules in rules/install.md for more information if the user is not logged in.

If not authenticated, run:

CODEBLOCK2

Get your API key from anakin.io/dashboard.

Organization

Create a .anakin/ folder in the working directory unless it already exists to store results. Add .anakin/ to the .gitignore file if not already there. Always use -o to write directly to file (avoids flooding context):

CODEBLOCK3

Capabilities

1. Scrape a Single URL

Extract content from a single web page in multiple formats.

When to use:

• - Extracting content from a single web page
• Converting a webpage to clean markdown
• Extracting structured data from one URL
• Getting full raw API response with metadata

Basic usage:

CODEBLOCK4

Advanced options:

CODEBLOCK5

2. Batch Scrape Multiple URLs

Scrape up to 10 URLs at once for efficient parallel processing.

When to use:

• - Scraping multiple web pages simultaneously
• Comparing products across different sites
• Collecting multiple articles or pages
• Gathering data from several sources at once

Basic usage:

CODEBLOCK6

For large lists (>10 URLs):

CODEBLOCK7

Output format: JSON file with combined results, each URL's status (success/failure), content, metadata, and any errors.

3. AI-Powered Web Search

Run intelligent web searches to find pages, answer questions, and discover sources.

When to use:

• - Finding pages on a specific topic
• Answering questions with web sources
• Discovering relevant sources for research
• Gathering links before scraping specific pages
• Quick factual lookups

Basic usage:

CODEBLOCK8

Follow-up workflow:

CODEBLOCK9

Output format: JSON file with search results including titles, URLs, snippets, relevance scores, and metadata.

4. Deep Agentic Research

Run comprehensive autonomous research that explores the web and returns detailed reports.

When to use:

• - Comprehensive research on complex topics
• Market analysis requiring multiple sources
• Technical deep-dives across documentation and articles
• Comparison research (products, technologies, approaches)
• Questions requiring synthesis from many sources

Basic usage:

CODEBLOCK10

⏱️ Important: Deep research takes 1-5 minutes and runs autonomously. Always inform the user about this duration before starting.

What it does:

• - Autonomously searches for relevant sources
• Scrapes and analyzes multiple pages
• Synthesizes information across sources
• Generates comprehensive reports with citations
• Provides key insights and conclusions

Output format: JSON file with executive summary, detailed report by subtopics, key insights, citations with URLs, confidence scores, and related topics.

Decision Guide

Use anakin scrape when:

• - You have a single specific URL to extract
• You need content in markdown, JSON, or raw format
• The page is static or JavaScript-heavy (use --browser)

Use anakin scrape-batch when:

• - You have 2-10 URLs to scrape simultaneously
• You need efficient parallel processing
• You want combined results in one file

Use anakin search when:

• - You need to find relevant URLs first
• You want quick factual lookups
• You need results in under 30 seconds
• You know what you're looking for

Use anakin research when:

• - You need comprehensive analysis across 5+ sources
• The topic is complex and requires deep exploration
• You want a synthesized report with insights
• You can wait 1-5 minutes for autonomous research
• The question requires comparing multiple perspectives

Guardrails

URL Handling

• - Always quote URLs to prevent shell interpretation of ?, &, # characters
• Example: anakin scrape "https://example.com?param=value" not INLINECODE14

Output Management

• - Always use -o <file> to save output to a file rather than flooding the terminal
• Choose appropriate output filenames based on content type

Format Selection

• - Default to markdown for readability unless user explicitly asks for JSON or raw
• Use --format json for structured data processing
• Use --format raw for full API response with HTML

Special Cases

• - Use --browser only when standard scrape returns empty or incomplete content
• For batch scraping: Maximum 10 URLs per command — split larger lists
• For research: Always warn about 1-5 minute duration before starting

Rate Limiting

• - On HTTP 429 errors (rate limit), wait before retrying
• Do not loop immediately on rate limit errors

Authentication

• - On HTTP 401 errors, re-run anakin login rather than retrying the same command

Error Handling

Output Formats

Markdown (default for scrape)

• - Clean, readable text stripped of navigation and ads
• Best for human reading and summarization
• File extension: INLINECODE26

JSON (structured)

• - Structured data with title, content, metadata
• Best for processing and parsing
• File extension: INLINECODE27

Raw (full response)

• - Full API response including HTML, links, images, metadata
• Best for debugging or accessing all available data
• File extension: INLINECODE28

Examples

Example 1: Article extraction

Example 2: Product comparison

Example 3: Find and scrape

Example 4: Market research

Example 5: JavaScript-heavy site

Example 6: Geo-targeted content

Best Practices

1. 1. Start simple: Try basic scrape first, add flags only if needed
2. Be specific: Use clear, specific search queries and research topics
3. Quote URLs: Always wrap URLs in quotes
4. Save output: Always use -o flag to save results to files
5. Check status: Run anakin status before starting work
6. Batch wisely: Group similar URLs together, max 10 per batch
7. Wait on rate limits: Don't retry immediately on 429 errors
8. Choose the right tool:

Troubleshooting

Authentication issues

Empty or incomplete content

• - Add --browser flag for JavaScript-heavy sites
• Increase timeout with INLINECODE36
• Check if the site requires specific geo-location with INLINECODE37

Rate limiting

• - Wait before retrying (don't loop immediately)
• Consider spacing out requests for large batch operations
• Check your API plan limits at anakin.io/dashboard

Resources

• - Anakin Website
• Anakin Dashboard - Get API keys and check usage
• anakin-cli on PyPI
• Support