XCrawl

Overview

This skill is the default XCrawl entry point when the user asks for XCrawl directly without naming a specific API or sub-skill.
It currently targets single-page extraction through XCrawl Scrape APIs.
Default behavior is raw passthrough: return upstream API response bodies as-is.

Routing Guidance

• - If the user wants to extract one or more specific URLs, use this skill and default to XCrawl Scrape.
• If the user wants site URL discovery, prefer XCrawl Map APIs.
• If the user wants multi-page or site-wide crawling, prefer XCrawl Crawl APIs.
• If the user wants keyword-based discovery, prefer XCrawl Search APIs.

Required Local Config

Before using this skill, the user must create a local config file and write XCRAWL_API_KEY into it.

Path: INLINECODE1

CODEBLOCK0

Read API key from local config file only. Do not require global environment variables.

Credits and Account Setup

Using XCrawl APIs consumes credits.
If the user does not have an account or available credits, guide them to register at https://dash.xcrawl.com/.
After registration, they can activate the free 1000 credits plan before running requests.

Tool Permission Policy

Request runtime permissions for curl and node only.
Do not request Python, shell helper scripts, or other runtime permissions.

API Surface

• - Start scrape: INLINECODE6
• Read async result: INLINECODE7
• Base URL: INLINECODE8
• Required header: INLINECODE9

Usage Examples

cURL (sync)

CODEBLOCK1

cURL (async create + result)

CODEBLOCK2

Node

CODEBLOCK3

Request Parameters

Request endpoint and headers

• - Endpoint: INLINECODE10
• Headers:
• INLINECODE11
• INLINECODE12

Request body: top-level fields

Field	Type	Required	Default	Description
INLINECODE13	string	Yes	-	Target URL
INLINECODE14

string | No | sync | sync or async | | proxy | object | No | - | Proxy config | | request | object | No | - | Request config | | js_render | object | No | - | JS rendering config | | output | object | No | - | Output config | | webhook | object | No | - | Async webhook config (mode=async) |

proxy

Field	Type	Required	Default	Description
INLINECODE25	string	No	INLINECODE26	ISO-3166-1 alpha-2 country code, e.g. US / JP / INLINECODE29
INLINECODE30

string | No | Auto-generated | Sticky session ID; same ID attempts to reuse exit |

request

Field	Type	Required	Default	Description
INLINECODE32	string	No	INLINECODE33	Affects INLINECODE34
INLINECODE35

string | No | desktop | desktop / mobile; affects UA and viewport | | cookies | object map | No | - | Cookie key/value pairs | | headers | object map | No | - | Header key/value pairs | | only_main_content | boolean | No | true | Return main content only | | block_ads | boolean | No | true | Attempt to block ad resources | | skip_tls_verification | boolean | No | true | Skip TLS verification |

js_render

Field	Type	Required	Default	Description
INLINECODE48	boolean	No	INLINECODE49	Enable browser rendering
INLINECODE50

string | No | load | load / domcontentloaded / networkidle | | viewport.width | integer | No | - | Viewport width (desktop 1920, mobile 402) | | viewport.height | integer | No | - | Viewport height (desktop 1080, mobile 874) |

output

Field	Type	Required	Default	Description
INLINECODE62	string[]	No	INLINECODE63	Output formats
INLINECODE64

string | No | viewport | full_page / viewport (only if formats includes screenshot) | | json.prompt | string | No | - | Extraction prompt | | json.json_schema | object | No | - | JSON Schema |

INLINECODE72 enum:

• - INLINECODE73
• INLINECODE74
• INLINECODE75
• INLINECODE76
• INLINECODE77
• INLINECODE78
• INLINECODE79

webhook

Field	Type	Required	Default	Description
INLINECODE81	string	No	-	Callback URL
INLINECODE82

object map | No | - | Custom callback headers | | events | string[] | No | ["started","completed","failed"] | Events: started / completed / failed |

Response Parameters

Sync create response (mode=sync)

Field	Type	Description
INLINECODE89	string	Task ID
INLINECODE90

string | Always scrape | | version | string | Version | | status | string | completed / failed | | url | string | Target URL | | data | object | Result data | | started_at | string | Start time (ISO 8601) | | ended_at | string | End time (ISO 8601) | | total_credits_used | integer | Total credits used |

INLINECODE101 fields (based on output.formats):

• - html, raw_html, markdown, links, summary, screenshot, INLINECODE109
• INLINECODE110 (page metadata)
• INLINECODE111
• INLINECODE112
• INLINECODE113

INLINECODE114 fields:

Field	Type	Description
INLINECODE115	integer	Base scrape cost
INLINECODE116

integer | Traffic cost |
| json_extract_cost | integer | JSON extraction cost |

Async create response (mode=async)

Field	Type	Description
INLINECODE119	string	Task ID
INLINECODE120

string | Always scrape | | version | string | Version | | status | string | Always pending |

Async result response (GET /v1/scrape/{scrape_id})

Field	Type	Description
INLINECODE126	string	Task ID
INLINECODE127

string | Always scrape | | version | string | Version | | status | string | pending / crawling / completed / failed | | url | string | Target URL | | data | object | Same shape as sync data | | started_at | string | Start time (ISO 8601) | | ended_at | string | End time (ISO 8601) |

Workflow

1. 1. Classify the request through the default XCrawl entry behavior.

• - If the user provides specific URLs for extraction, default to XCrawl Scrape.
• If the user clearly asks for map, crawl, or search behavior, route to the dedicated XCrawl API instead of pretending this endpoint covers it.

1. 2. Restate the user goal as an extraction contract.

• - URL scope, required fields, accepted nulls, and precision expectations.

1. 3. Build the scrape request body.

• - Keep only necessary options.
• Prefer explicit output.formats.

1. 4. Execute scrape and capture task metadata.

• - Track scrape_id, status, and timestamps.
• If async, poll until completed or failed.

1. 5. Return raw API responses directly.

• - Do not synthesize or compress fields by default.

Output Contract

Return:

• - Endpoint(s) used and mode (sync or async)
• INLINECODE147 used for the request
• Raw response body from each API call
• Error details when request fails

Do not generate summaries unless the user explicitly requests a summary.

Guardrails

• - Do not present XCrawl Scrape as if it also covers map, crawl, or search semantics.
• Default to scrape only when user intent is URL extraction.
• Do not invent unsupported output fields.
• Do not hardcode provider-specific tool schemas in core logic.
• Call out uncertainty when page structure is unstable.