XCrawl Crawl

Overview

This skill orchestrates full-site or scoped crawling with XCrawl Crawl APIs.
Default behavior is raw passthrough: return upstream API response bodies as-is.

Required Local Config

Before using this skill, the user must create a local config file and write XCRAWL_API_KEY into it.

Path: INLINECODE1

CODEBLOCK0

Read API key from local config file only. Do not require global environment variables.

Credits and Account Setup

Using XCrawl APIs consumes credits.
If the user does not have an account or available credits, guide them to register at https://dash.xcrawl.com/.
After registration, they can activate the free 1000 credits plan before running requests.

Tool Permission Policy

Request runtime permissions for curl and node only.
Do not request Python, shell helper scripts, or other runtime permissions.

API Surface

• - Start crawl: INLINECODE6
• Read result: INLINECODE7
• Base URL: INLINECODE8
• Required header: INLINECODE9

Usage Examples

cURL (create + result)

CODEBLOCK1

Node

CODEBLOCK2

Request Parameters

Request endpoint and headers

• - Endpoint: INLINECODE10
• Headers:
• INLINECODE11
• INLINECODE12

Request body: top-level fields

Field	Type	Required	Default	Description
INLINECODE13	string	Yes	-	Site entry URL
INLINECODE14

object | No | - | Crawler config | | proxy | object | No | - | Proxy config | | request | object | No | - | Request config | | js_render | object | No | - | JS rendering config | | output | object | No | - | Output config | | webhook | object | No | - | Async callback config |

crawler

Field	Type	Required	Default	Description
INLINECODE21	integer	No	INLINECODE22	Max pages
INLINECODE23

string[] | No | - | Include only matching URLs (regex supported) | | exclude | string[] | No | - | Exclude matching URLs (regex supported) | | max_depth | integer | No | 3 | Max depth from entry URL | | include_entire_domain | boolean | No | false | Crawl full site instead of only subpaths | | include_subdomains | boolean | No | false | Include subdomains | | include_external_links | boolean | No | false | Include external links | | sitemaps | boolean | No | true | Use site sitemap |

proxy

Field	Type	Required	Default	Description
INLINECODE36	string	No	INLINECODE37	ISO-3166-1 alpha-2 country code, e.g. US / JP / INLINECODE40
INLINECODE41

string | No | Auto-generated | Sticky session ID; same ID attempts to reuse exit |

request

Field	Type	Required	Default	Description
INLINECODE43	string	No	INLINECODE44	Affects INLINECODE45
INLINECODE46

string | No | desktop | desktop / mobile; affects UA and viewport | | cookies | object map | No | - | Cookie key/value pairs | | headers | object map | No | - | Header key/value pairs | | only_main_content | boolean | No | true | Return main content only | | block_ads | boolean | No | true | Attempt to block ad resources | | skip_tls_verification | boolean | No | true | Skip TLS verification |

js_render

Field	Type	Required	Default	Description
INLINECODE59	boolean	No	INLINECODE60	Enable browser rendering
INLINECODE61

string | No | load | load / domcontentloaded / networkidle | | viewport.width | integer | No | - | Viewport width (desktop 1920, mobile 402) | | viewport.height | integer | No | - | Viewport height (desktop 1080, mobile 874) |

output

Field	Type	Required	Default	Description
INLINECODE73	string[]	No	INLINECODE74	Output formats
INLINECODE75

string | No | viewport | full_page / viewport (only if formats includes screenshot) | | json.prompt | string | No | - | Extraction prompt | | json.json_schema | object | No | - | JSON Schema |

INLINECODE83 enum:

• - INLINECODE84
• INLINECODE85
• INLINECODE86
• INLINECODE87
• INLINECODE88
• INLINECODE89
• INLINECODE90

webhook

Field	Type	Required	Default	Description
INLINECODE92	string	No	-	Callback URL
INLINECODE93

object map | No | - | Custom callback headers | | events | string[] | No | ["started","completed","failed"] | Events: started / completed / failed |

Response Parameters

Create response (POST /v1/crawl)

Field	Type	Description
INLINECODE100	string	Task ID
INLINECODE101

string | Always crawl | | version | string | Version | | status | string | Always pending |

Result response (GET /v1/crawl/{crawl_id})

Field	Type	Description
INLINECODE107	string	Task ID
INLINECODE108

string | Always crawl | | version | string | Version | | status | string | pending / crawling / completed / failed | | url | string | Entry URL | | data | object[] | Per-page result array | | started_at | string | Start time (ISO 8601) | | ended_at | string | End time (ISO 8601) | | total_credits_used | integer | Total credits used |

INLINECODE121 fields follow output.formats:

• - html, raw_html, markdown, links, summary, screenshot, INLINECODE129
• INLINECODE130 (page metadata)
• INLINECODE131
• INLINECODE132
• INLINECODE133

Workflow

1. 1. Confirm business objective and crawl boundary.

• - What content is required, what content must be excluded, and what is the completion signal.

1. 2. Draft a bounded crawl request.

• - Prefer explicit limits and path constraints.

1. 3. Start crawl and capture task metadata.

• - Record crawl_id, initial status, and request payload.

1. 4. Poll GET /v1/crawl/{crawl_id} until terminal state.

• - Track pending, crawling, completed, or failed.

1. 5. Return raw create/result responses.

• - Do not synthesize derived summaries unless explicitly requested.

Output Contract

Return:

• - Endpoint flow (POST /v1/crawl + GET /v1/crawl/{crawl_id})
• INLINECODE142 used for the create request
• Raw response body from create call
• Raw response body from result call
• Error details when request fails

Do not generate summaries unless the user explicitly requests a summary.

Guardrails

• - Never run an unbounded crawl without explicit constraints.
• Do not present speculative page counts as final coverage.
• Do not hardcode provider-specific tool schemas in core logic.
• Highlight policy, legal, or website-usage risks when relevant.