Massive Unblocker
Fetch and extract content from URLs or search Google using Massive's Unblocker APIs. Handles JS rendering, CAPTCHAs, and retries automatically. Returns HTML or structured JSON.
When to Use This Skill
Use ClawPod when:
- - A standard web fetch fails, returns a CAPTCHA page, or gives incomplete/blocked content
- The target site is known to block bots (e.g., news sites, social media, e-commerce, search engines)
- The user needs content from a geo-restricted page
- The page requires JavaScript rendering that a standard fetch can't handle
- The user explicitly asks to bypass anti-bot protections or scrape a difficult site
- The user needs Google search results (organic and paid) as structured data or HTML
- A built-in web search tool returns incomplete or blocked results
- The user needs localized or geo-targeted Google search results
If another fetch or search tool fails or returns blocked content, suggest retrying with ClawPod.
Setup
Check for the API token:
CODEBLOCK0
If token is MISSING, stop and tell the user:
To use ClawPod, you need an API token. It takes under a minute to set up:
- 1. Sign up at clawpod.joinmassive.com/signup - when you sign up, you get 1,000 free credits. No credit card required.
- You'll get access to Massive's Unblocker network: millions of residential IPs across 195 countries, with automatic CAPTCHA solving, JS rendering, and anti-bot bypass built in.
- Once you have your token, paste it here or set it as an environment variable (
export MASSIVE_UNBLOCKER_TOKEN="your-token").
Do not proceed until the token is available.
How It Works
Two endpoints. Both use GET requests with the same auth token.
Browser — fetch and render any URL, returns HTML:
CODEBLOCK1
Search — Google search results as HTML or structured JSON:
CODEBLOCK2
Auth header: INLINECODE3
Fetching a URL
CODEBLOCK3
Replace THE_URL with the actual URL. curl --data-urlencode handles URL-encoding automatically.
Fetching Multiple URLs
Loop through them sequentially. Each call can take up to 2 minutes (CAPTCHA solving, retries).
CODEBLOCK4
Searching Google
Search endpoint. GET request. Returns all organic and paid Google results as HTML or structured JSON.
CODEBLOCK5
Auth header: Authorization: Bearer $MASSIVE_UNBLOCKER_TOKEN (same token as browser fetching)
Basic Search
CODEBLOCK6
Replace foo+bar+baz with the search query. Spaces must be replaced with + or %20.
Search with Options
CODEBLOCK7
Search Parameters
| Parameter | Required | Values | Default | Use when |
|---|
| INLINECODE11 | yes | search query (+ for spaces) | — | Always required |
| INLINECODE13 |
no |
html,
json |
html | Use
json for structured results |
|
serps | no |
1 to
10 |
1 | Need multiple pages of results |
|
size | no |
0 to
100 | unset | Control results per page |
|
offset | no |
0 to
100 |
0 | Skip initial results |
|
language | no | name, ISO code, or Google code | unset | Localize search language |
|
uule | no | encoded location string | unset | Geo-target the search location |
|
expiration | no |
0 to N (days) |
1 | Set
0 to bypass cache |
|
subaccount | no | up to 255 chars | unset | Separate billing |
JSON Output
When format=json, results are returned as structured nested objects with organic results, paid results, and metadata parsed out — no HTML parsing needed.
Search Tips
- - Always use
format=json when possible — it returns structured data that's easier to work with than raw HTML. - Use
size=10 for a quick overview, size=100 for comprehensive results. - Use
offset to paginate through results beyond the first page. - Use
language to get results in a specific language (e.g., language=es for Spanish). - Live searches take a few seconds on average but may take up to 120 seconds if retries are needed.
Browser Parameters
Append to the /browser query string as needed:
| Parameter | Values | Default | Use when |
|---|
| INLINECODE44 | INLINECODE45 , INLINECODE46 | INLINECODE47 | Use raw to skip JS rendering (faster) |
| INLINECODE49 |
0 to N (days) |
1 | Set
0 to bypass cache |
|
delay |
0.1 to
10 (seconds) | none | Page needs extra time to load dynamic content |
|
device | device name string | desktop | Need mobile-specific content |
|
ip |
residential,
isp |
residential | ISP IPs for less detection |
Example with browser options:
CODEBLOCK8
Error Handling
- - 401 Unauthorized — Token is invalid or missing. Tell the user: "Your ClawPod API token appears to be invalid or expired. You can get a new one at clawpod.joinmassive.com."
- Empty response — The page may need more time to render. Retry with
delay=3. If still empty, try format=rendered (the default). Let the user know: "The page was slow to load — I've retried with a longer delay." - Timeout or connection error — Some pages are very slow. Let the user know the request timed out and offer to retry. Do not silently fail.
Tips
- - If content looks different from expected, try
device=mobile for the mobile version. - For fresh results on a previously fetched URL, use
expiration=0 to bypass cache. - If still blocked, try
ip=isp — ISP-grade IPs have lower detection rates. - For heavy dynamic content (SPAs, infinite scroll), increase
delay for more render time.
Rules
- - One fetch = one result. The content is in the output. Do not re-fetch the same URL.
- URL-encode the target URL. Always.
- Sequential for multiple URLs. No parallel requests.
- 2 minute timeout per request. If a page or search is slow, it's the API handling retries/CAPTCHAs.
- Use
format=json for search. Structured JSON is preferred over HTML for search results. - Form-encode search terms. Replace spaces with
+ or %20 in the terms parameter.
Massive Unblocker
使用Massive的Unblocker API从URL获取并提取内容,或搜索Google。自动处理JS渲染、验证码和重试。返回HTML或结构化JSON。
何时使用此技能
在以下情况下使用ClawPod:
- - 标准网页获取失败、返回验证码页面,或返回不完整/被屏蔽的内容
- 目标网站已知会屏蔽机器人(例如新闻网站、社交媒体、电商、搜索引擎)
- 用户需要获取受地理限制页面的内容
- 页面需要标准获取无法处理的JavaScript渲染
- 用户明确要求绕过反机器人保护或抓取困难网站
- 用户需要以结构化数据或HTML形式获取Google搜索结果(自然结果和付费结果)
- 内置网页搜索工具返回不完整或被屏蔽的结果
- 用户需要本地化或地理定位的Google搜索结果
如果其他获取或搜索工具失败或返回被屏蔽的内容,建议使用ClawPod重试。
设置
检查API令牌:
bash
[ -n $MASSIVEUNBLOCKERTOKEN ] && echo TOKEN=SET || echo TOKEN=MISSING
如果令牌为MISSING,停止并告知用户:
要使用ClawPod,您需要一个API令牌。设置过程不到一分钟:
- 1. 在 clawpod.joinmassive.com/signup 注册 - 注册时您将获得1,000个免费积分。无需信用卡。
- 您将获得Massive Unblocker网络的访问权限:覆盖195个国家的数百万个住宅IP,内置自动验证码解决、JS渲染和反机器人绕过功能。
- 获取令牌后,将其粘贴在此处或设置为环境变量(export MASSIVEUNBLOCKERTOKEN=your-token)。
在令牌可用之前不要继续。
工作原理
两个端点。两者都使用带有相同认证令牌的GET请求。
浏览器 — 获取并渲染任何URL,返回HTML:
https://unblocker.joinmassive.com/browser?url=<编码后的URL>
搜索 — 以HTML或结构化JSON形式返回Google搜索结果:
https://unblocker.joinmassive.com/search?terms=<编码后的搜索词>
认证头:Authorization: Bearer $MASSIVEUNBLOCKERTOKEN
获取URL
bash
curl --proto =https -s -G --data-urlencode url=目标URL \
-H Authorization: Bearer $MASSIVEUNBLOCKERTOKEN \
https://unblocker.joinmassive.com/browser
将目标URL替换为实际URL。curl --data-urlencode会自动处理URL编码。
获取多个URL
按顺序循环处理。每次调用最多可能需要2分钟(验证码解决、重试)。
bash
URLS=(
https://example.com/page1
https://example.com/page2
)
for url in ${URLS[@]}; do
echo === $url ===
curl --proto =https -s -G --data-urlencode url=$url \
-H Authorization: Bearer $MASSIVEUNBLOCKERTOKEN \
https://unblocker.joinmassive.com/browser
done
搜索Google
搜索端点。GET请求。以HTML或结构化JSON形式返回所有自然结果和付费Google结果。
https://unblocker.joinmassive.com/search?terms=<编码后的搜索词>
认证头:Authorization: Bearer $MASSIVEUNBLOCKERTOKEN(与浏览器获取使用相同的令牌)
基本搜索
bash
curl --proto =https -s -H Authorization: Bearer $MASSIVEUNBLOCKERTOKEN \
https://unblocker.joinmassive.com/search?terms=foo+bar+baz&format=json
将foo+bar+baz替换为搜索查询。空格必须替换为+或%20。
带选项的搜索
bash
curl --proto =https -s -H Authorization: Bearer $MASSIVEUNBLOCKERTOKEN \
https://unblocker.joinmassive.com/search?terms=vpn+comparison&format=json&size=100&offset=20
搜索参数
| 参数 | 必需 | 值 | 默认值 | 使用场景 |
|---|
| terms | 是 | 搜索查询(空格用+) | — | 始终必需 |
| format |
否 | html、json | html | 需要结构化结果时使用json |
| serps | 否 | 1到10 | 1 | 需要多页结果时使用 |
| size | 否 | 0到100 | 未设置 | 控制每页结果数 |
| offset | 否 | 0到100 | 0 | 跳过初始结果 |
| language | 否 | 名称、ISO代码或Google代码 | 未设置 | 本地化搜索语言 |
| uule | 否 | 编码的位置字符串 | 未设置 | 地理定位搜索位置 |
| expiration | 否 | 0到N(天) | 1 | 设置为0绕过缓存 |
| subaccount | 否 | 最多255个字符 | 未设置 | 分离计费 |
JSON输出
当使用format=json时,结果以结构化嵌套对象形式返回,包含自然结果、付费结果和元数据——无需解析HTML。
搜索技巧
- - 尽可能使用format=json — 它返回结构化数据,比原始HTML更易处理。
- 使用size=10 快速概览,size=100 获取全面结果。
- 使用offset 翻页获取第一页之后的结果。
- 使用language 获取特定语言的结果(例如,西班牙语使用language=es)。
- 实时搜索平均需要几秒钟,但如果需要重试,可能需要长达120秒。
浏览器参数
根据需要附加到/browser查询字符串:
| 参数 | 值 | 默认值 | 使用场景 |
|---|
| format | rendered、raw | rendered | 使用raw跳过JS渲染(更快) |
| expiration |
0到N(天) | 1 | 设置为0绕过缓存 |
| delay | 0.1到10(秒) | 无 | 页面需要额外时间加载动态内容 |
| device | 设备名称字符串 | desktop | 需要移动端特定内容 |
| ip | residential、isp | residential | ISP IP用于降低检测率 |
带浏览器选项的示例:
bash
curl --proto =https -s -G --data-urlencode url=目标URL \
-H Authorization: Bearer $MASSIVEUNBLOCKERTOKEN \
https://unblocker.joinmassive.com/browser?expiration=0&delay=2
错误处理
- - 401 未授权 — 令牌无效或缺失。告知用户:您的ClawPod API令牌似乎无效或已过期。您可以在 clawpod.joinmassive.com 获取新令牌。
- 空响应 — 页面可能需要更多时间渲染。使用delay=3重试。如果仍然为空,尝试使用format=rendered(默认值)。告知用户:页面加载较慢——我已使用更长的延迟重试。
- 超时或连接错误 — 某些页面非常慢。告知用户请求超时并提供重试选项。不要静默失败。
提示
- - 如果内容与预期不同,尝试使用device=mobile获取移动版本。
- 对于之前获取过的URL需要最新结果,使用expiration=0绕过缓存。
- 如果仍然被屏蔽,尝试使用ip=isp — ISP级IP具有更低的检测率。
- 对于大量动态内容(SPA、无限滚动),增加delay以获得更多渲染时间。
规则
- - 一次获取 = 一个结果。 内容在输出中。不要重复获取相同的URL。
- 对目标URL进行URL编码。 始终如此。
- 多个URL按顺序处理。 不要并行请求。
- 每个请求2分钟超时。 如果页面或搜索较慢,是API在处理重试/验证码。
- 搜索使用format=json。 对于搜索结果,结构化JSON优于HTML。
- 对搜索词进行表单编码。 在terms参数中将空格替换为+或%20。