Handsfree Windows Control
A guide skill for automating native Windows apps (UIA) and web browsers (Playwright)
via the handsfree-windows CLI.
First use: auto-setup
On first use, run setup before issuing any other commands:
CODEBLOCK0
This will:
- 1. Clone
handsfree-windows from GitHub into ~/.handsfree-windows/cli/ (public repo, read-only) - Install it via
pip install -e (standard pip editable install) - Install Playwright + Chromium browser binaries (~200 MB one-time download from cdn.playwright.dev)
- Run
check_setup.py to verify everything is working
To skip browser installation (browser-* commands will not work):
CODEBLOCK1
To install to a custom directory:
CODEBLOCK2
Already installed? Verify anytime:
CODEBLOCK3
What is written to disk (transparent)
- - CLI source code:
~/.handsfree-windows/cli/ (or --install-dir) - pip editable link: standard site-packages egg-link (pip managed)
- Browser persistent profiles: INLINECODE6
Contains cookies and login sessions. Delete to reset browser auth.
- - Browser session state:
~/.handsfree-windows/browser-state.json (last visited URL) - Playwright browser binaries:
~\AppData\Local\ms-playwright\ (~800 MB, Windows)
To fully remove everything:
pip uninstall handsfree-windows -y
Remove-Item -Recurse -Force "$env:USERPROFILE\.handsfree-windows"
Core rules
- - Do not guess UI controls. Run
hf tree or hf inspect first, then act on what is actually there. - Do not type credentials. Navigate to login screens; let the human complete auth.
- Prefer UIA selectors (name + control_type) over raw coordinates.
- Use
drag-canvas only for canvas/ink surfaces (Paint, drawing apps, etc.). - For destructive actions (delete, submit, send), ask the human for confirmation first.
Workflow: Desktop app (UIA)
CODEBLOCK5
Workflow: Browser (Playwright)
CODEBLOCK6
Mixed macro (desktop + web in one YAML)
CODEBLOCK7
Run with: hf run macro.yaml
References
- - Full command reference + selector schema: references/api_reference.md
技能名称: handsfree-windows-control
详细描述:
Handsfree Windows 控制
通过 handsfree-windows CLI 自动化原生 Windows 应用（UIA）和网页浏览器（Playwright）的指南技能。
首次使用：自动设置
首次使用时，在发出任何其他命令前运行设置：
powershell
python scripts/setup.py
这将执行以下操作：
- 1. 从 GitHub 克隆 handsfree-windows 到 ~/.handsfree-windows/cli/（公共仓库，只读）
- 通过 pip install -e 安装（标准 pip 可编辑安装）
- 安装 Playwright + Chromium 浏览器二进制文件（约 200 MB 一次性下载，来自 cdn.playwright.dev）
- 运行 check_setup.py 验证一切正常
跳过浏览器安装（browser-* 命令将无法使用）：
powershell
python scripts/setup.py --no-browser
安装到自定义目录：
powershell
python scripts/setup.py --install-dir C:\your\preferred\path
已安装？随时验证：
powershell
python scripts/check_setup.py
写入磁盘的内容（透明）
- - CLI 源代码：~/.handsfree-windows/cli/（或 --install-dir）
- pip 可编辑链接：标准 site-packages egg-link（由 pip 管理）
- 浏览器持久配置文件：~/.handsfree-windows/browser-profiles//
包含 cookies 和登录会话。删除以重置浏览器认证。
- - 浏览器会话状态：~/.handsfree-windows/browser-state.json（最后访问的 URL）
- Playwright 浏览器二进制文件：~\AppData\Local\ms-playwright\（约 800 MB，Windows）
完全删除所有内容：
powershell
pip uninstall handsfree-windows -y
Remove-Item -Recurse -Force $env:USERPROFILE\.handsfree-windows
核心规则
- - 不要猜测 UI 控件。先运行 hf tree 或 hf inspect，然后根据实际存在的内容操作。
- 不要输入凭据。导航到登录屏幕；让人完成认证。
- 优先使用 UIA 选择器（名称 + 控件类型），而非原始坐标。
- 仅对画布/墨水表面（画图、绘图应用等）使用 drag-canvas。
- 对于破坏性操作（删除、提交、发送），先请求人工确认。
工作流程：桌面应用（UIA）
powershell
启动任何已安装的应用
hf start --app Outlook
查找窗口
hf list-windows --json
发现控件（无需猜测）
hf tree --title-regex Outlook --depth 10 --max-nodes 30000
根据发现的结果操作
hf click --title Outlook --name 新邮件 --control-type Button
检查光标下的元素
hf inspect --json
工作流程：浏览器（Playwright）
powershell
打开 URL - 登录会话自动保存在配置文件中
hf browser-open --url https://example.com
操作前检查页面
hf browser-snapshot --fmt text
操作
hf browser-click --text 登录
hf browser-type --selector #email --text user@example.com
验证
hf browser-screenshot --out result.png
混合宏（桌面 + 网页，一个 YAML 文件）
yaml
args:
app: 我的桌面应用
args:
url: https://app.example.com
headless: false
args:
text: 开始使用
args:
seconds: 1
运行方式：hf run macro.yaml
参考
- - 完整命令参考 + 选择器模式：references/api_reference.md
