Mac Compute Use
Control macOS GUI through the Accessibility API via MCP.
Setup
- 1. Install the MCP server:
CODEBLOCK0
- 2. Register with mcporter:
CODEBLOCK1
- 3. Grant Accessibility permission:
System Settings → Privacy & Security → Accessibility → add INLINECODE0
- 4. Verify:
CODEBLOCK2
Tools
All tools are called via mcporter call macos-use.<tool> key=value.
openapplicationand_traverse
Open/activate an app and get its UI tree.
mcporter call macos-use.macos-use_open_application_and_traverse identifier="Google Chrome"
mcporter call macos-use.macos-use_open_application_and_traverse identifier="com.apple.finder"
mcporter call macos-use.macos-use_open_application_and_traverse identifier="TextEdit"
Returns: PID, element count, visible interactive elements, and a JSON file path with full UI tree.
clickandtraverse
Click at coordinates (from UI tree) and get updated state.
mcporter call macos-use.macos-use_click_and_traverse pid=408 x=701 y=73 width=102 height=41
- -
x, y: top-left corner of the element (from traversal) - INLINECODE4 ,
height: optional, when provided click lands at center
typeandtraverse
Type text into the focused app.
CODEBLOCK5
presskeyand_traverse
Press a key with optional modifiers.
mcporter call macos-use.macos-use_press_key_and_traverse pid=408 keyName=Return
mcporter call macos-use.macos-use_press_key_and_traverse pid=408 keyName=a modifierFlags='["Command"]'
mcporter call macos-use.macos-use_press_key_and_traverse pid=408 keyName=Tab
mcporter call macos-use.macos-use_press_key_and_traverse pid=408 keyName=Escape
Valid modifiers: CapsLock, Shift, Control, Option, Command, Function, NumericPad, Help.
scrollandtraverse
Scroll within an app window.
mcporter call macos-use.macos-use_scroll_and_traverse pid=408 x=500 y=400 deltaY=3
mcporter call macos-use.macos-use_scroll_and_traverse pid=408 x=500 y=400 deltaY=-3
- -
deltaY positive = scroll down, negative = scroll up - INLINECODE7 optional, for horizontal scroll
refresh_traversal
Get current UI state without performing any action.
CODEBLOCK8
Workflow Pattern
Typical automation flow:
- 1. Open app → get PID and visible elements
- Read the visible_elements in the response summary — these are interactive elements with coordinates
- Click/type/press using coordinates from the UI tree
- Read the response — it shows what changed (diff) and new visible elements
- Repeat until task is complete
Reading the Response
Each tool returns a compact summary with:
- -
status: success/error - INLINECODE9 : process ID (use for subsequent calls)
- INLINECODE10 : path to full JSON with all elements (use
grep or python3 to search) - INLINECODE13 : key interactive elements currently visible, with coordinates
When you need to find a specific element, grep the JSON file:
CODEBLOCK9
Or parse with Python:
CODEBLOCK10
Tips
- - Always use
--output json for machine-readable results when chaining commands - After clicking, wait a moment then
refresh_traversal if the UI didn't update in the diff - Use app name ("Google Chrome"), bundle ID ("com.google.Chrome"), or path to open apps
- Coordinates are absolute screen positions — if the window moves, refresh the traversal
- The server writes traversal JSON to
/tmp/macos-use/ — these files are temporary
Mac Compute Use
通过MCP使用辅助功能API控制macOS图形用户界面。
设置
- 1. 安装MCP服务器：
bash
brew tap reedburns/mcp-server-macos-use
brew install mcp-server-macos-use
- 2. 注册到mcporter：
bash
mcporter config add macos-use --transport stdio --command $(which mcp-server-macos-use)
- 3. 授予辅助功能权限：
系统设置 → 隐私与安全性 → 辅助功能 → 添加 mcp-server-macos-use
- 4. 验证：
bash
mcporter list macos-use --schema
工具
所有工具均通过 mcporter call macos-use.<工具名> key=value 调用。
openapplicationand_traverse
打开/激活应用并获取其UI树。
bash
mcporter call macos-use.macos-use
openapplication
andtraverse identifier=Google Chrome
mcporter call macos-use.macos-use
openapplication
andtraverse identifier=com.apple.finder
mcporter call macos-use.macos-use
openapplication
andtraverse identifier=TextEdit
返回：PID、元素数量、可见交互元素，以及包含完整UI树的JSON文件路径。
clickandtraverse
点击指定坐标（来自UI树）并获取更新后的状态。
bash
mcporter call macos-use.macos-use
clickand_traverse pid=408 x=701 y=73 width=102 height=41
- - x, y：元素左上角坐标（来自遍历结果）
- width, height：可选参数，提供时点击位置为元素中心
typeandtraverse
在焦点应用中输入文本。
bash
mcporter call macos-use.macos-use
typeand_traverse pid=408 text=Hello world
presskeyand_traverse
按下按键（可附带修饰键）。
bash
mcporter call macos-use.macos-use
presskey
andtraverse pid=408 keyName=Return
mcporter call macos-use.macos-use
presskey
andtraverse pid=408 keyName=a modifierFlags=[Command]
mcporter call macos-use.macos-use
presskey
andtraverse pid=408 keyName=Tab
mcporter call macos-use.macos-use
presskey
andtraverse pid=408 keyName=Escape
有效修饰键：CapsLock、Shift、Control、Option、Command、Function、NumericPad、Help。
scrollandtraverse
在应用窗口内滚动。
bash
mcporter call macos-use.macos-use
scrolland_traverse pid=408 x=500 y=400 deltaY=3
mcporter call macos-use.macos-use
scrolland_traverse pid=408 x=500 y=400 deltaY=-3
- - deltaY 正数 = 向下滚动，负数 = 向上滚动
- deltaX 可选，用于水平滚动
refresh_traversal
获取当前UI状态而不执行任何操作。
bash
mcporter call macos-use.macos-use
refreshtraversal pid=408
工作流模式
典型的自动化流程：
- 1. 打开应用 → 获取PID和可见元素
- 读取响应摘要中的visible_elements — 这些是带有坐标的交互元素
- 使用UI树中的坐标进行点击/输入/按键操作
- 读取响应 — 显示变化内容（差异）和新的可见元素
- 重复操作 直到任务完成
读取响应
每个工具返回一个简洁的摘要，包含：
- - status：成功/错误
- pid：进程ID（用于后续调用）
- file：包含所有元素的完整JSON文件路径（可使用grep或python3搜索）
- visible_elements：当前可见的关键交互元素及其坐标
当需要查找特定元素时，使用grep搜索JSON文件：
bash
grep -i 搜索文本 /tmp/macos-use/<文件名>.json
或使用Python解析：
bash
python3 -c
import json
with open(/tmp/macos-use/<文件名>.json) as f:
data = json.load(f)
for e in data.get(traversal,{}).get(elements,[]):
text = (e.get(text) or ).strip()
if text and 搜索 in text.lower():
print(f[{e[\role\]}] ({e[\x\]},{e[\y\]} {e.get(\width\,\?\)}x{e.get(\height\,\?\)}) {text})
提示
- - 在链式命令中始终使用 --output json 获取机器可读的结果
- 点击后，如果UI在差异中没有更新，请稍等片刻再执行 refresh_traversal
- 使用应用名称（Google Chrome）、包ID（com.google.Chrome）或路径来打开应用
- 坐标是绝对屏幕位置 — 如果窗口移动，请刷新遍历结果
- 服务器将遍历JSON写入 /tmp/macos-use/ — 这些文件是临时文件
