Desktop Agent Ops

Use this skill as a main-agent operating manual for desktop GUI tasks.

---

MANDATORY: Auto-setup gate (FIRST ACTION, every time)

CODEBLOCK0

If "ready": false, run setup (installs EVERYTHING automatically):

CODEBLOCK1

Auto-installs on first run:

1. 1. Platform detection (macOS / Windows / Linux)
2. INLINECODE1 + tesseract (macOS via brew; Linux guide printed)
3. OCR language packs auto-detected from system locale (中文→chi_sim, 日本語→jpn, etc.)
4. Python venv + pillow, pyautogui, pytesseract, opencv-python, numpy (via uv or pip)
5. OS permissions (Screen Recording, Accessibility, Automation) with auto-open System Settings
6. Smoke test (screenshot + mouse move verification)

After setup, set $PY for ALL subsequent calls:
CODEBLOCK2

Do NOT proceed if setup is not ready.

---

Core Execution Loop

Every desktop task follows this loop. No exceptions.

CODEBLOCK3

Key principles:

• - One action at a time. Never chain blind actions.
• Always verify after each action. If verification fails, recapture and retry — do NOT guess.
• Always work within a specific window. Never click based on full-screen assumptions.

---

Window-Scoped Targeting (THE CORRECT WAY)

NEVER do OCR or clicking on a full-screen screenshot. Always scope to the target app window.

The 6-Step Pipeline

CODEBLOCK4

Shortcut (RECOMMENDED for most targeting):

CODEBLOCK5

This single command: focuses app → gets bounds → OCR within window → returns best_candidate with {x, y, within_window}.

Why window-scoped matters:

Approach	Risk
❌ Full-screen OCR	"搜索" in WeChat AND Chrome → clicks wrong app
✅ Window-scoped

"搜索" ONLY in WeChat window → correct click |

---

Failure Recovery (CRITICAL)

When something fails, follow these rules:

OCR finds nothing

1. 1. Re-focus the app: INLINECODE6
2. Re-get bounds: front-window-bounds --app "AppName" (window may have moved/resized)
3. Take a fresh screenshot and read it visually
4. Try a different region label (e.g. content_area instead of bottom_input)
5. Try lowering OCR confidence: INLINECODE10

Click doesn't work

1. 1. Screenshot with cursor to check cursor position
2. The window may have moved — re-get bounds
3. Try clicking a few pixels offset from the OCR center
4. Check if a dialog/popup is blocking the target

App state changed (login screen, dialog, etc.)

1. 1. ALWAYS re-get window bounds after any major UI change
2. ALWAYS re-run OCR after navigation or state change
3. Never reuse old coordinates — they may be stale

General retry rule

• - Maximum 3 retries per action
• Each retry must recapture fresh state
• If 3 retries fail, report the failure with screenshots and stop

---

Generalization: How to Apply This to ANY App

The pipeline works for any desktop application. Here is how to reason about new apps:

Step-by-step for ANY new app:

1. 1. Identify the app name exactly as it appears in the system (e.g. "Google Chrome", "微信", "System Settings")
2. Focus and get bounds — this tells you the window's exact position
3. Screenshot the window — look at what's on screen
4. Identify the target — what text, button, or area do you need to interact with?
5. Use OCR to find it — INLINECODE11
6. Verify and click

Common patterns across apps:

Task	How to do it
Click a button	OCR find text → verify → click
Type in a field

OCR find field label → click field → type --text | | Search for something | OCR find search box → click → type query → press return | | Scroll a list | Get window bounds → scroll at window center with --x --y | | Switch between apps | focus-app --name "OtherApp" → re-get bounds | | Handle a dialog | Screenshot → OCR for dialog buttons → click appropriate one | | Navigate menus | Click menu item → wait → screenshot → OCR new menu → click | | Select from dropdown | Click dropdown → wait → OCR options → click selection | | Read screen content | OCR the window → extract all text boxes | | Verify an action | Screenshot before and after → compare or OCR for expected text |

App-specific adaptations:

App type	Special considerations
Chat apps (WeChat, Slack, etc.)	Verify conversation title before typing; use insert-newline for multi-line; verify send mechanism
Browsers (Chrome, Safari, etc.)

Address bar at top; content area varies; may need to handle tabs | | System Settings | Deep navigation; panels change; re-get bounds after each navigation | | File managers (Finder, Explorer) | Sidebar + content area; double-click to open; path bar for navigation | | Editors (VS Code, TextEdit, etc.) | Tab bar + editor area; use hotkeys for save/undo; type in editor area |

---

Text Input and Send Rules

Typing text

$PY scripts/desktop_ops.py type --text "your message"

• - Uses clipboard paste as primary method on all platforms — reliable for all languages including CJK
• macOS: set the clipboard to + Cmd+V (single osascript call)
• Windows: PowerShell Set-Clipboard + Ctrl+V (falls back to clip.exe)
• Linux: xclip + INLINECODE22
• First click on the input field to focus it before typing

Multi-line messages

$PY scripts/desktop_ops.py type --text "first line"
$PY scripts/desktop_ops.py insert-newline
$PY scripts/desktop_ops.py type --text "second line"

• - Use insert-newline for literal line breaks
• Do NOT use \n in type --text — it may trigger send in some apps

Sending a message

1. 1. Preferred: Look for a visible send button (e.g., 发送) via OCR, then click it
2. Alternative: Use press --key return ONLY when the app is verified to use Enter-to-send
3. Never guess which send method to use — verify first

Backend priority (macOS)

Operation	Primary	Fallback
INLINECODE28	Clipboard paste	cliclick (ASCII only)
INLINECODE29

AppleScript key code | cliclick kp: |

| hotkey | cliclick kd:/t:/ku: | pyautogui | | click | cliclick | pyautogui |

Important: cliclick kp:return is NOT recognized by WeChat — always use AppleScript for key press.
Important: cliclick t: silently drops CJK characters — always use clipboard paste for text input.

---

DPI / HiDPI / Retina (All Platforms)

Handled automatically. No manual DPI work needed.

Platform	Common scales	Detection method
macOS Retina	2.0x	screenshot pixels ÷ logical screen bounds
Windows HiDPI

1.25x, 1.5x, 2.0x | screenshot pixels ÷ pyautogui.size() |
| Linux X11 | 1.0x, 1.5x, 2.0x | screenshot pixels ÷ pyautogui.size() |

OCR output: box = logical (use for mouse), pixel_box = raw pixels, dpi_scale = factor.

---

CLI Quick Reference (EXACT parameter names)

CRITICAL: Use EXACTLY these names. Do NOT guess.

desktop_ops.py

CODEBLOCK8

ocr_text.py

CODEBLOCK9

target_resolver.py

CODEBLOCK10

taskcontext.py / cleanuptask.py

CODEBLOCK11

window_regions.py

CODEBLOCK12

Labels: top_search, left_sidebar, left_sidebar_top, title_header, content_area, toolbar_row, bottom_input, primary_action

---

Workflow Examples

Example 1: Click a button by text (any app)

CODEBLOCK13

Example 2: Type and search

CODEBLOCK14

Example 3: Send a chat message (WeChat, Slack, etc.)

CODEBLOCK15

Example 4: Scroll a list and find an item

CODEBLOCK16

Example 5: Handle an unexpected dialog

CODEBLOCK17

---

Reference Documents

Load as needed:

Document	When to read
INLINECODE48	Core 8-step closed loop
INLINECODE49

macOS-specific tools and permissions |
| references/platform-windows.md | Windows setup |
| references/platform-linux.md | Linux X11/Wayland setup |
| references/operation-patterns.md | Reusable task templates |
| references/validation-patterns.md | Two-stage validation |
| references/precise-targeting.md | 5-layer precision targeting |
| references/target-providers.md | Provider ordering and fallback contract |
| references/coordinate-reconstruction.md | Rebuild click coordinates from screenshot evidence |
| references/chat-app-macos.md | Chat app workflow |
| references/app-wechat-desktop.md | Cross-platform WeChat guidance |
| references/cleanup-rules.md | Cleanup timing and scope |
| references/collaboration-rules.md | When multi-agent collaboration is justified |
| references/example-cases.md | Repeatable task examples |
| references/reproducible-setup.md | Host bring-up checklist |

Scope

Use this skill for: chat apps, browsers, file managers, editors, office apps, system settings, any closed desktop software with no usable API.

Hard Rules

1. 1. Always run auto-setup gate first
2. Always use EXACT parameter names from CLI reference — never guess
3. Always scope OCR to the target app window — NEVER full-screen OCR
4. Always: focus-app → front-window-bounds → OCR within window → verify → act
5. Always pass --python $PY to ocrtext.py and targetresolver.py
6. Always verify coordinates are within window bounds before clicking
7. Always re-get window bounds after any UI state change (login, dialog, navigation)
8. Use insert-newline for line breaks; never use \n in type --text
9. For send actions: prefer visible send button; use press --key return only when verified
10. One action at a time; verify after each
11. Maximum 3 retries per action; each retry must recapture fresh state
12. Cleanup is mandatory at task end
13. If verification fails, recapture and rebuild — do not retry blindly