PyAutoGUI Controller

Use this skill when the user needs local desktop automation on this Windows machine and normal browser tools are not enough.

Use for

• - Open desktop apps and interact with their windows
• Take screenshots
• Type Chinese/English text into native apps
• Click buttons/inputs on screen
• Run short natural-language automation commands through the local controller
• Browser actions that require screen-based control instead of DOM/web APIs

Avoid for

• - Normal web reading/searching → use web_fetch, browser, or search skills
• Mobile automation
• High-assurance unattended long workflows

Project location

The controller project lives at:

INLINECODE2

Primary invocation

Run the local controller through the wrapper script:

CODEBLOCK0

The wrapper calls the real project entrypoint:

INLINECODE3

Recommended workflow

1. 1. Prefer a single compound command over multiple split commands
2. Start with simple actions
3. Inspect returned JSON for success, results, runtime, and INLINECODE7
4. If the command fails, simplify the wording and retry

Good command examples

CODEBLOCK1

Practical notes

• - Compound commands are more reliable than split follow-ups
• OCR / DOM readiness may affect accuracy; check INLINECODE8
• Move the mouse to the top-left corner if PyAutoGUI fail-safe is triggered
• This skill is local-machine specific; it assumes the project path above exists