UI Element Ops

Parse one or more screenshots into a machine-readable JSON schema with:

• - type (normalized UI element type)
• INLINECODE1 and INLINECODE2
• INLINECODE3 (OCR/caption content when available)
• INLINECODE4 flag
• optional overlay image with labeled boxes
• desktop actions via scripts/operate_ui.py (click/type/key/hotkey/screenshot)
• element query and orchestration via scripts/operate_ui.py (find, wait)
• coordinate calibration profile for multi-display/DPI/window offset (calibrate)

Quick Start

1. 1. Prepare runtime once per machine:

CODEBLOCK0

1. 2. Parse one screenshot:

CODEBLOCK1

1. 3. Read outputs:

• - INLINECODE10
• INLINECODE11

1. 4. One-step capture + parse with randomized names:

CODEBLOCK2

Workflow

1. 1. Confirm screenshot path and desired output path.
2. Run scripts/bootstrap_omniparser_env.sh when .venv or OmniParser weights are missing.
3. Run scripts/run_parse_ui.sh for standard parsing.
4. Report absolute output paths and summary counts: total, clickable, by_type.
5. Call out obvious quality risks for tiny text or dense icon layouts.
6. Execute desktop actions when requested:

- list elements: python3 skills/ui-element-ops/scripts/operate_ui.py list --elements <json> - find elements: python3 skills/ui-element-ops/scripts/operate_ui.py find --elements <json> --type button --text-contains login - wait for appear/disappear: python3 skills/ui-element-ops/scripts/operate_ui.py wait --elements <json> --state appear --text-contains continue - click by id: python3 skills/ui-element-ops/scripts/operate_ui.py click --elements <json> --id e_0001 - screenshot: python3 skills/ui-element-ops/scripts/operate_ui.py screenshot (defaults to user tmp dir) - calibrate coordinates: INLINECODE23

Tunables

• - Edit type mapping keywords in references/type_rules.example.json.
• Use advanced parser args via scripts/parse_ui.py --help.
• Use --use-paddleocr only when paddleocr/paddlepaddle are installed.

Outputs

• - Main JSON output:

- schema_version, pipeline, image, counts, elements - each element has id, type, bbox_px, bbox_norm, text, clickable

• - Overlay PNG output:

- same screenshot with labeled detection boxes

Failure Handling

• - Missing dependencies or weights: run bootstrap script again.
• Permission/cache errors under $HOME: keep temporary caches under /tmp (handled by run script).
• CPU-only machine: expect slower inference.
• Performance note: parse/capture-and-parse commands are heavy; avoid very tight loops and reuse recent elements.json when possible.
• Headless environment limitation:

- usable without GUI: parse/list/find/wait/calibrate on existing files. - requires GUI session: click/click-xy/type/key/hotkey/screenshot/screen-info.