ADB Phone Control

Control Android devices through ADB with a structured observe-locate-act-verify loop.

Requirements

• - adb — Android Debug Bridge, must be in PATH
• python3 — Required for INLINECODE0
• ADBOUTPUTDIR (optional env var) — Directory for saving screenshots and UI dumps; defaults to current working directory

Permissions Used

This skill executes the following on the connected Android device:

• - adb shell input — tap, swipe, text input
• INLINECODE2 — UI hierarchy extraction
• INLINECODE3 — screen capture
• INLINECODE4 — ADBKeyboard IME input (for CJK text)
• INLINECODE5 — clipboard-based text input fallback

Prerequisites

Before any operation, verify device connection:

CODEBLOCK0

If no device found, instruct the user to:

1. 1. Connect via USB and enable USB Debugging
2. Or connect wirelessly: INLINECODE6

Core Principle

NEVER guess coordinates from screenshots. ALWAYS use UI hierarchy as the primary locator.

Screenshots are for human-readable context and visual verification. UI dumps give exact pixel bounds.

Operation Loop

Every interaction follows this cycle:

CODEBLOCK1

Do NOT skip the VERIFY step. UI transitions may take time; always confirm before proceeding.

Helper Functions

Source the helper script before starting any operation session:

CODEBLOCK2

Available Functions

Element Lookup Details

INLINECODE30 and adb_tap_id work by:

1. 1. Running adb_dump to get fresh UI hierarchy
2. Parsing the XML for matching text= or resource-id= attributes
3. Extracting the bounds="[x1,y1][x2,y2]" attribute
4. Computing center point: INLINECODE36
5. Executing INLINECODE37

If multiple matches are found, the function taps the first match and prints a warning.
If no match is found, the function prints an error — fall back to adb_screenshot + Read tool for visual inspection.

Standard Operating Procedure

Phase 1: Setup

CODEBLOCK3

Phase 2: Navigate & Operate

For each interaction step:

CODEBLOCK4

Phase 3: Text Input

CODEBLOCK5

Critical Rules

1. UI Dump First, Screenshot Second

• - uiautomator dump gives exact bounds, element states (enabled/focused/clickable), text content, and resource IDs
• Screenshots only for: visual verification, understanding layout context, or when UI dump fails (e.g., animations, WebView content)
• When UI dump returns elements with NAF="true", the element has No Accessible Framework info — use screenshot + coordinates as fallback

2. Keyboard Awareness

• - Always hide keyboard before tapping non-input elements. The keyboard shifts the layout, making UI dump bounds stale.
• After typing, call adb_hide_keyboard then adb_dump before tapping anything else.
• If uiautomator dump returns ERROR: could not get idle state, the keyboard animation may still be running — wait 1s and retry.

3. Wait Strategy

• - After tap: wait 1s before next dump/screenshot
• After launching app: wait 2-3s
• After page navigation: wait 2s
• After typing: wait 0.5s
• If UI hasn't changed after action: wait longer, up to 5s, then re-check
• Never blindly chain actions without verification

4. Chinese / CJK Text Input

INLINECODE45 does not support CJK characters natively. The helper adb_input_text handles this by:

• - Using adb shell am broadcast with ADBKeyboard if available
• Falling back to clipboard-based input: copy to clipboard via adb shell service call clipboard, then paste

If ADB Keyboard IME is installed (com.android.adbkeyboard), enable it:
CODEBLOCK6

5. Coordinate System

• - All coordinates are in physical pixels matching the device resolution
• INLINECODE50 returns the canonical resolution (e.g., 1080x2340)
• Screenshot pixel dimensions may differ from device resolution — never estimate coordinates from screenshot pixel positions
• Always derive coordinates from uiautomator dump bounds

6. Handling Failures

If an action doesn't produce the expected result:

1. 1. Re-dump UI hierarchy — the element may have moved or state changed
2. Take a screenshot — visual context may reveal popups, loading states, or errors
3. Check if the element is enabled="true" and clickable="true" before tapping
4. If element is not found by text, try partial match or search by resource-id
5. If the app is in a WebView, UI dump may not capture web elements — use screenshot + coordinate estimation as fallback

7. App Launch

Prefer adb_find_package + adb_launch_app over monkey command:
CODEBLOCK7

Limitations

• - uiautomator dump doesn't work during animations — wait for idle state
• WebView/Flutter/game content may not appear in UI hierarchy — use screenshot-based approach
• Some custom views may have empty text and no resource-id — use bounds + screenshot cross-reference
• Maximum ~100 actions per task is a reasonable limit to avoid infinite loops