ui-automation

UI Automation

You are automating a desktop UI interaction. Follow the observe-act-verify loop for every action.

1. Observe — Understand Current State

Before any interaction, see what is on screen:

• Use mcp__hijak__screenshot to capture the current state.
• Use mcp__hijak__accessibility_tree to get the structured element hierarchy of the target application.
• Identify the target element and its current state (enabled, focused, visible, value).

2. Locate — Find the Target Element

Use the most reliable identification strategy, in this priority order:

1. Accessibility find (preferred): Use mcp__hijak__accessibility_find to locate elements by role, title, or value. This is the most robust method and survives layout changes.
2. OCR text matching: Use mcp__hijak__ocr_window to find text on screen, then derive coordinates from the OCR bounding boxes. Useful for non-standard UI controls.
3. Coordinates (last resort): Only use raw coordinates from the screenshot if neither a11y nor OCR can locate the element. Coordinate-based targeting is fragile and breaks when windows move or resize.