Human-Priority Desktop Control

Capability Boundary

Use this skill for local Windows desktop automation from shell-accessible Codex sessions. It provides a reusable helper for guarded Win32 mouse/keyboard/window/screenshot actions plus Windows UI Automation control-tree discovery and control-pattern actions.

Do not claim that the model itself has native, always-on desktop control. This skill only works when the current Codex environment can run local commands and the process has permission to inject input into the active Windows session.

The bundled helper is best-effort human-priority automation. It installs low-level keyboard/mouse hooks while running, ignores its own injected input when Windows marks it as injected, checks recent input, cursor drift, and currently pressed keys/buttons, and resumes after the configured human-idle window. It is still not a trusted kernel driver or accessibility service.

UI Automation support uses Windows' built-in System.Windows.Automation APIs through scripts/desktop_uia.ps1. It works best for Win32, WPF, WinForms, Qt, Chromium/Electron, and other apps that expose accessibility metadata. Some custom canvases, games, remote desktops, and GPU-only surfaces may expose only a coarse pane; use screenshots and guarded coordinates for those cases.

Workflow

1. State the task type as Feature unless the user is reporting a broken automation.
2. Confirm the target app/window and the exact user-visible goal.
3. Prefer app-native APIs, files, CLI commands, COM automation, browser automation, or MCP/app connectors when those can complete the task more safely.
4. For GUI interaction, prefer this order: UI Automation controls, window-relative clicks, screenshot-guided coordinates, raw screen coordinates.
5. Run one status check near the start of a desktop-control session, not before every action: