Game automation: the agent's hands and eyes

The real game is the oracle. The loop is launch → navigate → set up the scene → act → screenshot and read logs → decide. Build it once per game, then every change you make can be checked in a minute.

Windows (native or from WSL): um win

um win setup                                   # once: PowerShell tools + an ffmpeg with gfxcapture, picks NVENC/AMF/QSV/x264
um win ps                                      # windowed processes: pid, name, title
um win launch --steam 105600                   # or: um win launch "C:\Games\Foo\Foo.exe" -- -windowed
um win shot --exe Terraria.exe shot.png --scale 0.33   # full frame + a 1/3 copy that's cheap to look at
um win drive --proc Terraria "focus" "click 640 360" "key 0x1B" "type hello" "hold 0x44 1500"
um win drive --proc Terraria idle              # seconds since the user last touched mouse/keyboard
um win kill <pid>                              # exact PID only
um win reg get "HKCU\Software\..."             # registry (reg set backs the key up first)

From Python (for longer scripts), from um.win import Drive, shot, Recorder, then d = Drive("AoE2DE_s"), d.focus(), d.click(x, y), d.key("0x0D").