Python Reliable Subprocess Handling

Prerequisites

This skill extends myai's engineering-principles. Load that first. writing-python-code is assumed to be already loaded. Start with writing-python-code for the baseline subprocess rules (no shell=True, create_subprocess_exec, handle TimeoutError); this skill covers the traps that only show up once a timeout, a kill, or a process tree is involved.

Ready-made implementation + tests. this repo ships a self-contained version of the reference pattern at shared/subprocess.py, with real-process tests at shared_tests/test_subprocess_utils.py and the flag-driven child-process stub they spawn at shared_tests/fixtures/tool_stub.py. Copy them into a project and adapt. The shipped runner covers the full set: timeout, process-tree kill, caller-cancellation cleanup, bounded EOF waits on both paths, an output cap, an optional graceful-kill window, optional stderr merging, and DEVNULL stdin. The text below explains why that code is shaped the way it is, so you can change it on purpose instead of rediscovering the traps.

Status: working memo, not a guide. Everything below came from one real debugging session; treat it as field notes. Verify against your runtime before relying on it, and expect corrections — some of this is version-sensitive behavior of asyncio internals, not documented contracts.

The big trap: communicate() loses output when cancelled

asyncio.subprocess.Process.communicate() drains both streams into its own local buffer while it runs. If you wrap it in asyncio.timeout(...) and the timeout fires, the cancellation throws that buffer away. The data is gone:

• it was already consumed from the OS pipe into the coroutine's locals;
• after you kill() the process, the pipes hit EOF, so any later communicate() / read() returns b"".

Symptom we hit: "timeout worked, but the recovered output is empty even though the installer had clearly printed progress." Confirmed empirically — do not trust a second communicate() after cancellation to recover anything.