File-Based IPC & Events

The Pattern

Problem: You have multiple processes (a web server and a container worker, or a host process and a spawned subprocess) that need to exchange messages and stream events. They don't share memory, and you don't want the complexity of a message broker.

Approach: Use the filesystem as the message bus. JSON files as request/response pairs, JSONL append-only logs as event streams, and atomic state.json snapshots for current status. Bridge async code with asyncio.Future objects that resolve when response files appear.

Pattern proven in production across multiple Python CLI tools and web services.

Key Design Decisions

1. JSONL append-only event log + atomic state.json snapshot

The EventEmitter writes two complementary files:

• events.jsonl — append-only, one JSON object per line, flushed immediately so tail -f works
• state.json — atomic overwrite of current status, always a complete snapshot