The Pipeline (Pipes and Filters) Paradigm

When to Employ This Paradigm

• When data must flow through a fixed sequence of discrete transformations, such as in ETL jobs, streaming analytics, or CI/CD pipelines.
• When reusing individual processing stages is needed, either independently or to scale bottleneck stages separately from others.
• When failure isolation between stages is a critical requirement.

When NOT To Use

• Interactive request/response systems (use archetypes:architecture-paradigm-client-server)
• Stages that must share mutable state, which the pattern cannot express

Adoption Steps

1. Define Filters: Design each stage (filter) to perform a single, well-defined transformation. Each filter must have a clear input and output data schema.
2. Connect via Pipes: Connect the filters using "pipes," which can be implemented as streams, message queues, or in-memory channels. validate these pipes support back-pressure and buffering.
3. Maintain Stateless Filters: Where possible, design filters to be stateless. Any required state should be persisted externally or managed at the boundaries of the pipeline.
4. Instrument Each Stage: Implement monitoring for each filter to track key metrics such as latency, throughput, and error rates.
5. Orchestrate Deployments: Design the deployment strategy to allow each stage to be scaled horizontally and upgraded independently.