Apache Beam Core Concepts

The Beam Model

Evolved from Google's MapReduce, FlumeJava, and Millwheel projects. Originally called the "Dataflow Model."

Protos

The /model directory defines the official, language-agnostic Protocol Buffer (.proto) and gRPC service specifications that establish the Beam Model and the Beam Portability Framework.

Why /model Exists (Portability & Decoupling)

Without a standardized model representation, supporting $N$ SDK languages across $M$ execution runners would require $N \times M$ separate translation layers. By defining all core pipeline concepts, data encodings, metrics, and worker RPC protocols as Protobuf messages and gRPC services, /model acts as the universal lingua franca:

• SDKs compile user pipelines into standardized Runner API protobuf graphs.

• Runners inspect, optimize, and distribute these graphs without needing SDK-specific language runtimes.

• Workers (SDK Harnesses) execute user code (DoFns) and communicate with runners over standardized Fn API gRPC channels.