streamling source plugin — Agent Skill

A source is the head of a pipeline: it owns its schema, produces RecordBatches on demand, and drives checkpointing. Sources are pull-based — streamling calls generate_batch() repeatedly and stops when is_running() is false or the pipeline drains.

Prerequisite: streamling-plugin-basics (crate setup, registration, lifecycle, errors).

The trait

#[async_trait]
pub trait SourcePlugin: SupportsGracefulShutdown + Send + Sync {
    async fn initialize(&self) -> Result<(), PluginError>;
    fn output_schema(&self) -> Result<SchemaRef, PluginError>;
    fn labels(&self) -> Vec<PluginLabel> { Vec::new() }            // optional identity
    async fn generate_batch(&self) -> Result<RecordBatch, PluginError>;
    async fn process_checkpoint_marker(&self, epoch: CheckpointEpoch) -> Result<(), PluginError>;
    async fn process_checkpoint_finalizer(&self, epoch: CheckpointEpoch) -> Result<(), PluginError>;
}