LLM Streaming

Pick the right abstraction boundary first, then add resilience, instrumentation, and SSE streaming on top of it.

Trust boundary

Build privileged instructions and trusted history on the server. Do not decode a public request directly into the internal Request below: clients cannot select system/developer roles, replace stored tool results, or raise server-owned limits. Authorize conversation and retrieval access per caller and tenant.

Keep retrieved documents, user content, and tool results separate from trusted instructions. Treat generated tool arguments as proposals: validate and authorize them at execution, outside the model. Structured output and prompt delimiters do not enforce permissions. Shared defenses and adversarial checks belong to [[z-security-hardening]].

Validate output before it reaches a tool, renderer, or external destination. If policy requires whole-response review, buffer until review completes. Once streamed, content cannot be recalled. Treat Markdown links and images as possible outbound requests; prevent unauthorized destinations and secret disclosure. Exclude raw prompts, responses, credentials, and provider bodies from routine telemetry.

Interface: decorate the SDK, don't wrap it again

When a provider SDK is already in play (eino, an OpenAI client, etc.), decorate its own interface directly instead of adding a redundant custom abstraction. The real case here decorates eino's own model.ToolCallingChatModel: