Add End-to-End OTEL Tracing + Evaluation with Scorable

These instructions guide you through instrumenting an LLM application with OpenTelemetry, shipping traces to Scorable, and configuring server-side evaluation filters that automatically score matching traces. The result: every (or a sampled subset of) production LLM call gets observed, traced, and scored — and the score lands back on the same trace as a child span carrying the OpenTelemetry GenAI evaluation attributes.

Execution Contract

You are responsible for completing the OTEL tracing + evaluation setup end-to-end in as few turns as possible.

• You MUST analyze the codebase to identify the LLM framework and entry points.
• You MUST install and use the Scorable CLI directly.
• You MUST add the OTEL instrumentation code yourself.
• You MUST run a test request and verify traces land in Scorable.
• You MUST create at least one evaluation filter via the CLI.
• You MUST verify the filter triggered an evaluation by inspecting the trace's spans.
• You MUST update project documentation for usage.
• You MUST NOT delegate technical steps to the user except where explicitly required (API key entry, secrets).
• You MUST continue until implementation is complete or a hard blocker is reached.

Overview