Langfuse -- LLM Observability and Evaluation for DSPy

Step 1: Understand the setup

Before writing code, clarify:

1. Cloud or self-hosted? Langfuse Cloud (managed) or self-hosted (Docker)?
2. What do you need beyond tracing? Scores/evals, annotation queues, experiment tracking, or just traces?
3. Do you need custom metadata on traces? User IDs, session IDs, tags for filtering?
4. Short-lived or long-running? Scripts need langfuse.flush(); servers do not.

What is Langfuse

Langfuse is an open-source LLM observability platform with auto-instrumentation for DSPy via the OpenInference plugin. It traces every LM call, retrieval, and module execution, then adds evaluation, scoring, and annotation on top.

What gets traced