Evaluate AI code with Logfire

How This Works

Python's pydantic_evals and Node.js's logfire/evals run a task against cases, apply evaluators, and return a report. An active Logfire or OpenTelemetry provider may export inputs and outputs even if this skill did not configure it. Intentional upload needs logfire.configure() in Python or a configured Node.js exporter.

Span-based evaluators inspect the task's OpenTelemetry span tree. Without working Logfire instrumentation, Python reports "No span tree available"; Node.js HasMatchingSpan can produce no evaluator result at all. Treat either signal as a setup failure, not evidence about the agent.

Step 1: Check for an Existing Braintrust Suite First

Cheap check, before anything else: does this repo already have an existing Braintrust suite — actual Eval(...) calls or from braintrust import Eval in source, not just a braintrust dependency listed without any real usage? This path needs no CLI auth at all — don't run Step 2 for it. The compatibility endpoint is documented only for Logfire Cloud US and EU; for any other supplied origin, use the native evaluation path instead.

Keep the existing Eval() code (Python braintrust>=0.30.1 / TypeScript braintrust>=3.24.0 — verified versions) and redirect its next run to Logfire by changing environment variables only, no pydantic_evals involved:

export BRAINTRUST_APP_URL="https://logfire-us.pydantic.dev/v1/braintrust"  # EU: logfire-eu.pydantic.dev
export BRAINTRUST_API_KEY="<logfire-project-api-key>"                     # Settings -> API Keys
unset BRAINTRUST_API_URL BRAINTRUST_PROXY_URL  # these override the endpoint above if set — the #1 "it still hit Braintrust" cause