AI Evals Observability

Own production evidence and operational signals. Do not reconstruct or override authoritative eval decisions.

Observe deployed behavior

1. Define the operational question, target, environment, sampling policy, retention, privacy boundary, and response owner.
2. Instrument provider-neutral traces for model calls, retrieval, tools, handoffs, errors, retries, latency, usage, and cost. Preserve target, prompt, tool, and configuration versions.
3. Normalize evidence without erasing source provenance. Correlate traces and score observations with stable request, session, scenario, and eval row IDs where available.
4. Build monitors and dashboards for operational health, drift, score distributions, failure clusters, cost, and latency. Distinguish missing coverage from measured zero.
5. Alert on declared thresholds and route incidents to the owning engineering lane. Keep diagnostic signals separate from release-authoritative gates.
6. Export selected, redacted evidence to ai-evals for dataset curation, failure attribution, calibration, and regression evaluation.

Preserve the boundary