Agent Evals and Observability

Evaluation asks whether behavior meets a defined criterion on a declared dataset or production sample. Observability supplies traces, logs, metrics, correlations, and diagnostic context. Use both; neither proves what the other does.

Workflow

1. Define the decision, risk, task contract, trajectory contract, and unacceptable outcomes. Select evidence by harm, reversibility, and deployment stage, not a staged completeness scale.
2. Create an immutable dataset version and manifest before comparing versions. Declare provenance, rights/consent, slices, fixtures, expected side effects, contamination risk, limitations, retention, and changelog.
3. Select complementary graders that observe the claimed property. Use deterministic checks for observable mechanics; use execution/environment checks for state and side effects; use human, model-judge, pairwise/ranking, domain, and safety review where appropriate.
4. Run the candidate and baseline under comparable conditions. Preserve run configuration, stochastic repeats where variability affects the decision, failures/timeouts, and trajectory evidence.
5. Report a multidimensional profile and uncertainty. Use paired comparisons where possible; inspect slices, missingness, base rates, confounders, effect sizes, and multiple comparisons. Non-significance is not equivalence.
6. Apply a risk-tiered release gate: hard safety, privacy, authorization, and side-effect invariants cannot be averaged away. Record authority, insufficient-evidence outcomes, rollback, and follow-up. Load release-engineering when this evidence must be incorporated into artifact promotion, deployment, rollback, or a broader release train.
7. Instrument production with minimized, redacted telemetry. Feed verified incidents and near misses into cases after consent, transformation, and contamination review.

When an evaluation needs intervention detail, keep the existing aggregate count separate from optional typed events. Absence of event detail means it was not collected; it does not mean zero. Collect only fields needed for the question, exclude raw messages and hidden reasoning by default, and do not treat lower counts alone as success. See trajectory review.