nightly-eval-investigation

Installation
SKILL.md

Nightly Evaluation Investigation

This skill automates the retrieval and multi-agent comparison of remote nightly evaluation runs from Google Cloud Storage (GCS) to diagnose system-wide guidance health, task drift, and over-prescribed guides (and also covers single-guide evaluation diagnosis via gd eval).

Core Objectives

  1. Cross-Agent Diagnostics: Compare results across three distinct, modern agents (Claude Code, Codex CLI, and Jetski CLI) to locate patterns that are agent-agnostic.
  2. Guide Discovery Audit: Catch cases where agents skip the expected guide or over-retrieve irrelevant guides.
  3. Health Thresholding: Flag tasks that are over-prescriptive or already baseline-known (meaning unguided runs already pass easily), and surface common guided check failures as informational diagnostics without setting a guided pass-rate target.
  4. Structured Reporting: Produce reliable Markdown and JSON artifacts that downstream automation can easily ingest.

[!IMPORTANT] CRITICAL CONSTRAINT - READ-ONLY INVESTIGATION ONLY In bulk nightly investigation mode, this skill is strictly for diagnostics, investigation, and suggesting recommendations. The agent MUST NOT under any circumstances edit, modify, create, delete, or touch any files under the guides/ directory (including task files, guides, graders, expectations, or demos). All suggestions for fixes must be documented exclusively in the generated investigation report markdown file under the "Actionable Recommendations" section. No automated or manual code remediation should be performed during a bulk nightly audit. (When asked to diagnose and fix a specific single guide's eval locally, check the 8 red lines in .agents/skills/project-evals/SKILL.md first and never change guide.md to fit a grader.)


Quick Reference: Remote Dashboard Results (GCS) & Local Eval Artifacts

Installs
49
GitHub Stars
9.0K
First Seen
Jun 12, 2026