grader-replay

Pass

Audited by Gen Agent Trust Hub on Nov 14, 2026

Risk Level: SAFEEXTERNAL_DOWNLOADSCOMMAND_EXECUTIONINDIRECT_PROMPT_INJECTIONDYNAMIC_EXECUTION
Full Analysis
  • [EXTERNAL_DOWNLOADS]: The skill provides instructions to download historical evaluation data and suite configurations from Google Cloud Storage using gcloud storage rsync. These resources are fetched from gs://guidance-evals/, which is managed by the vendor's infrastructure.
  • [COMMAND_EXECUTION]: The scripts/replay.ts file uses execFileSync to execute local git commands. This is used to automatically detect which grader files have been modified in the current branch compared to origin/main to scope the replay process.
  • [INDIRECT_PROMPT_INJECTION]: The skill processes untrusted data in the form of historical agent solutions (agent.patch and index.html) which could theoretically contain malicious instructions.
  • Ingestion points: Agent solution files are read from local suite directories (~/guidance-evals/ or results/suites/) in scripts/replay.ts.
  • Boundary markers: Absent. The solutions are passed directly to the grading runner without explicit injection delimiters.
  • Capability inventory: Includes local command execution via execFileSync, dynamic code execution via runPlaywright, and extensive filesystem access.
  • Sanitization: Absent. The script assumes the integrity of the historical data fetched from the internal evaluation bucket.
  • [DYNAMIC_EXECUTION]: The replay script uses a runPlaywright utility to dynamically apply agent patches to base applications and execute them within a Playwright-controlled browser environment. This is the primary mechanism for verifying that graders correctly evaluate different implementation patterns.
Audit Metadata
Risk Level
SAFE
Analyzed
Nov 14, 2026, 11:15 PM