grader-replay
Pass
Audited by Gen Agent Trust Hub on Nov 14, 2026
Risk Level: SAFEEXTERNAL_DOWNLOADSCOMMAND_EXECUTIONINDIRECT_PROMPT_INJECTIONDYNAMIC_EXECUTION
Full Analysis
- [EXTERNAL_DOWNLOADS]: The skill provides instructions to download historical evaluation data and suite configurations from Google Cloud Storage using
gcloud storage rsync. These resources are fetched fromgs://guidance-evals/, which is managed by the vendor's infrastructure. - [COMMAND_EXECUTION]: The
scripts/replay.tsfile usesexecFileSyncto execute localgitcommands. This is used to automatically detect which grader files have been modified in the current branch compared toorigin/mainto scope the replay process. - [INDIRECT_PROMPT_INJECTION]: The skill processes untrusted data in the form of historical agent solutions (
agent.patchandindex.html) which could theoretically contain malicious instructions. - Ingestion points: Agent solution files are read from local suite directories (
~/guidance-evals/orresults/suites/) inscripts/replay.ts. - Boundary markers: Absent. The solutions are passed directly to the grading runner without explicit injection delimiters.
- Capability inventory: Includes local command execution via
execFileSync, dynamic code execution viarunPlaywright, and extensive filesystem access. - Sanitization: Absent. The script assumes the integrity of the historical data fetched from the internal evaluation bucket.
- [DYNAMIC_EXECUTION]: The replay script uses a
runPlaywrightutility to dynamically apply agent patches to base applications and execute them within a Playwright-controlled browser environment. This is the primary mechanism for verifying that graders correctly evaluate different implementation patterns.
Audit Metadata