evaluating-llms-harness
Pass
Audited by Gen Agent Trust Hub on Sep 15, 2026
Risk Level: SAFEINDIRECT_PROMPT_INJECTIONDYNAMIC_EXECUTIONCOMMAND_EXECUTION
Full Analysis
- [INDIRECT_PROMPT_INJECTION]: The skill facilitates the ingestion of evaluation datasets from external sources which may contain malicious instructions designed to influence model behavior or exploit the evaluation environment. (1) Ingestion points: Dataset paths in SKILL.md and custom-tasks.md (e.g., .jsonl files). (2) Boundary markers: Jinja2 templates are used for prompt formatting but offer no security isolation between data and instructions. (3) Capability inventory: The harness supports code execution via the --allow_code_execution flag and allows subprocess calls in utility scripts. (4) Sanitization: No sanitization of ingested dataset content is documented.
- [DYNAMIC_EXECUTION]: Documentation in references/custom-tasks.md instructs users on how to use the !function YAML tag to dynamically load and execute Python logic from external files for metric calculation and document processing. This mechanism allows the execution of arbitrary Python logic defined in local configuration scripts.
- [COMMAND_EXECUTION]: The skill includes shell scripts and Python code snippets that utilize os.system and subprocess to automate the evaluation pipeline. These examples demonstrate the execution of system-level commands to manage checkpoints, run evaluation tasks, and process results.
Audit Metadata