evaluating-llms-harness
Pass
Audited by Gen Agent Trust Hub on Sep 17, 2026
Risk Level: SAFEINDIRECT_PROMPT_INJECTIONCOMMAND_EXECUTIONDYNAMIC_EXECUTION
Full Analysis
- [COMMAND_EXECUTION]: The documentation describes the
--allow_code_executionflag, which enables the harness to execute model-generated code for specific benchmarks like HumanEval. This is a standard and documented feature required for functional correctness evaluation. - [DYNAMIC_EXECUTION]: The skill documents the ability to define custom tasks that load and execute arbitrary Python logic from a local
utils.pyfile using the!functionYAML constructor. - [INDIRECT_PROMPT_INJECTION]: The tool ingests external data from datasets specified by
dataset_path, creating a surface for indirect prompt injection. Untrusted content is interpolated into model prompts via Jinja2 templates without explicit sanitization, while the harness possess capabilities such as writing to the file system, network access via APIs, and code execution. - [SAFE]: Documentation for API evaluation follows best practices by using environment variables for credential management and provides placeholders instead of hardcoded secrets.
Audit Metadata