evaluating-llms-harness
Warn
Audited by Gen Agent Trust Hub on Jun 29, 2026
Risk Level: MEDIUMCOMMAND_EXECUTIONREMOTE_CODE_EXECUTION
Full Analysis
- [COMMAND_EXECUTION]: The skill utilizes
os.systemwithinSKILL.mdto automate evaluation tasks during training loops. This involves interpolating variables such as training steps directly into shell commands, which could be exploited if those variables are maliciously manipulated. - [REMOTE_CODE_EXECUTION]: The documentation explicitly instructs users to use the
--allow_code_executionflag for benchmarks like HumanEval. This flag enables the runtime execution of code generated by the LLMs being tested, which is a high-risk operation when evaluating untrusted models. - [COMMAND_EXECUTION]: The
references/custom-tasks.mdguide demonstrates the implementation of custom Python utilities that utilize thesubprocessmodule to execute and verify code outputs during the evaluation process. - [REMOTE_CODE_EXECUTION]: The skill provides instructions for loading and executing arbitrary Python logic via the
!functionYAML tag in custom task configurations, allowing for the execution of external script logic within the evaluation harness.
Audit Metadata