The Agent Skills Directory

[COMMAND_EXECUTION]: The skill provides Python script examples (Workflow 2) that utilize os.system() to execute external shell scripts for tracking training progress and running periodic evaluations.
[DYNAMIC_EXECUTION]: The evaluation harness supports a plugin architecture where custom tasks can be defined in YAML and linked to Python logic in a utils.py file using the !function tag. This allows for the execution of arbitrary Python code as part of the data processing or metric calculation pipeline.
[DYNAMIC_EXECUTION]: The skill documents the use of benchmarks like HumanEval and MBPP, which involve the functional execution of model-generated Python code. The harness mitigates this risk by requiring an explicit --confirm_run_unsafe_code flag from the user.
[INDIRECT_PROMPT_INJECTION]: Evaluating third-party or untrusted models introduces an indirect prompt injection surface where model outputs are ingested and parsed by the tool.
Ingestion points: Model outputs generated during benchmark tasks (e.g., MMLU, GSM8K, HumanEval) are read into the agent context for scoring.
Boundary markers: The skill relies on standardized prompts and task-specific delimiters (e.g., fewshot_delimiter, target_delimiter) defined in YAML configurations.
Capability inventory: The skill environment supports shell command execution (os.system) and functional code execution (for programming benchmarks).
Sanitization: The harness uses rule-based parsers and metric-specific verification (like Exact Match) to process outputs, and requires manual confirmation before running potentially unsafe code generated by models.

evaluating-llms-harness