The Agent Skills Directory

[COMMAND_EXECUTION]: The skill provides numerous shell commands and Python snippets that use os.system() to invoke the lm_eval CLI for running model evaluations. Evidence in SKILL.md: workflow examples use os.system() to execute evaluation scripts during training loops.
[DYNAMIC_EXECUTION]: The harness utilizes a custom task system using the !function YAML tag to execute arbitrary Python code defined in a local utils.py file. Evidence in references/custom-tasks.md: configuration examples include process_docs: !function utils.process_docs and `metric_list:
metric: !function utils.execute_code`.
[INDIRECT_PROMPT_INJECTION]: The skill is designed to ingest external benchmark datasets (e.g., from HuggingFace or local files), creating a surface where malicious instructions in the data could influence the agent or the model being evaluated. Ingestion points: external datasets specified via dataset_path in task configurations (e.g., SKILL.md, references/custom-tasks.md). Boundary markers: uses Jinja2 templates (doc_to_text) to structure inputs, but these lack robust protection against embedded instructions. Capability inventory: the skill documents the use of os.system() and the --allow_code_execution flag for executing model-generated code during benchmarks. Sanitization: relies on basic filters and the internal logic of the lm-evaluation-harness framework.

evaluating-llms-harness