The Agent Skills Directory

[COMMAND_EXECUTION]: The skill provides numerous examples of CLI commands for installing dependencies and running the evaluation harness.
Evidence: Use of lm_eval, pip install, and accelerate launch commands in SKILL.md and referenced guides.
Evidence: Python snippets in SKILL.md and references/custom-tasks.md using os.system and subprocess to automate benchmarking workflows.
[EXTERNAL_DOWNLOADS]: The skill instructs on installing several well-known and trusted Python packages from standard registries.
Evidence: Installation of lm-eval, transformers, vllm, and human-eval using pip.
[REMOTE_CODE_EXECUTION]: The skill documents the --allow_code_execution flag, which is a specific feature of the lm_eval tool required for functional correctness benchmarks like HumanEval.
Evidence: Instructions in SKILL.md and references/benchmark-guide.md explaining how to enable code execution for evaluation.
[CREDENTIALS_UNSAFE]: The guides explain how to provide authentication for API-based models using standard environment variables.
Evidence: References to OPENAI_API_KEY and ANTHROPIC_API_KEY in references/api-evaluation.md as part of the configuration setup.

evaluating-llms-harness