evaluating-code-models
Pass
Audited by Gen Agent Trust Hub on Jul 16, 2026
Risk Level: SAFEEXTERNAL_DOWNLOADSCOMMAND_EXECUTIONREMOTE_CODE_EXECUTION
Full Analysis
- [EXTERNAL_DOWNLOADS]: Downloads the official evaluation harness from the BigCode Project repository on GitHub and pulls Docker images from the GitHub Container Registry for multi-language evaluation.
- [COMMAND_EXECUTION]: Executes standard CLI operations for environment setup and model benchmarking, including
git clone,pip install,accelerate launch, anddocker runcommands. - [REMOTE_CODE_EXECUTION]: Leverages the
--allow_code_executionfeature of the harness to evaluate model-generated solutions against unit tests. This behavior is documented as a necessary step for functional benchmarking, and the skill explicitly suggests using Docker containers to isolate the execution environment from the host system. - [CREDENTIALS_UNSAFE]: Mentions the use of Hugging Face authentication tokens via
--use_auth_tokenfor accessing gated models and datasets. These references are standard practices for model evaluation and do not involve credential exfiltration.
Audit Metadata