modelopt-launching-evals
Pass
Audited by Gen Agent Trust Hub on Nov 13, 2026
Risk Level: SAFECOMMAND_EXECUTIONINDIRECT_PROMPT_INJECTIONDYNAMIC_EXECUTIONEXTERNAL_DOWNLOADS
Full Analysis
- [INDIRECT_PROMPT_INJECTION]: The skill ingests untrusted data from multiple sources including evaluation client logs, server logs, results.yml files, and benchmark-specific output files like tasks.jsonl. These files contain model-generated content which could contain malicious instructions. The skill lacks explicit boundary markers or sanitization for this content while maintaining capabilities for remote command execution (SSH/rsync) and tool invocation.
- [COMMAND_EXECUTION]: The skill performs shell command execution to manage evaluations on Slurm clusters. This includes using
sshfor remote inspection,rsyncfor artifact retrieval, anduv runto execute thenemo-evaluator-launcherCLI tool. - [DYNAMIC_EXECUTION]: A Python script is embedded in the benchmark documentation to perform deduplication and status counting of JSONL-formatted task records. This script is executed at runtime via the
python3 -ccommand. - [EXTERNAL_DOWNLOADS]: The skill retrieves troubleshooting documentation from NVIDIA's official GitHub repository (
NVIDIA-NeMo/Evaluator). It also provides commands to install the launcher and dependencies (likehuggingface_hub) from remote sources.
Audit Metadata