nemo-evaluator-sdk
Pass
Audited by Gen Agent Trust Hub on Oct 29, 2026
Risk Level: SAFEEXTERNAL_DOWNLOADSCOMMAND_EXECUTIONDATA_EXFILTRATIONDYNAMIC_EXECUTIONINDIRECT_PROMPT_INJECTION
Full Analysis
- [EXTERNAL_DOWNLOADS]: The skill facilitates the installation of the
nemo-evaluator-launcherpackage and the retrieval of containerized benchmark harnesses from the NVIDIA Container Registry (nvcr.io), which are well-known and reputable sources. - [COMMAND_EXECUTION]: Orchestrates evaluation workflows by executing CLI commands for local Docker management and remote Slurm cluster operations, including sbatch and sacct for job lifecycle management.
- [DATA_EXFILTRATION]: Documentation includes configuration templates that reference sensitive local file paths, such as the user's SSH private key (
~/.ssh/id_rsa), which is a legitimate requirement for authenticating with remote HPC head nodes. - [DYNAMIC_EXECUTION]: The SDK supports an extensible adapter system that can dynamically discover and load custom interceptor modules from user-defined local directories or Python packages.
- [INDIRECT_PROMPT_INJECTION]: As an evaluation platform, the skill processes untrusted model outputs and benchmark datasets.
- Ingestion points: Benchmark task data (e.g., MMLU, IFEval) and responses from evaluated LLM API endpoints.
- Boundary markers: Includes a
system_messageinterceptor to wrap inputs in clear instructional boundaries. - Capability inventory: Performs file system writes (results), network requests (LLM APIs), and shell command execution (subprocess calls for Docker/Slurm).
- Sanitization: Employs container-based isolation (Docker) and cluster job isolation (Slurm) to ensure evaluation workloads remain segregated from the execution environment.
Audit Metadata