nemo-evaluator-plugin

Warn

Audited by Socket on Jul 22, 2026

3 alerts found:

AnomalySecurityx2
AnomalyLOW
assets/specs/exact_match_benchmark.json

No direct indicators of classic malware (e.g., reverse shells, file destruction, or exfiltration to unknown domains) are visible in this configuration fragment. The dominant supply-chain/security concern is the embedding of metric logic as cloudpickle payloads: if the runtime deserializes these blobs, arbitrary code execution is possible during unpickling. Treat the cloudpickle blobs and the consumer runtime as a trusted boundary—verify provenance/integrity and ensure deserialization is safe/locked down. Otherwise, this artifact functions as an LLM evaluation harness that sends prompts to a specified external API using a secret reference.

Confidence: 63%Severity: 67%
SecurityMEDIUM
assets/specs/llm_as_judge.json

No explicit malicious network targets beyond the legitimate NVIDIA chat-completions endpoint are visible in the JSON, and there is no direct evidence of file/system modification. However, the artifact embeds an opaque cloudpickle payload intended for runtime deserialization, which is a well-known arbitrary code execution vector if trust boundaries or integrity verification are missing. Combined with authenticated outbound API usage, this creates a high supply-chain/runtime execution risk that should be treated as untrusted until the consumer’s deserialization path and provenance/integrity protections are confirmed.

Confidence: 62%Severity: 85%
SecurityMEDIUM
assets/specs/exact_match_metric.json

No explicit malicious behavior is visible in the snippet itself, but the artifact contains an embedded opaque cloudpickle payload that is intended for deserialization/execution by the consumer. If payload.blob is not cryptographically verified and the consumer unpickles it with standard cloudpickle/pickle loaders, it creates a significant supply-chain risk of arbitrary code execution. Treat this package as potentially dangerous unless the deserialized content is strictly trusted and loaded using hardened controls (signature/allowlisting and sandboxing).

Confidence: 65%Severity: 75%
Audit Metadata
Analyzed At
Jul 22, 2026, 07:47 AM
Package URL
pkg:socket/skills-sh/NVIDIA-NeMo%2Fnemo-platform%2Fnemo-evaluator-plugin%2F@8ba4d50ff9bafe60ac923b28f895bc366e43701a3119b8592e94610a013c2205
Security Audit — socket — nemo-evaluator-plugin