The Agent Skills Directory

[INDIRECT_PROMPT_INJECTION]: The skill is designed to ingest and process untrusted agent trajectory data (traces) from multiple external sources, which could contain embedded malicious instructions intended to influence the evaluation task generation process.
Ingestion points: Traces are ingested from various formats including MLflow, Intake, OpenTelemetry (OTLP), and NeMo Gym rollouts (e.g., lookup.atif.json and intake-trace.json).
Boundary markers: The workflow mandates the creation of a scrubbed 'safe copy' (safe/trace.atif.json) for making candidacy decisions, and uses separate private directories for raw vs. processed evidence.
Capability inventory: The skill can execute shell commands via Bash, perform file writes, and trigger containerized runs through Harbor and Docker using qualification_runtime.py.
Sanitization: The skill includes a 'prepare' command that redacts common secrets, personal identifiers (PII), and sensitive data classes (emails, SSNs, IP addresses), as well as an 'audit-evidence' tool to detect ambiguous or conflicting data markers.
[COMMAND_EXECUTION]: The script scripts/qualification_runtime.py (intended for use in isolated verifier environments) uses subprocess.Popen to execute arbitrary command arrays (argv) defined in a JSON manifest.
Evidence: In scripts/qualification_runtime.py, the run_probe function executes subprocess.Popen(probe["argv"], ...) using data loaded from qualification-probes.json. This is used to verify that the generated evaluation environment is reachable and correctly configured.
[DYNAMIC_EXECUTION]: The skill generates and executes diagnostic probes and grading scripts at runtime to verify environment integrity and score submissions.
Evidence: scripts/review_runtime.py dynamically assesses submission bytes using a provided callback, and scripts/qualification_runtime.py performs runtime reachability observations based on probe definitions generated during the authoring process.

eval-author-trace-environment