The Agent Skills Directory

[INDIRECT_PROMPT_INJECTION]: The skill processes untrusted external content, such as Datadog traces and user-authored free text, to evaluate model performance. This content is interpolated into judge prompts, creating a potential injection surface.
Ingestion points: Data is pulled from trace_ids, ml_app, annotation_queue_id, and local_dataset_path as defined in SKILL.md.
Boundary markers: The skill implements explicit delimiters in references/eval_harness_template.py and references/eval_harness_template.mjs, including <evaluators>, <domain_notes>, <datapoint_input>, and <datapoint_output> tags.
Capability inventory: The skill uses sub-agents, writes to the filesystem, and executes shell commands for Git, Datadog CLI (pup), and script execution.
Sanitization: A _seal function is provided in the harness templates to defang homoglyphs and tag-shaped text by inserting zero-width spaces, and explicit framing instructions are provided to the judge LLM to treat content as data to be scored rather than instructions.
[DYNAMIC_EXECUTION]: The skill generates an evaluation harness locally (.auto_experiment/eval_harness.py or .mjs) by populating internal templates with user-defined rubrics and domain notes. This generated script is then executed by the agent to perform the scoring iterations.
[COMMAND_EXECUTION]: The skill invokes several shell-based tools to manage the experiment workflow, including git for branch management and commits, and the pup CLI for interacting with Datadog LLM-Obs APIs. These actions are consistent with the skill's primary purpose and author (datadog-labs).

agent-observability-auto-experiment