The Agent Skills Directory

[INDIRECT_PROMPT_INJECTION]: The skill is susceptible to indirect prompt injection as it ingests untrusted production trace data (user queries and LLM responses) to generate new evaluator prompts and Python SDK code. An attacker who can influence production trace content could potentially inject instructions that are then incorporated into the generated rubrics or code.
Ingestion points: Retrieves production data via search_llmobs_spans, get_llmobs_span_details, and get_llmobs_span_content in Phase 1.
Boundary markers: The skill does not explicitly describe the use of boundary markers when interpolating trace content into generated evaluator prompts, although it does instruct the agent to ground prompts in observed patterns.
Capability inventory: The skill has access to Bash (via pup CLI), file writing (via Write tool), and Datadog-specific MCP tools.
Sanitization: The skill includes mandatory PII scrubbing (redacting emails, phones, SSNs, and API keys) to mitigate data exposure during dataset and code generation.
[DYNAMIC_EXECUTION]: The skill generates Python scripts using the Datadog Evals SDK and writes them to the local filesystem (./evals/). These scripts include logic for BaseEvaluator and LLMJudge classes.
Evidence: Phase 3A provides detailed rules for generating .py files containing evaluator suites based on analyzed traces.
[COMMAND_EXECUTION]: The skill executes the pup CLI tool via Bash to perform search and retrieval operations when MCP tools are unavailable or pup mode is requested.
Evidence: Commands such as pup llm-obs spans search and pup llm-obs spans get-content are used to interact with Datadog services.
[EXTERNAL_DOWNLOADS]: The skill interacts with Datadog's official services and well-known endpoints to fetch trace data and publish evaluators.
Evidence: References the Datadog MCP server at https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs. These references to well-known services are documented neutrally.

agent-observability-eval-bootstrap