Output Validation Skill

Goal

Make LLM/agent outputs safe and machine-reliable through schema validation and PII masking BEFORE they are used downstream or written to logs. Structured outputs must be typed and validated so a side-effecting tool never acts on malformed model text, and PII must be masked before it crosses a trust boundary (external API, log, or observability sink). Grounded in OWASP LLM05:2025 (Improper Output Handling) and LLM02:2025 (Sensitive Information Disclosure).

Steps

1. Define an output schema for every structured output

   • Pydantic in Python, Zod in TypeScript, or JSON Schema as the language-neutral contract.
   • The schema is the boundary contract — no structured output leaves the model layer without one.

2. Prefer constrained / native structured output

   • Use constrained decoding, native structured-output modes, or tool-calling so the model is guided toward valid output at generation time rather than corrected after the fact.

3. Validate, then retry-on-validation-failure (bounded)

   • Validate the raw output against the schema.
   • On failure, run a RETRY-ON-VALIDATION-FAILURE loop: feed the validation error plus the schema back to the model and regenerate, with a bounded retry count.
   • Only after exhausting retries do you surface an error — never pass invalid output downstream.