AI Hallucination SLO Doc Skill

Overview

The AI complement to the SaaS SLO doc. Treats hallucination, citation, abstention, and safety violations as first-class SLIs with their own error budgets and rollback rules.

Core Instructions

Step 1: SLI inventory per AI feature

Required SLIs:

• Factuality SLI — % of responses on a production-sample where judge-LLM marks all claims as supported.
• Citation accuracy SLI (RAG only) — % of citations matching source spans within tolerance.
• Abstention precision SLI — % of abstain responses that correctly should have abstained.
• Abstention recall SLI — % of should-abstain inputs that did abstain.
• Safety violation SLI — count of outputs hitting content-policy or PII filters per million calls.
• Latency SLI (already in parent SLO doc, restated for AI clarity).
• Cost-per-call SLI (cross-link to cost runbook).