Drift Detection

Monitor LLM quality degradation and input/output distribution shifts in production.

Overview

• Detecting input distribution drift (data drift)
• Monitoring output quality degradation (concept drift)
• Implementing statistical methods (PSI, KS, KL divergence)
• Setting up dynamic thresholds with moving averages
• Integrating Langfuse scores with drift analysis

Quick Reference

Population Stability Index (PSI)

import numpy as np