Longitudinal Measurement

AI products change over time — models get updated, usage patterns shift, and quality can drift without anyone noticing. Longitudinal measurement is how you track quality across time and catch degradation before users do.

What Changes Over Time

• Model updates: New model versions may improve some capabilities and regress others
• Prompt drift: System prompts accumulate edits that may interact in unexpected ways
• Usage evolution: Users discover new use cases that weren't tested for
• Data drift: The real-world inputs diverge from what was tested
• Expectation drift: Users' expectations change as they become more experienced

What to Measure Longitudinally

• Quality scores: Track rubric scores on a consistent test set over time
• Task success rates: Monitor whether users are completing tasks at the same rate
• Satisfaction signals: Track trends in explicit and implicit satisfaction
• Error rates: Monitor failure frequency and type distribution
• Latency: Response time changes can indicate degradation
• Engagement patterns: Changes in usage frequency, depth, and breadth

Measurement Infrastructure

• Golden test sets: A fixed set of inputs evaluated regularly to detect quality changes
• Automated evaluation: Run golden test sets automatically on a schedule
• Dashboards: Visualise trends and set alerts for significant changes
• Regression detection: Statistical methods to distinguish real changes from noise