Event Deduplication

Canonical selection with reputation scoring and hash-based grouping for multi-source data.

When to Use This Skill

• Aggregating data from multiple sources (news, events, products)
• Same content appears from different outlets/sources
• Need to pick the "best" version from duplicates
• Tracking deduplication metrics for optimization

Core Concepts

Simple URL deduplication isn't enough. Production needs:

• Grouping by semantic similarity (same story, different outlets)
• Canonical selection (pick the "best" version)
• Reputation scoring (prefer authoritative sources)
• Both ID-based and content-based deduplication