auto-data-discovery
Installation
SKILL.md
Automated PII Discovery and Classification
Overview
Automated data discovery tools scan structured and unstructured data repositories to identify, classify, and catalogue personal data across the enterprise. Manual data inventories cannot keep pace with the volume, velocity, and variety of modern data processing. Automated discovery provides continuous visibility into where personal data resides, how it flows, and whether it is classified and protected according to policy. This skill covers implementation patterns for four leading platforms — Microsoft Purview, BigID, OneTrust DataDiscovery, and AWS Macie — with focus on scanning configuration, accuracy optimisation, and integration with privacy compliance workflows.
Platform Comparison
| Capability | Microsoft Purview | BigID | OneTrust DataDiscovery | AWS Macie |
|---|---|---|---|---|
| Structured data scanning | SQL Server, Azure SQL, Synapse, Cosmos DB, Oracle, PostgreSQL, MySQL, Teradata | 100+ connectors including all major RDBMS, NoSQL, data warehouses | 200+ connectors, pre-built integrations with SaaS applications | S3, DynamoDB, RDS (via Lambda) |
| Unstructured data scanning | SharePoint, OneDrive, Exchange, Azure Blob, Azure Files, AWS S3, GCP Storage | File shares, email, SharePoint, cloud storage, Slack, Teams, Confluence | File shares, email, cloud storage, collaboration platforms | S3 buckets (primary focus) |
| Classification method | 300+ built-in sensitive information types (SITs), trainable classifiers, exact data match (EDM), custom regex | ML-based NER, correlation analysis, pattern matching, custom classifiers | Pattern matching, NER, contextual analysis, custom rules | ML-based pattern matching, custom data identifiers, managed data identifiers |
| GDPR-specific classifiers | EU national ID formats, EU passport numbers, EU debit/credit card numbers, EU tax ID numbers per Member State | GDPR personal data taxonomy, Art. 9 special category detection, cross-regulation mapping | Pre-built GDPR data subject types, purpose mapping, lawful basis tagging | EU personal data identifiers (limited — primarily financial and identity patterns) |
| Accuracy tuning | Confidence levels (low/medium/high), custom keyword dictionaries, EDM for exact matching, document fingerprinting | ML model retraining, feedback loop, confidence thresholds, correlation rules | Confidence scoring, validation rules, exception management | Custom data identifiers with regex and keyword proximity, severity scoring |
| Deployment model | SaaS (Microsoft 365/Azure), hybrid with Purview governance | SaaS, on-premises, hybrid | SaaS, on-premises agent | AWS-native SaaS |
| Pricing model | Per information protection unit (Azure), per Microsoft 365 licence tier (E5 includes advanced) | Per data source connector, per TB scanned | Per data source module, per connector | Per S3 bucket evaluated, per GB scanned |