databricks-synthetic-data-gen
Installation
SKILL.md
Catalog and schema are always user-supplied — never default to any value. If the user hasn't provided them, ask. For any UC write, always create the schema if it doesn't exist before writing data.
Databricks Synthetic Data Generation
Generate realistic, story-driven synthetic data for Databricks using Spark + Faker + Pandas UDFs (strongly recommended).
Quick Reference
| Topic | Guide | When to Use |
|---|---|---|
| Setup & Execution | references/1-setup-and-execution.md | Setting up environment, choosing compute, installing dependencies |
| Generation Approaches | references/2-generation-approaches.md | Choosing Spark UDFs vs Polars local, writing generation code |
| Data Patterns | references/3-data-patterns.md | Creating realistic distributions, referential integrity, time patterns |
| Domain Guidance | references/4-domain-guidance.md | E-commerce, IoT, financial, support/CRM domain patterns |
| Output Formats | references/5-output-formats.md | Choosing output format, saving to volumes/tables |
| Troubleshooting | references/6-troubleshooting.md | Fixing errors, debugging issues |
| Example Script | scripts/generate_synthetic_data.py | Complete Spark + Pandas UDF example |