This creates:

datapackage.json with required metadata
generate_schema.py for schema generation
schema.yaml (auto-generated after database creation)
ingest/, source/, assets/ directories
README.md and PROCESS.md templates

Fixing an Existing Dataset

If tables don't show in the Zo UI:

Verify datapackage.json exists
Ensure data.duckdb is a valid database
Re-generate schema: python3 generate_schema.py

See TROUBLESHOOTING.md for common issues.

Dataset Structure

A valid Zo Dataset requires:

dataset-name/
├── datapackage.json          # Required: Dataset metadata
├── schema.yaml                # Required: Auto-generated from database
├── data.duckdb                # Required: DuckDB database
├── generate_schema.py         # Required: Script to generate schema
├── README.md                  # Recommended: Dataset documentation
├── PROCESS.md                 # Recommended: Ingestion instructions
├── ingest/                    # Optional: Ingestion scripts
├── source/                    # Optional: Raw source files
└── assets/                    # Optional: Generated files

Critical: Schema Format

ALWAYS auto-generate schema.yaml from the database. Never write it manually.

The Zo UI expects this format (list of tables with name: keys):

tables:
- name: my_table
  row_count: 10
  columns:
  - name: id
    type: VARCHAR
  - name: title
    type: VARCHAR
  - name: created_at
    type: TIMESTAMP

This format is generated by generate_schema.py.

Do NOT use manual YAML format (nested dictionaries):

# WRONG - Will not display in Zo UI
tables:
  my_table:
    columns:
      id:
        type: VARCHAR

Using the Scripts

`create_dataset.py`

Create a new dataset scaffold:

python3 scripts/create_dataset.py my-dataset

Creates the dataset in Datasets/my-dataset/ with all required files.

`generate_schema.py`

Generate schema from an existing database:

cd /home/workspace/Datasets/my-dataset
python3 generate_schema.py

When to run:

After creating data.duckdb
After modifying table structure (add/remove columns, create tables)
After changing COMMENT annotations
After any database schema changes

`validate_dataset.py`

Validate a dataset structure:

python3 scripts/validate_dataset.py /home/workspace/Datasets/my-dataset

Checks:

datapackage.json exists and is valid
data.duckdb exists and is readable
schema.yaml exists and is in correct format
Tables in schema match tables in database

Workflow Examples

Example 1: New Dataset from CSV

# 1. Create scaffold
python3 scripts/create_dataset.py sales-data

# 2. Copy CSV to source/
cp sales.csv /home/workspace/Datasets/sales-data/source/

# 3. Create database and import data
cd /home/workspace/Datasets/sales-data
python3 -c "
import duckdb
con = duckdb.connect('data.duckdb')
con.execute(\"CREATE TABLE sales AS SELECT * FROM 'source/sales.csv'\")
con.execute(\"COMMENT ON TABLE sales IS 'Monthly sales data'\")
con.close()
"

# 4. Generate schema
python3 generate_schema.py

# 5. View in Zo UI at /?t=datasets

Example 2: Fixing Broken Dataset

# 1. Validate to identify issues
python3 scripts/validate_dataset.py /home/workspace/Datasets/broken-dataset

# 2. If schema format is wrong, re-generate
cd /home/workspace/Datasets/broken-dataset
python3 generate_schema.py

# 3. Validate again
python3 scripts/validate_dataset.py /home/workspace/Datasets/broken-dataset

Example 3: Adding Comments to Tables

import duckdb

con = duckdb.connect('data.duckdb')

# Add table comment
con.execute("COMMENT ON TABLE videos IS 'YouTube videos from watchlist playlist'")

# Add column comments
con.execute("COMMENT ON COLUMN videos.title IS 'Video title from YouTube'")
con.execute("COMMENT ON COLUMN videos.view_count IS 'Total view count'")

con.close()

# Re-generate schema to include comments
# (run: python3 generate_schema.py)

Best Practices

Use COMMENT Annotations

Add inline comments to tables and columns — they're extracted into schema.yaml:

CREATE TABLE videos (
    id VARCHAR PRIMARY KEY COMMENT 'YouTube video ID',
    title VARCHAR COMMENT 'Video title from YouTube metadata',
    published_at TIMESTAMP COMMENT 'When the video was published'
) COMMENT 'Collection of videos from the watchlist playlist'

Keep Schema in Sync

Always re-run generate_schema.py after database changes:

Adding/removing columns
Changing column types
Creating new tables
Updating COMMENT annotations

Use Descriptive Names

Tables: playlist_videos, user_sessions, transactions
Columns: video_id, created_at, total_amount
Use snake_case consistently

Document in README.md

Include:

Purpose of the dataset
What data it contains
How to query it
Example queries
Business rules and caveats

Common Pitfalls

See TROUBLESHOOTING.md for detailed solutions to:

Tables not showing in Zo UI
Invalid schema.yaml format
Missing datapackage.json
Stale schema after database changes
Locked database files

Reference Materials

TROUBLESHOOTING.md - Common issues and solutions
SCHEMA_GUIDE.md - Schema formatting details
DATAPACKAGE_SPEC.md - datapackage.json reference
DUCKDB_BASICS.md - DuckDB usage examples

zo-dataset-creator

Zo Dataset Creator

Quick Start

Creating a New Dataset