Exploring Data

0. Route by size FIRST

ls -la <filepath>   # or: wc -l for row estimate

• < 200MB and < ~5M rows → ydata-profiling path (section A). Exact stats, interactive HTML.
• Larger → large-file path (section B). ydata-profiling loads everything into pandas and will crawl or OOM; the DuckDB/sketch path runs in fixed memory at any size.
• Task-specific ops (any size): duplicates, join feasibility, drift → section C.

A. Standard path (ydata-profiling)

1. Check if installed (instant)

bash /mnt/skills/user/exploring-data/scripts/check_install.sh

Returns: installed or not_installed