SQLite for Structured Data

Decision Check

Before writing any data analysis code, evaluate:

1. Will the data be queried more than once? -> Use SQLite
2. Are GROUP BY, COUNT, AVG, or JOIN operations needed? -> Use SQLite
3. Is custom Python/jq parsing code about to be written? -> Use SQLite instead
4. Is the dataset >100 records? -> Use SQLite

If the answer to any question above is YES, use SQLite. Do not write custom parsing code.

# Custom code for every query:
cat data.json | jq '.[] | select(.status=="failed")' | jq -r '.error_type' | sort | uniq -c

# SQL does the work:
sqlite3 data.db "SELECT error_type, COUNT(*) FROM errors WHERE status='failed' GROUP BY error_type"