Harvard Artifacts Data Engineering Pipeline

Skill by ara.so — Data Skills collection.

This project provides an end-to-end data engineering solution for collecting, transforming, storing, and analyzing artifact data from the Harvard Art Museums API. It demonstrates production-ready ETL pipelines, relational database design, SQL analytics, and interactive Streamlit dashboards.

What This Project Does

• API Integration: Fetches artifact data from Harvard Art Museums API with pagination and rate limiting
• ETL Pipeline: Extracts nested JSON, transforms into relational schema, loads into SQL database
• Database Design: Implements normalized tables (artifactmetadata, artifactmedia, artifactcolors)
• SQL Analytics: Executes 20+ predefined analytical queries
• Visualization: Interactive Plotly charts rendered through Streamlit

Installation