SciDataCopilot: Agentic Scientific Data Preparation

This skill enables Claude to build end-to-end agentic data preparation pipelines for heterogeneous scientific datasets. Based on the SciDataCopilot framework, it implements a four-agent architecture (Data Access, Intent Parsing, Data Processing, Data Integration) that transforms raw experimental data — CSV, JSON, FASTA, pickle, NumPy arrays, HDF5, and domain-specific formats — into unified, analysis-ready datasets. The core insight is treating data readiness as a first-class operational primitive organized around scientific tasks rather than model-centric formatting.

When to Use

• When the user has raw scientific data in multiple formats (e.g., protein sequences in FASTA + reaction data in CSV + metadata in JSON) and needs a unified dataset
• When building a pipeline to ingest, validate, and normalize experimental data from a lab or public repository
• When the user asks to "make data AI-ready" or prepare heterogeneous data for downstream ML models
• When integrating data across scientific domains (e.g., combining genomics, proteomics, and metabolomics data)
• When the user needs automated data quality checks, audit trails, and reproducible preparation scripts for scientific workflows
• When normalizing messy tabular data with inconsistent headers, units, temporal resolutions, or missing values across files
• When the user wants to parse a natural-language research question into structured data requirements (variables, constraints, task type)

Key Technique

SciDataCopilot introduces the Scientific AI-Ready data paradigm, which shifts data preparation from ad hoc, format-driven munging to a principled, task-conditioned approach. Three principles govern it: (1) task-conditioned organization — the scientific question determines which data units, variables, and constraints are needed; (2) downstream compatibility — outputs satisfy specific model input contracts (tensor shapes, column schemas, sampling rates); (3) cross-integration ability — heterogeneous modalities (sequence, tabular, spatial-temporal, spectral) are aligned through explicit constraint decomposition rather than forced into a single embedding space.

The framework operates through four coordinated agents over a shared knowledge base K = {D, T, C} where D is a Data Lake of normalized dataset descriptors, T is a Tool Lake of processing tools with I/O contracts, and C is a Case Lake of historical solutions. The Data Access Agent recursively explores directories, detects file types, and generates atomic data unit descriptors (shapes, types, value ranges, missingness). The Intent Parsing Agent translates a natural-language query into structured requirements R = {Obj, Var, Con, Task}, retrieves similar past solutions from the Case Lake, and produces a validated execution plan. The Data Processing Agent refines the plan into runnable code, executes it in a self-repairing loop (error capture, diagnosis, code regeneration, bounded retries), and generates audit reports. The Data Integration Agent decomposes cross-unit constraints into tool-constraint matches and orchestrates a merge pipeline that produces the final unified dataset.