exploratory-data-analysis
Exploratory Data Analysis for Scientific Data
Overview
Exploratory data analysis (EDA) is the systematic examination of scientific data files to understand their structure, content, quality, and characteristics before formal analysis. This knowhow covers methodology for detecting file types, selecting appropriate analysis approaches, assessing data quality, and generating comprehensive reports across all major scientific data domains.
Key Concepts
Scientific Data Type Categories
| Category | Common Formats | Typical Analysis | Key Libraries |
|---|---|---|---|
| Tabular | CSV, TSV, XLSX, Parquet | Summary statistics, distributions, correlations, missing values | pandas, polars |
| Sequence | FASTA, FASTQ, SAM/BAM | Length distribution, quality scores, GC content, alignment stats | BioPython, pysam |
| Image/Microscopy | TIFF, ND2, CZI, DICOM | Dimensions (XYZCT), intensity stats, metadata, calibration | tifffile, aicsimageio, nd2reader |
| Spectral | mzML, SPC, JCAMP, FID | Peak detection, baseline, S/N ratio, resolution | pymzml, nmrglue, pyteomics |
| Structural | PDB, CIF, MOL, SDF | Atom counts, bond validation, B-factors, completeness | BioPython, RDKit, MDAnalysis |
| Array/Tensor | NPY, HDF5, Zarr, NetCDF | Shape, dtype, value range, NaN/Inf check, chunk structure | numpy, h5py, zarr, xarray |
| Omics | H5AD, MTX, VCF, BED | Feature/sample counts, sparsity, annotation completeness | scanpy, pyranges, cyvcf2 |
More from jaechang-hits/sciagent-skills
scientific-brainstorming
Structured ideation methods: SCAMPER, Six Thinking Hats, Morphological Analysis, TRIZ, Biomimicry, plus more. Decision framework for picking methods by challenge type (stuck, improving, systematic exploration, contradiction). Use when generating research ideas or exploring interdisciplinary connections.
12gene-database
Query NCBI Gene via E-utilities for curated gene records across 1M+ taxa. Retrieve official gene symbols, aliases, RefSeq accessions, summary descriptions, genomic coordinates, GO annotations, and interaction data. Use for gene ID resolution, cross-species queries, and gene function summaries. For sequence retrieval use Ensembl; for expression data use geo-database.
10snakemake-workflow-engine
Python-based workflow management system for reproducible, scalable pipelines. Define rules with file-based dependencies; Snakemake automatically determines the execution order and parallelism. Supports local, SLURM, LSF, AWS, and Google Cloud execution via profiles; per-rule conda/Singularity environments. Use for bioinformatics NGS pipelines, ML training workflows, and any multi-step file-processing analysis. Use Nextflow instead for Groovy-based dataflow pipelines or when nf-core ecosystem integration is required.
10esm-protein-language-model
Protein language models (ESM3, ESM C) for sequence generation, structure prediction, inverse folding, and protein embeddings. Use when designing novel proteins, extracting sequence representations for downstream ML, or predicting structure from sequence. Local GPU or EvolutionaryScale Forge cloud API. For traditional structure prediction use AlphaFold; for small-molecule cheminformatics use RDKit.
10matchms-spectral-matching
Mass spectrometry spectral matching and metabolite identification with matchms. Use for importing spectra (mzML, MGF, MSP, JSON), filtering/normalizing peaks, computing spectral similarity (cosine, modified cosine, fingerprint), building reproducible processing pipelines, and identifying unknown metabolites from spectral libraries. For full LC-MS/MS proteomics pipelines, use pyopenms instead.
10chembl-database-bioactivity
Query ChEMBL via Python SDK. Search compounds by structure/properties, retrieve bioactivity (IC50, Ki, EC50), find target inhibitors, run SAR, access drug mechanism/indication data.
10