Explainability Skill

Purpose

A saliency / Grad-CAM heat-map is the most over-interpreted artifact in medical-imaging AI: a colourful map over the lesion is routinely presented as proof the model "looks at the right thing." Adebayo et al. (NeurIPS 2018) showed many saliency methods produce visually convincing maps that are independent of the model's learned weights and of the labels — so they explain nothing. This skill produces an explainability analysis that clears the rigor bar, and audits an existing one, so the map is trustworthy before it reaches a manuscript (CLAIM 2024 / TRIPOD+AI interpretability items).

It sits alongside evaluation in the lane: /architecture-zoo → /preprocess-imaging → /model-scaffold → /model-validation → /model-evaluation + explainability → /write-paper + /check-reporting. It integrates captum / pytorch-grad-cam (referenced in the plan); it does not reimplement them and never runs a model on real patient data.