DEFT Mining and Embedding Skill

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

You are the operator of the DEFT embed-then-mine workflow for VCN AOI. Your job is to take a parquet of weak target images (the gap-analysis or routing output) and a source pool, then produce a deduplicated parquet of mined source images that look similar to the targets — ready to feed into the next training round.

The workflow is fixed and deterministic: embed the targets, embed the source pool, then mine nearest neighbours. Each step's output parquet is the next step's input. There is no iterative search, no clustering pass, no human-in-the-loop selection — depth comes from picking the right encoder and the right topn, not from a multi-phase investigation.

The whole skill is a thin wrapper around three direct docker run invocations against the pinned TAO data-services image (see versions.yaml (resolved at runtime — see Setup). The container's entrypoint takes <category> <action> -e <spec.yaml> [hydra overrides...] — pass embedding image_embeddings -e <embedding_spec.yaml> … for embedding and tmm nearest_neighbors -e <mining_spec.yaml> … for mining. The -e flag points at a YAML that supplies default values for the subtask's schema; anything afterward is a bare Hydra override (key=value) that selectively overrides spec fields per run. (There is no dataset keyword inside the container — that's the TAO launcher's pillar prefix and is dropped here.) Pull the image once if it isn't cached: docker pull "$DS_IMAGE" (after resolving $DS_IMAGE per Setup).

Schema keys can rename between data-services releases (the RCA skill saw inference_csv → inference_results_dir, output_dir → results_dir). When in doubt, introspect the actual schema once per image: docker run --rm "$DS_IMAGE" embedding image_embeddings --cfg=job and ... tmm nearest_neighbors --cfg=job.

Inputs