ATFT Training Skill

Mission

• Launch production-grade training for the Graph Attention Network forecaster with correct dataset/version parity.
• Tune hyper-parameters (LR, batch size, horizons, latent dims) exploiting 80GB GPU headroom.
• Safely resume, stop, or monitor long-running jobs and record experiment metadata.

Engagement Triggers

• Requests to “train”, “fine-tune”, “HP optimize”, “resume training”, or “monitor training logs”.
• Need to validate new dataset compatibility with model code.
• Investigations into training stalls, divergence, or GPU under-utilization.

Preflight Safety Checks

1. Dataset freshness: ls -lh output/ml_dataset_latest_full.parquet then python scripts/utils/dataset_guard.py --assert-recency 72.
2. Environment health: tools/project-health-check.sh --section training.
3. GPU allocation: nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv (target >60% util, <76GB used baseline).
4. Git hygiene: git status --short ensure working tree state is understood (avoid accidental overrides during long runs).

Training Playbooks