Training Check

Periodically read WandB metrics during training to catch problems early. Do not wait until training finishes to discover it was a waste of GPU time.

Context: $ARGUMENTS

Constants

• WANDB_RUN - Read from project notes or pass as entity/project/run_id.
• CHECK_INTERVAL - Starts at 10 minutes, then gradually increases if consistently healthy: 10 min -> 20 min -> 30 min -> 60 min (cap).
• REVIEWER_MODEL = gpt-5.4 - Used via a secondary Codex agent for ambiguous cases only.

When to Use

• After training is confirmed running (session alive, loss decreasing for the first few steps)
• When the user wants recurring health checks during training
• This skill checks training QUALITY, not process HEALTH. Process health (session alive, GPU utilization) belongs to watchdog-style monitoring.

Workflow