Monitor Training — Autonomous Training Pipeline

Set up cron-based monitoring for long-running YOLO training sessions. Automatically detects completion, reports results, and continues to the next pipeline phase.

When to Use

• After launching a training run that will take hours
• When you want to chain multiple pipeline phases autonomously (train → analyze → clean → retrain → tune → export)
• To monitor training progress without manual polling

Workflow

1. Identify Training State

Determine what's currently running:

• Check for active GPU processes: nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader
• Check for results CSV: tail -5 <training_dir>/results.csv
• Check background task status if task ID is known