F5-TTS Mining Rig Monitor Skill

This skill provides instructions for ADA to safely monitor the ongoing F5-TTS training process on the 9-GPU mining rig (Local-LLM), without interfering with the data or environment.

IMPORTANT:

1. The training dataset and checkpoints are strictly located on the HDD of the mining rig at /mnt/toshiba/projects/F5-TTS/.
2. Do not attempt to run training locally on asus-z170k.
3. Use uv exclusively when interacting with the Python environment on the mining rig.

Steps to Monitor Training

1. Check GPU Utilization

To ensure all 9 GPUs are actively training and not bottlenecked or OOMed, run the following command via SSH (remember to use pseudo-terminal if using watch):

ssh Local-LLM "nvidia-smi"

You should see 9 python3 processes consistently consuming ~11GB of VRAM each.