Spark Training Gotchas

DGX Spark's GB10 chip (Grace Blackwell, SM121, 128GB unified memory, aarch64) has ten recurring failure modes across launch, memory, thermals, bandwidth, and precision. Each is named G1–G10 so it can be checked by number — the numbering is load-bearing for tooling that runs these checks. Read this before a long run, not after hour six.

When to Use This Skill

• A training run fails to start, with an import error or a segfault that doesn't point at the real cause.
• A run OOMs while nvidia-smi still shows headroom.
• Throughput degrades partway through a run that started fine.
• Before any multi-hour or multi-epoch job on GB10.
• Wiring two Sparks together, before picking a parallelism strategy.
• Choosing between FP8 and NVFP4 for a Spark-hosted run.