NVSHMEM Troubleshooting and Bug Reporting

Purpose

Diagnose NVSHMEM runtime failures from high-signal evidence. Match documented problems conservatively, verify version-sensitive claims with live official documentation, separate NVSHMEM defects from application or environment failures, and prepare a portable report when escalation is useful.

Prerequisites

• Require no credentials, API keys, or privileged access.
• Work from the user's exact launch command, complete output, and generated logs. Use direct read-only inspection only when the user supplies access and asks for it.
• Use $nvshmem-docs when it is registered to verify version-specific behavior, and $nvshmem-install when it is registered for installation or build handoffs. Otherwise use the live official documentation linked from the bundled knowledge base.
• Keep troubleshooting-knowledge-base.md available for diagnosis and bug-report-template.md available for escalation.

Inputs

• Required: Obtain the exact command or complete batch-script launch stanza and complete stdout and stderr from every PE, including lines before the first error and any per-PE logs.
• Conditional: Obtain output that confirms INFO-level NVSHMEM logging before diagnosing an in-process NVSHMEM failure. Apply the documented pre-start exception only when NVSHMEM demonstrably cannot emit logs.
• Optional: Obtain the full NVSHMEM version, CUDA and driver versions, launcher and bootstrap, transport settings, PE/node/GPU counts, rank-to-GPU mapping, topology, smallest failing scale, and last known-good configuration only when they distinguish plausible causes.
• Source precedence: Prefer raw artifacts from the current run, then explicit values in the current user prompt or invocation, then non-conflicting agent context, then focused follow-up answers. Report conflicts and prefer the current raw artifact unless the user confirms a correction.