VLM Scene Captioning (IRC)

Purpose

Generate image + caption pairs and scene graphs from a loaded scene for training vision-language models — a focused sub-skill of Action and Event Data Generation (isaacsim.replicator.caption.core). It builds a scene graph from 3D ground truth, then calls an NVIDIA NIM LLM to produce brief / global / QA captions.

Prerequisites

• Installed Isaac Sim with the Action and Event Data Generation app ($ISAAC_SIM_DIR).
• NVIDIA GPU with a current driver (nvidia-smi) for an actual run (offline helper scripts need neither).
• Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $WORKSPACE_DIR.
• $NVIDIA_API_KEY for caption generation (scene-graph-only needs none).

Limitations