Renderer In The Simulation Loop

Purpose

A renderer inside a physics/RL loop is updated, not reloaded every step, and is fed by a physics engine whose conventions are not the renderer's. This skill is the contract between the two: how per-step state flows in, how time is handled, how frames are paced, and the failure modes that silently feed the policy a wrong or empty observation. Pairs with renderer-realtime-sensor (the sensor-output side) and renderer-gpu-interop (the zero-copy hand-off).

The transform convention is the single most expensive bug

• Physics→render transform plumbing must agree on both row-vector vs column-vector and row-major vs column-major. The catastrophic, silent failure: applying a transpose under the wrong assumption picks up the always-zero last row of a rigid transform, zeroing every instance's translation — every body renders stacked at the origin, the sensor sees nothing, and there is no error. In the field this shows up as "the TLAS has N instances but the depth AOV reports 0 ray hits."
• Establish the producer's convention once (e.g. column-vector row-major, as numpy/Warp emit), apply one named conversion at the renderer boundary, and verify with a translate-and-re-render test — never a static frame, which passes tautologically. See nanousd-renderer-scene-extraction for the generic transform trap; this is its sharpest sim-loop manifestation.

Per-step updates flow through narrow dirty paths, never a reload

• A transform / visibility / color / time edit sets a narrow dirty bit and updates in place; never rebuild the scene or reload geometry per step (see renderer-performance-architecture for the dirty-type separation). The non-obvious residue at multi-environment scale: when the TLAS is partitioned per environment, globals shared across all envs are refitted only via env 0's slot — because identical entries already populate every env's partition, env-0's transform copy is correct for all of them. Mishandling this re-scatters or drops shared static geometry.

Time is an explicit input with an "authored default" sentinel

• A renderer on a physics clock needs a current-time input, but its default must mean "evaluate at the USD-authored default time," not literally t = 0 — defaulting to 0.0 silently re-times every time-sampled attribute. Use a NaN (or equivalent) sentinel for "unset → authored default," and re-evaluate time-varying transforms / geometry / visibility whenever the time changes.