embedded-video-pip-smooth-playback
1. Problem scenario
When you build videos with code-driven renderers (e.g. Remotion, AE scripts, complex FFmpeg filter graphs), you often need picture-in-picture: one main composition with another video embedded inside it.
Typical symptom: In the exported file, motion on the main layer (translation, scale, etc.) looks smooth, but the embedded clip stutters badly, drops frames, or even freezes for long stretches.
2. Root cause: sparse keyframes
Modern codecs (H.264/H.265) save space by storing full pictures only at scene cuts or every few seconds (keyframes / I-frames). Frames in between (P-frames / B-frames) only store differences from neighbors.
Engines like Remotion export frame by frame. To render frame N, the embedded clip must seek to the matching timestamp.
If the embedded file has almost no keyframes (e.g. one I-frame at the start of a 10 s clip), the decoder often has to decode from frame 0 forward to reach frame N. That leads to:
- Very slow seeks: Decoding takes so long that the renderer times out and grabs a frame before the decode finishes.
- Repeated frames: The decoder cannot keep up, so several consecutive captures show the same old image—stutter in the final output.