Gaussian Splatting Scene Description

Overview

gaussian_splatting_scene_description bridges 3D lab reconstruction and natural language understanding. Given a small set of lab photos or short video clips, it builds a 3D Gaussian Splatting (3DGS) scene representation and then generates a structured natural language description of the spatial layout — instrument positions, sample locations, bench topology, and relational predicates (e.g., "pipette is left of tube rack", "centrifuge is behind the operator"). The output is designed for downstream consumption by VLMs, spatial reasoning models, or LabOS skills (protocol_video_matching, detect_common_wetlab_errors, realtime_protocol_guidance_prompts) that need a persistent, queryable representation of the lab environment for context-aware guidance, error detection, or AR overlay anchoring.

When to Use This Skill

Use this skill when any of the following conditions are present:

• Spatial context for protocol guidance: A protocol step references "the tube on your left" or "the centrifuge behind you"; the agent needs a 3D-aware scene description to resolve spatial references and generate accurate realtime_protocol_guidance_prompts.
• Lab layout documentation: A new lab setup or bench configuration must be documented in natural language for onboarding, SOP writing, or remote collaboration — "The pipette is on the right side of the bench; the tube rack is centered; the centrifuge is 2 m behind."
• VLM / spatial model pre-training or fine-tuning: A VLM or spatial intelligence model requires structured scene descriptions as training data; the skill produces consistent, schema-aligned text from real lab imagery.
• AR overlay anchoring: AR overlays (e.g., step indicators, deviation highlights) need to be anchored to 3D positions; the scene description provides object labels and approximate coordinates for overlay placement.
• Multi-session consistency: The same lab is imaged at different times; 3DGS + description enables comparison ("centrifuge was moved from left to right since last scan").
• Error detection context: detect_common_wetlab_errors or protocol_video_matching benefits from knowing the canonical layout — e.g., "tube A1 is in position (x, y) of the rack" — to disambiguate observations.
• Robotic or automation planning: A lab robot (Opentrons, Hamilton) or future automation system needs a natural language map of the workspace for path planning or object localization.
• Virtual lab tours or training: Generate descriptive text for a 3D lab model used in VR training, virtual tours, or remote supervision.