Protocol Video Matching

Overview

protocol_video_matching bridges the physical bench and the digital protocol by continuously aligning a first-person XR headset video stream (e.g., Meta Quest, HoloLens 2, Magic Leap) against structured protocol text in real time. The skill parses each protocol step into a semantic action graph, tracks operator gestures and reagent interactions through a Vision-Language Model (VLM), detects when execution diverges from the ground-truth procedure, and surfaces instant corrective guidance as spatial overlays — turning every researcher into a compliant, self-auditing one-person lab.

When to Use This Skill

Use this skill when any of the following conditions are present:

• Live XR-assisted experiments: An operator wearing a first-person XR headset is executing a wet-lab protocol (PCR, cell culture, CRISPR editing, RNA extraction, Western blot, etc.) and needs real-time step-by-step guidance or compliance validation.
• Protocol compliance auditing: A lab manager needs post-hoc or live documentation showing whether a protocol was followed exactly — including timing, reagent volumes, temperature set-points, and action sequence.
• Deviation interception: The agent must interrupt or warn the operator the moment a step is skipped, performed out of order, or executed with incorrect parameters (wrong pipette volume, wrong incubation time, incorrect tube labeling).
• Training and onboarding: A trainee is learning a complex protocol and requires spatial annotations, step-completion confirmations, and error explanations anchored to their field of view.
• GMP / GLP documentation: A regulated workflow (clinical sample processing, diagnostic assay) requires a timestamped, frame-accurate audit trail of every protocol action for regulatory submission.
• Remote expert supervision: A remote PI or supervisor needs a live or recorded feed where protocol adherence is automatically annotated so they can intervene selectively.
• Autonomous lab robot verification: A robotic arm (Opentrons, Hamilton) is executing the protocol and the XR feed from an overhead or wrist-mounted camera must be validated against the digital twin protocol in real time.

Core Capabilities