Egocentric Lab Video Dataset

Laboratories are one of the highest-value target environments for robotic automation. Pharmaceutical companies, biotech startups, and academic research labs spend billions annually on repetitive manual tasks — pipetting, plate handling, reagent preparation, centrifugation — that are ideal candidates for robotic automation. Training models for these tasks requires video data captured in real lab environments with the precision and sterility constraints that define laboratory work.

The egocentric viewpoint is essential because it captures the operator's perspective during complex protocols. A head-mounted camera records exactly what a lab technician sees: pipette tips at close range, microplate wells partially occluded by the operator's hands, small-volume liquid transfers that are invisible from a room-scale camera, and the constant context-switching between bench areas, instruments, and reagent storage that characterizes real lab workflows.

Lab environments present unique perception challenges that kitchen or warehouse data cannot address. Transparent vessels (glass flasks, plastic tubes, microplates) create reflection and refraction artifacts. Small objects (pipette tips, microcentrifuge tubes, slides) require high spatial resolution. Color-coded reagents and labels demand accurate color perception. Sterile technique imposes movement constraints that differ fundamentally from everyday object manipulation.

Claru collectors wear lightweight head-mounted cameras while performing genuine laboratory protocols in their own facilities. This includes wet lab procedures (cell culture, PCR preparation, ELISA assays, buffer preparation), dry lab tasks (sample weighing, powder handling, equipment calibration), and cleanroom operations (wafer handling, sterile assembly). Each session follows a real protocol rather than scripted actions, generating authentic data with natural timing, error recovery, and the multi-step reasoning that lab work demands.

Collection sessions range from 20 minutes to 2 hours of continuous activity, depending on the protocol. Collectors are recruited from diverse lab types — university research labs, pharmaceutical QC labs, clinical diagnostics labs, and biotech startups — ensuring the dataset covers variation in equipment brands, bench layouts, PPE requirements, and protocol standards. This diversity is critical for training models that generalize beyond a single laboratory setup.

Raw video is captured at 1080p resolution at 30fps. Each session includes metadata: lab type classification, equipment inventory, protocol name and description, and PPE worn. For a subset of collections, we deploy RealSense depth cameras and wrist-mounted IMU sensors to capture depth and motion data alongside RGB video.

Lab automation companies building robotic pipetting and sample handling systems need demonstrations that capture the dexterity, precision, and protocol-awareness of expert technicians. Claru's lab dataset provides the visual observation sequences these systems need, with temporal annotations aligned to standard lab protocol taxonomies that researchers can map to robotic action spaces.

Vision systems for lab instrument monitoring benefit from the equipment state annotations. A model trained on hours of centrifuge, incubator, and hood interactions learns to recognize instrument states, predict when operations complete, and coordinate multi-instrument workflows — the perception backbone for autonomous lab orchestration systems.

Pharmaceutical and biotech companies developing digital twin simulations of their lab processes use this data to validate simulation fidelity. By comparing real egocentric video of a protocol against simulated execution, teams can identify where their digital models diverge from physical reality — particularly for liquid handling dynamics, which remain difficult to simulate accurately.