How to Create Safety-Labeled Robot Data

Before any annotation begins, create a comprehensive safety taxonomy that covers all hazard types relevant to your robot's deployment environment. The taxonomy should be hierarchical: top-level categories (collision, force violation, workspace violation, speed violation, environmental hazard), mid-level subcategories (collision with object, collision with human, self-collision), and leaf-level severity ratings (minor/moderate/critical).

Start from established standards: ISO 10218 (industrial robot safety) defines hazard categories including crushing, shearing, cutting, entanglement, impact, and ejection. ISO 15066 (collaborative robots) adds force and pressure limits for human contact: maximum quasi-static force of 150N for the chest, 110N for the abdomen, 65N for the hand. Map these standards to your robot and task — a kitchen robot handling knives has different safety priorities than a warehouse robot moving boxes.

For each leaf-level hazard, define: (1) detection criteria (what sensor readings or visual observations indicate this hazard), (2) severity level (minor: no damage or injury risk, moderate: potential equipment damage, critical: potential human injury), (3) expected frequency in normal operation (rare, occasional, common), and (4) the desired policy response (slow down, stop, retract, alert human). Document this taxonomy in a living document that annotators reference during labeling. The taxonomy typically contains 15-30 leaf-level hazard types for a manipulation robot.

Validate the taxonomy by reviewing 100 random episodes from your existing dataset and checking that every observed safety-relevant event maps cleanly to exactly one leaf-level category. If events fall between categories or do not fit, refine the taxonomy before starting production annotation.

Automated pre-labeling identifies obvious safety events using sensor thresholds, reducing human annotation effort by 60-80%. Build detection scripts for each measurable hazard category.

Force violations: threshold the compensated F/T signal at task-specific limits. For a collaborative manipulation task, flag any frame where contact force exceeds 30N (below the ISO 15066 limit of 65N for hands but above the expected task force range). For insertion tasks, flag frames where lateral force exceeds 15N (indicating misalignment rather than controlled insertion). Use a debounced threshold: the force must exceed the limit for at least 3 consecutive frames (100ms at 30Hz) to avoid flagging transient spikes from sensor noise.

Workspace violations: compute the robot's full kinematic chain at each timestep and check if any link or the end-effector exits the defined safe workspace boundary (a convex polytope defined in the robot base frame). Use the URDF collision geometry for accurate link positions. Flag any frame where the minimum distance between any robot link and the workspace boundary is negative (penetration) or below a 5cm warning threshold.

Speed violations: compute joint velocities from encoder data and end-effector Cartesian velocity from the Jacobian. Flag frames where any joint velocity exceeds 80% of the rated maximum (allowing 20% margin for measurement noise) or where the end-effector speed exceeds the task-specific safe limit (typically 0.25 m/s for collaborative tasks near humans, 1.5 m/s for unattended operation).

Self-collision: check pairwise distances between non-adjacent robot links at each timestep using the collision geometry. Flag frames where any non-adjacent link pair distance falls below 5mm.

Output pre-labels as a JSON file per episode: a list of {start_frame, end_frame, category, severity, sensor_evidence} entries. These pre-labels are the starting point for human review, not the final labels.

Natural failures from regular data collection provide some negative examples, but they are biased toward common failure modes (dropped objects, slight misalignments) and underrepresent rare but critical failures (collisions with humans, catastrophic force events). Supplement with deliberately collected negative demonstrations.

Design a negative demonstration protocol for each safety category: (1) Collision demonstrations — the operator deliberately moves the robot into gentle contact with various surfaces (table edge, wall, objects) at controlled speeds of 0.05-0.2 m/s. Use a padded environment and enable the robot's protective stop at 50N to prevent hardware damage. Collect 50+ collision demonstrations varying the contact point (gripper, wrist, forearm), contact surface (hard, soft, edge, flat), and approach angle. (2) Force violation demonstrations — the operator attempts insertion tasks with deliberate misalignment, producing sustained lateral forces of 15-40N. (3) Workspace boundary demonstrations — the operator reaches toward the workspace limits until the safety controller activates. (4) Recovery demonstrations — after each deliberate failure, the operator demonstrates the correct recovery behavior (retract, re-approach, slow down).

Label each deliberate negative demonstration with: the safety category, the intended failure mode, the actual outcome (did the safety controller activate? was the failure controlled?), and the recovery strategy demonstrated. These deliberate demonstrations should constitute 10-15% of the final dataset, with the remaining negative examples coming from natural failures during regular collection.

For simulation-generated negatives, use MuJoCo or Isaac Gym to replay demonstrations with perturbations (shifted object positions, added external forces, modified friction) and identify which perturbations cause safety violations. Simulation can generate thousands of negative examples per hour, but they must be validated against real-world physics — check that simulated force profiles during collisions are within 2x of real-world measurements before including them in the training set.

Human reviewers correct automated pre-labels and annotate events that sensor-based detection misses: near-misses (the robot almost collided but corrected at the last moment), intention violations (the robot completed the task but used an unsafe strategy), and environmental hazards (objects fell from shelves, liquids spilled, a person entered the frame unexpectedly).

Set up the annotation workflow in CVAT or Label Studio with a custom schema matching your safety taxonomy. Each reviewer sees: the episode video from all camera angles, the F/T trace overlaid on the video timeline, the automated pre-labels as initial annotations, and the taxonomy reference document. The reviewer's tasks are: (1) confirm or dismiss each pre-label, (2) adjust segment boundaries if the automated detection is imprecise, (3) add labels for events the automation missed, and (4) assign severity ratings based on the taxonomy.

Calibrate reviewers before production annotation: have 3 reviewers independently label the same 50 episodes, compute inter-annotator agreement (Cohen's kappa), and resolve disagreements through discussion. Target kappa > 0.75 for category agreement and kappa > 0.65 for severity agreement. If agreement is lower, the taxonomy definitions are ambiguous and need clarification. After calibration, assign single reviewers to episodes with spot-checking: a senior reviewer re-annotates 10% of episodes to catch drift.

Budget 5-8 minutes per episode for safety review (at 2x playback speed with pre-labels), compared to 15-20 minutes per episode for annotation from scratch. For a 5,000-episode dataset, this is approximately 400-650 person-hours of review effort. The pre-labeling automation saves an estimated 1,000-2,000 person-hours.

After annotation is complete, validate that the safety labels provide sufficient coverage for training robust safety-aware policies. Compute coverage metrics across three dimensions.

Category coverage: count the number of labeled events per safety category. Every leaf-level category in your taxonomy should have at least 30 examples (for statistical reliability in training). Categories with fewer than 30 examples should be augmented through deliberate collection or simulation. Common coverage gaps: self-collision events (hard to induce during normal teleoperation), human proximity violations (require a human presence during collection), and rare environmental hazards (object breakage, liquid spills).

Severity distribution: for each category, verify that minor, moderate, and critical severity levels are all represented. A common imbalance is that minor events dominate (because they are common and easy to collect) while critical events are underrepresented (because they are rare and dangerous to induce). If critical events are underrepresented, supplement with simulation-generated data or carefully staged demonstrations.

Spatial and temporal distribution: verify that safety events occur across the full workspace (not concentrated in one area) and across the full task timeline (not only at the beginning or end of episodes). Plot a heatmap of safety event locations in the workspace and a histogram of safety event timing within episodes. Uniform distributions indicate thorough coverage; clusters indicate that your collection protocol systematically avoids certain regions or phases.

Run a train-test split validation: train a simple safety classifier (binary: safe/unsafe per frame) on 80% of the data and evaluate on the held-out 20%. The classifier should achieve F1 > 0.85 on the test set. If F1 is lower, the labels may be inconsistent (poor inter-annotator agreement) or the safety events may be too subtle for the available sensor data.

Integrate safety labels into the training dataset in a format that supports multiple training paradigms: constraint-aware imitation learning, safe reinforcement learning, and safety monitor training.

For constraint-aware imitation learning (e.g., constrained Diffusion Policy), add a binary safety mask to each timestep in the observation: 1 for safe, 0 for unsafe. The policy is trained to maximize action likelihood on safe timesteps and minimize action likelihood on unsafe timesteps (or to avoid action distributions that lead to unsafe future states). Include the safety category and severity as additional metadata fields for curricula that weight critical safety violations more heavily.

For safe RL fine-tuning, convert safety labels into cost signals: cost = 0 for safe timesteps, cost = severity_weight for unsafe timesteps (e.g., minor = 0.1, moderate = 0.5, critical = 1.0). The RL objective becomes maximizing task reward subject to keeping the expected cost below a threshold (constrained MDP formulation). Store cost signals as a separate data stream aligned with the observation-action pairs.

For safety monitor training, create a dedicated dataset of short windows (0.5-2 seconds) centered on each labeled safety event. Each window includes: the observation sequence leading up to the event, the safety label at each timestep, and the event category and severity. The safety monitor is a lightweight classifier that runs in parallel with the policy and triggers a protective stop when it predicts an imminent safety violation.

Store all safety annotations in a sidecar file per episode (not embedded in the main data file) so that the safety labels can be updated without reprocessing the entire dataset. The sidecar file should reference the main data file by episode ID and contain the safety event list with frame indices, categories, severities, and annotator IDs. Include a schema version number so that downstream consumers can verify compatibility.