How to Annotate Robot Failure Modes in Demonstration Data

Before any annotation, formalize a hierarchical failure taxonomy specific to your robot platform, task suite, and downstream use case. A taxonomy that is too coarse (just 'success' vs. 'failure') wastes annotation effort because the model cannot learn failure-specific recovery strategies. A taxonomy that is too fine (30+ subcategories) causes inter-annotator disagreement to spike because the boundaries between similar failure modes become subjective.

Start with the five-category framework used by the RoboCasa and LIBERO benchmarks: grasp failure, placement failure, collision, perception failure, and planning failure. Under each, define 2-4 subcategories with precise operational definitions. For grasp failure, distinguish between: (a) 'missed grasp' where the gripper closed without contacting the object (gripper aperture went to minimum without force sensor spike), (b) 'slip during lift' where initial contact was made but the object fell during the lift phase (detected by sudden gripper force drop followed by no object in the gripper), and (c) 'unstable grasp' where the object remained grasped but rotated more than 15 degrees from its initial orientation during transport. Each definition must reference observable signals, not subjective judgment.

Create a JSON Schema for the annotation format. Each episode annotation should include: `episode_id` (string), `outcome` (enum: success, partial_success, failure), `failure_category` (top-level enum or null if success), `failure_subcategory` (subcategory enum or null), `failure_onset_frame` (integer, the first frame where the failure begins), `failure_obvious_frame` (integer, the first frame where the failure is visually unambiguous), `recovery_attempted` (boolean), `recovery_start_frame` (integer or null), `recovery_outcome` (enum: recovered, failed_recovery, no_attempt), `root_cause` (free text, 10-50 words), and `annotator_confidence` (float 0-1). Validate every annotation against this schema programmatically before accepting it into the dataset.

Before human annotators see any episode, run automated detectors that flag likely failures and pre-populate the annotation fields. This hybrid approach (automated pre-annotation plus human correction) reduces per-episode annotation time from 3-5 minutes to 60-90 seconds for clear cases.

Implement four signal-based detectors. First, a task success detector that checks the final state against the task's success criterion. For pick-and-place tasks, verify that the target object's final position (from the last frame's object detection or known goal coordinates) is within the success threshold (typically 3 cm Euclidean distance from the goal). Use OpenCV template matching or a pre-trained object detector (YOLOv8-nano, which runs at 200+ FPS on CPU) to localize the object in the final frame. Episodes that fail this check are automatically labeled as failures.

Second, a grasp detector that analyzes the gripper aperture and force/torque signals. Parse the proprioceptive data stream and extract gripper width over time. A successful grasp shows: gripper width decreasing, stabilizing at a value above minimum (object is between fingers), and remaining stable during transport. A missed grasp shows gripper width decreasing to minimum without the characteristic force spike (typically 2-10 N for tabletop objects). A slip shows force spike followed by sudden force drop and gripper closing to minimum. Implement these as simple threshold detectors: `is_slip = (force_peak > 2.0) and (force_after_peak < 0.5) and (gripper_width_after < 0.005)`.

Third, a collision detector that monitors joint torque residuals. Compute the expected torques from the robot's dynamic model using the recorded joint positions and velocities (`tau_expected = M(q) * qddot + C(q, qdot) * qdot + g(q)`), then compare against measured torques. Residuals exceeding 5 Nm for more than 100 ms indicate external contact, likely a collision. Libraries like Pinocchio (`pip install pin`) compute inverse dynamics efficiently.

Fourth, a trajectory smoothness detector that flags episodes with abnormal jerk (third derivative of position). Compute per-timestep jerk magnitude: `jerk = np.diff(accel, axis=0) * hz`. Episodes where peak jerk exceeds 3 standard deviations from the dataset mean often contain operator errors, emergency stops, or controller instabilities.

Set up an annotation workflow that presents each episode as a synchronized multi-modal display: video playback, time-series plots of key signals, and pre-populated failure labels from Step 2 that the annotator confirms or corrects.

The best tool for this is rerun.io, which renders 3D robot state, camera feeds, and time-series signals in a synchronized timeline viewer. Log the following per episode: camera RGB frames as an image stream, end-effector position as a 3D scatter trail, gripper aperture as a scalar time series, force/torque magnitude as a scalar time series, and the automated detector outputs as binary event markers on the timeline. Annotators scrub through this view and can instantly see where force spikes coincide with grasp attempts, where collisions occur, and where the trajectory deviates from nominal.

If rerun.io is not available, build a lightweight annotation interface using Label Studio with a custom video annotation template. Pre-render each episode as a side-by-side video: left panel shows the camera feed (primary and wrist views if available), right panel shows matplotlib plots of gripper aperture, force magnitude, end-effector velocity, and collision detector residual. Render these using FFmpeg to combine the video and signal overlay: `ffmpeg -i camera.mp4 -i signals.mp4 -filter_complex hstack annotated.mp4`. Upload the combined videos to Label Studio and configure a labeling template with: a classification block for episode outcome and failure category (dropdowns pre-populated from Step 2), a temporal span block for marking failure onset and recovery attempt intervals, and a text block for root cause description.

Organize the annotation workflow in three stages. Stage 1 (triage): annotator reviews each episode at 2x speed and confirms or rejects the automated outcome label. Average time: 20 seconds per episode. Stage 2 (detailed annotation): for episodes flagged as failures, annotator plays at normal speed, marks the failure onset frame and subcategory, and writes the root cause. Average time: 90-120 seconds per failure episode. Stage 3 (review): a senior annotator reviews 20% of all annotations, including all disagreements between automated and human labels.

Beyond labeling the failure type, annotate the causal chain that led to the failure. This additional annotation layer transforms your dataset from a simple failure classifier training set into a resource for learning causal reasoning about manipulation outcomes.

For each failure episode, annotators answer three structured questions. First, 'what was the proximal cause?' -- the immediate physical event that caused the failure. Examples: 'gripper contacted object 2 cm off-center', 'object rolled on curved surface during approach', 'depth sensor returned zero values on the metallic object surface'. This is selected from a controlled vocabulary of 15-20 proximal causes specific to your platform. Second, 'what was the distal cause?' -- the upstream decision or condition that led to the proximal cause. Examples: 'approach angle was too steep (>45 degrees from vertical)', 'object was placed on a tilted surface', 'lighting caused specular reflection on metal lid'. Third, 'was this failure preventable with the available observations?' -- a binary yes/no that indicates whether the failure could have been avoided by a better policy given the same sensory input, or whether it was fundamentally caused by unobservable factors (e.g., unknown object mass, hidden surface texture).

Store these causal annotations as structured fields, not free text. Define enum types for proximal and distal causes. This enables programmatic analysis: you can compute that 35% of grasp failures are caused by off-center contact, which tells engineering to improve the grasp pose prediction model, while 20% are caused by unobservable object mass, which suggests adding a force-feedback exploration primitive. Group causal annotations into actionable clusters using pandas: `df.groupby(['failure_subcategory', 'proximal_cause']).size().sort_values(ascending=False)` reveals the highest-impact failure modes to address.

For complex failure episodes where multiple causes interact (e.g., a slight grasp offset led to instability, which led to a slip during transport, which led to a collision with another object), annotate the full causal chain as an ordered list of (cause, effect) pairs. This is more time-intensive (3-5 minutes per complex episode vs. 90 seconds for simple failures) so reserve it for the top 10% most informative failure episodes selected by the automated detectors' confidence scores.

Run systematic quality assurance on all failure annotations before they enter the training pipeline. Failure annotation has inherently lower inter-annotator agreement than success labeling because failure causes involve subjective causal reasoning, making QA especially important.

Measure inter-annotator agreement on the 20% overlap subset across three dimensions. For episode outcome (success/partial/failure), compute Cohen's kappa targeting kappa > 0.85. For failure category (5 top-level classes), compute Fleiss' kappa across all annotators targeting kappa > 0.75. For failure onset frame, compute the mean absolute frame difference between annotators targeting agreement within 5 frames (500 ms at 10 Hz). Implement these computations using scikit-learn: `from sklearn.metrics import cohen_kappa_score; kappa = cohen_kappa_score(annotator_a, annotator_b)`. Generate a per-annotator quality dashboard showing their kappa scores, average annotation time, and disagreement rate with the automated detectors.

Run automated consistency checks. Cross-reference failure annotations against the signal-based detectors from Step 2. If a human annotated 'grasp slip' but the force signal shows no contact was ever made (peak force < 0.5 N throughout the episode), flag the annotation for review. If a human annotated 'success' but the automated task success detector found the object more than 5 cm from the goal, flag it. Compute the human-vs-automated disagreement rate per annotator. Annotators who disagree with the automated detectors more than 25% of the time likely misunderstand part of the taxonomy.

Validate the failure distribution against expectations. For teleoperated datasets, typical failure rates are 10-20% for well-practiced operators on simple tasks and 30-50% for complex or novel tasks. If your annotated failure rate is below 5%, the success criterion may be too lenient. If above 60%, the task may be too difficult for the teleoperation interface or the operators need more training. Plot failure category distribution as a histogram and check for pathological imbalances. If one category accounts for more than 50% of all failures, consider subdividing it or collecting additional episodes that exercise other failure modes.

Compute the temporal distribution of failure onset within episodes. Most failures should cluster in the manipulation phase (middle third of the episode), not at the start (which suggests setup problems) or very end (which suggests overly strict success criteria). A bimodal distribution with peaks at the grasp moment and the placement moment is healthy for pick-and-place tasks.

Package the validated failure annotations into formats that integrate with robot learning frameworks. The goal is to enable three downstream use cases: (1) filtering failures out for standard behavioral cloning, (2) including labeled failures for robust policy training, and (3) training dedicated failure detection and recovery models.

For the robomimic/HDF5 convention, add annotation groups to each episode: `/data/demo_N/annotations/outcome` (string: success, partial_success, failure), `/data/demo_N/annotations/failure_category` (string or null), `/data/demo_N/annotations/failure_subcategory` (string or null), `/data/demo_N/annotations/failure_onset_frame` (int or -1), `/data/demo_N/annotations/recovery_attempted` (bool), `/data/demo_N/annotations/root_cause` (string or empty), `/data/demo_N/annotations/causal_chain` (JSON string). Also add a top-level `/metadata/failure_taxonomy` group containing the taxonomy as a JSON string, so the dataset is self-documenting. Use `compression='gzip'` for string arrays and `compression_opts=4` for numeric arrays.

For RLDS format, add failure annotations as episode-level metadata in the `episode_metadata` feature. Add per-step features: `is_failure_onset` (bool, True at exactly the failure onset frame), `failure_phase` (string enum: pre_failure, failure_onset, during_failure, recovery, post_recovery, normal). These per-step labels let sequence models attend to failure-relevant timesteps during training.

Generate three dataset splits designed for failure-related evaluation. Split A (standard): stratified by task, 80/10/10, where the test set has the same failure rate as training. Split B (robustness): training set contains only successes, test set contains a mix of successes and failures -- this evaluates whether the trained policy can handle novel perturbations. Split C (failure detection): balanced 50/50 success/failure split for training a binary failure classifier, with the test set held out by failure subcategory (train on slips and collisions, test on placement failures) to evaluate generalization to unseen failure types.

Provide a Python dataloader class that accepts a `filter` parameter: `TrajectoryDataset(path, filter='success_only')`, `filter='failure_only'`, `filter='all'`, or `filter='with_recovery'`. Include a visualization script that renders any episode with failure annotations overlaid: red tint on frames after failure onset, green tint during recovery attempts, and a text overlay showing the failure category. This visualization is the single most useful deliverable for debugging model behavior on failure cases.