How to Annotate Manipulation Trajectories

Before annotating any trajectories, formalize what annotation layers you need and the vocabulary for each. Manipulation trajectory annotation typically involves three layers at different temporal granularities, and attempting to annotate all three simultaneously leads to inconsistency.

Layer 1 -- Dense action labels: per-timestep annotations of the action being executed. At minimum, verify that the recorded actions (joint velocities, end-effector deltas, or gripper commands) are correctly paired with observations. For teleoperated data, this means confirming synchronization between the controller input and the robot state recording. Check timestamp alignment by plotting commanded vs. actual end-effector position and verifying the lag is consistent (typically 30-100ms for VR teleoperation systems like GELLO or UMI). Record the constant lag offset so downstream users can shift actions to compensate.

Layer 2 -- Phase segmentation: divide each trajectory into semantic phases. Define a phase vocabulary appropriate for your domain. For tabletop pick-and-place: {approach, pre-grasp, grasp, lift, transport, pre-place, place, release, retract}. For kitchen tasks: {reach, grasp, manipulate, pour, stir, cut, release, idle}. Each phase must have a clear start condition (e.g., "grasp begins at the first timestep where gripper width decreases below the fully-open threshold") and end condition documented with examples.

Layer 3 -- Waypoint annotation: mark 3-10 key poses per trajectory that capture the essential shape of the motion. Waypoints should include: the approach pose (end-effector positioned above the target), the contact pose (first contact with the object), the manipulation pose (peak of the manipulation action, e.g., highest point of a lift), and the goal pose (object in final desired configuration). Store each waypoint as a full robot state (joint positions + end-effector SE(3) pose + gripper state) with a semantic label.

Write the annotation schema as a JSON Schema document and validate all annotations against it programmatically. This catches format errors at annotation time rather than during training.

Before human annotators see the data, run automated pre-processing to ensure trajectory quality and generate visualizations that make annotation efficient. Start by loading trajectories from their recorded format. For rosbag2 data: `from rosbags.rosbag2 import Reader` followed by extracting `/joint_states`, `/tf`, and `/camera/color/image_raw` topics. For HDF5 (the robomimic convention): load from `/data/demo_N/obs/`, `/data/demo_N/actions/`, `/data/demo_N/rewards/`.

Run forward kinematics to compute end-effector pose from joint states if not already recorded. Using robotics-toolbox-python: `robot = rtb.models.Panda()`, then `T = robot.fkine(q)` for each timestep's joint configuration `q`. Verify that the computed end-effector pose matches the recorded one (if available) within 1mm and 0.5 degrees. Discrepancies indicate calibration errors or URDF inaccuracies that will corrupt downstream training.

Compute derived signals that aid annotation: end-effector velocity magnitude (`v = np.linalg.norm(np.diff(ee_pos, axis=0) * hz)`), gripper aperture over time, and force/torque magnitude if available. These derived signals make phase boundaries obvious -- velocity near zero indicates a pause or pre-grasp alignment, gripper aperture change indicates grasp/release, and force spikes indicate contact events.

Generate a synchronized visualization for each trajectory. The best tool for this is rerun.io (`pip install rerun-sdk`), which renders 3D robot state, camera feeds, and time-series plots in a single synchronized timeline viewer. Log the robot URDF mesh, joint states per timestep, end-effector trajectory as a 3D line, camera images, and derived signals. Annotators scrub through this visualization to identify phase boundaries. An alternative is to render a side-by-side video: left panel shows the camera view, right panel shows end-effector position/velocity/gripper plots over time with a moving cursor indicating the current frame.

Filter out obviously failed trajectories before annotation. Implement automatic failure detection: episodes where the gripper never closes (no grasp attempted), episodes where the end-effector exits the workspace bounds, episodes shorter than the minimum expected duration, and episodes where the task success signal (if available) is false. Log filtered episodes for review rather than deleting them -- some 'failures' contain useful recovery behaviors.

Set up the annotation workflow for human annotators to segment trajectories into phases and assign skill labels. The most efficient interface for this task is a timeline-based editor where the annotator sees the trajectory visualization (from Step 2) and places segment boundaries by clicking on the timeline.

Configure Label Studio with a video labeling template that includes a custom timeline component. Upload the pre-rendered visualization videos from Step 2. Define the labeling interface with a `<Labels>` block containing your phase vocabulary, and a `<TimeSeriesLabels>` block for annotating temporal segments on the derived-signal plots. Each segment annotation consists of: start_timestep, end_timestep, phase_label, and an optional free-text note for ambiguous cases.

For the annotation workflow, implement a two-pass strategy. In the first pass, the annotator identifies major phase boundaries by scrubbing through the trajectory at 2-4x speed, placing markers at obvious transitions (approach-to-grasp, grasp-to-lift, transport-to-place). This takes 30-60 seconds per 20-second trajectory. In the second pass, the annotator reviews each boundary at normal speed, adjusting to frame-level precision and adding the phase label. Total annotation time per trajectory should be 2-4 minutes for a standard pick-and-place and 5-10 minutes for complex multi-step tasks.

For waypoint annotation, extend the interface to allow the annotator to mark specific timesteps as waypoints. At each marked timestep, the full robot state is automatically extracted and stored. The annotator assigns a waypoint type from a dropdown: {pre-grasp, grasp, post-grasp, via-point, pre-place, place, goal}. Constrain waypoint placement: at minimum, every trajectory must have a grasp waypoint and a goal waypoint. Warn if the annotator marks fewer than 3 waypoints for a pick-and-place or fewer than 5 for a multi-step task.

Batch trajectories by task type so annotators build expertise on one task before switching. An annotator who labels 50 pick-and-place trajectories consecutively develops strong intuition for boundary placement and produces more consistent labels than one who switches between pick-and-place, pouring, and stacking.

Add natural language annotations that make trajectories consumable by vision-language-action models (RT-2, Octo, OpenVLA). Each trajectory needs at least two levels of language annotation: a task-level instruction and per-phase descriptions.

The task-level instruction is a single sentence describing the full task from start to goal state: "Pick up the red block and place it on the blue plate." This is the instruction that a VLA model conditions on at inference time. Write instructions that are specific enough to be unambiguous but general enough to cover the natural variation in demonstrations. Bad: "Move the arm to the right, close the gripper, lift, move left, open." Good: "Pick up the mug from the counter and place it in the dish rack." The instruction should describe the intent, not the motion.

Per-phase descriptions provide finer-grained language aligned to the phase segmentation from Step 3. For each phase segment, write a 3-8 word description: "approaching the mug handle", "grasping the mug", "lifting the mug", "moving to the dish rack", "placing the mug in the rack", "releasing the mug". These per-phase descriptions are used by hierarchical models and for phase-conditioned trajectory retrieval.

Standardize the language vocabulary across all annotators. Create a controlled vocabulary document with approved verbs (pick, place, push, pull, slide, pour, twist, press, open, close, stack, insert), approved spatial references (left, right, above, below, near, on, in, next to), and approved object attributes (color, size, material). Reject annotations that use non-standard terms. This consistency is critical because language conditioning in VLA models is sensitive to vocabulary -- a model trained on "pick up" may not generalize to "grab" without explicit vocabulary normalization.

For multi-step tasks, also annotate the task decomposition as an ordered list of sub-goals: ["open the drawer", "pick up the spoon from the drawer", "close the drawer", "place the spoon on the plate"]. This hierarchical annotation supports task planners like SayCan that decompose high-level instructions into executable skill sequences.

Budget 1-2 minutes per trajectory for language annotation. This is fast relative to phase segmentation because the language directly maps to already-annotated phases.

Run a comprehensive quality assurance pipeline that catches annotation errors before they contaminate training data. Trajectory annotations have several types of errors that are detectable programmatically.

Phase boundary validation: verify that phase labels follow a legal ordering. Define a finite state machine (FSM) for valid phase transitions. For pick-and-place: approach -> pre_grasp -> grasp -> lift -> transport -> pre_place -> place -> release -> retract. Any annotation with an illegal transition (e.g., grasp directly to release without lift) is flagged. Implement this as a transition matrix check: `valid_transitions = {('approach', 'pre_grasp'), ('pre_grasp', 'grasp'), ...}` then verify each consecutive pair of phases is in the set.

Phase-action consistency: during the 'grasp' phase, the gripper should be closing (aperture decreasing). During 'release', the gripper should be opening. During 'transport', the end-effector should be moving (velocity > threshold). Compute these consistency scores per phase segment: `grasp_consistency = np.mean(np.diff(gripper_aperture[start:end]) < 0)`. A grasp phase where the gripper opens more than 30% of the time is flagged.

Waypoint validation: verify that annotated waypoints are kinematically reachable and that the trajectory between consecutive waypoints is physically plausible. Compute the distance between consecutive waypoints in Cartesian space and flag pairs that are unreachably far apart (> 0.5m for tabletop manipulation) or impossibly close (< 1mm, likely a duplicate annotation). Verify that joint configurations at waypoints do not violate joint limits: `assert np.all(q >= joint_lower) and np.all(q <= joint_upper)`.

Language consistency: verify that language annotations reference the correct objects. If the trajectory involves a red mug (from metadata or object detection), the language instruction should mention 'red', 'mug', or both. Cross-reference entity nouns in the language annotation with the object labels in the scene metadata. Flag mismatches where the instruction mentions objects not present in the scene.

Inter-annotator agreement: on the 20% overlap subset, compute phase boundary agreement as the mean absolute difference in boundary frame indices between two annotators. Target: within 5 frames (500ms at 10Hz). For phase labels, compute Cohen's kappa on the frame-level labels. Target: kappa > 0.80. For waypoint locations, compute the mean Cartesian distance between corresponding waypoints. Target: < 20mm for precision tasks, < 50mm for coarse tasks.

Package the validated annotations into formats consumed by major robot learning frameworks. The goal is that a researcher can load your dataset and begin training within 30 minutes, with zero format-wrangling.

For the robomimic/HDF5 convention (consumed by Diffusion Policy, ACT, BC-Z): store each trajectory as `/data/demo_N/obs/joint_pos` (float64, T x 7 for Franka), `/data/demo_N/obs/ee_pos` (float64, T x 3), `/data/demo_N/obs/ee_quat` (float64, T x 4), `/data/demo_N/obs/gripper_state` (float64, T x 1), `/data/demo_N/actions` (float64, T x A where A matches the action dimension), and `/data/demo_N/rewards` (float64, T). Add annotation groups: `/data/demo_N/annotations/phase_labels` (string array, T), `/data/demo_N/annotations/waypoint_indices` (int array, K), `/data/demo_N/annotations/language_instruction` (string attribute), `/data/demo_N/annotations/phase_descriptions` (string array per segment). Use `compression='gzip', compression_opts=4` for float arrays and `chunks=(min(T, 256), D)` for efficient partial reads.

For RLDS/TFRecord format (consumed by RT-X, Octo, OpenVLA): structure each trajectory as an episode with steps. Each step contains `observation` (image, proprioception), `action`, `reward`, `is_terminal`, `language_instruction`, and custom fields for phase label and waypoint flag. Use the RLDS dataset builder (`tfds.rlds.RLDSDatasetBuilder`) with a `DatasetConfig` specifying feature types. Validate the output by loading one episode with `tfds.load()` and checking tensor shapes match expectations.

For zarr format (consumed by newer Diffusion Policy implementations): create a zarr group per trajectory with chunked arrays. Zarr's async writing and partial reads make it faster than HDF5 for large datasets. Use `zarr.open('dataset.zarr', mode='w')`, then `root.create_dataset('demo_0/actions', data=actions, chunks=(256, 7), dtype='float32')`.

Regardless of format, include: (1) a `dataset_info.json` with total trajectories, action space definition, observation space definition, phase vocabulary with definitions, waypoint types, and train/val/test split assignments; (2) a Python dataloader class with `__getitem__` returning a dictionary of numpy arrays; (3) a visualization script that renders any trajectory with phase color-coding and waypoint markers; (4) normalization statistics (per-feature mean and standard deviation) computed on the training split only.

Generate train/val/test splits at the task-variant level: all trajectories for a given (object, initial_position, goal_position) tuple go in the same split. This prevents the trivially high accuracy that comes from near-duplicate initial conditions appearing in both train and test.