Manipulation Trajectories: The Core Training Signal for Robot Learning

A manipulation trajectory is a temporally ordered sequence of (observation, action, metadata) tuples recorded while a robot performs a manipulation task. Each trajectory represents one episode — one attempt at a task from start to completion (or failure). The trajectory captures not just what happened (the observations) but what the robot did (the actions), enabling supervised learning methods like behavioral cloning to train policies that replicate the demonstrated behavior.

The observation component of a trajectory typically includes multiple synchronized sensor streams. Visual observations from one or more cameras provide the primary perceptual input. Proprioceptive observations (joint positions, velocities, and sometimes torques from motor encoders) provide the robot's internal state. Tactile observations from force/torque sensors or tactile arrays provide contact information. Each sensor stream has its own recording frequency, and temporal alignment across streams is critical — a 50ms misalignment between camera frames and action labels can degrade trained policy performance by 10-20%.

The action component records what the human operator (during teleoperation) or the robot controller commanded at each timestep. Action representations vary: some datasets record absolute joint positions, others record delta joint positions (change from current), others record end-effector poses (Cartesian position and quaternion orientation), and some record motor torques. The choice of action representation affects which learning algorithms can consume the data and how well policies transfer across robot platforms. Delta end-effector poses are the most transferable representation across different robot arms.

Trajectory metadata includes the task description (what the robot was supposed to do), the episode outcome (success or failure), the robot configuration (URDF, camera calibration), the environment state (which objects are present, their initial positions), and any human annotations (action segmentation boundaries, quality scores, natural language narrations). Rich metadata enables data filtering, stratification, and conditional training that improve policy quality.

Designing a manipulation trajectory dataset requires decisions about task selection, environment setup, collection protocol, and storage format that should be made before collection begins.

Task selection determines what the trained policy can do. Tasks should be specified with enough precision that operators can execute them consistently but enough flexibility that demonstrations are diverse. "Pick up the red cup and place it on the coaster" is better than "pick up something" (too vague) or "pick up the red cup at position (0.35, 0.12, 0.05) with a pinch grasp using fingers 1 and 3" (too specific). Each task should have 100+ demonstrations at minimum, with 500+ for tasks with high variability.

Environment variation is the key to generalization. For each task, collect demonstrations with varied object positions (randomize initial placement), varied objects (different cups, different coasters), varied backgrounds (clean table, cluttered table), and varied lighting (daylight, overhead fluorescent, dim). Without this variation, the policy memorizes a specific scene layout rather than learning the underlying task skill.

Recording infrastructure should be tested end-to-end before collection begins. Verify that all sensor streams are temporally synchronized by recording a known event (a clap, a dropped object) and checking alignment across streams. Verify that the action recording matches the robot's actual joint state by replaying a trajectory on the physical robot and checking for drift. Implement automatic validation that runs after each episode to catch recording errors immediately.

Storage format affects downstream training pipeline compatibility. RLDS (TensorFlow Datasets for robotics) is the standard for JAX/TensorFlow pipelines and is used by Octo, OpenVLA, and other foundation models. HDF5 is common in PyTorch pipelines and used by robomimic. Zarr provides cloud-native storage with efficient partial reads for large datasets. Converting between formats is straightforward but should be done once at the dataset level, not per training run.

Manipulation trajectories are Claru's primary deliverable. We collect teleoperation demonstrations on client robot hardware with full quality validation: temporal synchronization verified to within 10ms, trajectory smoothness metrics computed per episode, task success labeled, and data delivered in the client's preferred format with loading scripts and dataset documentation.

Our datasets include the environment diversity that production policies require — demonstrations collected across varied object sets, scene layouts, lighting conditions, and operator strategies. With 386,000+ annotated clips in our catalog and the capacity to collect custom datasets of 10,000-50,000 trajectories on any standard robot platform, Claru provides manipulation trajectory data at the scale needed for production VLA and policy training.