How to Collect Multimodal Robot Data

Different VLA architectures consume different modality combinations, and mismatches between your dataset and the target model are the most common source of wasted collection effort. Start by reading the model's dataloader code (not the paper — papers omit normalization details). RT-2 expects a single 320x256 RGB image and a natural language instruction. Octo expects a primary RGB image (256x256), an optional wrist image (128x128), proprioceptive state as a 7-DoF end-effector pose, and a language instruction tokenized with T5. Diffusion Policy expects multi-view RGB images (2-3 cameras at 96x96 to 256x256) and joint-space proprioception and actions. OpenVLA expects a single 224x224 RGB image, language instruction, and 7-DoF delta end-effector action.

Create a modality specification table with columns: modality name, tensor shape, dtype, units, coordinate frame, sampling rate, and which target models require it. For example: rgb_primary: (256,256,3) uint8, BGR order (OpenCV default), 10 Hz, base frame, required by all. depth_primary: (256,256) float32, meters, 10 Hz, camera frame, required by 3D-aware models. joint_positions: (7,) float64, radians, 100 Hz, joint frame. ee_pose: (7,) float64, [meters, quaternion xyzw], 100 Hz, base frame. force_torque: (6,) float64, [N, Nm], 1000 Hz, sensor frame. language_instruction: string, one per episode. Freeze this spec before configuring any hardware. If you plan to train on multiple architectures, define the union of all required modalities — it is far cheaper to record extra modalities you may not immediately need than to re-collect data when you switch architectures.

Install and calibrate each sensor in the modality spec. For RGB-D cameras, mount an Intel RealSense D435i at the primary viewpoint (typically 45-degree angle, 0.5-1.0m from workspace) and optionally a D405 on the robot's wrist. Calibrate intrinsics with a ChArUco board (target reprojection error <0.3 px) and extrinsics via hand-eye calibration. For force-torque sensing, mount the ATI Mini45 or Robotiq FT 300 between the robot's wrist flange and the gripper, zero the sensor after mounting (with gripper attached, before any payload), and record the gravity compensation offset. For proprioception, subscribe to the robot's /joint_states topic at 100+ Hz and compute forward kinematics for the end-effector pose using the official URDF.

Build the synchronization pipeline as a ROS2 node. Use message_filters.ApproximateTimeSynchronizer for sensor topics running at different rates. The synchronizer groups messages within a configurable slop window — set this to half the period of your slowest sensor (e.g., 16 ms for 30 Hz cameras). On each synchronized callback, pack all modalities into a single observation dictionary. For high-rate sensors (force-torque at 1000 Hz, joint states at 100 Hz), downsample to the target control frequency (typically 10-20 Hz) using the most recent reading before the synchronized timestamp. Store the raw high-rate streams as separate datasets for researchers who need them. Test the full pipeline by recording 10 episodes and verifying: zero NaN values, no all-black image frames, force-torque readings within sensor range (not saturated), and maximum cross-modality timestamp discrepancy below 20 ms.

Define the task suite and, crucially, the language annotation protocol. For VLA model training, every episode needs a natural language instruction that describes the task goal. Write a task catalogue with 3 language template variations per task to prevent models from memorizing specific phrasings: for a pick-and-place task, templates might be 'Pick up the {object} and place it on the {target}', 'Move the {object} to the {target}', and 'Put the {object} on top of the {target}'. Randomize which template is used for each episode.

For richer language supervision, add step-level narrations. Train annotators to narrate episodes post-hoc using a standardized vocabulary: 'reach toward the {object}', 'close gripper on {object}', 'lift {object} to height', 'transport {object} toward {target}', 'lower {object} onto {target}', 'open gripper'. These narrations become the training signal for hierarchical policy learning and language-conditioned sub-goal prediction. Design 15-30 tasks spanning increasing complexity: single-object pick-and-place, multi-object sorting, tool use (e.g., scooping with a spoon), articulated object manipulation (opening drawers, turning faucets), and multi-step sequential tasks (set a table: place plate, then fork, then knife). For each task, specify the required modalities — force-torque is critical for insertion and wiping tasks but optional for open-space transport. Create a task-modality matrix to guide collection scheduling.

Run collection sessions with real-time monitoring of all modality streams. A multimodal collection station has more failure modes than a vision-only setup: force-torque sensors drift after thermal cycling (re-zero every 30 minutes), depth cameras produce holes on reflective surfaces (monitor depth completion rate), and high-rate proprioception recording can drop samples under CPU load (verify actual vs. expected sample count). Build a Streamlit dashboard with one panel per modality showing: RGB stream thumbnail, depth completion percentage, force-torque magnitude plot, joint position trace, and recording health indicators.

For each episode, the recording pipeline should: (1) display the language instruction template to the operator, (2) wait for the operator to confirm the scene is set up correctly, (3) record a 0.5-second pre-episode buffer (captures initial state), (4) record the episode until the operator signals completion or timeout, (5) run immediate validation — check all modalities present, no NaN values, frame counts match expected duration, force-torque not saturated. If any check fails, alert the operator to re-record immediately. Store each episode as a single HDF5 file with group structure: /observations/{modality_name}, /actions, /language_instruction (as string attribute), and /metadata (JSON with task_id, operator_id, success, timestamps). At production pace, expect 15-25 multimodal episodes per hour — slower than vision-only due to additional sensor checks and environment resets.

After collection, run a post-processing pipeline that aligns all modalities to a uniform timeline and computes derived features. The raw data contains modalities at different rates (images at 30 Hz, proprioception at 100 Hz, force-torque at 1000 Hz) that were approximately synchronized during recording. Now resample everything to the target control frequency (typically 10 or 20 Hz). For downsampled modalities, take the nearest-neighbor sample before the target timestamp. For derived features, compute: end-effector velocity via Savitzky-Golay differentiation of the EE pose trajectory (window=11, order=3), grasp state changes (open-to-close and close-to-open transitions from gripper width thresholding), contact detection from force-torque magnitude thresholding (>2N typically indicates contact), and per-frame language relevance if using dense narrations.

Run cross-modal consistency validation: (1) project the end-effector position into each camera frame using the calibrated extrinsics and verify it falls near the visible gripper (catches extrinsic calibration drift). (2) Check that force spikes temporally coincide with visible contact events in the RGB stream. (3) Verify that joint velocities are consistent with observed end-effector motion (catches URDF or FK bugs). (4) For depth data, verify that depth values at the gripper location are consistent with the forward-kinematics-computed distance. Flag episodes that fail cross-modal consistency for manual review. Expect to flag 5-15% of episodes; about half will be recoverable via recalibration of the alignment, the rest should be discarded.

Convert the post-processed dataset to your target training format. For RLDS (Octo, RT-X, OpenVLA): implement a TFDS DatasetBuilder with a features_dict that includes all modalities. Map image modalities to tfds.features.Image (use PNG encoding for RGB, float32 for depth), proprioceptive state to tfds.features.Tensor, language to tfds.features.Text, and add a modality_mask boolean tensor indicating which modalities are present in each episode (critical for mixed-embodiment datasets). For HDF5 (Diffusion Policy, ACT): your existing HDF5 files likely need only normalization. Compute per-modality statistics (mean, std) across the training split and store them as a JSON sidecar.

Generate stratified train/val/test splits (80/10/10) stratified by task variant and modality availability — ensure each split contains examples with all modality combinations. Final validation: load 100 episodes through the target model's dataloader and verify that a forward pass produces outputs of the expected shape (this catches subtle dtype mismatches, incorrect normalization, and modality key name mismatches). For multimodal datasets, additionally verify that masking a modality (replacing it with zeros and setting the mask to False) does not cause NaN outputs — models must gracefully handle missing modalities. Ship the dataset with: modality specification document, calibration files (camera intrinsics/extrinsics, FT zero offsets), normalization statistics, a working dataloader example, and a visualization script that renders all modalities for a single episode as a synchronized multi-panel video.