How to Collect Warehouse Robot Data for Training

Warehouse environments contain dozens of distinct manipulation and navigation tasks. Start by building a task taxonomy document that enumerates every task variant you intend to support: single-item pick-and-place from shelf to tote, multi-item bin picking, mixed-case palletizing, depalletizing, conveyor singulation, package scanning and sorting, and AMR navigation with dynamic obstacle avoidance. For each task, specify the observation space (which cameras, what resolution, whether depth is required), the action space (Cartesian end-effector deltas vs. joint position targets, gripper width or binary open/close), the control frequency (typically 10-20 Hz for manipulation, 5-10 Hz for navigation), and the expected episode length.

Create a formal data spec document using a standardized template that covers: target model architecture and its exact input tensor shapes, required coordinate frames and transforms, object categories with unique IDs for each SKU, environment configuration parameters (shelf heights, aisle widths, conveyor speeds), and minimum diversity requirements — for example, 50 unique SKU models, 5 lighting conditions, 3 shelf configurations. Share this document with all stakeholders (ML team, robotics engineers, collection operators) and sign off before spending money on hardware or scheduling collection sessions. Changes to the data spec mid-collection are the single most expensive mistake teams make.

Warehouse data collection demands a sensor rig that captures both the robot's workspace in high fidelity and the broader environment context. A proven configuration for mobile manipulators uses three camera streams: a wrist-mounted Intel RealSense D455 (640x480 at 30 fps, depth + RGB) pointed at the gripper workspace, a head or shoulder-mounted D435i (640x480 at 30 fps) providing the agent's egocentric view, and an overhead or third-person ZED 2i (1280x720 at 15 fps) for supervision and debugging. For proprioception, record the full joint state vector (positions, velocities, efforts) from the robot's /joint_states topic at 100 Hz, plus end-effector pose from the forward kinematics at the same rate.

Calibrate each camera's intrinsics using a ChArUco board with OpenCV's cv2.aruco.calibrateCameraCharuco — aim for reprojection error below 0.3 pixels. Compute extrinsics (camera-to-base transforms) using hand-eye calibration: the eye-in-hand method for the wrist camera (cv2.calibrateHandEye with the Tsai-Lenz solver), and PnP-based registration for fixed cameras using known AprilTag positions in the workspace. Store all calibration data as YAML files alongside the dataset. For depth quality in warehouse environments, note that RealSense infrared stereo struggles with reflective shrink-wrap and dark conveyor belts — enable the D455's built-in IMU and use the high-accuracy preset. Test depth quality on your actual SKU inventory before committing to a collection schedule.

Sensor synchronization is non-negotiable for policy learning — a 50 ms misalignment between an RGB frame and the corresponding joint state renders the observation-action pair nearly useless. Implement hardware synchronization by connecting all RealSense cameras to a shared hardware trigger line using the D455's GPIO sync port. For joint state recording, use a real-time ROS2 node running in a dedicated executor thread with SCHED_FIFO priority. Timestamp all sensor streams using a single NTP-synchronized clock source; avoid mixing ROS time with camera-internal timestamps.

Build a recording manager node in Python that subscribes to all sensor topics via message_filters.ApproximateTimeSynchronizer with a slop tolerance of 10 ms. On each synchronized callback, pack the observation into a dictionary: {"rgb_wrist": (H,W,3) uint8, "depth_wrist": (H,W) float32 in meters, "rgb_head": (H,W,3) uint8, "joint_positions": (N,) float64, "ee_pose": (7,) float64 as [x,y,z,qx,qy,qz,qw], "gripper_width": float64}. Write episodes to HDF5 using h5py with gzip compression (level 4 is a good speed/size tradeoff). Each episode is one HDF5 file containing datasets for obs/{modality}, action, and metadata. Implement a post-episode validation hook that checks: no NaN values in joint states, no all-zero depth frames, frame count matches expected duration, and file size is within expected range. Reject and re-record any episode that fails validation before the operator moves to the next trial.

Write a Standard Operating Procedure (SOP) document for each task in your taxonomy. The SOP should be detailed enough that a new operator can follow it with zero verbal instruction. For a palletizing task, the SOP specifies: (1) Initial state setup — place N boxes of specified dimensions on the source conveyor or staging area in a randomized arrangement, position the pallet at the target location, ensure the robot is at its home configuration. (2) Task execution — the teleoperator picks each box in a specified order (or randomized order, depending on the protocol variant) and places it on the pallet following a specified packing pattern. (3) Episode termination criteria — all boxes placed successfully (success) or timeout after 180 seconds (failure). (4) State randomization between episodes — shuffle box positions, vary the number of boxes (3-8 per episode), rotate pallet orientation by 0 or 90 degrees.

Critically, define the teleoperator interface. For warehouse manipulation, a SpaceMouse (3Dconnexion SpaceMouse Pro) controlling end-effector velocity in Cartesian space is the most common choice — it offers intuitive 6-DoF control and operators reach proficiency after 2-3 hours of practice. Configure deadzone thresholds (typically 0.05 for translation, 0.02 for rotation) to filter hand tremor. Set velocity scaling to 0.15 m/s max translation and 0.8 rad/s max rotation for safe operation around shelving. Run a pilot session of 30 episodes with 2-3 operators to measure: average episode duration, success rate, and inter-operator consistency (compute DTW distance between trajectories for the same task). Use pilot results to refine the SOP, adjust speed limits, and estimate total collection time.

Run data collection in structured shifts with dedicated roles: one teleoperator, one environment resetter (who randomizes objects between episodes), and one quality monitor watching a live dashboard. Build the dashboard using a simple Streamlit app that displays: episodes completed vs. target (progress bar), rolling success rate over the last 50 episodes, distribution of episode durations (histogram), depth frame dropout rate, and joint state recording gaps. Flag any episode where depth dropout exceeds 2% of frames or where joint state gaps exceed 50 ms for immediate re-recording.

For warehouse-scale collection targeting 5,000+ episodes, organize work into collection campaigns of 500 episodes each. At the end of each campaign, run an automated quality audit: compute action velocity statistics and flag outliers (episodes where max joint velocity exceeds safety limits suggest teleoperation glitches), verify observation space coverage (e.g., at least 80% of target SKUs appear in the campaign), and check for recording artifacts (duplicate timestamps, zero-length episodes, corrupted HDF5 files). Maintain a collection log spreadsheet tracking: campaign ID, date, operator, robot serial number, environment configuration, episodes recorded, episodes passing QA, and any hardware issues. This log becomes essential for debugging dataset problems months later when training reveals unexpected failures.

After collection completes, run a comprehensive post-processing pipeline. First, apply consistent image preprocessing: resize all camera streams to the target model's input resolution (typically 256x256 or 224x224), convert depth from raw millimeters to normalized float32 meters, and apply the camera intrinsics to correct any lens distortion. Second, compute derived features if your model requires them: end-effector velocity by finite differencing the pose trajectory with a Savitzky-Golay filter (window=11, order=3 from scipy.signal), wrist force/torque if available, and binary contact labels from force thresholds.

For deduplication, compute a trajectory embedding for each episode by encoding the full joint state sequence through a lightweight 1D-CNN or by computing DTW distance between all episode pairs. Flag episodes with DTW distance below a threshold (calibrate on your pilot data — typically episodes from the same operator performing the same task variant will cluster). Remove exact duplicates and near-duplicates, keeping the highest-quality instance (measured by smoothness of the action trajectory and depth frame completeness). Generate stratified train/validation/test splits (80/10/10) ensuring that splits are stratified by task variant, operator ID, and environment configuration — never leak episodes from the same session across splits. Finally, convert to your target format. For RLDS, implement a custom TFDS DatasetBuilder: define the features_dict matching your observation and action spec, implement _split_generators to map your train/val/test HDF5 files, and run tfds build --data_dir=/path/to/output. Validate the RLDS output by loading 100 random episodes through the standard tfds.load pipeline and visually inspecting observation-action alignment in a Jupyter notebook.