HDF5 (Hierarchical Data Format 5): Complete Guide for Robotics Data

HDF5 organizes data hierarchically: /episodes/episode_0/observations/images, /episodes/episode_0/actions, etc. Each group can contain datasets (multi-dimensional arrays) with arbitrary dtypes. Chunked storage enables partial reads, and built-in compression (gzip, lz4, zstd) reduces file size 2-5x. Metadata is stored as HDF5 attributes on groups or datasets.

The HDF5 specification defines two core primitives: groups (analogous to filesystem directories) and datasets (analogous to files containing N-dimensional arrays). Groups can be nested to arbitrary depth, and both groups and datasets can carry key-value metadata as attributes. For robotics, the typical convention uses top-level groups for episodes, with each episode containing sub-groups for observations (further split by modality), actions, rewards, and metadata. The robomimic convention, widely adopted in manipulation research, structures files as /data/demo_0/obs/{camera_name, joint_positions, gripper_state} and /data/demo_0/actions, with a top-level /mask group defining train/val splits.

HDF5's chunking mechanism is critical for training performance. When you create a dataset with chunks=(1, 480, 640, 3), each single-frame image occupies one chunk on disk, enabling O(1) random access to any frame without reading the entire array. The chunk size determines the minimum I/O granularity, so aligning chunks with your access pattern (per-step reads for training, per-episode reads for evaluation) is essential. Virtual datasets (VDS), introduced in HDF5 1.10, allow you to create a unified view across multiple physical files without copying data, which is valuable for combining datasets from different collection campaigns into a single logical dataset.

The h5py library is the standard Python interface for HDF5. Reading a robomimic-style dataset is straightforward: open the file with h5py.File('dataset.hdf5', 'r'), then access episodes as f['data/demo_0/obs/agentview_image'][:] to get a NumPy array of all frames. For training, use f['data/demo_0/actions'][start:end] to read action slices without loading the entire episode. The key performance optimization is to keep the file handle open across batches rather than opening and closing it per sample, and to use HDF5's built-in chunked reading rather than loading entire arrays into memory.

Writing a new robotics HDF5 dataset follows a pattern: create the file with h5py.File('output.hdf5', 'w'), then for each episode, create a group with f.create_group(f'data/demo_{i}'), and write observation arrays with f.create_dataset('data/demo_0/obs/image', data=images, chunks=(1, H, W, 3), compression='lz4'). Always store metadata as attributes: f['data/demo_0'].attrs['num_samples'] = len(actions). For large datasets, the hdf5plugin package provides access to fast compressors like LZ4 and Blosc that are not included in the default HDF5 installation. The command h5dump -H dataset.hdf5 (from the HDF5 command-line tools) lets you inspect the structure without loading data, and h5stat dataset.hdf5 reports file-level statistics including chunk utilization.

The most common performance mistake with HDF5 robotics datasets is poor chunk alignment. If your training loop reads one step at a time but your chunks span 100 steps, every read loads 100x more data than needed. For imitation learning, set image dataset chunks to (1, H, W, C) and state/action dataset chunks to (1, state_dim) or (1, action_dim). If your training loop reads fixed-length windows (e.g., action chunking with horizon=16), align chunks to (16, action_dim) for optimal throughput.

For distributed training, HDF5's file-locking mechanism can become a bottleneck when multiple workers read the same file. The solution is to set the HDF5_USE_FILE_LOCKING environment variable to FALSE and ensure workers access the file in read-only mode. Alternatively, shard the dataset into multiple HDF5 files (one per N episodes) and assign different shards to different workers. The robomimic framework provides a SequenceDataset class that handles this worker-to-shard assignment automatically.

When migrating from raw video files to HDF5, consider your compression strategy carefully. Storing raw uint8 frames with LZ4 compression gives the fastest decompression (important for GPU-bound training), while JPEG-compressed bytes in an opaque dataset reduce file size by 5-10x at the cost of decode overhead and lossy quality. For proprioceptive data (joint positions, forces), float32 with no compression is typically fastest since these arrays are small. Always validate your HDF5 files after creation using h5py's built-in checksumming: create datasets with fletcher32=True to detect corruption.

Claru delivers HDF5 files with standardized group hierarchy: /episodes/{id}/observations/{modality} and /episodes/{id}/actions. Chunked storage with lz4 compression balances read speed and file size. Metadata attributes include camera calibration, robot URDF paths, and task descriptions.

Every HDF5 delivery is validated against the robomimic SequenceDataset loader to ensure compatibility with the most widely used imitation learning framework. We provide a Python verification script that checks chunk alignment, compression settings, dtype consistency, and attribute completeness. For teams using D4RL-style flat layouts, ManiSkill episode structures, or custom schemas, we adapt our output format to match your existing data pipeline. Large deliveries (100K+ episodes) are sharded across multiple HDF5 files with a manifest JSON that maps episode IDs to file paths.