Zarr (Chunked Array Storage): Complete Guide for Robotics Data

Zarr stores N-dimensional arrays in chunked format with per-chunk compression. A dataset is a directory (or zip file, or cloud object store) containing chunks named by their indices (e.g., 0.0.0, 0.0.1). Metadata in .zarray and .zattrs JSON files describes dtype, shape, chunks, and compressor. Groups organize arrays hierarchically like HDF5.

The fundamental advantage of zarr's architecture is that each chunk is an independent file (or object in a cloud store), which means different chunks can be read, written, or compressed completely independently. This is in stark contrast to HDF5, where the entire file is a single monolithic container with internal addressing. For robotics datasets, this translates to true parallel writes (multiple robots can write different episodes simultaneously to the same zarr store without locking), native cloud access (each chunk is a separate S3/GCS object that can be fetched independently), and incremental updates (adding new episodes does not require rewriting existing data).

Zarr v3, the latest specification, introduces a more flexible metadata system with zarr.json replacing the separate .zarray/.zattrs files, support for variable-length data types (useful for storing natural language instructions alongside fixed-size arrays), and a codec pipeline that allows chaining multiple transformations (e.g., delta encoding followed by Blosc compression) per chunk. The v3 spec also standardizes the storage-agnostic interface, making it easier to swap between local filesystem, S3, GCS, and Azure Blob backends without changing application code.

Reading a zarr dataset is straightforward with the zarr-python library. Open a local store with z = zarr.open('dataset.zarr', mode='r'), then access arrays as z['episodes/0/observations/image'][:] for a NumPy array. For cloud access, use z = zarr.open('s3://bucket/dataset.zarr', mode='r', storage_options={'anon': False}) with fsspec handling authentication. Slicing works lazily: z['actions'][100:200] only downloads the chunks covering that range, making it efficient to sample random timesteps from a large cloud-hosted dataset.

Writing a new zarr robotics dataset follows the group-creation pattern: root = zarr.open('output.zarr', mode='w'), then for each episode, create groups with ep = root.create_group(f'episodes/{i}') and write data with ep.create_dataset('observations/image', data=images, chunks=(1, 480, 640, 3), compressor=zarr.Blosc(cname='lz4', clevel=5)). The zarr.convenience.copy_store function enables efficient format conversion from HDF5 to zarr (or vice versa) without loading all data into memory. For Diffusion Policy-style datasets, the convention stores observations and actions as flat arrays indexed by timestep with separate episode_ends arrays marking boundaries, which the Diffusion Policy codebase reads via its ReplayBuffer class.

Migrating from HDF5 to zarr is one of the simplest format conversions in the robotics data ecosystem because zarr was explicitly designed with HDF5 compatibility in mind. The zarr library can read HDF5 files directly via zarr.open(h5py.File('data.hdf5', 'r'), mode='r'), which presents the HDF5 hierarchy as a zarr group tree. For a full conversion, zarr.convenience.copy_store copies all groups, datasets, attributes, and compression settings from HDF5 to a zarr store in a single call. The converted zarr store maintains the same group hierarchy, so code that accesses data by path (e.g., root['episodes/0/observations/image']) works identically on both formats.

The primary migration decision is chunk layout. HDF5 and zarr use the same chunking concept, but zarr's per-chunk-file architecture means that very small chunks (e.g., 1 KB each) create millions of tiny files that overwhelm filesystem metadata operations. For robotics data on S3, the optimal chunk size is 1-10 MB per chunk, which typically means chunking images as (32, H, W, 3) rather than (1, H, W, 3) to avoid excessive S3 GET requests during training. The zarr.rechunk function (from the rechunker package) can re-chunk an existing zarr store without loading all data into memory.

After migration, update your data loading code to use zarr.open() instead of h5py.File(). For Diffusion Policy users, the framework already expects zarr stores and no code changes are needed. For robomimic users, the ZarrDataset class provides a drop-in replacement for the HDF5-based SequenceDataset. Test your training pipeline end-to-end after migration, paying attention to data loading throughput: zarr's parallel chunk reads can improve throughput 2-4x over HDF5 on multi-core machines.

Claru delivers zarr datasets with optimized chunk sizes for your access patterns -- contiguous episode reads or random step access. Cloud-hosted zarr stores on S3 enable direct streaming without local download.

Every zarr delivery is tuned for your specific infrastructure. For cloud training clusters, we host zarr stores on S3 with chunk sizes optimized for your batch size and number of data-loading workers, typically targeting 2-8 MB per chunk. For local workstations, we deliver consolidated zarr stores (zarr.convenience.consolidate_metadata) that eliminate the per-chunk metadata lookups that can slow down initial dataset scanning. We include a zarr-compatible DataLoader implementation that handles parallel chunk fetching, episode shuffling, and prefetching, tested to saturate GPU utilization on A100 and H100 training rigs.