WebDataset (Tar-based Shards): Complete Guide for Robotics Data

WebDataset, created by Thomas Breuel (formerly at Google Brain and NVIDIA), stores training samples as consecutive entries in standard POSIX tar archives. Each sample consists of multiple files sharing a common key prefix: 000042.jpg (image), 000042.json (metadata), 000042.actions.npy (action vector), 000042.state.npy (proprioceptive state). The key is everything before the first dot in the filename, and the extension determines the data type. This convention means that any file format can be embedded in a WebDataset shard: JPEG images, NumPy arrays, JSON metadata, PNG depth maps, protobuf messages, or raw binary blobs. Shards are typically 100 MB to 1 GB tar files named with a pattern like shard-{000000..001000}.tar, and the library supports brace expansion for specifying shard ranges in URLs.

The naming convention is the core design insight of WebDataset. Within a tar shard, files are grouped into samples by their shared key prefix. The webdataset Python library (pip install webdataset) handles this grouping automatically during iteration, yielding dictionaries with extension-based keys: {'jpg': image_bytes, 'json': metadata_dict, 'actions.npy': action_array, 'state.npy': state_array}. Built-in decoders automatically handle common formats: JPEG/PNG images are decoded to PIL Images or NumPy arrays, .npy files are loaded as NumPy arrays, .json files are parsed to dictionaries, and .txt files are read as strings. Custom decoders can be registered for domain-specific formats via wds.decode(custom_handler).

For distributed training, WebDataset's shard-based architecture provides a natural parallelism unit. Each GPU worker loads a different subset of shards, and the ResampledShards or ShardList classes handle shard-to-worker assignment with configurable shuffling granularity. Because tar files support efficient sequential streaming, data can flow directly from S3, GCS, or HTTP endpoints to GPU memory without intermediate disk writes, using standard Python HTTP libraries. The webdataset.ShardWriter class creates new shards automatically when the current shard exceeds a configurable maximum size (default 1 GB) or sample count (default 100,000), distributing samples evenly across shards for balanced distributed loading. This design achieves near-linear scaling: 8 GPU workers reading 8 shards simultaneously achieve 8x the throughput of a single worker.

Reading a WebDataset is a single pipeline expression: dataset = wds.WebDataset('shards/shard-{000000..000099}.tar').decode('pil').to_tuple('jpg', 'json'). This loads 100 shards, decodes JPEG images as PIL Images, and yields (image, metadata) tuples. For PyTorch training: dataloader = wds.WebLoader(dataset, batch_size=32, num_workers=4) provides a standard DataLoader interface with automatic shard resampling for distributed training. The pipeline supports chaining operations: .shuffle(1000) shuffles samples within a 1000-sample buffer, .batched(32) creates batches, and .map(transform_fn) applies arbitrary transformations.

Writing WebDataset shards uses the ShardWriter class: with wds.ShardWriter('output/shard-%06d.tar', maxcount=10000, maxsize=1e9) as sink: for each sample, sink.write({'__key__': f'{i:06d}', 'jpg': image_bytes, 'json': metadata_dict, 'actions.npy': action_array}). The __key__ field sets the sample key prefix, and other dictionary entries are written as files with the corresponding extension. ShardWriter automatically creates new shards when the current shard exceeds maxsize bytes or maxcount samples. For images, pass raw JPEG/PNG bytes (not decoded arrays) to avoid re-encoding overhead. For NumPy arrays, the .npy extension triggers automatic np.save encoding.

For cloud-native training, WebDataset supports streaming from any URL that tar files can be served from. Reading from S3: dataset = wds.WebDataset('pipe:aws s3 cp s3://bucket/shard-{000..099}.tar -') streams shards through the AWS CLI. For HTTP: dataset = wds.WebDataset('https://host/shards/shard-{000..099}.tar') fetches shards via standard HTTP GET requests. The pipe: prefix enables arbitrary shell commands as data sources, supporting custom authentication, caching, and compression. For maximum throughput on cloud storage, the wds.WebDataset class supports parallel shard fetching via the shardshuffle parameter, overlapping network I/O with GPU computation.

WebDataset has become the format of choice for large-scale vision model training, and this approach is increasingly adopted for robotics foundation models. The LAION-5B dataset (5.85 billion image-text pairs, ~240 TB) is stored entirely as WebDataset shards, and the OpenCLIP training pipeline reads these shards at over 10,000 samples per second per GPU node using WebDataset's streaming architecture. For robotics, the same pattern applies when training vision-language-action models or large-scale perception models on millions of trajectory steps: each step becomes a WebDataset sample with image, proprioceptive state, action, and language instruction as separate files within the shard.

The key performance advantage of WebDataset over random-access formats (HDF5, Zarr) is its sequential I/O pattern. Modern storage systems (NVMe SSDs, cloud object stores) achieve maximum throughput on sequential reads: NVMe SSDs deliver 3-7 GB/s sequential versus 0.5-1 GB/s random, and S3 delivers roughly 100 MB/s per stream with practically unlimited parallel streams. WebDataset exploits this by reading each shard as a single sequential stream, achieving near-theoretical-maximum storage throughput. For a training cluster reading 64 shards in parallel from S3, this translates to 6+ GB/s aggregate throughput, which is sufficient to keep 8 A100 GPUs fully utilized on image-based training.

For multi-modal robotics data, WebDataset's file-per-field approach maps naturally to the heterogeneous data types in robot demonstrations. A single WebDataset sample for a manipulation demonstration step might contain: 000042.cam0.jpg (overhead camera, JPEG compressed), 000042.cam1.jpg (wrist camera), 000042.depth.png (16-bit depth map), 000042.state.npy (7-DoF joint positions as float32), 000042.action.npy (7-DoF action vector), 000042.language.txt (natural language instruction), and 000042.meta.json (episode ID, timestamp, task label). The webdataset decoder handles each format appropriately based on extension, and custom decoders can handle domain-specific formats like compressed point clouds.

Claru delivers WebDataset shards optimized for your training cluster: shard sizes tuned to your storage backend (100 MB for NFS, 500 MB-1 GB for S3/GCS), consistent sample counts per shard for balanced distributed loading, and S3-compatible URLs for direct streaming with no local download required. Each shard contains multi-modal samples with JPEG-compressed images, NumPy arrays for proprioceptive state and actions, JSON metadata, and optional language instruction text files.

Every delivery includes a shard manifest (JSON file listing all shard URLs, total sample count, and per-shard statistics) that integrates with distributed training coordinators. For teams using NVIDIA DALI, we provide DALI-compatible shard specifications. For teams using PyTorch DataPipes, we include a pre-built DataPipe configuration. Shard contents are validated for consistency (all samples have the same set of extensions, all arrays have consistent shapes) and a sample verification script is provided. For incremental data deliveries (adding new collection batches to an existing training set), new shards follow the existing naming convention and the manifest is updated atomically.