Apache Arrow / Parquet: Complete Guide for Robotics Data

Apache Arrow defines a language-independent columnar memory format designed for efficient analytical operations. Data is organized in record batches where each column is a contiguous array of a single type, enabling SIMD (Single Instruction, Multiple Data) vectorized processing and zero-copy reads between processes. Arrow supports rich nested types including structs, lists, maps, and union types, making it capable of representing complex robotics data structures like variable-length joint trajectories and nested observation dictionaries. The in-memory layout uses 64-byte alignment for cache-line efficiency and maintains validity bitmaps for null handling.

Apache Parquet is the complementary on-disk format, storing Arrow-compatible columnar data with efficient compression. Parquet files are organized into row groups (typically 64-128 MB each), where each row group contains column chunks that can be independently compressed with codecs like Snappy (fast, ~2x compression), ZSTD (balanced, ~4x compression), or LZ4 (low-latency, ~2.5x compression). Column-level statistics (min, max, null count) in the file footer enable predicate pushdown, allowing queries to skip entire row groups that do not match filter conditions. For robotics tabular data such as joint trajectories, sensor readings, and episode metadata, this column-oriented layout enables reading only the specific columns needed for a given analysis without scanning irrelevant data.

The Arrow ecosystem includes Arrow IPC (Inter-Process Communication) format for streaming between processes, Arrow Flight for high-throughput network data transfer, and the Feather format (Arrow IPC with file metadata) for fast local persistence. In robotics pipelines, Arrow IPC is particularly valuable for zero-copy data sharing between a data collection process and a real-time monitoring dashboard, or between a preprocessing step and a training loop running in separate processes. The pyarrow library provides the Python interface, while arrow-rs (Rust), Arrow C++ (libarrow), and Arrow Java provide native implementations for other languages.

The pyarrow library (pip install pyarrow) is the standard Python interface for both Arrow and Parquet. Reading a Parquet file is straightforward: pq.read_table('episodes.parquet') returns an Arrow Table that can be converted to pandas with .to_pandas() or to NumPy with .to_pydict() for individual columns. For large files, use pq.ParquetFile('episodes.parquet').read_row_group(i) to load specific row groups, or pq.read_table('episodes.parquet', columns=['joint_positions', 'timestamp']) to read only needed columns. Memory-mapped reading via pq.read_table(..., memory_map=True) avoids copying data into Python's heap, which is critical for datasets that approach available RAM.

Writing robotics data to Parquet follows a pattern of building Arrow arrays and tables. For a trajectory dataset: create arrays with pa.array(joint_positions_list, type=pa.list_(pa.float32())), build a table with pa.table({'timestamp': timestamps, 'joint_positions': joints, 'episode_id': episodes}), then write with pq.write_table(table, 'episodes.parquet', compression='zstd', row_group_size=65536). The row_group_size parameter controls the granularity of predicate pushdown and should be tuned to your query patterns: smaller row groups enable finer-grained skipping but add metadata overhead. For time-series robotics data, sorting by timestamp before writing ensures that temporal range queries can skip most row groups.

The Hugging Face datasets library wraps pyarrow with higher-level APIs for ML workflows. Loading a LeRobot dataset returns a datasets.Dataset backed by memory-mapped Arrow files: dataset = datasets.load_dataset('lerobot/aloha_sim_transfer_cube'). The dataset supports .select(), .filter(), and .map() operations that execute lazily on Arrow arrays without materializing intermediate copies. For custom robotics datasets, datasets.Dataset.from_parquet('your_data.parquet') creates a HF-compatible dataset from any Parquet file, enabling integration with the Hugging Face Hub for versioned dataset hosting and streaming.

In modern robotics data pipelines, Parquet has emerged as the metadata and index layer that ties together multi-modal data stored across different formats. The LeRobot framework exemplifies this pattern: video frames are stored as MP4 files for compression efficiency, while all tabular data (timestamps, joint positions, action vectors, episode boundaries, language instructions) lives in Parquet files. The Parquet episode index enables efficient temporal lookups and episode filtering without scanning video data. This separation of concerns allows teams to query metadata at analytical speeds (millions of rows per second via DuckDB or Polars) while keeping bulk sensor data in format-appropriate containers.

Arrow Flight, the network-native data transfer protocol in the Arrow ecosystem, is gaining traction for real-time robotics data streaming. Flight provides gRPC-based endpoints that transfer Arrow record batches with zero serialization overhead, achieving throughput of 2-4 GB/s on 10 GbE networks. For robotics fleet management, a central Flight server can ingest telemetry from hundreds of robots simultaneously, with each robot's data landing as Arrow record batches that are immediately queryable. The Arrow Flight SQL extension adds SQL query capabilities over Flight endpoints, enabling analysts to query live robot fleet data with standard SQL tools.

For dataset versioning and reproducibility, table formats like Delta Lake and Apache Iceberg build on Parquet to add transactional guarantees. These formats maintain a transaction log alongside Parquet data files, enabling schema evolution (adding new sensor columns without rewriting existing data), time travel (querying the dataset as it existed at any past point), and atomic updates (safely adding new episodes while others are reading). For robotics teams managing evolving datasets across multiple collection campaigns, these table formats solve the practical problem of data versioning that raw Parquet files do not address.

Claru delivers tabular robotics data (joint states, sensor readings, episode metadata, action vectors, language instructions) in Parquet format with optimized row group sizes tuned for your training workload. Numeric columns use float32 precision by default with configurable dtype overrides, and categorical columns (task labels, robot IDs, environment names) use dictionary encoding for 5-10x compression on repetitive string data. All Parquet files include column-level statistics in the footer for efficient predicate pushdown during filtering operations.

For teams using the Hugging Face ecosystem, Claru provides LeRobot-compatible deliveries with Parquet metadata files and MP4 video files structured for direct upload to the Hugging Face Hub. For teams running analytical queries on large datasets, we additionally provide DuckDB-compatible Parquet with sorted columns and Bloom filter indexes for sub-second point queries on datasets with millions of rows. Every delivery includes a schema documentation file mapping column names to their semantic meaning, units, and coordinate frame conventions.