Aerial Manipulation Training Data

Claru collects aerial manipulation data using calibrated multirotor platforms equipped with global-shutter cameras, 200 Hz IMUs, GPS-RTK receivers, 6-axis force/torque sensors, and gripper state logging. Our collection protocol covers structured wind diversity (calm, moderate, gusty conditions logged via ground anemometer), multiple environment types (indoor arenas with motion capture ground truth, outdoor fields, urban delivery sites, and infrastructure inspection locations), and split teleoperation control (dedicated pilot plus manipulation operator). Each episode includes flight phase labels, wind classification, stability metrics, grasp quality assessments, and task-specific annotations — all verified through automated flight log analysis and 25% human spot-verification. We deliver in RLDS, HDF5, or custom formats with full sensor calibration, flight controller logs, and stratified splits by environment, wind condition, and task type.

Aerial manipulation combines the mobility of unmanned aerial vehicles (UAVs) with robotic end-effectors capable of physically interacting with the environment. Unlike ground-based manipulation where the robot base is rigidly fixed or moves on stable ground, aerial manipulators must simultaneously maintain stable flight while exerting forces through a gripper or tool. Every contact force generates an equal and opposite reaction on the airframe, creating coupled dynamics that ground robots never encounter. A simple 2 N grasp force on a 5 kg multirotor shifts the center of mass and induces a torque that the flight controller must immediately compensate. This tight coupling between manipulation and locomotion makes the control problem fundamentally harder and the data requirements fundamentally different.

The commercial and industrial applications are expanding rapidly. Wing (Alphabet) and Amazon Prime Air are deploying package delivery drones that must grasp, carry, and release payloads. Infrastructure inspection companies like Skydio and Flyability need drones that make physical contact with bridges, wind turbines, and power lines for ultrasonic thickness testing and surface sampling. Agricultural applications include precision spraying, fruit picking from tall canopies, and sensor placement. The global commercial drone market exceeded $30 billion in 2025, with manipulation-capable platforms representing the fastest-growing segment.

The data challenge is unique to aerial manipulation. Ground manipulation datasets like DROID or Bridge V2 are collected with stationary or slow-moving bases where camera viewpoints change slowly and predictably. In aerial manipulation, the camera viewpoint shifts continuously due to wind gusts, attitude corrections, and intentional repositioning. The observation space includes not just the workspace and objects but also the drone's own attitude, velocity, and motor states. Training policies must learn to decouple the visual motion caused by flight dynamics from the motion caused by object interaction, requiring data that captures both in sufficient diversity.

Early aerial manipulation research focused on fully actuated hexarotors with rigidly mounted grippers. The ARCAS project (EU FP7) demonstrated cooperative construction with multiple aerial manipulators, while the AEROARMS project (Ollero et al., 2018) built a dual-arm system on a hexarotor platform capable of valve turning on industrial pipes. These systems relied on classical control (PID cascades, impedance control) and required careful system identification rather than learned policies. The data requirements were minimal — hand-tuned controllers needed no demonstrations — but generalization was effectively zero. Each new task required weeks of parameter tuning.

The learning-based paradigm shift began with reinforcement learning in simulation. Zeng et al. (2020) trained quadrotor grasping policies in the Flightmare simulator, achieving sim-to-real transfer for simple pick-and-place tasks. However, the sim-to-real gap in aerial manipulation is severe: rotor wash effects on lightweight objects, ground effect during low-altitude operations, wind turbulence near structures, and actuator latency on real flight controllers are all poorly modeled. ETH Zurich's Aerial Robotics Lab has demonstrated that policies trained purely in simulation achieve 40-60% lower success rates than those fine-tuned with real flight data, particularly for contact-rich tasks like inspection and perching.

Current state-of-the-art approaches use a two-stage pipeline: pretrain a vision-based policy in high-fidelity simulation (Isaac Sim with AeroGym or Gazebo with PX4 SITL), then fine-tune with 500-2,000 real-world flight episodes. Kim et al. (ICRA 2024) demonstrated that 1,000 real teleoperated grasping flights — collected via a VR headset controlling a hexarotor with a magnetic gripper — enabled robust outdoor package pickup in winds up to 15 km/h. The key insight is that real aerial data must capture the full spectrum of disturbance conditions (calm, gusty, sustained crosswind) and lighting (dawn, noon, dusk, overcast) to build a robust policy. Controlled indoor data alone is insufficient for outdoor deployment.

Multi-agent aerial manipulation is an emerging frontier. Transporting large or heavy objects with multiple cooperating drones requires coordination data: cable tension profiles, formation geometry logs, and synchronized trajectory data across all agents. The LARICS group (University of Zagreb) collected multi-UAV cooperative transport datasets with up to four drones carrying a shared payload via cables. These datasets require microsecond-level time synchronization across all platforms and annotations for formation quality, cable sag estimation, and payload swing damping — annotation categories that do not exist in single-robot datasets.

The sensor configuration for aerial manipulation data collection is more complex than ground robotics because of the need to capture both manipulation state and flight state simultaneously. The core stack includes: onboard RGB cameras (minimum one forward-facing and one downward-facing, ideally global shutter to avoid rolling shutter artifacts during rapid attitude changes), an IMU sampling at 200 Hz or higher (the PX4 flight controller natively logs at 250 Hz), GPS-RTK for outdoor positioning with centimeter accuracy (required for delivery tasks but not indoor inspection), a 6-axis force/torque sensor between the arm and the gripper for contact tasks, gripper servo positions and current draw, and motor RPM or ESC telemetry from all rotors. For contact inspection, add the inspection sensor output (ultrasonic thickness readings, thermal camera frames). All sensors must be temporally synchronized to a common clock — on PX4-based systems, this is achieved through the uORB message bus with hardware timestamping.

Collection environments fall into four categories. Indoor flight arenas with motion capture (OptiTrack, Vicon) provide millimeter-accurate ground truth pose at 120-360 Hz, making them ideal for policy training where precise end-effector trajectories matter. These arenas should be at least 10 m x 10 m x 5 m to allow realistic approach trajectories. Outdoor open fields provide natural wind disturbances but require GPS-RTK for positioning. Urban delivery environments add obstacles (mailboxes, porches, vehicles) and realistic drop-off surfaces. Infrastructure inspection sites (bridges, cell towers, building facades) provide the surface textures, geometries, and access constraints that inspection policies must handle.

Teleoperation for aerial manipulation is challenging because the operator must simultaneously fly the drone and control the arm/gripper. The preferred approach is split control: one operator flies the drone using a standard RC transmitter or FPV goggles while a second operator controls the arm via a SpaceMouse or VR controller. Alternatively, the drone can fly autonomously to a pre-specified waypoint (using GPS or visual servoing), and the operator controls only the manipulation. This decoupled approach produces cleaner manipulation demonstrations and is the protocol used by most labs collecting aerial grasping data. Each episode captures: the full flight trajectory from takeoff to landing, the arm joint trajectory, gripper commands, all sensor streams, and metadata including wind speed (from a ground anemometer), ambient light level, and payload weight.

Annotation requirements for aerial manipulation data go beyond standard manipulation labels. Each episode needs: approach phase labels (transit, approach, hover-stabilize, reach, grasp, retract, transit-out), wind disturbance classification (calm, light gust, sustained crosswind, turbulent), flight stability metrics (maximum attitude deviation during manipulation, position hold error in meters), grasp quality (success/failure, force at contact, slip detection), and task-specific labels (for delivery: landing accuracy in centimeters; for inspection: contact duration and force profile quality; for perching: perch stability duration). Claru's protocol applies automated flight log analysis to extract stability metrics and human annotation for task-level quality assessments, with 25% spot-verification and inter-annotator agreement tracking.

Simulation plays a larger role in aerial manipulation than in ground robotics because real flight data is expensive (battery life limits sessions to 10-20 minutes), risky (crashes destroy hardware), and slow to collect (regulatory approvals, weather windows). However, the sim-to-real gap is also larger due to several factors unique to aerial systems. Rotor wash — the downward airflow from propellers — affects lightweight objects on the ground in ways that are computationally expensive to simulate accurately. Ground effect, which increases lift when the drone is within one rotor diameter of the surface, changes the vehicle dynamics during pickup and delivery tasks. Wind turbulence near structures (buildings, bridges) creates complex flow fields that CFD can approximate but not reproduce in real time. And motor response latency on real ESCs (5-20 ms) introduces dynamics that idealized models ignore.

The practical approach is structured domain randomization in simulation combined with real-world fine-tuning. In simulation, randomize: wind magnitude (0-20 km/h) and direction, motor response delay (5-30 ms), payload mass (0.5x to 2x nominal), camera position and orientation (to account for vibration-induced shifts), and lighting conditions. Collect 50,000-100,000 simulated episodes with this randomization, then fine-tune the policy with 1,000-3,000 real flights. The real flights should specifically target the conditions where simulation is weakest: gusty wind, low-altitude ground effect, and contact with real-world surfaces of varying compliance. This hybrid approach reduces the real-data requirement by 5-10x compared to training from scratch on real flights.