How to Evaluate Sim-to-Real Transfer Performance

Before transferring any policy to the real world, establish comprehensive sim-only baselines. Evaluate the policy on 1,000+ simulation episodes across the full task distribution: all object categories, initial condition variations, and distractor configurations. Record per-task success rates, failure mode distributions, and action trajectory statistics. A sim success rate below 90% indicates the policy is not yet ready for transfer — fix sim training first, as real-world performance will always be lower.

Compute sim observation statistics (image mean/std, joint state ranges, action value distributions) and compare against known real-world statistics from a calibration dataset. Large distributional differences (e.g., sim images have mean brightness of 120 while real images average 80) indicate visual domain shift that domain randomization should address before transfer. Rank your tasks by expected transfer difficulty: tasks with simple geometries, limited contact, and consistent lighting transfer best; tasks with deformable objects, tool use, or precise insertion are hardest. Start real-world evaluation with the easiest tasks to validate your evaluation protocol before investing in the difficult ones.

A rigorous evaluation protocol ensures results are reproducible and statistically valid. Define: the exact object set (use the same objects as simulation, with 3D-printed or purchased physical copies), the initial state distribution (how objects are placed — use placement templates with marked positions for reproducibility), the number of trials per configuration (minimum 20 per object, 100+ total), the success criteria (identical to sim: object reaches target within 2cm tolerance, held for 3 seconds), and the failure taxonomy (same categories as sim evaluation).

Design the evaluation workspace to minimize uncontrolled variation: consistent overhead lighting (LED panels at 5000K, positioned to eliminate shadows), fixed camera mounts (never adjust between trials), temperature-controlled room (actuator behavior changes with temperature), and anti-vibration table surface. Between trials, reset the environment using a standardized protocol: return robot to home configuration, place objects using placement templates, verify camera views against reference images. Have a human observer log each trial with: success/failure, failure type if applicable, any anomalies (e.g., object slipped unexpectedly, robot hit joint limit). Run 10 warm-up trials that are not counted to verify the full system works end-to-end before beginning scored evaluation.

Run the evaluation in controlled batches. For each batch of 50 trials, execute the sim-trained policy on the real robot and record: high-level success/failure, the robot's executed state trajectory (joint positions at 100 Hz), camera observations (images at the policy's input frequency), and any force/torque readings if available. Simultaneously, for each real trial, run the same initial condition in simulation and record the sim trajectory. This paired sim-real data is essential for gap attribution.

Compute diagnostic metrics after each batch: (1) Success rate gap: sim_success - real_success. If this exceeds 20%, investigate before continuing. (2) Trajectory divergence: for each paired trial, compute the RMSE between sim and real joint trajectories at matched timesteps. Identify the timestep where trajectories diverge — this is usually the point where the gap manifests (e.g., at first contact, during a rotation, or when releasing). (3) Visual input diagnostics: save the real images the policy received and compute CLIP embedding distance from the sim training distribution — outlier frames indicate visual domain shift. (4) Action distribution comparison: plot histograms of predicted actions in sim versus real — distribution shift indicates the policy is encountering observations outside its training distribution.

Use the paired sim-real diagnostic data to systematically attribute each failure to a gap source. For each failed real trial: (1) Check if the same trial succeeded in sim. If yes, the gap is in transfer, not the policy. If no, the failure exists in sim too and is a policy bug. (2) For transfer failures, examine the trajectory divergence point. If divergence occurs at the observation level (the policy outputs different actions for real vs. sim images of the same scene), the visual gap is the cause. If the policy outputs the same actions but the robot reaches different states, the dynamics gap is the cause. (3) For visual gap failures, further classify: is it a texture/appearance issue (domain randomization can help), a lighting issue (adjustable), or a geometric fidelity issue (sim objects do not match real object shapes accurately).

Create a gap attribution report with: total trials, success rate with confidence intervals, failure count by gap source (visual: N, dynamics: N, observation: N, policy bug: N), specific failure examples with paired sim-real videos, and recommended mitigations for each gap source. Prioritize mitigations by impact: if 60% of failures are visual, improving domain randomization or adding real-data fine-tuning will have more impact than tuning physics parameters. This report is the primary deliverable of the evaluation — it directly informs the next training iteration.

Based on the gap attribution report, implement targeted improvements. For visual gap: expand domain randomization ranges for the identified failure modes — if lighting caused failures, randomize light intensity (0.3-3.0x), color temperature (3000-7000K), and shadow direction. If texture caused failures, randomize object textures, table surface, and background. Use NVIDIA Isaac Sim's built-in randomization APIs or implement custom randomization in MuJoCo's XML. For dynamics gap: measure real-world physics parameters (object mass with a scale, friction coefficient with an inclined plane test, actuator response with step-input characterization) and update the simulation model. For residual gaps that randomization cannot close: collect 100-500 real demonstrations on the failing task variants and fine-tune the sim-trained policy using DAgger (dataset aggregation) or simple behavioral cloning fine-tuning with a reduced learning rate (10x lower than sim training).

After each improvement iteration, re-run the full evaluation protocol with the same trial count. Compare success rates using a two-proportion z-test to verify the improvement is statistically significant (p < 0.05). Track the gap attribution breakdown across iterations — a healthy trajectory shows the dominant failure source shifting (first visual, then dynamics, then edge cases) as each gap is progressively closed. Typically, 2-3 iterations of evaluate-diagnose-improve bring a well-randomized sim policy from 40-60% real success to 80-90%.

Package the evaluation results into a reproducible report. Include: sim baseline performance (per-task success rates, 95% CI), real-world performance per iteration (success rates, CI, trial counts), gap attribution per iteration (visual/dynamics/observation breakdown with examples), the final domain randomization configuration, any real data used for fine-tuning (episode count, collection protocol), and a statistical comparison between the final policy and the sim baseline (z-test p-value, effect size).

Establish ongoing evaluation infrastructure so that future sim-trained policies can be evaluated consistently. Create a standardized evaluation kit: the physical object set (stored in a labeled case), placement templates, camera mount positions (marked on the table), lighting configuration, and a scripted evaluation workflow that an operator can follow without researcher supervision. Build an automated evaluation pipeline: the operator triggers a batch of N trials, the system records all sensor data and logs success/failure per trial, and a post-processing script generates the diagnostic metrics and gap attribution. Store all evaluation data (trajectories, images, logs) in a versioned archive — this becomes invaluable for regression testing when you update the simulation or policy architecture months later.